This post is more than 5 years old
7 Posts
1
12282
AtmosApi.readObjectStream returns empty stream.
Hello,
Please do let me know if this is not the right forum to ask this question.
That being said, I'm having an issue with using the Java Atmos Client API (ver 2.1.5). I want to to stream the content to the Servlet response instead of first reading the byte array because that loads all the content of the file to memory.
Here is the code so far:
ReadObjectResponse response = this.esuAPI.readObjectStream(new ObjectId(atmosObjectId), null);
after this executes the response.getContentLength() method prints 0.
I have verified following details before posting:
1. Connection credentials are correct (host, port, uid and secret key)
2. File does have proper data on the server. I'm able to download the file via AtmosChrome extension.
I'm not sure why the response would be empty?
JasonCwik
281 Posts
0
October 27th, 2015 11:00
You cannot pass null to the second argument of readObject. This is the class type you want back (I'm not sure how it even compiled). You should pass InputStream.class if you want an InputStream, e.g.
InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), InputStream.class);
What is the size of the object if you getSystemMetadata() ?
JasonCwik
281 Posts
1
October 26th, 2015 18:00
getContentLength() will return zero for objects over 1MB in size. This is because for objects greater than 1MB Atmos uses Transfer-Encoding: chunked to "stream" the response and does not send a Content-Length header in the response. You should simply read the stream until you reach EOF. If you need to know the size of the object before reading it, we suggest you call getSystemMetadata() first to read the value of the "size" field.
SNShiva
7 Posts
0
October 26th, 2015 21:00
Thank you for the response Jason!
Initially I had used the InputStream returned to create a file and the file was of 0kb, and for that reason I wanted to print the content length.
Here is what my code looks like for creating a file (this code writes out 0kb file as the control would not execute the while loop even once):
FileOutputStream output = new FileOutputStream(new File("C:/Sample.pdf"));
byte[] buff = new byte[1024];
int bytesRead;
boolean contentRead = false;
while ((bytesRead = inputStream.read(buff)) > 0) {
output.write(buff, 0, bytesRead);
contentRead = true;
}
output.flush();
If I use readObject method instead of readObjectStream:
InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), null);
I'm getting null pointer exception shown below. I cant seem to get any download working with the Atmos Client 2.1.5 jar.
java.lang.NullPointerException
at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.isReadable(AbstractRootElementProvider.java:92)
at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:345)
at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:315)
at com.sun.jersey.core.spi.factory.MessageBodyFactory.getMessageBodyReader(MessageBodyFactory.java:294)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:617)
at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:586)
at com.emc.atmos.api.jersey.AtmosApiClient.readObject(AtmosApiClient.java:196)
at com.emc.atmos.api.AbstractAtmosApi.readObject(AbstractAtmosApi.java:53)
...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:82)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:240)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:180)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
SNShiva
7 Posts
0
October 27th, 2015 14:00
The getSystemMetadata() returns the size 21122.
If I use the InputStream.class parameter, I get the stream object back. But when I try to read content from Stream - I get Stream Closed error:
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
Another stupid mistake I had done is to use different credentials, basically the object Id I was looking to download existed in a different subtenant. I am surprised why I didn't receive errorCode=1003 and HTTP Status Code 404. Anyways once the issue with credentials is corrected, I used readObjectStream() again and this time it worked like a charm!
. I can see this method returns ByteArrayInputStream by default.
Surprising thing is the readObject() method still has issues. For now since I got readObjectStream() working, I'm going to continue using that. However I would need to circle back to this if I run into trouble handling large files (about 200 MB size) with out-of-memory issues.
Thank you once again for your time!
JasonCwik
281 Posts
0
October 28th, 2015 07:00
Glad you're up and running. I'm not sure what the issue is that you're seeing with readObject, but I would recommend upgrading to the latest atmos-client (2.2.2). If you're using a Maven-compatible system, the artifact is:
com.emc.vipr
atmos-client
2.2.2
Otherwise, you can download the latest from this page.
SNShiva
7 Posts
0
October 28th, 2015 08:00
We did try to use the atmos-client jar 2.2.2 version, but the distributed jar requires Java 7+ but our systems are still using Java 6 and it involves spending additional effort (and money
) trying to upgrade our systems at this point, for that reason we resorted to version 2.1.5 which requires Java 6 as minimum requirement. I believe from version 2.1.6 the minimum requirement is Java 7+.
I had run into same issue of empty stream again yesterday with both readObject and readObjectStream methods. This time when I verified the file via Chrome extension I could see that the Object was created but there was no content in it. Content-length was 0. I also confirmed this by calling getSystemMetadata() and that prints size 0 too.
When I went through the log entries when upload is done I can see following message in the logs:
Content request with input stream and zero-length will not send any data
So I believe the content was never transferred to Cloud. I'm using below Code to upload the file:
FileInputStream inputStream = new FileInputStream(new File("C:/SampleFile.docx"));
CreateObjectRequest req = new CreateObjectRequest();
req.content(inputStream);
req.setAcl(new Acl()); //This may not be required
CreateObjectResponse resp = this.atmosAPI.createObject(req);
ObjectId objectId = resp.getObjectId();
Do I have to use code similar to what we have in com.emc.esu.api.rest.UploadHelper.createObject(InputStream, Acl, MetadataList, boolean) as this uploads the file content as small chunks?
chris_arnett
110 Posts
1
October 29th, 2015 17:00
If you're uploading a file, you can use the File object as the content. That should solve you're problem.
Whenever you upload a stream, you must specify a content-length as well. The Atmos API requires a content-length header and, aside from buffering the entire stream in memory, there is no other way for the client to know what that is.
SNShiva
7 Posts
0
November 2nd, 2015 09:00
Thank you for the response! you were spot on the issue.
Jason's response to this post was the hint to get me to the solution.
It was easy for us to find out the content-length because our source of data was SQL Server (files are stored in a column with image data type). I could run a simple SQL query to get the length of the image column in bytes and set that to the CreateObjectRequest-object.
We did not use any temporary storage and were able to implement a 'pure streaming' from SQL Server to Atmos and successfully transferred few thousand files in our testing. Looking forward to migrate all the data on production (it has about 1.5 million files of various sizes)
It would not have been possible without all of your help! thank you!
JasonCwik
281 Posts
0
March 8th, 2016 09:00
Hi Vandan,
You can look at our UploadHelper class. You can use an ArraySegment to specify data to upload. You'll first create the object with the first segment then append in subsequent segments.
https://github.com/EMCECS/atmos-dotnet/blob/master/EsuApiLib/UploadHelper.cs#L245
Alternatively, if you know the length of the stream beforehand, there's a CreateObjectFromStream where you can give us the whole stream.
atmos-dotnet/EsuApi.cs at master · EMCECS/atmos-dotnet · GitHub
The implementation is here where we actually buffer the objects from the source stream and write them out to the HTTP stream.
https://github.com/EMCECS/atmos-dotnet/blob/master/EsuApiLib/Rest/EsuRestApi.cs#L481
VandanKumar
9 Posts
0
March 8th, 2016 09:00
SNShiva and Stu Arnett,
I am in the same situation where I have to write a large file onto Atmos.
The solution explained here is what I am looking for. I just need one more bit of detail.
How did you implement the "pure streaming" from your source location to Atmos? I have questions about how to write the content in a loop where we will stream only certain number of bytes. And make the receiving side receive it the same way and return only after the whole content is read.
Could you please explain that part?
Thanks,
Vandan
VandanKumar
9 Posts
0
March 9th, 2016 10:00
Jason,
Thanks a lot for the explanation and the source links.
Our application is Java based. I use the atmos-client-2.2.2.jar to talk to Atmos. Based on the dot net code that you shared, the logic at the high level is to create the object first and then keep appending the content in a loop until done. I did this in Java and it did work.
Here is the sample code fragment showing the same.
AtmosApi atmos = getAtmosInterface();
byte[] buffer = new byte[1024 * 1024];
File file = new File("C:\\testfile.txt");
FileInputStream fis = new FileInputStream(file);
int len = fis.read(buffer);
if (len != -1) {
byte[] bytes = Arrays.copyOfRange(buffer, 0, len);
//Create the Object with initial content read from the file
ObjectId objectid = atmos.createObject(bytes, "application/octet-stream");
long contentLength = len;
//Read the rest of the content in a loop and append it to the object
len = fis.read(buffer);
while (len != -1) {
bytes = Arrays.copyOfRange(buffer, 0, len);
//Append the object with additional content
atmos.updateObject(objectid, bytes, new Range(contentLength, (contentLength + len - 1)));
contentLength += len;
len = fis.read(buffer);
}
}
fis.close();
Hope this logic is correct. Please let me know if there is any other better way to do the same.
Thanks,
Vandan
JasonCwik
281 Posts
0
March 10th, 2016 08:00
Yes, that's about right. One optimization: instead of copying the array you can wrap it in a BufferSegment object: atmos-java-client/BufferSegment.java at master · EMCECS/atmos-java-client · GitHub
There's a more complete sample here where we test the checksumming: https://github.com/EMCECS/atmos-java-client/blob/master/src/test/java/com/emc/atmos/api/test/AtmosApiClientTest.java#L1609
VandanKumar
9 Posts
0
March 10th, 2016 14:00
Thanks Jason. I have used the BufferSegment and it does work fine and eliminates the unnecessary copying of the array content.
Thanks,
Vandan
VandanKumar
9 Posts
0
September 1st, 2016 09:00
All,
What would be recommended size of the buffer for doing the createObject() and for each successive updateObject() calls?
In our case we have some files as large as a few GB that we want to store on Atmos, while most of the other files are just few MB. I would like to know the appropriate size of the chunks that can be written and appended in a loop.
The applications where we are planning to use Atmos is a Web based as well as batch. The peak connections that we have currently is about 7000 users per hour. And about 90% will be READONLY calls.
Please let me know your recommendation.
Thanks,
Vandan
JasonCwik
281 Posts
0
September 6th, 2016 08:00
The internal buffer size on Atmos is 1105920 (1080 kiB), so a multiple of this value would result in the most efficient read/write path.