AtmosApi.readObjectStream returns empty stream.

getContentLength() will return zero for objects over 1MB in size.  This is because for objects greater than 1MB Atmos uses Transfer-Encoding: chunked to "stream" the response and does not send a Content-Length header in the response.  You should simply read the stream until you reach EOF.  If you need to know the size of the object before reading it, we suggest you call getSystemMetadata() first to read the value of the "size" field.

Thank you for the response Jason!

Initially I had used the InputStream returned to create a file and the file was of 0kb, and for that reason I wanted to print the content length.

Here is what my code looks like for creating a file (this code writes out 0kb file as the control would not execute the while loop even once):

FileOutputStream output = new FileOutputStream(new File("C:/Sample.pdf"));

  byte[] buff = new byte[1024];

  int bytesRead;

  boolean contentRead = false;

  while ((bytesRead = inputStream.read(buff)) > 0) {

  output.write(buff, 0, bytesRead);

  contentRead = true;

  }

  output.flush();

If I use readObject method instead of readObjectStream:

InputStream stream = this.atmosAPI.readObject(new ObjectId(atmosObjectId), null);

I'm getting null pointer exception shown below. I cant seem to get any download working with the Atmos Client 2.1.5 jar.

java.lang.NullPointerException

  at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.isReadable(AbstractRootElementProvider.java:92)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:345)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory._getMessageBodyReader(MessageBodyFactory.java:315)

  at com.sun.jersey.core.spi.factory.MessageBodyFactory.getMessageBodyReader(MessageBodyFactory.java:294)

  at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:617)

  at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:586)

  at com.emc.atmos.api.jersey.AtmosApiClient.readObject(AtmosApiClient.java:196)

  at com.emc.atmos.api.AbstractAtmosApi.readObject(AbstractAtmosApi.java:53)

...

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

  at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:74)

  at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:82)

  at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:72)

  at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:240)

  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)

  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)

  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)

  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)

  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)

  at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)

  at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)

  at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

  at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:180)

  at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)

  at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)

  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

The getSystemMetadata() returns the size 21122.

If I use the InputStream.class parameter, I get the stream object back. But when I try to read content from Stream - I get Stream Closed error:

java.io.IOException: Stream closed

  at java.io.BufferedInputStream.getBufIfOpen(Unknown Source)

  at java.io.BufferedInputStream.read(Unknown Source)

  at java.io.FilterInputStream.read(Unknown Source)

  at java.io.FilterInputStream.read(Unknown Source)

Another stupid mistake I had done is to use different credentials, basically the object Id I was looking to download existed in a different subtenant. I am surprised why I didn't receive errorCode=1003 and HTTP Status Code 404. Anyways once the issue with credentials is corrected, I used readObjectStream() again and this time it worked like a charm! . I can see this method returns ByteArrayInputStream by default.

Surprising thing is the readObject() method still has issues. For now since I got readObjectStream() working, I'm going to continue using that. However I would need to circle back to this if I run into trouble handling large files (about 200 MB size) with out-of-memory issues.

Thank you once again for your time!

Glad you're up and running.  I'm not sure what the issue is that you're seeing with readObject, but I would recommend upgrading to the latest atmos-client (2.2.2).  If you're using a Maven-compatible system, the artifact is:

    com.emc.vipr  

    atmos-client  

    2.2.2  

Otherwise, you can download the latest from this page.

We did try to use the atmos-client jar 2.2.2 version, but the distributed jar requires Java 7+ but our systems are still using Java 6 and it involves spending additional effort (and money ) trying to upgrade our systems at this point, for that reason we resorted to version 2.1.5 which requires Java 6 as minimum requirement. I believe from version 2.1.6 the minimum requirement is Java 7+.

I had run into same issue of empty stream again yesterday with both readObject and readObjectStream methods. This time when I verified the file via Chrome extension I could see that the Object was created but there was no content in it. Content-length was 0. I also confirmed this by calling getSystemMetadata() and that prints size 0 too.

When I went through the log entries when upload is done I can see following message in the logs:

Content request with input stream and zero-length will not send any data

So I believe the content was never transferred to Cloud. I'm using below Code to upload the file:

FileInputStream inputStream = new FileInputStream(new File("C:/SampleFile.docx"));

CreateObjectRequest req = new CreateObjectRequest();

req.content(inputStream);

req.setAcl(new Acl()); //This may not be required

CreateObjectResponse resp = this.atmosAPI.createObject(req);

ObjectId objectId = resp.getObjectId();

Do I have to use code similar to what we have in com.emc.esu.api.rest.UploadHelper.createObject(InputStream, Acl, MetadataList, boolean) as this uploads the file content as small chunks?

If you're uploading a file, you can use the File object as the content.  That should solve you're problem.

Whenever you upload a stream, you must specify a content-length as well.  The Atmos API requires a content-length header and, aside from buffering the entire stream in memory, there is no other way for the client to know what that is.

Thank you for the response! you were spot on the issue.

Jason's response to this post was the hint to get me to the solution.

It was easy for us to find out the content-length because our source of data was SQL Server (files are stored in a column with image data type). I could run a simple SQL query to get the length of the image column in bytes and set that to the CreateObjectRequest-object.

We did not use any temporary storage and were able to implement a 'pure streaming' from SQL Server to Atmos and successfully transferred few thousand files in our testing. Looking forward to migrate all the data on production (it has about 1.5 million files of various sizes)

It would not have been possible without all of your help! thank you!

Hi Vandan,

You can look at our UploadHelper class.  You can use an ArraySegment to specify data to upload.  You'll first create the object with the first segment then append in subsequent segments.

https://github.com/EMCECS/atmos-dotnet/blob/master/EsuApiLib/UploadHelper.cs#L245

Alternatively, if you know the length of the stream beforehand, there's a CreateObjectFromStream where you can give us the whole stream.

atmos-dotnet/EsuApi.cs at master · EMCECS/atmos-dotnet · GitHub

The implementation is here where we actually buffer the objects from the source stream and write them out to the HTTP stream.

https://github.com/EMCECS/atmos-dotnet/blob/master/EsuApiLib/Rest/EsuRestApi.cs#L481

SNShiva and Stu Arnett,

I am in the same situation where I have to write a large file onto Atmos.

The solution explained here is what I am looking for. I just need one more bit of detail.

How did you implement the "pure streaming" from your source location to Atmos? I have questions about how to write the content in a loop where we will stream only certain number of bytes. And make the receiving side receive it the same way and return only after the whole content is read.

Could you please explain that part?

Thanks,
Vandan

Jason,

Thanks a lot for the explanation and the source links.

Our application is Java based. I use the atmos-client-2.2.2.jar to talk to Atmos. Based on the dot net code that you shared, the logic at the high level is to create the object first and then keep appending the content in a loop until done. I did this in Java and it did work.

Here is the sample code fragment showing the same.

AtmosApi atmos = getAtmosInterface();

byte[] buffer = new byte[1024 * 1024];

File file = new File("C:\\testfile.txt");

FileInputStream fis = new FileInputStream(file);

int len = fis.read(buffer);

if (len != -1) {

   byte[] bytes = Arrays.copyOfRange(buffer, 0, len);

   //Create the Object with initial content read from the file

   ObjectId objectid = atmos.createObject(bytes, "application/octet-stream");

   long contentLength = len;

   //Read the rest of the content in a loop and append it to the object

   len = fis.read(buffer);

   while (len != -1) {

      bytes = Arrays.copyOfRange(buffer, 0, len);

      //Append the object with additional content

      atmos.updateObject(objectid, bytes, new Range(contentLength, (contentLength + len - 1)));

      contentLength += len;

      len = fis.read(buffer);

   }

}

fis.close();

Hope this logic is correct. Please let me know if there is any other better way to do the same.

Thanks,
Vandan

Yes, that's about right.  One optimization: instead of copying the array you can wrap it in a BufferSegment object: atmos-java-client/BufferSegment.java at master · EMCECS/atmos-java-client · GitHub

There's a more complete sample here where we test the checksumming: https://github.com/EMCECS/atmos-java-client/blob/master/src/test/java/com/emc/atmos/api/test/AtmosApiClientTest.java#L1609

Thanks Jason. I have used the BufferSegment and it does work fine and eliminates the unnecessary copying of the array content.

Thanks,
Vandan

All,

What would be recommended size of the buffer for doing the createObject() and for each successive updateObject() calls?

In our case we have some files as large as a few GB that we want to store on Atmos, while most of the other files are just few MB. I would like to know the appropriate size of the chunks that can be written and appended in a loop.

The applications where we are planning to use Atmos is a Web based as well as batch. The peak connections that we have currently is about 7000 users per hour. And about 90% will be READONLY calls.

Please let me know your recommendation.

Thanks,

Vandan

The internal buffer size on Atmos is 1105920 (1080 kiB), so a multiple of this value would result in the most efficient read/write path.