Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

13542

May 18th, 2018 04:00

getting Caused by: java.net.SocketException: Connection reset by peer: socket write error

I am getting Caused by: java.net.SocketException: Connection reset by peer: socket write error

when using multi-part upload with SSECustomerKey using Amazon s3 client sdk

                // Create the request to upload a part.

                UploadPartRequest uploadRequest = new UploadPartRequest()//.withSSECustomerKey(SSE_KEY)

                        .withBucketName(bucketName)

                        .withKey(keyName)

                        .withUploadId(initResponse.getUploadId())

                        .withPartNumber(i)

                        .withInputStream(is)

                        .withPartSize(partSize)

                        .withSSECustomerKey(SSE_KEY);

                UploadPartResult uploadResult = S3_CLIENT.uploadPart(uploadRequest);

while upload works fine with single part upload

PutObjectRequest putRequest = new PutObjectRequest(bucketName, keyName, is, metadata).withSSECustomerKey(SSE_KEY);

S3_CLIENT.putObject(putRequest);

why is dell emc ecs throwing java.net.SocketException: Connection reset by peer: socket write error

110 Posts

May 23rd, 2018 06:00

Thanks for posting that code.  This is exactly what we needed to see.  The problem is that you are specifying SSE (with server-managed key) in the init request, but specifying SSE-C (customer-managed key) in the part uploads.  If you want to use SSE-C (where you manage your own keys), you need to change your init call to the following:

// DELETE THIS LINE

//objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);

// Initiate the multipart upload.

InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName,

        objectMetadata);

initRequest.setSSECustomerKey(SSE_KEY); // ADD THIS LINE

InitiateMultipartUploadResult initResponse = S3_CLIENT.initiateMultipartUpload(initRequest);

If you want to use SSE (with server-managed keys), then remove the customer managed key assignment in the part uploads (it will inherit the setting from the init request).

16 Posts

May 18th, 2018 09:00

If you are using ECS 3.2.0.0, this is a known bug with Multi-part uploads.

The fix will be in v3.2.0.1, which should be released 22-May.

Normal uploads will work as will multi-part uploads if the URL is not percent-encoded. For 3.2.0.0, the work-around is to avoid using percent-encoding in multi-part uploads.

May 18th, 2018 10:00

ok thanks. I'll wait for v3.2.0.1.


"For 3.2.0.0, the work-around is to avoid using percent-encoding in multi-part uploads."

-- But I am using ip and port directly like to create AmazonS3 client.

EndpointConfiguration endpointConfiguration = new EndpointConfiguration("*.*.*.*:9021", "US_WEST_2");

Which URL should be percent-encoded?

16 Posts

May 21st, 2018 12:00

I'm talking about the URL that is the MPU request. For instance:

POST /Bucket1/object.name?uploads

It is common to percent-encode the 'dot' in the URL above:

POST /Bucket1/object%2Ename?uploads

If the URL does contain a percent-encoded character (such as the %2E above), the MPU will fail until you upgrade to ECS 3.2.0.1.

May 22nd, 2018 05:00

Yes but I am doing using aws-java-sdk-s3  and not through rest url and my keyname and bucketName doesn't contain any special characters. As said before that It works for me without .withSSECustomerKey(SSE_KEY) but with server side encryption S3_CLIENT.uploadPart fails 

16 Posts

May 22nd, 2018 06:00

Your original note stated that the upload failed as a multi-part, but worked as a single upload. In BOTH cases you were using withSSECustomerKey(SSE_KEY). It never states that you were successful without "withSSECustomerKey(SSE_KEY)"


From reading the bug report, there should be no association of this error with SSE. Please verify this behavior, and be careful that any request that has no percent-encoded will cause the multi-part upload to succeed.

May 22nd, 2018 09:00

my bad if post is misinterpreted . I meant uploadpart is successful without "withSSECustomerKey(SSE_KEY)". I have tried it number of times but upload part is continuously failing with withSSECustomerKey(SSE_KEY). I am not sure whether withSSECustomerKey(SSE_KEY) introduces any percent-encoded due to encryption but same request is passing without withSSECustomerKey(SSE_KEY).

110 Posts

May 22nd, 2018 15:00

Since you are using the AWS SDK, you have access to the TransferManager class.  Using this to upload MPU with SSE-C against ECS Test Drive (ECS v3.2.0) works for me.  Here is my code excerpt:

        // Create an encryption key.

        KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");

        keyGenerator.init(256, new SecureRandom());

        SSECustomerKey encKey = new SSECustomerKey(keyGenerator.generateKey());

        ObjectMetadata metadata = new ObjectMetadata();

        metadata.setContentLength(data.length);

        // only way to use SSE-C is to use a POR:

        PutObjectRequest putRequest = new PutObjectRequest(testBucket, key, dataStream, metadata);

        putRequest.withSSECustomerKey(encKey);

        // set MPU threshold to 1 to make sure we use MPU

        TransferManager tm = TransferManagerBuilder.standard().withS3Client(client).withMultipartUploadThreshold(1L).build();

        tm.upload(putRequest).waitForUploadResult();

May 23rd, 2018 02:00

But how will TransferManager work with UploadPartRequest object for I don't have single stream but each part is coming as seperate stream of length say 5MB each. Is there something we can do similar to com.emc.vipr.services.s3.model.AppendObjectRequest or com.emc.vipr.services.s3.model.UpdateObjectRequest in com.emc.vipr s3-client package for aws-s3-sdk as well for ViPRS3Client doesn't work with  withSSECustomerKey


The current code is

long partSize = 5 * 1024 * 1024; // Set part size to 5 MB.

// Create a list of ETag objects. You retrieve ETags for each object part

// uploaded,

// then, after each individual part has been uploaded, pass the list of ETags to

// the request to complete the upload.

List partETags = new ArrayList ();


ObjectMetadata objectMetadata = new ObjectMetadata();

// objectMetadata.setContentType("application/x-www-form-urlencoded;

// charset=utf-8");

objectMetadata.setContentLength(contentLength);


objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);


// Initiate the multipart upload.

InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucketName, keyName,

objectMetadata);

InitiateMultipartUploadResult initResponse = S3_CLIENT.initiateMultipartUpload(initRequest);


// Upload the file parts.

long filePosition = 0;

for (int i = 1; filePosition < contentLength; i++) {

// Because the last part could be less than 5 MB, adjust the part size as

// needed.

partSize = Math.min(partSize, (contentLength - filePosition));


byte[] subArray = Arrays.copyOfRange(contentBytes, (int) filePosition, (int) (filePosition + partSize));


// Create the request to upload a part.

UploadPartRequest uploadRequest = new UploadPartRequest()// .withSSECustomerKey(SSE_KEY)

.withBucketName(bucketName).withKey(keyName).withUploadId(initResponse.getUploadId())

.withPartNumber(i).withInputStream(new ByteArrayInputStream(subArray)).withPartSize(partSize)

.withSSECustomerKey(SSE_KEY);//works without withSSECustomerKey

//but fails with Unable to execute HTTP request: Connection reset by peer: socket write error for withSSECustomerKey

boolean isError = false;

while (!isError) {

try {

// Upload the part and add the response's ETag to our list.

UploadPartResult uploadResult = S3_CLIENT.uploadPart(uploadRequest);

partETags.add(uploadResult.getPartETag());

} catch (Exception e) {

System.out.println(e.getMessage());

continue;

}

System.out.println("done upload part");

isError = true;

}

filePosition += partSize;

}


// Complete the multipart upload.

CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucketName, keyName,

initResponse.getUploadId(), partETags);

S3_CLIENT.completeMultipartUpload(compRequest);

May 23rd, 2018 07:00

Thanks for pointing out the mistake that I was doing. I was doing objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION); and then passing it to initRequest but going by suggestion to use

  1. initRequest.setSSECustomerKey(SSE_KEY); // ADD THIS LINE and removing objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);

worked for me and I am able to upload file successfully using UploadPartRequest with .withSSECustomerKey(SSE_KEY) attached screenshot. thanks for quick suggestions.dellemc.png

No Events found!

Top