This post is more than 5 years old
14 Posts
0
10585
Atmos-php EMC Storage Object Checksum Value
We tried to locate the checksum value on object's metadata resides on EMC storage without luck.
Thus, we tried to leverage on the Atmos-php method to ReadObject from EMC storage to perform the checksum value calculation as below:
$ck = new Checksum( 'SHA0' ); | |
$ck->update( $_files->ReadObject($id, null, null, null) ); |
Unfortunately, this is not an efficient/acceptable way especially on huge file. As this will incur bandwidth to download the object to perform such checksum calculation.
Please advise is there a way to perform this checksum calculation at EMC storage end without download it back locally?
Thank you.
Jason Cwik
JasonCwik
281 Posts
1
May 17th, 2013 10:00
Unfortunately, there's no way to request Atmos to compute a checksum for existing objects. However, you can use the checksum feature when uploading objects and that checksum will be validated by Atmos and can be viewed at a later time through the object's metadata.
HybridCloud
14 Posts
0
May 17th, 2013 13:00
Thanks Jason! As always!
By the way, are you able to give some pointer at which section/function/script of Atmos-php that I could provide a checksum value to Atmos for validation? Is there checksum mechanism to follow? Such as must be MD5 or SHA1 etc.
Understand at Atmos end the MD5 checksum values are stored in Berkeley DB which is not exposing to external application.
Greatly appreciate your further enlightenment.
Thank you.
JasonCwik
281 Posts
1
May 17th, 2013 13:00
Sure, you use the same Checksum object when uploading to Atmos. Atmos <2.1 supports only SHA0, but with 2.1 we added SHA1 and MD5.
http://code.google.com/p/atmos-php/source/browse/trunk/src/EsuRestApiTest.php#1241
Then, when you read the object back, you should get the x-emc-wschecksum to validate. You can also get it by reading the system metadata for the object.
HybridCloud
14 Posts
0
May 20th, 2013 09:00
Thanks Jason,
I have tested it works, but the correct syntax to used is 'checksum' instead of 'x-emc-wschecksum' at least for atmos-php, for reference.
$onlinecs = $cs->getMetadata( 'checksum' )->getValue();
Due to the following code in EsuRestApi.php line #651
if (! empty($value))
{ $m = new Metadata( 'checksum', $value, false ); $meta->addMetadata( $m ); }
Thank you.
HybridCloud
14 Posts
0
May 21st, 2013 09:00
Jason Cwik
Hi Jason,
I am experiencing updateObject challenge as it doesn't recognize upload resume as a object as follow:
1st attempt of upload obtained SHA0 value SHA0/0/f96cea198ad1dd5617ac084a3d92c6107708c0ef and partially completed
2nd attempt of resume upload submit same SHA0 value and hits with following error
PHP Fatal error: Call to a member function update() on a non-object in /home/atmos-php/src/EsuRestApi.php on line 430
The input of 2nd attempt of resume upload as follow:
Extent: offset: 4194304 size: 236978176
SHA0: SHA0/0/f96cea198ad1dd5617ac084a3d92c6107708c0ef
The code hits error at $checksum->update( $data ); with non-object error
Are you able extend your help to give some pointer here?
P.S.: When I perform an upload resume on a non-SHA0-enable file with updateObject() it works without error.
Thank you.
JasonCwik
281 Posts
1
May 23rd, 2013 13:00
The problem is that the UploadHelper currently isn't smart enough to resume an upload. To properly resume an upload, you must:
1) Query the object's metadata to see what the current checksum algorithm, offset, and value are.
2) Re-initialize the checksum object.
3) Replay the source data into the checksum object up to the server's reported offset.
4) Validate that the local value matches the remote value.
5) Resume uploading from the offset, using the new Checksum object.
You could probably do steps 1-4 yourself and set the checksum object directly on the UploadHelper without setting computeChecksums = true.
HybridCloud
14 Posts
0
June 7th, 2013 12:00
Hi Jason Cwik ,
We are suffering upload performance issue with turning on the checksum feature for SHA0 (due to Atmos version) as follow:
EsuHelpers.php (at Line 77)
private $computeChecksums = true;
Upload result for a 116MB file took 14 minutes.
private $computeChecksums = false;
Upload result for a 235MB file took less than a minute.
As you can see from above, there are a huge different in both with and without SHA0 checksum feature.
Are you able to extend your insight how we could achieve both upload performance and at the same time able to validate the checksum value at both local and online object without re-download it back for calculation?
Is other API language able to achieve that, please advice, in case is the limitation of Atmos-php in handling the checksum feaure?
Thank you!
The upload test result for reference:
Without Checksum
2013-06-07 10:55:44 0MB/230MB 0%
2013-06-07 10:55:44 4MB/230MB 2%
2013-06-07 10:55:45 8MB/230MB 3%
2013-06-07 10:55:45 12MB/230MB 5%
2013-06-07 10:55:45 16MB/230MB 7%
2013-06-07 10:55:46 20MB/230MB 9%
2013-06-07 10:55:47 24MB/230MB 10%
2013-06-07 10:55:48 28MB/230MB 12%
2013-06-07 10:55:48 32MB/230MB 14%
2013-06-07 10:55:48 36MB/230MB 16%
2013-06-07 10:55:50 40MB/230MB 17%
2013-06-07 10:55:50 44MB/230MB 19%
2013-06-07 10:55:51 48MB/230MB 21%
2013-06-07 10:55:51 52MB/230MB 23%
2013-06-07 10:55:52 56MB/230MB 24%
2013-06-07 10:55:53 60MB/230MB 26%
2013-06-07 10:55:53 64MB/230MB 28%
2013-06-07 10:55:54 68MB/230MB 30%
2013-06-07 10:55:55 72MB/230MB 31%
2013-06-07 10:55:55 76MB/230MB 33%
2013-06-07 10:55:55 80MB/230MB 35%
2013-06-07 10:55:56 84MB/230MB 37%
2013-06-07 10:55:57 88MB/230MB 38%
2013-06-07 10:55:57 92MB/230MB 40%
2013-06-07 10:55:58 96MB/230MB 42%
2013-06-07 10:55:58 100MB/230MB 43%
2013-06-07 10:55:59 104MB/230MB 45%
2013-06-07 10:56:00 108MB/230MB 47%
2013-06-07 10:56:00 112MB/230MB 49%
2013-06-07 10:56:01 116MB/230MB 50%
2013-06-07 10:56:01 120MB/230MB 52%
2013-06-07 10:56:03 124MB/230MB 54%
2013-06-07 10:56:03 128MB/230MB 56%
2013-06-07 10:56:05 132MB/230MB 57%
2013-06-07 10:56:05 136MB/230MB 59%
2013-06-07 10:56:06 140MB/230MB 61%
2013-06-07 10:56:06 144MB/230MB 63%
2013-06-07 10:56:07 148MB/230MB 64%
2013-06-07 10:56:07 152MB/230MB 66%
2013-06-07 10:56:08 156MB/230MB 68%
2013-06-07 10:56:08 160MB/230MB 70%
2013-06-07 10:56:09 164MB/230MB 71%
2013-06-07 10:56:09 168MB/230MB 73%
2013-06-07 10:56:11 172MB/230MB 75%
2013-06-07 10:56:12 176MB/230MB 77%
2013-06-07 10:56:12 180MB/230MB 78%
2013-06-07 10:56:12 184MB/230MB 80%
2013-06-07 10:56:14 188MB/230MB 82%
2013-06-07 10:56:14 192MB/230MB 83%
2013-06-07 10:56:14 196MB/230MB 85%
2013-06-07 10:56:17 200MB/230MB 87%
2013-06-07 10:56:17 204MB/230MB 89%
2013-06-07 10:56:18 208MB/230MB 90%
2013-06-07 10:56:18 212MB/230MB 92%
2013-06-07 10:56:19 216MB/230MB 94%
2013-06-07 10:56:19 220MB/230MB 96%
2013-06-07 10:56:20 224MB/230MB 97%
2013-06-07 10:56:20 228MB/230MB 99%
2013-06-07 10:56:21 230MB/230MB 100%
With Checksum
2013-06-07 11:10:49 0MB/230MB 0%
2013-06-07 11:10:49 4MB/230MB 2%
2013-06-07 11:11:20 8MB/230MB 3%
2013-06-07 11:11:50 12MB/230MB 5%
2013-06-07 11:12:20 16MB/230MB 7%
2013-06-07 11:12:49 20MB/230MB 9%
2013-06-07 11:13:21 24MB/230MB 10%
2013-06-07 11:13:51 28MB/230MB 12%
2013-06-07 11:14:21 32MB/230MB 14%
2013-06-07 11:14:51 36MB/230MB 16%
2013-06-07 11:15:21 40MB/230MB 17%
2013-06-07 11:15:51 44MB/230MB 19%
2013-06-07 11:16:21 48MB/230MB 21%
2013-06-07 11:16:52 52MB/230MB 23%
2013-06-07 11:17:22 56MB/230MB 24%
2013-06-07 11:17:51 60MB/230MB 26%
2013-06-07 11:18:21 64MB/230MB 28%
2013-06-07 11:18:51 68MB/230MB 30%
2013-06-07 11:19:21 72MB/230MB 31%
2013-06-07 11:19:52 76MB/230MB 33%
2013-06-07 11:20:24 80MB/230MB 35%
2013-06-07 11:20:53 84MB/230MB 37%
2013-06-07 11:21:24 88MB/230MB 38%
2013-06-07 11:21:55 92MB/230MB 40%
2013-06-07 11:22:25 96MB/230MB 42%
2013-06-07 11:22:55 100MB/230MB 43%
2013-06-07 11:23:25 104MB/230MB 45%
2013-06-07 11:23:55 108MB/230MB 47%
2013-06-07 11:24:25 112MB/230MB 49%
2013-06-07 11:24:55 116MB/230MB 50%
JasonCwik
281 Posts
1
June 7th, 2013 12:00
Are you on Atmos 2.1? If so, please try "SHA1" or "MD5". The problem is that SHA0 is not widely available as a native algorithm so it needs to be implemented in software. PHP has some limitations with its integer math that compounds this issue. Using the other algorithms should help since native implementations can be used. If you're not yet on 2.1, using a different language should provide better checksumming performance even with SHA0.
HybridCloud
14 Posts
0
June 10th, 2013 07:00
Thanks Jason Cwik as always!
Managed to use other language for the checksumming operation and saw significant improvement over the native PHP code handling of the checksum operation.
Also, managed to replay the file and make resume operation fully functional.
Cheers!
amarcionek
56 Posts
0
October 18th, 2013 11:00
Can we get an official word as to whether SHA-1 is supported in Atmos 2.1? The API documentation 300-013-493.pdf for 2.1 does not list SHA-1.
chris_arnett
110 Posts
0
October 18th, 2013 12:00
Yes, Atmos 2.1 supports SHA0, SHA1 and MD5 checksums.
testing111
1 Message
0
September 12th, 2019 10:00