After create life-cycle policy (that has no prefix) on the bucket, GET or PUT or DELETE object requests return 500 internal error. List bucket works fine.
Checking existing policy for object xxx with prefix null shows this in dataheadsvc.log.
2018-04-07T03:30:48,744 [qtp597531831-21452-0a20a979:162908dae63:2fef:17-s3-xx.xx.xx.xx] INFO ObjectExpirationUtil.java (line 33) Checking existing policy for object testfile1 with prefix null
2018-04-07T03:30:48,745 [qtp597531831-21452-0a20a979:162908dae63:2fef:17-s3-xx.xx.xx.xx] ERROR GeneralExceptionMapper.java (line 37) internal error
java.lang.NullPointerException
at java.lang.String.startsWith(String.java:1405)
at java.lang.String.startsWith(String.java:1434)
at com.emc.storageos.data.object.expiration.ObjectExpirationUtil.getObjectExpirationDetails(ObjectExpirationUtil.java:42)
at com.emc.storageos.data.object.RESTAccess.DataAccessLayer.getObjectInfo(DataAccessLayer.java:870)
at com.emc.storageos.data.api.service.impl.resource.s3.operation.S3ObjectOperations.getObject(S3ObjectOperations.java:926)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
... ...
2018-04-07T03:30:48,751 [qtp597531831-21452] INFO RequestLog.java (line 83) xx.xx.xx.xxx GET //xx.xx.xx.xx:9020/test/testfile1 HTTP/1.1 500 9 - 111
For AWS, each rule has <Filter> element which has children element: <Prefix>, <Tag>, <Filter> is required XML element, <Prefix>, and <Tag> is optional (apply to all objects in buckets).
For ECS, we only support filter <prefix> and do not have <Filter> element designed in request XML. <prefix> is not required parameter, but some place code uses prefix value directly(null point) which cause issue here.
If seeing error log like show in "Issue" field, confirm whether life cycle is set on the bucket and no prefix configured. See the
ECS Data Access Guide for further reference of using s3curl.pl.
# ./s3curl.pl --id=test -- http://<node data ip>:9020/<bucket>/?lifecycle |xmllint --format -
# ./s3curl.pl --id=test -- http://<node data ip>:9020/<bucket>/?lifecycle |xmllint --format -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 285 100 285 0 0 23538 0 --:--:-- --:--:-- --:--:-- 23750
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID/>
<Status>Enabled</Status>
<Expiration>
<Days>10</Days>
</Expiration>
<NoncurrentVersionExpiration/>
</Rule>
</LifecycleConfiguration>
If you confirm that hit this issue, the workaround prior to 3.2 is to add <prefix/> in lifecycle.
This issue is fixed in 3.2.
Workaround:
1. Get current lifecycle setting and save to local file
# ./s3curl.pl --id=test -- http://<data node ip>:9020/<bucket>/?lifecycle -o lifecycle.xml
# ./s3curl.pl --id=test -- http://<data node ip>:9020/<bucket>/?lifecycle -o lifecycle.xml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 268 100 268 0 0 11926 0 --:--:-- --:--:-- --:--:-- 12181
2. Modify the lifecycle.xml and add prefix
# vi lifecycle.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID/>
<Prefix/> -->add this line
<Status>Enabled</Status>
<Expiration>
<Days>10</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
3. Update lifecycle
# ./s3curl.pl --id=test --put=lifecycle.xml http://xx.xx.xx.xx:9020/<bucket>/?lifecycle --calculateContentMd5 --debug
# ./s3curl.pl --id=test --put=lifecycle.xml http://xx.xx.xx.xx:9020/fog-bbr-backup/?lifecycle --calculateContentMd5 --debug
s3curl: Found the url: host=xx.xx.xx.xx; port=9020; uri=/fog-bbr-backup/; query=lifecycle;
s3curl: replaced string: lifecycle
s3curl: ordinary endpoint signing case
s3curl: StringToSign='PUT\na4EIw7LnRPKpiAfSy/YHIQ==\n\nSat, 07 Apr 2018 05:11:32 +0000\n/fog-bbr-backup/?lifecycle'
s3curl: exec curl -v -H 'Date: Sat, 07 Apr 2018 05:11:32 +0000' -H 'Authorization: AWS pcf-fog-sys:4NFcmpx/3i0FBoJsFUJSabAlmGI=' -L -H 'content-type: ' -H 'Content-MD5: a4EIw7LnRPKpiAfSy/YHIQ==' -T lifecycle.xml http://xx.xx.xx.xx:9020/fog-bbr-backup/?lifecycle
* Hostname was NOT found in DNS cache
* Trying xx.xx.xx.xx...
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 9020 (#0)
> PUT /fog-bbr-backup/?lifecycle HTTP/1.1
> User-Agent: curl/7.37.0
> Host: xx.xx.xx.xx:9020
> Accept: */*
> Date: Sat, 07 Apr 2018 05:11:32 +0000
> Authorization: AWS pcf-fog-sys:4NFcmpx/3i0FBoJsFUJSabAlmGI=
> Content-MD5: a4EIw7LnRPKpiAfSy/YHIQ==
> Content-Length: 284
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Sat, 07 Apr 2018 05:11:32 GMT
* Server ViPR/1.0 is not blacklisted
< Server: ViPR/1.0
< x-amz-request-id: 0a23a338:1628835dec8:5cb3:19
< x-amz-id-2: a52c5da7cb5555375cef4f9172b03532652afc6cf67528d42451b0a3a20aad4a
< Content-Length: 0
<
* Connection #0 to host xx.xx.xx.xx left intact
4. Confirm lifecycle is updated.
# ./s3curl.pl --id=test -- http://xx.xx.xx.xx:9020/<bucket>/?lifecycle |xmllint --format -
# ./s3curl.pl --id=test -- http://xx.xx.xx.xx:9020/fog-bbr-backup/?lifecycle |xmllint --format -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 285 100 285 0 0 23538 0 --:--:-- --:--:-- --:--:-- 23750
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID/>
<Prefix/>
<Status>Enabled</Status>
<Expiration>
<Days>10</Days>
</Expiration>
<NoncurrentVersionExpiration/>
</Rule>
</LifecycleConfiguration>
5. Retry read/write/delete.