ECS: NFS Remote I/O errors; Change of bucket owner for FS enabled bucket may lead to applications and or users unable to access NFS files
Summary: Previous Bucket Owner is not allowed or restricted ObjectControllerException: Method updateObjectInternal not allowed for previous bucket owner
This article applies to
This article does not apply to
Symptoms
Change made by user to the bucket owner page on the user interface:
This issue applies to NFS enabled buckets and a change of the bucket owner by the user interface. This may cause applications or users connected to loss access to the bucket on the Linux file system. Even if we revert the change back to the original owner again access will not be possible leading to DU.
In this example:
The bucket owner was changed to "sham2" using the user interface. Due to a limitation in ECS, even after changing the bucket owner name back to "sham1." The ECS will not change the bucket owner back to "sham1" using the user interface. This can only be done for now with the CLI using an API with payload to resetowner flag to true.
Ways to identify the issue, on the Linux machine ask user to touch a file for example:
This issue applies to NFS enabled buckets and a change of the bucket owner by the user interface. This may cause applications or users connected to loss access to the bucket on the Linux file system. Even if we revert the change back to the original owner again access will not be possible leading to DU.
In this example:
The bucket owner was changed to "sham2" using the user interface. Due to a limitation in ECS, even after changing the bucket owner name back to "sham1." The ECS will not change the bucket owner back to "sham1" using the user interface. This can only be done for now with the CLI using an API with payload to resetowner flag to true.
Ways to identify the issue, on the Linux machine ask user to touch a file for example:
admin@node1~>touch file
touch: setting times of `file': Remote I/O error
se svc_log with the string "method updateObjectInternal "
Command:
# svc_log -a -sr dataheadsvc | grep "method updateObjectInternal"
Example:
admin@node1~>svc_log -a -sr dataheadsvc | grep "method updateObjectInternal" svc_log v1.0.22 (svc_tools v1.5.3) Started 2019-06-06 10:45:04 Running on nodes: <All nodes> Time range: 2019-06-05 10:45:04 - 2019-06-06 10:45:04 Filter string(s): <All messages> Show nodename(s): True Search reclaim logs (if any): False com.emc.storageos.data.object.exception.ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner sham1 Caused by: com.emc.storageos.data.object.exception.ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner sham1
Cause
Creating a bucket with a specific user as owner, followed by a change of the ownership of the bucket. Lastly, giving the original owner full control using the ACLs page fails with the ECS log exception:
ObjectControllerException: method updateObjectInternal not allowed for previous bucket owner <ownerid> This is a known issue currently being evaluated by Dell EMC at this time.
Resolution
The workaround is to change the bucket owner using the API through CLI with payload to resetowner flag to true.
1. Determine the current bucket owner.
2. Create a simple xml file by using the vi editor. In the example below it is called, /tmp/bucket-owner.xml. This, is a two-step process. We have to set it to a new owner temporarily of sham2. As in this example below, before setting back to original owner sham1, confirm the output:
. 7. Once the config change is done, we should not see the same error any more
1. Determine the current bucket owner.
Require the user interface root password to generate the TOKEN. For example:
admin@ecsnode1:~> tok=$(curl -iks https://XX.XX.XX.XX:4443/login -u 'root:ChangeMe' | grep X-SDS-AUTH-TOKEN)
Verify the current bucket owner (substitute bucket and namespace in your situation):
admin@node1:~> curl -s -k -X GET -H "$tok" https://XX.XX.XX.XX:4443/object/bucket/sham_bk_nfs/info?namespace=degreat_nfs | xmllint --format - | grep '<owner>' <owner>sham2</owner>
This confirms parameter reset_previous_owners is required to be set to true. The reverted bucket owner is on the user interface, but the API through CLI confirms that the ECS is still seeing the bucket owner as "sham2."
2. Create a simple xml file by using the vi editor. In the example below it is called, /tmp/bucket-owner.xml. This, is a two-step process. We have to set it to a new owner temporarily of sham2. As in this example below, before setting back to original owner sham1, confirm the output:
admin@node1:~ # vi /tmp/bucket-owner.xml admin@ecsnode1:~ # cat /tmp/bucket-owner.xml <object_bucket_update_owner> <namespace>degreat_nfs</namespace> <new_owner>sham2</new_owner> <reset_previous_owners>true</reset_previous_owners> </object_bucket_update_owner> 3. Change the bucket owner to the temporary owner.
The API syntax required to change the bucket owner to "sham2" through the xml file is as follows:
admin@ecsnode1:~> curl -v -k -X "POST" "https://xx.xx.xx.xx:4443/object/bucket/sham_bk_nfs/owner" -H "$tok" -H "Content-Type: application/xml" -H "ACCEPT:application/xml" -d @/tmp/bucket-owner.xml -v * Hostname was NOT found in DNS cache * Trying xx.xx.xx.xx... * Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 4443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS unknown, Certificate Status (22): * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, Certificate (11): * SSLv3, TLS handshake, Server key exchange (12): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * Server certificate: * subject: CN=localhost * start date: 2019-03-25 09:53:41 GMT * expire date: 2029-03-22 09:53:41 GMT * issuer: CN=localhost * SSL certificate verify result: self signed certificate (18), continuing anyway. > POST /object/bucket/sham_bk_nfs/owner HTTP/1.1 > User-Agent: curl/7.37.0 > Host: xx.xx.xx.xx:4443 > X-SDS-AUTH-TOKEN: BAAcUy9KYlhxTlVYb2M0bnF3bTNscEsvSEdDeWhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmJhOGQ3ZTkzLTMyMGYtNDNmNy05Y2FkLWM4YWQzMWFiMzY1MAIADTE1NTk3Mzk3OTA2MDgDAC51cm46VG9rZW46YjQ4NGNiZjEtNTkwNy00YWI3LTgzYTctM2Y3OGRhM2RiY2NiAgAC0A8= > Content-Type: application/xml > ACCEPT:application/xml > Content-Length: 179 > * upload completely sent off: 179 out of 179 bytes < HTTP/1.1 200 OK < Date: Thu, 06 Jun 2019 10:56:08 GMT < Content-Length: 0 < Connection: keep-alive < * Connection #0 to host xx.xx.xx.xx left intact 4. Edit the simple.xml file previously created in step 2 and this time insert original owner of sham1
admin@node1:~ # vi /tmp/bucket-owner.xml admin@ecsnode1:~ # cat /tmp/bucket-owner.xml <object_bucket_update_owner> <namespace>degreat_nfs</namespace> <new_owner>sham1</new_owner> <reset_previous_owners>true</reset_previous_owners> </object_bucket_update_owner> 5. Change the bucket owner back to the original owner The API syntax required to change the bucket owner back to "sham1" through the xml file is as follows:
admin@ecsnode1:~> curl -v -k -X "POST" "https://xx.xx.xx.xx:4443/object/bucket/sham_bk_nfs/owner" -H "$tok" -H "Content-Type: application/xml" -H "ACCEPT:application/xml" -d @/tmp/bucket-owner.xml -v * Hostname was NOT found in DNS cache * Trying xx.xx.xx.xx... * Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 4443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS unknown, Certificate Status (22): * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, Certificate (11): * SSLv3, TLS handshake, Server key exchange (12): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * Server certificate: * subject: CN=localhost * start date: 2019-03-25 09:53:41 GMT * expire date: 2029-03-22 09:53:41 GMT * issuer: CN=localhost * SSL certificate verify result: self signed certificate (18), continuing anyway. > POST /object/bucket/sham_bk_nfs/owner HTTP/1.1 > User-Agent: curl/7.37.0 > Host: xx.xx.xx.xx:4443 > X-SDS-AUTH-TOKEN: BAAcUy9KYlhxTlVYb2M0bnF3bTNscEsvSEdDeWhJPQMAjAQASHVybjpzdG9yYWdlb3M6VmlydHVhbERhdGFDZW50ZXJEYXRhOmJhOGQ3ZTkzLTMyMGYtNDNmNy05Y2FkLWM4YWQzMWFiMzY1MAIADTE1NTk3Mzk3OTA2MDgDAC51cm46VG9rZW46YjQ4NGNiZjEtNTkwNy00YWI3LTgzYTctM2Y3OGRhM2RiY2NiAgAC0A8= > Content-Type: application/xml > ACCEPT:application/xml > Content-Length: 179 > * upload completely sent off: 179 out of 179 bytes < HTTP/1.1 200 OK < Date: Thu, 06 Jun 2019 10:56:08 GMT < Content-Length: 0 < Connection: keep-alive < * Connection #0 to host xx.xx.xx.xx left intact 6. Confirm the bucket owner change is reflected.
Confirm that the change of bucket owner is now "sham1."
admin@ecsnode1:~> curl -s -k -X GET -H "$tok" https://XX.XX.XX.XX:4443/object/bucket/sham_bk_nfs/info?namespace=degreat_nfs | xmllint --format - | grep '<owner>' <owner>sham1</owner>
Once the bucket owner is reverted on the API, confirm that the host can now access the bucket on the Linux file system.
. 7. Once the config change is done, we should not see the same error any more
svc_log -f "method updateObjectInternal not allowed" -start "20 hour ago" -sr all -sh -st hour svc_log v1.0.22 (svc_tools v1.6.8) Started 2020-01-23 09:28:17 Running on nodes: <All nodes> Time range: 2020-01-22 13:28:17 - 2020-01-23 09:28:17 Filter string(s): 'method updateObjectInternal not allowed' Show nodename(s): True Search reclaim logs (if any): False Count of message occurrences per hour: 2020-01-22 13:xx - 5066 2020-01-22 14:xx - 9580 2020-01-22 15:xx - 9574 2020-01-22 16:xx - 9580 2020-01-22 17:xx - 9570 2020-01-22 18:xx - 9576 2020-01-22 19:xx - 9564 2020-01-22 20:xx - 9576 2020-01-22 21:xx - 9576 2020-01-22 22:xx - 9572 2020-01-22 23:xx - 9564 2020-01-23 00:xx - 9586 2020-01-23 01:xx - 9574 2020-01-23 02:xx - 9572 2020-01-23 03:xx - 4564 2020-01-23 04:xx - 0 2020-01-23 05:xx - 0 2020-01-23 06:xx - 0 2020-01-23 07:xx - 0 2020-01-23 08:xx - 0 2020-01-23 09:xx - 0 Dell EMC is aware of this issue and are working on a fix in a future release.
Additional Information
Related NFS KB:
You can subscribe to updates by following the instructions in the Knowledge Article below:
DELL EMC: How to subscribe to Product Pages - Dell Support?
- ECS: How to create a basic NFS export and mount it on a client
- ECS: NFS unable to write or delete objects
- ECS: Large NFS writes from ESXi may fail after 3.2 upgrade
- ECS: NFS Export Configuration Reset After Adding New Export in Multi-VDC Environment
- ECS: dataheadsvc log streaming error: NFSv3 procedure LINK not supported in request ReadLinkRequest
- ECS: Duplicate cookie error when listing NFS
- ECS: NFS mount failing with No such file or directory or ERROR_OBJECT_NOT_FOUND
- ECS: File listing of NFS bucket containing more than 2 million files may be slow or fail
- ECS: Sub folder or directory created using S3 Browser is not listed by NFS Client/Linux machine but sub folder content appears in main or root directory
- ECS: Remote I/O error received when attempting to write to a bucket with NFS
- ECS: NFS Remote I/O errors; Change of bucket owner for FS enabled bucket may lead to applications/users unable to access NFS files
- ECS: NFS write throws I/O error after a certain amount of data.
- ECS: Using NFS file share from ECS with a VMware NFS datastore
- ECS: Best practices for mounting ECS NFS exports
- ECS: How to mount NFS share on Windows client
- ECS: NFS fails to mount after changing file export settings in the UI
- ECS: Is Oracle WebCenter Content (WCC) compatible with ECS?
You can subscribe to updates by following the instructions in the Knowledge Article below:
DELL EMC: How to subscribe to Product Pages - Dell Support?