- Notes, cautions, and warnings
- S3
- Amazon S3 API support in ECS
- S3 API supported and unsupported features
- Bucket policy support
- Object Tagging
- S3 Object Lock
- Object lifecycle management
- S3 Extensions
- Metadata Search
- S3 and Swift Interoperability
- Create and manage secret keys
- Authenticating with the S3 service
- Using s3curl with ECS
- Use SDKs to access the S3 service
- ECS S3 error codes
- Hadoop S3A for ECS
- Enabling data2 IP in ECS S3
- Cloud DVR
- ECS IAM for S3
- OpenStack Swift
- EMC Atmos
- CAS
- Setting up CAS support in ECS
- Cold Storage
- Compliance
- CAS retention in ECS
- Advanced retention for CAS applications: event-based retention, litigation hold, and the min/max governor
- Set up namespace retention policies
- Create and set up a bucket for a CAS user
- Set up a CAS object user
- Set up bucket ACLs for CAS
- ECS Management APIs that support CAS users
- Content Addressable Storage (CAS) SDK API support
- ECS CAS error codes
- Enabling data2 IP in CAS
- ECS Management REST API
- ECS HDFS
- ECS HDFS Introduction
- Configuring Hadoop to use ECS HDFS
- Hadoop authentication modes
- Migration from a simple to a Kerberos Hadoop cluster
- File system interaction
- Supported Hadoop applications
- Integrate a simple Hadoop cluster with ECS HDFS
- Install Hortonworks HDP using Ambari
- Create a bucket for HDFS using the ECS Portal
- Plan the ECS HDFS and Hadoop integration
- Obtain the ECS HDFS installation and support package
- Deploy the ECS HDFS Client Library
- Configure ECS client properties
- Set up Hive
- Verify Hadoop access to ECS
- Secure the bucket
- Relocate the default file system from HDFS to an ECS bucket
- Integrate a secure Hadoop cluster with ECS HDFS
- Verify that AD/LDAP is correctly configured with a secure Hadoop cluster
- Pig test fails: unable to obtain Kerberos principal
- Permission denied for AD user
- Permissions errors
- Failed to process request
- Enable Kerberos client-side logging and debugging
- Debug Kerberos on the KDC
- Eliminate clock skew
- Configure one or more new ECS nodes with the ECS service principal
- Workaround for Yarn directory does not exist error
- Set up the Kerberos KDC
- Configure AD user authentication for Kerberos
- Secure bucket metadata
- Hadoop core-site.xml properties for ECS HDFS
- Hadoop core-site.xml properties for ECS S3
- External key management
- Document feedback
ECS 3.6.2 Data Access Guide
Example Hadoop and ECS bucket permissions
Examples are provided in this topic to demonstrate the relationship between Hadoop users/groups and the users/groups that are assigned permission to access the bucket using ECS User ACLs and Custom Group ACLs.
When a bucket is created, ECS automatically assigns ACLs to the bucket owner and to the default group, which is the group assignment for the bucket when accessed using HDFS. A bucket must always have an owner, however, a bucket does not require an assigned default group. Users and groups other than the bucket owner, that is, Custom Groups, can be assigned ACLs on the bucket. ACLs assigned in this way translate to permissions for Hadoop users.
| Hadoop users and groups | Bucket permissions | Bucket/file system access |
|---|---|---|
| Bucket access using Group ACL | ||
|
|
Custom Group ACLs must be set on the bucket in the ECS Portal assign Full Control on the bucket/root file system to the hadoop,users, hive, and spark groups. This example assumes that hdfs is the superuser - the user that started the namenode. |
| Bucket created by s3 user - crosshead access | ||
|
|
Where you want objects written by an S3 user to be accessible as files from HDFS, a default group must be defined (hadoop) so that Hadoop users and services have permissions on the files due to group membership. The default group automatically has Custom Group ACLs on the bucket/file system. The following example shows that
hadoop has been set default group and the root file system permissions are 777:
drwxrwxrwx+ - s3user hadoop 0 2018-03-09 12:28 / You can give users access either by adding User ACLs or by adding Custom Group ACLs for the group to which the users belong. |
| Hadoop user | Bucket perrmissions | Bucket/file system access |
|---|---|---|
|
|
Custom Group ACLs set on the bucket in the Portal enable the hadoop and users group to have permissions on the bucket/root file system. User information from the Hadoop cluster must be available to ECS so that it can provide secure access to the bucket. This information is provided using bucket metadata and an example metadata file is provided in Secure bucket metadata. |
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\