Skip to main content
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Create and access a list of your products

OneFS Attempts to use the Apache Hadoop YARN node label feature fail

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.

Symptoms



Issue

When attempting to use the Apache Hadoop YARN node label feature, the action fails and OneFS generates an error message.


Symptoms

When attempting to use the YARN node label feature with OneFS 7.2.1.x and Apache Hadoop 2.7, the users may receive an  -appendToFile: Fatal internal error  message.
 

Environment

OneFS 7.2.1.x and Hadoop 2.7

Cause

Cause

By default, Hadoop stores node label information in a directory on the HDFS cluster.  Isilon OneFS 7.2.1.x does not recognize the  appendtofile parameter sent from Hadoop 2.7 and may generate errors when users attempt to use the YARN node label feature.

Resolution


Resolution

To resolve this issue, do one of the following:
  • OneFS 7.2.1.2

    • Implement the workaround provided below
  • OneFS 7.2.1.1

    • Apply patch-164059 first, and then implement the workaround provided below

Workaround

Implement the following workaround to manually create and use a YARN node label directory that resides on the OneFS cluster rather than the HDFS cluster.  Note: If you have ran the isilon_hadoop_tools script on GitHub, you do not need to follow this workaround, as the script automatically creates and permissions the correct directory.

Step 1. Prepare OneFS to store node label information
Perform the following steps on the OneFS cluster.
Note: In the following procedure, HDFS_ROOT_DIR refers to your HDFS root directory on the OneFS cluster.

  1. Create the new directory on OneFS to host the node label store.
    • mkdir -p HDFS_ROOT_DIR/system/yarn/node-labels
  2. Confirm that the user  yarn  exists.Note: If you have not already created a user called  yarn , create one now.  For information on creating users, see the  Managing local users and groups  section of the Isilon OneFS CLI Administration Guide.
    • isi auth users list
  3. Make the user  yarn  the owner of the new directory.
    • chown yarn HDFS_ROOT_DIR/system/yarn/node-labels
  4. Change the permission level of the directory to allow the owner to read, write, and execute.
    • chmod 700 HDFS_ROOT_DIR/system/yarn/node-labels
 
(Optional) Step 2: Direct Apache Hadoop to store node label information on OneFS
Perform the following steps on the HDFS cluster.
  1. Using a browser, log on to Ambari Manager on your HDFS cluster.
  2. From the top banner of the Ambari Dashboard, click Services.
  3. From the left panel, Click YARN.
  4. Click the Configs tab.
  5. Click the Settings tab.
  6. Under YARN Features, confirm node labels are Enabled.
  7. Click Save.
  8. Click on the Advanced tab.
  9. Expand the Advanced yarn-site section.
  10. In the yarn.node-labels.fs-store.root-dir field, confirm that the default  /system/yarn/node-labels  is listed.
  11. Click Save.

Additional Information

Additional information

Apache information on the Hadoop 2.7 issue:
issues.apache.org/jira/browse/HDFS-9290
 
Apache documentation on how to use Yarn s node label feature:
hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html
 
Hortonworks documentation on how to use Yarn s node label feature:
docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/ch_node_labels.html

Affected Products

Isilon

Products

Isilon
Article Properties
Article Number: 000052203
Article Type: Solution
Last Modified: 11 Oct 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.