Ga naar hoofdinhoud
  • Snel en eenvoudig bestellen
  • Bestellingen en de verzendstatus bekijken
  • Een lijst met producten maken en openen

OneFS Attempts to use the Apache Hadoop YARN node label feature fail

Dit artikel is van toepassing op Dit artikel is niet van toepassing op Dit artikel is niet gebonden aan een specifiek product. Niet alle productversies worden in dit artikel vermeld.

Symptomen



Issue

When attempting to use the Apache Hadoop YARN node label feature, the action fails and OneFS generates an error message.


Symptoms

When attempting to use the YARN node label feature with OneFS 7.2.1.x and Apache Hadoop 2.7, the users may receive an  -appendToFile: Fatal internal error  message.
 

Environment

OneFS 7.2.1.x and Hadoop 2.7

Oorzaak

Cause

By default, Hadoop stores node label information in a directory on the HDFS cluster.  Isilon OneFS 7.2.1.x does not recognize the  appendtofile parameter sent from Hadoop 2.7 and may generate errors when users attempt to use the YARN node label feature.

Oplossing


Resolution

To resolve this issue, do one of the following:
  • OneFS 7.2.1.2

    • Implement the workaround provided below
  • OneFS 7.2.1.1

    • Apply patch-164059 first, and then implement the workaround provided below

Workaround

Implement the following workaround to manually create and use a YARN node label directory that resides on the OneFS cluster rather than the HDFS cluster.  Note: If you have ran the isilon_hadoop_tools script on GitHub, you do not need to follow this workaround, as the script automatically creates and permissions the correct directory.

Step 1. Prepare OneFS to store node label information
Perform the following steps on the OneFS cluster.
Note: In the following procedure, HDFS_ROOT_DIR refers to your HDFS root directory on the OneFS cluster.

  1. Create the new directory on OneFS to host the node label store.
    • mkdir -p HDFS_ROOT_DIR/system/yarn/node-labels
  2. Confirm that the user  yarn  exists.Note: If you have not already created a user called  yarn , create one now.  For information on creating users, see the  Managing local users and groups  section of the Isilon OneFS CLI Administration Guide.
    • isi auth users list
  3. Make the user  yarn  the owner of the new directory.
    • chown yarn HDFS_ROOT_DIR/system/yarn/node-labels
  4. Change the permission level of the directory to allow the owner to read, write, and execute.
    • chmod 700 HDFS_ROOT_DIR/system/yarn/node-labels
 
(Optional) Step 2: Direct Apache Hadoop to store node label information on OneFS
Perform the following steps on the HDFS cluster.
  1. Using a browser, log on to Ambari Manager on your HDFS cluster.
  2. From the top banner of the Ambari Dashboard, click Services.
  3. From the left panel, Click YARN.
  4. Click the Configs tab.
  5. Click the Settings tab.
  6. Under YARN Features, confirm node labels are Enabled.
  7. Click Save.
  8. Click on the Advanced tab.
  9. Expand the Advanced yarn-site section.
  10. In the yarn.node-labels.fs-store.root-dir field, confirm that the default  /system/yarn/node-labels  is listed.
  11. Click Save.

Extra informatie

Additional information

Apache information on the Hadoop 2.7 issue:
issues.apache.org/jira/browse/HDFS-9290
 
Apache documentation on how to use Yarn s node label feature:
hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html
 
Hortonworks documentation on how to use Yarn s node label feature:
docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/ch_node_labels.html

Getroffen producten

Isilon

Producten

Isilon
Artikeleigenschappen
Artikelnummer: 000052203
Artikeltype: Solution
Laatst aangepast: 11 okt. 2024
Versie:  3
Vind antwoorden op uw vragen via andere Dell gebruikers
Support Services
Controleer of uw apparaat wordt gedekt door Support Services.