Dell EMC Data Mover Admin Guide

Upgrade Data Mover from a version earlier than 2.0-87

When upgrading Data Mover from a version earlier than 2.0-87, additional steps are required to update the Data Mover configuration file.

1. Transfer the Data Mover installation package to a folder on the DataIQ host, for example, /root.

   cd /root/ 
   cp /yourdrive/data-mover-2.0-88-installer.tar.gz .

2. On the DataIQ host, extract the DataIQ installer file:

   tar -zxvf data-mover-2.0-88-installer.tar.gz 
   cd data-mover-2.0-88/

3. Save a copy of the configuration file:

   • UI method: From DataIQ, select Settings > Data management configuration. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Data Mover, and select Edit configuration. Copy the content of the configuration file and paste it into a text editor. Save this file to a location on the DataIQ host machine.

   • Command-line interface method: On the DataIQ host, save a copy of the configuration file:

     cp /opt/dataiq/maunakea/data/plugins/data-mover/.configs/ca.control original_config.txt

4. Delete the old version of Data Mover and install the new one.

   python data_mover.py delete
   python data_mover.py install

5. Open both the old and new configuration files in the text editor of your choice.

   • UI method: From DataIQ, select Settings > Data management configuration. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Data Mover, and select Edit configuration. Select all content of the configuration file, copy all and then paste into a text editor. Save this file to some location on the DataIQ host machine.

   • Command-line method: On the DataIQ host, save a copy of the active configuration file:

     cp /opt/dataiq/maunakea/data/plugins/data-mover/.configs/ca.control new_config.txt

6. Compare the old and new configuration files. Add back settings as needed.
   NOTE Use these instructions to modify configurations through the DataIQ UI. Do not modify the configuration file directly from the command line while DataIQ is running, because the changes may not be applied and could break the plugin functionality.

   It may be helpful to use a text-comparison tool such as Vim, to compare the two files. Example:

   vim -d original_config.txt new_config.txt

   The following settings often need to be copied over:

   • The lists of groups allowed to see and interact with the plugin.
   • The entire vol_attributes section.
   • The entire S3 buckets section.
   • password
   • error_handling
   • regex_exclude_list
   • Any other values which you want to keep that differ from the default.

   NOTE If you use any of these settings, we recommend the following updates:

   • task_status_timeout: Use a value of 120 (or more). The previous default of 60 might cause workers to be timed out too soon.
   • max_fs_transfer_tries and max_s3_transfer_tries: Use a value of 10 (or more) to increase job resiliency.
   • heavy_task_attempts: Use a value of 1000 (or more) to increase job resiliency.
   • s3_multipart_threshold and s3_multipart_chunksize: Use a value of 511 MiB (or less) to increase job performance and reduce chances of timeouts while transferring large files.
   • password: Do not use the default password. Change this password, even if no external workers are used. Keep this password safe using your organization's recommended security practices.

   Paste the new values into the UI. When you are finished editing settings, select Save.

7. From the DataIQ browser, select Settings > Data management configuration. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Data Mover, and select Enable.
8. Upgrade external worker nodes:

   Transfer the file: data-mover-workers-2.0-88.x86_x64.rpm to each worker node.

   Upgrade the worker processes:

   rpm -Uvh data_mover_workers-2.0-88.x86_64.rpm

   NOTE Although this process does not change the local configuration file (/usr/local/data_mover_workers/etc/workers.cfg), it is recommended to review the file to make sure that settings are the way you would like them to be.