Dell EMC Data Mover Admin Guide

Set up volumes for Data Mover

Add volume information to the DataIQ host and each external worker.

Connect volumes to DataIQ

Data Mover requires that volumes are connected with read/write permissions. To learn more, see the Dell EMC DataIQ Admin Guide.

Set up volumes

Whenever making updates to volumes, update both the DataIQ host and each external worker.

To update the DataIQ host: From the web UI, select Settings > Data management configuration. In the Plugins section, select the ⋮ (vertical ellipses) icon next to Data Mover, and select Edit configuration. The Data Mover configuration file opens.
To update external workers, see update the configuration file: /usr/local/data_mover_workers/etc/workers.cfg on each worker node.
After configuring a worker node, restart the file transfer service:
```
systemctl restart data_mover_workers
```

Modify the following values:

vol_attributes (required)

Settings for mounted volumes. Do not use vol_attributes to configure S3 buckets.

Format on the DataIQ host:

vol_attributes:
  value:
    "< DataIQ Volume Name >":  # Repeat this section for each volume
      "mount_point": < where the volume is mounted on the DataIQ host >
      "min_remaining_space": < size in bytes >
      "allow_preallocation": < True or False >
      "ignore_attributes": < True or False >

Format on external worker nodes:

vol_attributes:
    "< DataIQ Volume Name >":  # Repeat this section for each volume
      "mount_point": < where the volume is mounted on the DataIQ host >

value: (add to DataIQ host only): Contains volume information.
< DataIQ Volume Name > (add to DataIQ host and external worker nodes): The name of the volume within DataIQ. The name must exactly match the DataIQ volume name, and it must have a forward slash (/) at the beginning of the volume name.
mount_point (add to DataIQ host and external worker nodes): Path to where the volume is mounted locally. This value does not need to match on the host and external worker nodes. To learn more about mounting volumes, see Volumes in the Dell EMC DataIQ Admin Guide .
min_remaining_space (add to DataIQ host only): Size in bytes to ensure remains free on the destination volume. This overrides the setting: default_min_remaining_space.
allow_preallocation (add to DataIQ host only): Enable Data Mover to reserve (preallocate) space before copying bytes to the destination volume. Note: Preallocation on some file systems might be very slow or even impossible.
ignore_attributes (add to DataIQ host only): Chooses whether to preserve or ignore file attributes during transfers.; If either the source or destination volume has ignore_attributes set to True, then attributes are not carried from the source to the destination.; On POSIX-based systems: When this setting is False, Data Mover preserves file and folder attributes when copying files between POSIX-based systems. When transferring files to S3 storage, Data Mover preserves attributes as metadata. When the files are transferred back to a POSIX-based system, the attributes are restored. When this setting is True, attributes are not preserved.; On non-POSIX-based systems, set this to True, because Data Mover does not support preserving attributes between non-POSIX-based systems. If this setting is set to False, Data Mover generates errors when attempting to transfer files between POSIX-based and non-POSIX-based systems.; Note, Data Mover only supports preserving attributes for POSIX-based systems. For non-POSIX systems, attributes will not be preserved on transfer.

The following configuration example contains four file system volumes, their names, their mount points, and optional settings. The first volume uses the defaults for all values. The second volume overrides all the optional defaults. The third and fourth volumes only override a single default.

Example on the DataIQ host:

vol_attributes:
  value:
    "/san01":
      "mount_point": "/mnt/san01"
    "/san02":
      "mount_point": "/mnt/san02"
      "min_remaining_space": 70000
      "allow_preallocation": False
      "ignore_attributes": True
    "/archive01":
      "mount_point": "/mnt/archive01"
      "ignore_attributes": True
    "/archive02":
      "mount_point": "/mnt/archive02
      "allow_preallocation": True

Example on external worker nodes:

vol_attributes:
  "/san01":
    "mount_point": "/mnt/san01"
  "/san02":
    "mount_point": "/mnt/san02"
  "/archive01":
    "mount_point": "/mnt/archive01"
  "/archive02":
    "mount_point": "/mnt/archive02"

S3 buckets (optional)

Format:

"S3 buckets:"
  <On the DataIQ host, add "value:". Do not add "value:" on worker nodes>
    "<bucket_name>": # Repeat this section for each bucket
      "dataiq_volume": "< DataIQ Volume Name >"
      "access_key": "< bucket access key >"
      "secret_key": "< bucket secret key >"
      "endpoint": "< endpoint url >"

< bucket_name > (required): The name of the S3 bucket. The name must match the bucket name that is used on the S3 service. Buckets with the same name are not supported.
dataiq_volume (optional): The name of the volume within DataIQ. Only use this setting when the S3 bucket is already represented as a volume within DataIQ. The name must match the DataIQ volume name, and it must be prepended with a forward slash (/). If this value is not present, the bucket may have files that are transferred to it, but no transfer will be allowed from the bucket to the file system.
access_key (required): The access key for the S3 bucket.
secret_key (required): The secret key for the S3 bucket.
endpoint (required): The endpoint URL of the S3 bucket.

Example for Data Mover Host:

"S3 buckets:"
  value:
    "amazon-s3-bucket": 
      "dataiq_volume": "/amazon_s3"
      "access_key": "123234345456567678789890"
      "secret_key": "ABCDE12345"
      "endpoint": "s3.ap-northeast-1.amazonaws.com"
    "ecs-s3-bucket": 
      "dataiq_volume": "/ecs"
      "access_key": "dataiquser"
      "secret_key": "FGHIJ67890"
      "endpoint": "http://10.123.45.67:9090"
    "google-s3-bucket-3": 
      "dataiq_volume": "/gcp_on_us_west_4"
      "access_key": "12345abcde7890a1b2c3d4e5"
      "secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
      "endpoint": "https://us-west4.storage.googleapis.com"
    "google-s3-bucket-on-asia-east2": 
      "dataiq_volume": "/gcp_on_asia_east_2"
      "access_key": "12345abcde7890a1b2c3d4e5"
      "secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
      "endpoint": "https://asia-east2.storage.googleapis.com"

Example for Data Mover external worker nodes: (... other than the heading "value:", all other values should match the DataIQ host)

"S3 buckets:"
    "amazon-s3-bucket": 
      "dataiq_volume": "/amazon_s3"
      "access_key": "123234345456567678789890"
      "secret_key": "ABCDE12345"
      "endpoint": "s3.ap-northeast-1.amazonaws.com"

    (...etc...)

default_min_remaining_space (optional)

Use default_min_remaining_space to set the default for preserving free space on mounted volumes. This setting defines the minimum number of bytes to remain available on a volume after a data movement. A transfer job is rejected when the total byte size of a transfer, plus the minimum remaining space, is greater than the available space on the destination volume.

This value can be overridden on an individual volume with the setting vol_attributes: min_remaining_space.

NOTE This value may not be respected if multiple transfer jobs are transferring files to the same destination volume. If the total byte size of files from all transfers to the same destination exceed the available space on that destination volume, an error occurs and at least one transfer job to that destination fails.

Example for the DataIQ host:

    default_min_remaining_space:
      value: 262144000

For Data Mover external nodes, Data Mover uses the setting from the DataIQ host.

Welcome

Welcome to Dell