Add volume information to the DataIQ host and each external worker.
Connect volumes to DataIQ
Data Mover requires that volumes are connected with read/write permissions. To learn more, see the
Dell EMC DataIQ Admin Guide.
Set up volumes
Whenever making updates to volumes, update both the DataIQ host and each external worker.
To update the DataIQ host: From the web UI, select
Settings > Data management configuration. In the
Plugins section, select the
⋮ (vertical ellipses) icon next to
Data Mover, and select
Edit configuration. The Data Mover configuration file opens.
To update external workers, see update the configuration file:
/usr/local/data_mover_workers/etc/workers.cfg on each worker node.
After configuring a worker node, restart the file transfer service:
systemctl restart data_mover_workers
Modify the following values:
vol_attributes (required)
Settings for mounted volumes. Do not use
vol_attributes to configure S3 buckets.
Format on the DataIQ host:
vol_attributes:
value:
"< DataIQ Volume Name >": # Repeat this section for each volume
"mount_point": < where the volume is mounted on the DataIQ host >
"min_remaining_space": < size in bytes >
"allow_preallocation": < True or False >
"ignore_attributes": < True or False >
Format on external worker nodes:
vol_attributes:
"< DataIQ Volume Name >": # Repeat this section for each volume
"mount_point": < where the volume is mounted on the DataIQ host >
value: (add to DataIQ host only)
Contains volume information.
< DataIQ Volume Name > (add to DataIQ host and external worker nodes)
The name of the volume within DataIQ. The name must exactly match the DataIQ volume name, and it must have a forward slash (/) at the beginning of the volume name.
mount_point (add to DataIQ host and external worker nodes)
Path to where the volume is mounted locally. This value does not need to match on the host and external worker nodes. To learn more about mounting volumes, see
Volumes in the
Dell EMC DataIQ Admin Guide .
min_remaining_space (add to DataIQ host only)
Size in bytes to ensure remains free on the destination volume. This overrides the setting:
default_min_remaining_space.
allow_preallocation (add to DataIQ host only)
Enable Data Mover to reserve (preallocate) space before copying bytes to the destination volume. Note: Preallocation on some file systems might be very slow or even impossible.
ignore_attributes (add to DataIQ host only)
Chooses whether to preserve or ignore file attributes during transfers.
If either the source or destination volume has
ignore_attributes set to
True, then attributes are not carried from the source to the destination.
On POSIX-based systems: When this setting is
False, Data Mover preserves file and folder attributes when copying files between POSIX-based systems. When transferring files to S3 storage, Data Mover preserves attributes as metadata. When the files are transferred back to a POSIX-based system, the attributes are restored. When this setting is
True, attributes are not preserved.
On non-POSIX-based systems, set this to
True, because Data Mover does not support preserving attributes between non-POSIX-based systems. If this setting is set to
False, Data Mover generates errors when attempting to transfer files between POSIX-based and non-POSIX-based systems.
Note, Data Mover only supports preserving attributes for POSIX-based systems. For non-POSIX systems, attributes will not be preserved on transfer.
The following configuration example contains four file system volumes, their names, their mount points, and optional settings. The first volume uses the defaults for all values. The second volume overrides all the optional defaults. The third and fourth volumes only override a single default.
"S3 buckets:"
<On the DataIQ host, add "value:". Do not add "value:" on worker nodes>
"<bucket_name>": # Repeat this section for each bucket
"dataiq_volume": "< DataIQ Volume Name >"
"access_key": "< bucket access key >"
"secret_key": "< bucket secret key >"
"endpoint": "< endpoint url >"
< bucket_name > (required)
The name of the S3 bucket. The name must match the bucket name that is used on the S3 service. Buckets with the same name are not supported.
dataiq_volume (optional)
The name of the volume within DataIQ. Only use this setting when the S3 bucket is already represented as a volume within DataIQ. The name must match the DataIQ volume name, and it must be prepended with a forward slash (/). If this value is not present, the bucket may have files that are transferred to it, but no transfer will be allowed from the bucket to the file system.
Use
default_min_remaining_space to set the default for preserving free space on mounted volumes. This setting defines the minimum number of bytes to remain available on a volume after a data movement. A transfer job is rejected when the total byte size of a transfer, plus the minimum remaining space, is greater than the available space on the destination volume.
This value can be overridden on an individual volume with the setting
vol_attributes:
min_remaining_space.
NOTE This value may not be respected if multiple transfer jobs are transferring files to the same destination volume. If the total byte size of files from all transfers to the same destination exceed the available space on that destination volume, an error occurs and at least one transfer job to that destination fails.
Example for the DataIQ host:
default_min_remaining_space:
value: 262144000
For Data Mover external nodes, Data Mover uses the setting from the DataIQ host.
Data is not available for the Topic
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\