Dell EMC Data Mover Admin Guide

When Data Mover encounters a system file that it can't reliably move, this setting determines whether to treat the situation as an error, or to continue, replacing the file with a placeholder file that can be restored later manually.

Some UNIX-like system files cannot be reliably moved between file systems, including pipe files, socket files, character device files, and block device files.

• When set to True, Data Mover replaces the system file with a 0-byte placeholder file. For example:
  • sample.txt - pipe file by this name was at this location
  • sample.txt - socket file by this name was at this location
  • sample.txt - block device file by this name was at this location
  • sample.txt - character device file by this name was at this location
• When set to False, Data Mover reports an error when it encounters a system file it can't move. Depending on how error_handling is set, Data Mover either reports the job as a failure and aborts the job, or completes as much of the job as it can, and reports "Completed with errors", and reports the files on the original source that could not be moved. This is the default value.

    convert_system_files_to_stubs:
      value: False
      default: False

This setting determines whether Data Mover can continue in the situation where it is asked to transfer an empty folder with a path that is too long for an S3 volume.

S3 volumes do not support file and folder paths over 1024 bytes.

If Data Mover is asked to transfer an empty folder to an S3 volume, and the path would exceed this length, Data Mover choose whether to observe or ignore the error.

• When set to True, transfer jobs can continue even if there are empty folders with very long path names. Data Mover records the situation as a warning in the logs. Unlike other ignored paths, this situation does not prevent deleting the source data.
• When set to False, transfer jobs that create very long path names cause the job to fail, even if the folder is empty. This is the default value.

    ignore_empty_folder_if_path_too_long:
      value: False

List of folders and files that should be excluded from data transfer jobs. When found:

• Any folders or files that are excluded are not copied. Any remaining folders or files are copied.
• If the job is a move job, the job is converted to a copy job. The original source files will not be deleted.
• To see which folders and files were not transferred, select View Transfers and review the Excluded paths.

Each entry should be written as a Java Regular Expression (RegEx). Use online tools like RegEx Reference or the Regular Expression Analyzer to learn about and test RegEx expressions.

Each entry should be written on its own line, and should start with a dash and space (- ) aligned with the regex_exclude_list: category. Example:

    regex_exclude_list:
    - \.snapshot
    - Vol\d\/root
    - trash

This setting determines whether Data Mover encodes and decodes file and folder names when transferring files when the file path contains control characters or invalid unicode characters.

S3 volumes do not support file paths that contain control characters or invalid unicode characters. When transferring files that include control characters or non-UTF characters, Data Mover renames the files with an encoded path name. When transferring the file back from the object store, Data Mover restores the original path name.

In most cases, we recommend setting the value to True. However, the default setting is False, because altered path names can be considered a form of data loss. Admins should understand how some paths might change before altering the setting.

• When set to True, when Data Mover transfers the files back to DataIQ, Data Mover decodes the path name and uses it to restore the path.
• When set to False, Data Mover does no encoding or decoding of paths with control characters or invalid Unicode. If paths with control characters or invalid Unicode are encountered, they may generate errors, which could cause the job to fail.

    support_extended_characters:
      value: False
      default: False

This setting determines whether Data Mover shows warnings when files do not include all expected metadata during a transfer from an S3 volume.

S3 volumes typically do not include storage settings for the attributes: permissions, uid, gid, atime and mtime. When Data Mover transfers files from a POSIX-compliant volume to an S3 volume, the data for these attributes is transferred to the object metadata. When transferring the files back, Data Mover restores the original permissions and attributes.

If the files were transferred to S3 volumes through some other method, this metadata might not exist on the server. When the files are transferred back, by default, warnings about these missing attributes are included in the Data Mover logs. If this attribute metadata is not expected to exist, or if the warnings are not useful, set this value to True.

• When set to True, warnings about ownership, permissions, and attributes are suppressed.
• When set to False, warnings about ownership, permissions, and attributes are included. This is the default value.

    suppress_S3_ownership_permission_and_attribute_warnings:
      value: False
      default: False

Whether to translate symlink pointers to relative paths when the pointed-to path is contained in the selected source.

    translate_contained_src_link:
      value: True
      default: True

Whether to transfer symlinks.

• When set to True, symlinks are transferred.

• When set to False, then symlinks are not transferred.

  After successful transfer jobs, when Delete Source is set to True, symlinks are deleted from the source.

  (If there are errors in the transfer, or if source files are excluded using regex_exclude_list, then the original files including symlinks, remain on the source.)

    transfer_symlinks:
      value: True
      default: True

This setting determines whether to preserve the POSIX attributes: uid, gid, permissions, mtime, atime during a file transfer.

This option is not alterable by users. There is no option in the UI.

• When set to True, Data Mover attempts to preserve the POSIX attributes when adding files to destination files and folders.

• When set to False, the destination files and folders will not have the original POSIX attributes but instead will have whatever the destination machine provides.

The default value is True.