Dell
PowerScaleOneFS Datamover (also called SmartSync) enables you to transfer data between
PowerScale clusters and object stores (for example, ECS, AWS) using the Datamover transfer engine that is embedded in
OneFS. Datamover ensures that you have a consistent copy of your data on another
PowerScale cluster or cloud platform. Datamover allows you to control the frequency of data transfers at scheduled times using policies. Similar to the SyncIQ module, you can transfer data at the directory level.
The embedded Datamover feature provides data replication for file and object deployments on-premises or in the cloud. Datamover enables file-to-file transfers between
PowerScale clusters using RPC and file-to-object copy transfers to S3 (ECS, AWS) and Azure cloud systems.
Datamover provides the following primary functions:
Data protection
Data repurposing (copy)
Data archive
Datamover provides a flexible execution model of push/pull data transfers between systems. While SyncIQ allows administrators to push data from a source to a target cluster, Datamover also allows for a target cluster to pull data from a source cluster, resulting in reduced throughput and CPU impacts on the source cluster.
Datamover features
Faster data transfers than SyncIQ
Snapshot locking
Separation between Datamover datasets and user snapshots prevents accidental deletion of snapshots during transfers
Scalable run-time engine
Dataset "reconnect" allows systems with identical datasets to reconnect for instant incremental backups during failover scenarios
Namespace contention avoidance
Batch operations for efficient small file transfers
Bulk operations to address file ID-mapping contention
Smart scheduling
CPU and bandwidth throttling
Centralized management of policies and jobs
Replication between multiple sets of clusters
NANO(A)N: Not-All-Nodes-On-(All)-Networks detection. Active accounts are monitored on-the-fly by each node.
Nodes with no accessibility to an account do not participate in a transfer
Improved error handling and graceful crash recovery to ensure checkpointing stability
Data recovery
You can restore from a dataset that you replicated to another cluster. For example, you can copy a dataset from an archive tier to a production tier as a one-time copy. That copy will be read/write on arrival.
You can perform a one-time copy at any time to your archive tier or between any two clusters. A one-time copy provides the option to not make Datamover datasets, which means you can start using them read/write immediately.
File-to-file high-performance data transfers
Streamlined baseline and incremental file transfers
Namespace contention avoidance. Namespace creation is separated from data transfers
Batch transfers of small files, attributes, and data blocks
Asynchronous I/O backed by lightweight threads (fibers) allows for maximized parallel transfers
File-to-object content distribution "copy" format limitations
The following table lists current limitations in file-to-object transfers.
Limitation
Description
ADS files
Skipped when encountered
Hardlinks
Not supported. An object is created for each link (hard links are not preserved)
Symlinks
Skipped when encountered
Special files
Skipped when encountered
Metadata
Only the following POSIX attributes are copied: mode, UID, GID, atime, mtime, ctime
File name encoding
Encodings are converted to UTF-8
Large files
Errors are returned for files greater than the cloud provider's maximum object size
Sparse files
Sparse sections are not preserved; they are written out fully as zeros
CloudPools
Not supported
Compression in transit
Not supported
Copy back from the cloud
Not supported if the data was not created by Data Mover
Incremental transfers
Not supported for file-to-object transfers. Only one-time copy to cloud/copy back from cloud is supported
Credential requirements
Datamover must be hosted on all
PowerScale clusters where transfers are planned.
For
OneFS copy to cloud and copy back from cloud transfers, Datamover is installed on
OneFS but not on the cloud systems.
Datamover waits for the administrator to commit the
OneFS upgrade.
Users must have the
ISI_PRIV_DATAMOVER administrative (AIMA) privilege to configure the Datamover using the Rest APIs.
Inbound TCP port 7722 must be opened in firewalls.
Certificate requirements
The following Certificate Authorities (CA) and trust hierarchies are required.
Requirement
Description
TLS certificates
A mutually authenticated TLS handshake is required. Authorization, authentication, and encryption are provided by TLS certificates.
TLS certificates are always required for daemon startup and all communication between Datamover engines.
Encryption can be disabled, but authorization and authentication cannot be disabled. In other words, while data traffic might be configured to be unencrypted, a successful TLS handshake is still required before unencrypted data transfers can begin.
Certificate Authorities (CA)
The CA (the private signing key material) does not need to be on each Datamover system. The certificate of one or more CAs is required on each Datamover system but not the CA itself. The CA is required to verify that an identity certificate was signed by that CA. However, the CA key should not be distributed beyond the CA.
Dell recommends that customers use a new, Datamover-specific CA for signing Datamover identity certificates.
The CA that signs an identity certificate is not required be installed on the system that the identity certificate is installed on. Two systems trust each other if they have the CAs that signed each other's identity certificates.
Identity certificates
The certificate that provides authentication of the identity claimed.
Exactly one identity certificate must exist on each Datamover system.
Identity certificates are signed by one of the CAs deployed on the systems that the system is going to communicate with.
Trust hierarchies
Two systems trust each other if they have the CAs that signed each other's identity certificates.
There is no concept of unidirectional trust—trust is entirely mutual.
Reference documentation
The Datamover feature includes a full set of
isi dm command-line interface (CLI) commands and APIs in the
PowerScaleOneFS CLI Command Reference and
PowerScaleOneFS API Reference Guides.