PowerProtect Data Manager 19.17 Kubernetes User Guide

Performance and scale limits

The following limits have been tested successfully with PowerProtect Data Manager.

Factors that can impact backup performance

The following factors can impact the Kubernetes namespace and PVC backup performance:

• The number of Kubernetes resources in a namespace.
• The number of PVCs in a namespace.
• The performance of the storage system that contains the persistent volumes.

Enable concurrent multi-streaming in File System agent for protecting large volumes

The File System agent is the default data mover that PowerProtect Data Manager uses for protecting persistent volumes that are provisioned by CSI drivers other than the vSphere Container Storage Interface (CSI) driver. For volumes larger than 100 GB, PowerProtect Data Manager can protect PVCs using multiple File System agent streams. To enable this functionality, you can add the annotation ppdm.config.fsagent/streaming_mode: concurrent to the PVC, which the controller then passes to the cProxy. Adding this annotation can improve performance and reduce backup latency by up to 50%.

The number of required streams is calculated dynamically, and will not exceed eight streams to avoid impact to CPU performance. Configuration of the chunk size impacts the stream count calculation.

Add line to the custom-ports file when not using port 443 or 6443 for the Kubernetes API server

If a Kubernetes API server listens on a port other than 443 or 6443, an update is required to the PowerProtect Data Manager firewall to allow outgoing communication on the port being used. Before you add the Kubernetes cluster as an asset source, perform the following steps to ensure that the port is open:

1. Log in to PowerProtect Data Manager, and change the user to root.
2. Add a line to the file /etc/sysconfig/scripts/custom-ports that includes the port number that you want to open.
3. Run the command service SuSEfirewall2 restart.

This procedure should be performed after a PowerProtect Data Manager update, restart, or server disaster recovery.

Log locations for Kubernetes asset backup and restore operations and pod networking

All session logs for Kubernetes asset protection operations are pulled into the /logs/external-components/k8s folder on the PowerProtect Data Manager host.

Parallel backup and restore performance considerations

To throttle system performance, the default settings for PowerProtect Data Manager data protection are five parallel namespace backups and two parallel namespace restores per Kubernetes cluster. You can change the ppdm.backup.concurrency and ppdm.restore.concurrency settings in the controller configuration for the Kubernetes cluster asset source, as described in the section Controller configurations. You can queue up to 100 namespace backups across protection policies in PowerProtect Data Manager.

You can also enable the k8s.ppdm.enable.parallel.pvc.backup setting to concurrently back up a maximum of 5 PVCs per namespace. If parallel PVC backup is enabled with the k8s.ppdm.enable.parallel.pvc.backup setting, in the case of vSphere CSI volumes, one cProxy is started per namespace. For non-vSphere CSI volumes, up to five cProxies will be started per namespace.

Refer to the section Resource requirements of PowerProtect Data Manager components on Kubernetes cluster to determine the overhead on the Kubernetes cluster with the higher concurrency settings before making any changes.

Resource requirements of PowerProtect Data Manager components on Kubernetes cluster

At any time during backup, with the default backup and restore concurrency settings the typical footprint of PowerProtect Data Manager components (Velero, PowerProtect Controller, cProxy) is less than 2 GB RAM Memory and four CPU cores. Utilization of this resource only peaks during backups.

The following resource limits are defined on the PowerProtect PODs, which are part of the PowerProtect Data Manager stack:

• Velero maximum resource usage: 1 CPU core, 256 MB memory
• PowerProtect Controller maximum resource usage: 1 CPU core, 256 MB memory
• PowerProtect cProxy pods: Each cProxy pod typically consumes less than 300 MB memory and less than 0.8 CPU cores. These pods are created and terminated within the backup and restore job.
• One cProxy pod is started for each namespace with PVCs that is backed up or restored.

Only Persistent Volumes with VolumeMode Filesystem supported

Backup and recovery of Kubernetes cluster assets in PowerProtect Data Manager is only supported for Persistent Volumes with the VolumeMode Filesystem.

Objects using PVC scaled down before starting the restore

The following activities occur before a PVC restore to the original namespace or an existing namespace when PowerProtect Data Manager detects that the PVC is in use by a Pod, Deployment, StatefulSet, DaemonSet, ReplicaSet, or Replication Controller:

• PowerProtect Data Manager scales down any objects using the PVC.
• PowerProtect Data Manager deletes the daemonSet and any Pods using PVCs.

Upon completion of the PVC restore, any objects that were scaled down are scaled back up, and any objects that were deleted are re-created. Ensure that you shut down any Kubernetes jobs that actively use the PVC before running a restore.

Restore to a different existing namespace can result in mismatch between UID of pod and UID persistent volume files

A PowerProtect Data Manager restore of files in persistent volumes restores the UID and GID along with the contents. When performing a restore to a different namespace that already exists, and the pod consuming the persistent volume is running with restricted Security Context Constraints (SCC) on OpenShift, the UID assigned to the pod upon restore might not match the UID of the files in the persistent volumes. This UID mismatch might result in a pod startup failure.

For namespaces with pods running with restricted SCC, it is recommended to use one of the following restore options:

• Restore to a new namespace where PowerProtect Data Manager restores the namespace resource as well.
• Restore to the original namespace if this namespace still exists.