Avamar: How to throttle the backup client's CPU, network, IO and memory consumption
Summary: This article describes how to limit the physical resources consumed by an Avamar client while a backup is in progress.
Symptoms
This article discusses throttling and prioritization of CPU, network, and I/O resources consumed by the avtar process during a backup.
Avamar avtar process uses as much network bandwidth, CPU cycles, and I/O as are made available to it by the operating system. Avamar can be set with parameters which determine how aggressive it is when laying claim to some of these system resources.
CPU Utilization
An Avamar client machine may be a filer server, a mail server, a database or have some other role that requires system resources.
If a backup runs while those system resources are under heavy load, both the backup and the client's regular role may be impacted. If it is not possible to schedule the backup to run during a quiet period, you may want to throttle CPU usage by avtar to ensure that sufficient CPU cycles remain available for client's primary function.
Avtar CPU utilization can be reviewed in the Avamar client backup log as shown below:
avtar Info <5100>: Status 2009-04-10 20:37:02, 0 files, 25.00 GB (0 files, 473.7 MB, 1.85% new) 117 MB 99% CPU
The CPU percentage value may be seen to reach higher than 100% in environments where multiple CPU cores are available.
CPU Process Prioritization
Linux Behavior:
- The avagent process automatically sets all processes that it spawns to be at a "nice" level.
- In Linux, priority levels range from -20 to 20, where -20 is the highest priority.
- By default, processes are spawned with priority 0.
- Avtar is created with priority 10, which is lower than the default priority.
Windows Behavior:
- Windows manages priority levels ranging from 1 (lowest) to 31 (highest).
- Avtar is created with BELOW_NORMAL_PRIORITY_CLASS, which means that threads owned by avtar by default are created with priority 6.
- For reference, threads created by processes on Windows have a default priority of 8.
Network Utilization
Relative to traditional backup software, Avamar consumes little bandwidth. However, if an Avamar client is separated from an Avamar server by a low-bandwidth link it may be helpful to throttle network usage. This can prevent backup data from swamping a line and slowing down transmission of more time critical data.
I/O Prioritization
The I/O priority of Avamar client processes is not explicitly set.
Linux Behavior:
- I/O niceness ranges from 0 (highest priority) to 7 (lowest priority).
- The algorithm that is used is (cpu niceness + 20)/5 (source: http://linux.die.net/man/1/ionice)
- For avtar, I/O priority would be (10 + 20)/5 = 6, which is the second lowest I/O Priority.
- In comparison, a process s I/O priority with a default thread priority of 0 would be 4.
Windows Behavior:
For Windows, I/O prioritization is less clear. Microsoft claims that a process' I/O priority is tied to the thread priority. Avamar Engineering investigation for issue #240762 demonstrated that, in practice, this does not appear to be the case. Setting avtar to run with a lower priority does not change the I/O priority from the default.
Memory utilization
The amount of memory that is required during a backup depends the cache file sizing and the type of cache files in use.
Traditional 'monolithic' cache files load the entire cache files into memory whereas the paging cache files 'page' data into memory as needed.
Memory utilization and cache file sizing are beyond the scope of this KB article. See the Operational Best Practices Guide for more information.
Cause
Resolution
The following flags exist to throttle network and CPU usage of the avtar process:
--net-throttle (throttles network usage)
--cpu-throttle (throttles CPU usage)
To implement either of these flags, edit the 'avtar.cmd' file found in the Avamar client's avs/var directory.
Note: Create a new 'avtar.cmd' file if a file does not exist.
CPU Throttling
The flag should be in the following format, where '80' is the maximum average percentage of CPU that avtar should use.
--cpu-throttle=80
This flag was introduced before multiple core CPUs were widely used. The percentage to use may require experimentation. A backup that is carried out on a dual core desktop machine with avtar throttled to 10% appeared to use around 5% of the CPU resource.
Network Throttling
The entry that is made in the file should be in the following format:
--throttle=Mbit/sec
When implemented, avtar pauses if necessary after sending each packet. This ensures that network usage does not exceed the specified maximum bandwidth (specified in mega bits per second).
For example, --throttle=5 uses half of a 10Mbit/sec connection, --throttle=0.772 restricts usage to one-half of a T1 link.
After you have made the change, save the avtar.cmd file.
The changes should take effect the next time avagent spawns a new avtar process.
Alternatively, using the Options tab in Avamar Administrator, select the appropriate file system plug-in from the drop-down list (Windows File System, Linux File System, so forth).
Check the "Show Advanced Options" checkbox and scroll down until to find "Network rate throttle (Mbps)".
Fill this in with the required bandwidth in Mbps (decimals are fine, for example 1.2).
Avamar plug-in backups
For plug-in backups (Exchange, Oracle, so forth), in the Avamar Administrator user interface, select the plug-in from the menu drop-down, click the "More" button and enter the attribute and value by hand.
For the attribute, enter [avtar]net-throttle. For the attribute value, enter the required bandwidth. When happy with the setting, click the + sign under the fields to add the flag to the box below.
Click OK to save the changes.
The [avtar] in the flag tells the plug-in to pass the flag through to avtar which is the process that communicates with the server.
Additional Information
Use the throttle flags with CAUTION and note the following:
-
Processes run as fast as they can. If we attempt to lower the CPU or network utilization, the backup may run far slower than it otherwise would. Slow backups may not complete within a regular backup window.
-
If the backup is slowed down, other clients whose backups are queuing to start, are delayed by the lengthened backup time of this client.
-
If the client is performing other work at the time of the Avamar backup, this work is impacted less but for a longer time.
If in doubt about the suitability of applying the flags discussed below, contact your Dell Professional Services Representative.
A network graph showing traffic for a client with a throttle set to 0.2 (Megabits per second) may reveal that the data transmitted between client and server occurs in short bursts at rates of over 150 Mb/s.
The Avamar network throttling algorithm works in such a way that it transmits data in short bursts. After each burst, the algorithm queues data for an appropriate amount of time before continuing to send data. The data is sent at a rate equivalent to the throttling level which is specified by the throttle flag.
The level of perceived 'burstiness' correlates with the granularity of the monitoring. In other words, analysis of the transmission rates in subsecond increments shows bursty behavior but if we look at the total amount of data that is sent over the duration of a backup we should see that it is near to 0.2 Mb/s.
Below is a short-term graph for a backup, which is captured over a few minutes:
Frequently asked questions:
- How much bandwidth is used for a Burst.
- Maximum bandwidth of the link
- End to end latency across the network
- Quality of Service
- Any other regular bottlenecks that might apply
- How long does a burst last for?
- Can you explain what data is transmitted during a "burst"?
- Can network throttling be applied across groups of clients at a site?
Network throttling during Avamar backups to Data Domain
Network throttling of backups to Data Domain is supported where Avamar client is v7.1 and Data Domain is v5.5 or later.