For backup and recovery, we will care about whether the data can be backed up and restored up, as well as really care about backup’s speed, which is the backup performance.

This article will introduce the factors that may affect Avamar backup performance.

Avamar backup process is that client send data to Avamar server through network.

In this three-part, there are many little details will affect backup speed. Overall, the factors that affect backup performance are as follows:

1.     Cache file is not big enough.

2.     Underutilized CPU.

3.     The network connection between the client and the Avamar server is too slow.

4.     Poor performance of the client disk (disk I/O).

5.     Memory is too low (Less than 3GB).

6.     Avamar server performance problems.

Avamar backup defined the standard speed is (Except the first backup):

·         File system backup = approximately one million files per hour

·         Database backup = about 100GB per hour

So, let's look at the specifics:

1.    cache file is not big enough

First of all, you need to understand clients are mainly having two cache files. f_cache mainly used for file system backup, p_cache mainly used for database backup. The default f_cache size is the 1/8 of physical RAM; the default p_cache size is 1/16 of physical RAM. In addition, cache file size is growing exponentially, rather than increase little by little. Such as the current cache file size is 300MB, then the next increase its size is 600MB.

As we all know, Avamar is front-end deduplication backup mechanism which is mainly based on the cache files to do deduplication. When Avamar backup started, Avamar will compare the local files and cache files to determine what data needs to be sent to the Avamar server via the network, and which is already backed up data, without the need to send it again.

Based on the above picture, if the total size of the default cache or you defined cache file is too small, causing it cannot continue to be grow,the backup speed will be greatly affected. This time we need to check the current cache file size and its total size. Then decide whether to change its original value, and thus improve backup speed. According to check the following information exists in the backup log or not, we can determine whether the cache file size is sufficient or not.

f_cache:

2014-02-04 06:34:37 avtar Warning <6562>: CAPACITY WARNING: The file cache is full; filecachemax=256MB, 2097150 entries overflowed.

p_cache;

entries=2097152,added=709614, overflow=56849

Adjusting method:

Log on to the client, under the C:\Program files\avs\ folder, edit or create avtar.cmd file and add the following parameters:

--filecachemax = or

--hashcachemax = or

Note: Cache file size should not exceed 1/4 of physical RAM, as this will affect performance.

2.    Underutilized CPU

Example:

2013-02-07 08:12:59 avtar Info <8688>: Status 2013-02-07 08:12:59, 7,758,607 files, 156,089 folders, 380.4 GB (3,821,086 files, 128.5 GB, 33.80% new) 623MB  4% CPU  X:\images\IMAGES29\3704\

From the above results, we can see that the backup CPU utilization rate is only 4%. This generally means the client is busy (running other programs) or disk performance problems.

In addition, some third-party anti-virus software will also affect the backup speed.

3.    The network connection between the client and the Avamar server is too slow

In this case, we will see that there are following contents in the logs:

2013-03-08 02:33:20 avtar Info <8688>: Status 2013-03-08 02:33:20, 2,713 files, 695 folders, 1.198 GB (56 files, 809.2 KB, 0.06% new) 33MB   0% CPU  C:\Documents and Settings\cnwhite\ntuser.dat.LOG

2013-03-08 02:35:23 avtar Info <7694>: Server(mkeavbak05.mke.ra.rockwell.com) not responding (possible network congestion?) (600 seconds)

2013-03-08 02:39:39 avtar Info <7695>: Server(mkeavbak05.mke.ra.rockwell.com) is back (855 seconds)

2013-03-08 02:44:39 avtar Info <7694>: Server(mkeavbak05.mke.ra.rockwell.com) not responding (possible network congestion?) (300 seconds)

Actually, if the backup performance is really caused by slow network problems, it is out of the scope of Avamar technical support. But in order to help customers to resolve the problem, we will help to do some basic tests. We are usually using iperf tools to test the network bandwidth between the client and the server. Specific steps are as follows:

·         Log on to the client, open putty and connect to Avamar Server.

·         Run this command on putty: iperf -s -i 1

·         While the above command is running, also run the following iperfcommand on the client

·         iperf -c AvamarServerIP -i 1

·         After running for some time to interrupt the operation and then view the results.

We will also run a backup test, using randchunkparameters to generate a log, so we can detects backup bandwidth:

avtar -c --randchunk=10000 --compress=none  --status=60  --comstats –nocache--logfile=c:\randchunk.log  --id=MCUser --ap=MCUser1 --server=Avamar.emc.com --path=/clients/DaveTemplin.emc.com

If you want to test the network latency and packet loss, you can use ping and tcpdump to detect.

4.    Poor performance of the client disk (disk I/O)

As a reference, we will take the following measures to determine the problem:

1.     Use performance monitor which is windows own performance testing tools to check the disk performance.

2.     Manually copy the files from the client side to another client, and then check the time is normal or not.

3.     Run a backup test, add --degenerateparameter, such as:

avtar -c --degenerate --compress=none  --nocache--logfile=c:\degenerate.log --id=MCUser  --ap=MCUser1 --server=Avamar.emc.com --path=/clients/DaveTemplin.emc.com  E:\testdata

Using --degenerate parameter, the data is not really being sent to Avamar server over the network, so, according to the backup speed, we can determine whether there is a performance problem in the client side.

5.    Memory is too low (Less than 3GB)

This is client system resources problem. If system resources are too low, Avamar runtime usable memory will be less, it will affect the backup speed.

6.    Avamar server performance problems

Here also involves a lot of problems, such as Avamar server busy, poor performance of a node, gsan or mscproblems etc. If this is the case, all of your backups may be affected, or at least most of the backup will have problems. In this time, let us to check it.

We will analyze Avamar log, and then probably determine the problem. The classic approach is:

1.     Log on to the client, open the C:\Program Files\ avs\var\ folder.

2.     Create and edit avtar.cmd file, add the following parameters:

--stats

--status=60

--comstats

3.     Save this file and initiate a backup again. After the backup fails again, collect and analyze logs.

Note: --comstats parameter used to diagnose communication problems between avtar and gsan (per second)