Avamar: Backup performance impacted by anti-virus software real-time file scan

The backup performance of an Avamar client during the scanning of files is relatively low.

The following points are true:

• Status messages in the client log show low CPU usage by avtar.exe (see Dell article Avamar - How to interpret avtar backup log status lines)

• The backup is not bottlenecked due to network congestion.
• Anti-virus software is installed on the client.
• The anti-virus software is set to scan files in real time "on access" (when an application accesses files)

To diagnose or view the issue during a backup, use the Task Manager:

1. Processes tab > View > Select Columns
2. Enable the "I/O Read Bytes", and "I/O Reads" columns.
3. While a backup is in progress, monitor the anti-virus process.
4. If the "I/O Read Bytes" or "I/O Reads" numbers for the anti-virus application are heavily incremented during the backup, this may indicate an issue.

The anti-virus on access scanning feature contends with the avtar process for the storage I/O by intercepting files which avtar is trying to access.

The anti-virus may be scanning snapshot storage which contends with avtar process for access to the snapshot that was taken as part of the backup.

This extra load on the storage hardware reduces the potential file scan speed of the backup.

As a test, temporarily disable the anti-virus on-access or real-time scanning feature and rerun the backup.

Temporarily configure a test Virtual Machine (VM) to do client-level backup with no anti-virus installed record timing then install anti-virus and compare.

Compare the performance of the latest backup with earlier backups where real-time scanning was active. To do this, review a complete client log for Backed-up.

Manually add --status=120 in additional options to increase the frequency of status messages for testing.

avtar Info <6083>: Backed-up 344.3 GB in 862.43 minutes: 24 GB/hour (1,508,688 files/hour)

If backup performance is improved with on-access scanning disabled, configure the anti-virus software to trust application activity by Avamar binaries or to ignore the reads.

In addition to the avtar.exe, the executables to trust or safe-list from anti-virus ON ACCESS SCANNING are below. The files are typically installed in the c:\program files\avs\bin folder:

avexchglr.exe   Exchange Server GLR
avexvss.exe     Exchange Server
axionfs.exe     Exchange Server GLR
avoracle.exe    Oracle
avlotus.exe     Lotus Notes
avscc.exe       Desktop Icon
avagent.exe     Communication service between the client and Avamar server
avvss.exe       Disaster Recovery, BMR
avmossvss.exe   SharePoint Servers
avupdate.exe    Client update tool

After resolving this type of interference from anti-virus software, if backup performance is still low, see Dell article: Avamar slow backup performance - how to troubleshoot and identify bottlenecks (RESOLUTION PATH)

The behavior described occurs with any anti-virus program that performs "real-time" or "on-access" scanning.
Below are links to suggested articles of various anti-virus vendors' online documentation that discuss the behavior or how to mitigate it.
Some anti-virus programs keep hash or MD5 checksums of excluded or safe listed programs. When upgrading Avamar software the hash or MD5 checksum needs to be recalculated per software vendor requirements.

Symantec EndPoint Protection

• Broadcom Backup Exec or Backup Exec System Recovery: Backup and Restore job rates are slower than expected with Symantec EndPoint Protection 12.1 installed (although the article mentions Symantec's own backup software, the issue is the same for any backup solution.)
• Broadcom EndPoint Protection exclusions for Backup Exec 

Trend Micro Office scan

• Dell article How to check if TrendMicro anti-virus real-time scanning is impacting Avamar client performance (log in to Dell Support is required to view this article)
• Trend Recommended scan exclusion list for Trend Micro EndPoint products
• Trend Excluding Volume Shadow copies from the Real-Time Scan of APEX One

Sophos

• Sophos Process exclusion (Windows) 
• Dell article Poor backup performance for Avamar client running Sophos anti-virus (deprecated)
• Sophos Central Server: Set scanning exclusion for Volume Shadow Copies

Quick Heal Endpoint Security

• Dell article Avamar backup performance impacted due to virus protection software - Seqrite EndPoint Security by Quick Heal Technologies

ESET Endpoint anti-virus

• ESET Endpoint 7 Processes exclusions

Panda Security

• Panda What is the Application Control of Panda?

CrowdStrike

• CrowdStrike Whitelisting

Microsoft

• Microsoft Anti-virus policy for EndPoint security in Intune
• Microsoft Defender for EndPoint
• Microsoft How to add a file type or process exclusion to Windows Security

Carbon Black

• Broadcom Configuring Exclusions for Core Prevention Rules

Fortinet

• Fortinet FortiClient: Excluding applications from protection

Trellix (FireEye)
 

• Trellix EndPoint Security xAgent Administration Guide-Enabling a Custom Policy 

Cisco AMP

• Cisco Configure and Identify Secure EndPoint Exclusions 

Rapid7

• Rapid7 How to exclude activity from Endpoint Prevention 

Avamar Client for Windows, Avamar Desktop/Laptop Option, Avamar Plug-in for Exchange 2007, Avamar Plug-in for Exchange VSS, Avamar Plug-in for Hyper-V VSS, Avamar Plug-in for IBM DB2, Avamar Plug-in for Lotus Domino, Avamar Plug-in for Oracle , Avamar Plug-in for SAP with Oracle ... View More about warranties View Less about warranties