The purpose of this KB article is to discuss factors which determine the number of concurrent backup and restore connections which can be made to the Avamar server.
A typical symptom where the number of concurrent sessions has been exhausted is where backups fail to start running within their allotted backup schedule. At the end of the backup schedule, they are recorded on the activity monitor as "Timed out - start".
Client avtar logs may report the following:
YYYY-MM-DD HH:MM:SS avtar Info <5557>: No connections available
This is not a defect, it is as per design.
The total number of client connections which can be made to the Avamar server varies according to which operations are currently running, how the connection is made, and the number of data nodes which are online.
Various parameters in the /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml file govern this behavior.
Example from a v6.1 system:
<entry key="max_jobs_per_node" value="30" />
<entry key="percent_of_max_concurrent_jobs_hfscheck" value="10" />
<entry key="percent_of_max_concurrent_jobs" value="90" />
<entry key="max_concurrent_jobs" value="480" />
When HFScheck is not running
- The MCS queues up max_jobs_per_node * percent_of_max_concurrent_jobs / 100. The number of concurrent jobs for the whole grid does not exceed max_current_jobs.
During HFScheck
- The MCS queues up a maximum of max_jobs_per_node * percent_of_max_concurrent_jobs_hfscheck / 100.
One or more restore is always possible
- The MCS always reserves one connection per grid (not per data node) for the restore activity.
The MCS has no concept of "restricted" windows or "maintenance" windows. It does know when the grid is read-only, and it also knows when the grid is running hfscheck.
Based on the above, determine the capability for backup and restore jobs. The following examples are based upon the mcserver.xml parameters shown above which cover v5.x through v6.1.
During garbage collection and during checkpoint jobs
- No backup sessions allowed.
- Restore jobs are possible, potentially. 27 per node (v6.1) although testing should be carried out to confirm this.
- During HFScheck:
- v6.1: Three jobs per data node (less one for the overall grid for restore).
- One restore for the entire grid.
- At all other times:
- (v6.1) 27 jobs per data node (less one for the overall grid for restore) and a maximum of 480 for the grid.
- One restore for the entire grid
Information relating to v7.0 onwards
Refer to the appropriate release notes document on the
Avamar server product page at the time of writing the v7.0 and v7.1 release notes document.
The maximum number of concurrent connections for each storage node on the Avamar server depends on the Avamar Data Store version. For Avamar Data Store Gen4 and Gen4s storage nodes, the maximum is 80. For Avamar Data Store Gen3 storage nodes and Avamar Virtual Edition systems, the maximum is 27.
Example from a v7.1 system:
<entry key="max_jobs_per_node" value="obsolete7.0.0" />
<entry key="max_jobs_per_proxy" value="8" />
<entry key="percent_of_max_concurrent_jobs_hfscheck" value="10" />
<entry key="percent_of_max_concurrent_jobs" value="90" />
<entry key="max_concurrent_jobs" value="500" />
<entry key="activities_max_queue_size" value="1000" />
A job is considered to be either a backup or a restore session.
Replication source session counts as a restore.
A single node or AVE system can be considered to have one data node.
- These values are defined by the MCS and altering them can have a negative impact on the performance of the Avamar system. The values are not considered to be changeable unless advised by EMC Engineering.
- Backup sessions are initiated by avtar commands, bypass the MCS, and therefore, not obey the limitations defined above.