Integration Data Protection Appliance - Avamar password out of sync
Summary: Avamar password out of sync shown in IDPA dashboard UI
Symptoms
Password Out of Sync error message(s) show on IDPA ACM Dashboard complaining about Avamar Server and or AVProxy
Here is a Password Out of Sync example:
Here is a Password Out of Sync example:
Cause
ACM monitors and or reports all point-products which include its hardware and hypervisor platform health status. In order to do that, it keeps a copy of all current point-product login information in an encrypted password file. Constantly makes connection to those targets to check their health status. If it fails to log in any of the monitoring target machines, it would report a password out of sync error in the ACM dashboard.
Possible causes:
Possible causes:
- Password changed or reset directly at the Avamar server or proxy end instead of from the ACM dashboard
- Network latency from ACM to Avamar Server or AVProxy to query its status. (temporary issue)
Resolution
If update password in ACM UI unable to resolve your password out of sync issues, following this KB article:
Here are Resolutions for Avamar server password out of sync:
Scenario #1 password changed from point-product end instead of from ACM UI:
Scenario #2 Password out of sync error due to network latency when ACM trying to query its point-products:
Scenario #3 Even though password is in sync and works on Avamar, ACM shows pass out of sync for AV, due to SSH failure or test connection failure to AV. This could be due to ACM failing to login to AV due to SSH issues like recent changes made on av sshconfig, cipher negotiation, and so on
Scenario #4 Avamar MCUser or viewuser may show out of sync when ACM is unable to perform MCSDK call to Avamar to validate those user passwords. This can happen if the ACM MCSDK call fails to Avamar due to various reasons.
If the above provided scenarios and resolutions are unable to fix the issue, do the following:
Avamar password related KB references:
Here are Resolutions for Avamar server password out of sync:
Scenario #1 password changed from point-product end instead of from ACM UI:
- Avamar password out of sync (one or more of the following Avamar user passwords have changed)
- Avamar OS root user password
- Avamar OS admin user password
- Avamar Server root user password
- Avamar Server MCUser user password
- Avamar Server repluser user password
- Avamar PostgreSQL database viewuser user password
- Avamar vProxy OS root user password (Appliance Internal proxy VM)
- Avamar vProxy OS admin user password (Appliance Internal proxy VM)
Resolution:
- Avamar Server OS root password MUST be the same as OS admin password. Otherwise, you will be getting the error "Protection Software root user test connection failed. Please ensure root password is same as admin password." In ACM UI when trying to enter correct password. To verify, ssh login to Avamar server as admin user and password, then su to root user with the same password to see if the password can login both admin and root account.
- Ensure that you have valid Avamar Server root, MCUser, repluser passwords (Avamar OS root is different from Avamar Server root. OS root is a Linux OS level user, and Avamar Server root is an app level user)
To verify the Avamar Server root, MCUser and repluser passwords, SSH login Avamar machine and run:
# avmgr logn --id=MCUser --ap=<password>
1 Request succeeded
7161 privilege level (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired)
2 block type (directory)
To verify the Avamar db viewuser password, SSH to ACM machine and run (if you run this command from the Avamar machine itself, it will not prompt for password):
# psql -U viewuser -h <Avamar Server IP> -p 5555 mcdb -c "\d"
Password for user viewuser:
- If step 1 and or step 2 condition are not met, then go to step 6 directly to sync password in ACM dashboard.
- SSH login to Avamar server, first login as 'admin', then su to user 'root' and run "change-passwords" command to update or reset the Avamar passwords. Note that OS root and admin user passwords MUST be the same, and if needed, making changes to Avamar Server user passwords if you forgot those passwords or want to change those passwords. We recommend keeping all Avamar Server passwords to be the same as IDPA common password, however if there is a requirement to keep them different, then ensure Avamar OS 'admin' and 'root' users have same password.
When you update Avamar password(s), ensure the new password(s) align with the IDPA global password policy shown as below:
In the following exercise, we run change-passwords command (login as root user) to change both admin and root OS passwords but without changing ssh keys (it should not be changed unless there is a problem with the ssh keys), and also change root and or MCUser and or repluser and or viewuser passwords.
login as: admin
Password: xxxxx
admin@Avamar-svr:~/>: su -
Password: xxxxx
root@Avamar-svr:~/#:
root@Avamar-svr:~/#: change-passwords
[change-passwords version 2.1]
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Identity added: /root/.ssh/rootid-save (/root/.ssh/rootid-save)
Do you wish to specify one or more additional SSH passphrase-less
private keys that are authorized for root operations?
Answer n(o) here unless there are known inconsistencies in
~root/.ssh/authorized_keys files among the various nodes.
Note that the following keys will be used automatically (i.e., there is
no need to re-specify them here):
/root/.ssh/rootid
/root/.ssh/rootid-save
y(es), n(o), h(elp), q(uit/exit): no
--------------------------------------------------------
The following is a test of OS root authorization with the currently
loaded SSH key(s).
If the authorization test fails, then you might be missing an
appropriate private key, e.g., rootid or dpnid.
-> In that event, re-run this program and, when prompted,
specify as many SSH private key files as are necessary
in order to complete root operations.
Starting root authorization test with 600 second timeout...
End of root authorization test.
--------------------------------------------------------
Change OS (login) passwords?
y(es), n(o), q(uit/exit): yes
change-passwords: INFO: Each OS password will be changed locally without further prompting as soon as you have (twice) entered a valid password.
--------------------------------------------------------
Change OS password for "admin"?
y(es), n(o), q(uit/exit): yes
Change password for user "admin".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same OS user password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
BAD PASSWORD: it is too simplistic/systematic
Backup lockbox file
Backup keystore files
Backup SSV files
Flush backup
Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00
Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
Updated with new value under name "admin".
Backup lockbox file
Backup keystore files
Backup SSV files
Flush backup
Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00
Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup
change-passwords: INFO: The password for OS user admin has been updated on _this_ host.
change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases.
--------------------------------------------------------
Change OS password for "root"?
y(es), n(o), q(uit/exit): yes
Change password for user "root".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same OS user password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
BAD PASSWORD: it is too simplistic/systematic
change-passwords: INFO: The password for OS user root has been updated on _this_ host.
change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases.
--------------------------------------------------------
Generate new SSH keys?
y(es), n(o), h(elp), q(uit/exit): no
--------------------------------------------------------
Change Avamar Server passwords?
y(es), n(o), q(uit/exit): yes
--------------------------------------------------------
Please enter the CURRENT server password for "root"
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Checking Avamar Server root password (1200 second timeout)...
Avamar Server current root password accepted.
--------------------------------------------------------
Change Avamar Server password for "MCUser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "MCUser".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "MCUser".
--------------------------------------------------------
Change Avamar Server password for "root"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "root".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "root".
--------------------------------------------------------
Change Avamar Server password for "repluser"?
y(es), n(o), q(uit/exit): yes
Please enter a new Avamar Server password for user "repluser".
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Enter the same Avamar Server password again.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
Accepted Avamar Server password for "repluser".
--------------------------------------------------------
Change the viewuser password?
y(es), n(o), h(elp), q(uit/exit): yes
Checking Administrator Server status...
Enter the NEW viewuser password.
Enter ? or help for help.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
For verification, re-enter the NEW viewuser password.
Enter ? or help for help.
(Entering an empty (blank) line twice quits/exits.)
> xxxxx
--------------------------------------------------------
Do you wish to proceed with your changes on the selected node?
Answering y(es) will proceed to make changes.
Answering n(o) or q(uit) will not proceed.
y(es), n(o), q(uit/exit): yes
Changing OS passwords...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done changing OS passwords...
Changing Avamar Server passwords...
Suspending maintenance cron jobs
Checking Administrator Server status...
Stopping Administrator Server...
Changing the passwords for the local Avamar Server...
The passwords for the local Avamar Server have been changed.
Starting process of updating Administrator and Enterprise Manager configurations...
Running script to update Administrator and Enterprise Manager configurations on node 0.s...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating Administrator configuration on node 0.s...
Starting process of updating client configurations...
Running script to update client configuration on all+...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Updating client configuration on node 0.0...
Done updating client configuration on 0.0...
Starting process of updating mccli configuration files...
Running script to update mccli configuration files on node set "0.0"...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mccli configuration files on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...
Resuming maintenance cron jobs
Starting process of updating viewuser password...
Checking Administrator Server status...
Stopping Administrator Server...
Running script to update mcdb viewuser password on node 0.0...
[Logging to /usr/local/avamar/var/change-passwords.log...]
Done with updating mcdb viewuser password on node 0.0...
Checking Administrator Server status...
Starting Administrator Server...
Stopping EMT subsystem
Starting EMT subsystem
--------------------------------------------------------
Done.
NOTES:
- If mccli (the Administrator command line interface)
is used from any remote user accounts, then please update
the password in each remote account's copy of the mccli
preferences/configuration file, typically
~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.
- Please be sure to resume schedules via the
Administrator GUI or via 'dpnctl start sched'.
#: dpnctl start sched
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: Resuming backup scheduler...
dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
- If ACM UI shows Protection Software Proxy 'root / admin' user password is out of sync. Then SSH login to AVproxy and update its OS root and admin passwords. Importantly, the admin and root password must be the same. Otherwise it would report an error: Protection Software Proxy 'root' user out of sync. Please ensure 'root' password is same as 'admin' password
Here is an example of changing both proxy admin and root passwords (Ensure you login as root first, then change both admin and root passwords):
login as: admin
Password: xxxxx
su -
Password: xxxxx
# passwd admin
New password: xxxxx
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: is too simple
Retype new password: xxxxx
passwd: password updated successfully
# passwd root
New password: xxxxx
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: is too simple
Retype new password: xxxxx
passwd: password updated successfully
- Logout and log back in ACM again, click on the Out of Sync error message, it would prompt you to enter new password for a certain user, update the password accordingly. You may see password Out of Sync message still show up asking for a different user password in a scenario that you have multiple passwords out of sync, enter them accordingly.
(Sometimes it still will show password out of sync error after you entered correct password, wait for a few minutes and refresh your web browser page again. Root cause shows in Scenario #2).
Scenario #2 Password out of sync error due to network latency when ACM trying to query its point-products:
- This would be a temporary issue and typically can be resolved if you refresh the ACM page after 1-2 minutes. This is a known issue and the Dell engineering team is working on a fix in a future release.
Scenario #3 Even though password is in sync and works on Avamar, ACM shows pass out of sync for AV, due to SSH failure or test connection failure to AV. This could be due to ACM failing to login to AV due to SSH issues like recent changes made on av sshconfig, cipher negotiation, and so on
- Run a test SSH connection from ACM to Avamar server. If it fails, login Avamar server and restart ssh service:
# service sshd restart
- If this does not help, gather the error message, troubleshooting steps you have performed and raise a ticket with Dell technical support for further assistance.
Scenario #4 Avamar MCUser or viewuser may show out of sync when ACM is unable to perform MCSDK call to Avamar to validate those user passwords. This can happen if the ACM MCSDK call fails to Avamar due to various reasons.
- IDPA: ACM Reports Avamar Server Passwords Out of Sync Following Upgrade to 2.7.1 Upgrade or Application of Avamar 19.4 MCS Hotfix 333618
- IDPA: ACM UI shows error "Backup Server viewuser user password is out of sync. Please update the latest password" on Backup Server Dashboard
If the above provided scenarios and resolutions are unable to fix the issue, do the following:
- SSH Login ACM as root, and stop and start ACM web application service:
# service dataprotection_webapp restart
# service dataprotection_webapp statu
- Refresh the ACM web page and login, it would show "Appliance Startup progress." It would take some time to resync up with all the appliance components, and once done it would return to ACM dashboard. (This is not a process of restarting Appliance)
- If the issue still cannot be resolved, raise a support ticket with Dell Technologies.
Avamar password related KB references:
- Avamar - How to restart AVE in single user mode to reset root password
- IDPA: ChangeAvamarPasswords failed to execute change-passwords command on Avamar Utility Node
Affected Products
PowerProtect DP4400, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance SoftwareArticle Properties
Article Number: 000217330
Article Type: Solution
Last Modified: 11 Oct 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.
Article Properties
Article Number: 000217330
Article Type: Solution
Last Modified: 11 Oct 2023
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.