Skip to main content
Sign Out

Welcome to Dell

My Account
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
Sign In Create an Account Dell Financial Services Premier Sign In Partner Program Sign In
Contact Us

Your Dell.com Carts

Some article numbers may have changed. If this isn't what you're looking for, try searching all articles. Search articles

NetWorker: Bootstrap Backup and Authc Commands Fail on Clustered Linux Server After Upgrade to 19.7.0.3 or 19.8.0.2 and prior

Summary: The NetWorker server is deployed on a clustered Linux NetWorker server. After upgrading the NetWorker version to 19.7.0.3 or 19.8.0.2 and prior, the Server Protection bootstrap backup fails. Authentication service commands (nsrlogin, authc_config, authc_mgmt) also fail, the error returned is that the authentication service is not available with HTTP error 404 (not found). ...

This article applies to   This article does not apply to 
Check out resources for

Symptoms

NetWorker server is deployed on a Red Hat Pacemaker (pcs) High Availability cluster.

NetWorker was upgraded to 19.7.0.4.

NetWorker service startup succeeds, all cluster resources show as "started" on one of the cluster nodes: 
root@NWrhelNodeG:~# pcs resource * Resource Group: NW_group: * fs (ocf::heartbeat:Filesystem): Started NWrhelNodeG.emclab.local * ip (ocf::heartbeat:IPaddr): Started NWrhelNodeG.emclab.local * nws (ocf::EMC_NetWorker:Server): Started NWrhelNodeG.emclab.local
nsrlogin command fails with HTTP error 404 (not found): 
[admin@NWrhelNodeG linux_x86_64]$ nsrlogin -u Administrator 130136:nsrlogin: Please enter password: 117849:nsrlogin: Authentication library error: GET failed with HTTP-ERROR: 404 Server Message : Could not parse server-response from json string Server Message : Make sure that server is running
The 'Server Protection' bootstrap backup fails to backup the authcdb:
 
Bootstrap Backup Fails to Backup the authcdb
Figure 1: Bootstrap Backup Fails to Back up the authcdb

Cause

Changes to authc during upgrade were not committed to cluster shared authcdb. During the upgrade process, the pcs NWS resource is disabled or PCS cluster resource is stopped completely. When NWS is not running, the /nsr directory is symbolically linked (pointing) to /nsr.NetWorker.local instead of /nsr_share.

/nsr.NetWorker.local is the physical nodes /nsr directory and only contains client-related folders. A /nsr.NetWorker.local/authc folder exists but it does not contain any of the files specific to the NetWorker server's authcdb, this is located under /nsr_share/nsr/authc. 

After the NetWorker upgrade, authc is expecting "version B" of the authc files but is seeing "version A" of the files from before upgrade.

Resolution

Upgrade NetWorker to one of the following releases (or later):
  • 19.8.0.4
  • 19.9.0.2
If you are using an earlier release, upgrade to one of the versions recommended above. NetWorker packages can be downloaded from Dell Support Product Page for NetWorker.

If you are using 19.8.0.4 or 19.9.0.2 and later, perform the following:
  1. On each node in the cluster, rename the /opt/nsr/authc-server/conf/h2_db.properties:
mv /opt/nsr/authc-server/conf/h2_db.properties /opt/nsr/authc-server/conf/h2_db.properties.bak
  1. On each node, rerun the /opt/nsr/authc-server/scripts/authc_configure.sh to reconfigure authc. This does not delete any settings or configurations previously done in authc.
On the active node, this looks something like: 
root@NWrhelNodeH:~# /opt/nsr/authc-server/scripts/authc_configure.sh 

Specify the directory where the Java Standard Edition Runtime Environment (JRE) software is installed [/opt/nre/java/latest]: 

The installation process will install an Apache Tomcat instance. For optimum security, EMC NetWorker Authentication Service will use a non-root user (nsrtomcat) to start the Apache Tomcat instance. If your system has special user security requirements, ensure that proper operational permissions are granted to this non-root user (nsrtomcat). 
Please refer to NetWorker Installation Guide. 

WARNING: Port 9090 is already in use. 

Do you wish to specify a different port number [y]? n 

The Apache Tomcat will use "NWrhelNodeH.emclab.local" as the host name. The Apache Tomcat will use "9090" as the port number. 

The NetWorker Authentication Service requires a keystore file to configure encryption and to provide SSL support. 

EMC recommends that you specify a keystore password that has a minimum of six characters. 

Do you want to use the existing keystore /nsr/authc/conf/authc.keystore [y]? 

Specify password for the existing keystore: 
The install will use the existing certificate "emcauthctomcat" for Apache Tomcat. 
The install will use the existing certificate "emcauthcsaml" for Authentication Service. 

Creating the installation log in /opt/nsr/authc-server/logs/install.log.

Performing initialization. Please wait... 

The installation completed successfully.

On the passive nodes, this looks something like:
  
root@NWrhelNodeG:~# /opt/nsr/authc-server/scripts/authc_configure.sh 

Specify the directory where the Java Standard Edition Runtime Environment (JRE) software is installed [/opt/nre/java/latest]: 

The installation process will install an Apache Tomcat instance. For optimum security, EMC NetWorker Authentication Service will use a non-root user (nsrtomcat) to start the Apache Tomcat instance. If your system has special user security requirements, ensure that proper operational permissions are granted to this non-root user (nsrtomcat). 
Please refer to NetWorker Installation Guide. 

The Apache Tomcat will use "NWrhelNodeG.emclab.local" as the host name. The Apache Tomcat will use "9090" as the port number. 

The NetWorker Authentication Service requires a keystore file to configure encryption and to provide SSL support. 

EMC recommends that you specify a keystore password that has a minimum of six characters. 

Do you want to use the existing keystore /nsr/authc/conf/authc.keystore [y]? 

Specify password for the existing keystore: 
The install will use the existing certificate "emcauthctomcat" for Apache Tomcat. 
The install will use the existing certificate "emcauthcsaml" for Authentication Service. 

The NetWorker Authentication Service defines automatically an administrator user account named administrator in the NetWorker Authentication Service local database. This account is specific to the administration of the NetWorker Authentication Service, and is not related to other administrator accounts on this system. 

******************************************************************************************* 
Password criteria: Minimum required characters - 9 and Maximum allowed characters - 126 Minimum [alphabetic - 2, Uppercase - 1, Lowercase - 1, Numeric - 1, Special character - 1] 
******************************************************************************************** 

Specify an initial password for administrator: 
Confirm the password: 

Creating the installation log in /opt/nsr/authc-server/logs/install.log. 

Performing initialization. Please wait... 

The installation completed successfully.
 
NOTE: On the passive node, you are prompted to create a new password for the NetWorker Administrator account. This does not mean that the existing password is lost. This happens because the authcdb used by the cluster is under /nsr_share/nsr/authc which only exists on the active node. When a passive node becomes the new active node, it uses the shared authcdb. The authc_configure.sh script is run on each node to re-create the /opt/nsr/authc-server/conf/h2_db.properties which is local to each node.
  1. Restart the NWS resource:
pcs resource restart nws
  1. Confirm that the NWS resource has started:
pcs resource 

root@NWrhelNodeH:~# pcs resource 
   * Resource Group: NW_group: 
      * fs (ocf::heartbeat:Filesystem): Started 
NWrhelNodeH.emclab.local 
      * ip (ocf::heartbeat:IPaddr): Started 
NWrhelNodeH.emclab.local 
      * nws (ocf::EMC_NetWorker:Server): Started 
NWrhelNodeH.emclab.local

Nsrlogin attempts and bootstrap backups should succeed.

Additional Information

NetWorker: Red Hat Pacemaker Cluster How to Upgrade NetWorker Server and Best Practices
NetWorker: Troubleshooting Guide for Red Hat Cluster Service Issue

Affected Products

NetWorker

Products

NetWorker Family, NetWorker Series