PowerProtect Data Manager 19.16 Oracle RMAN User Guide

Agent registration

On Linux or AIX, if the agent fails to establish a connection with the PowerProtect Data Manager server, agent registration might fail with the following error message:

During a network connectivity test, the agent is unable to reach the PowerProtect Data Manager server by using ping and curl. 

1.	If the ping command is blocked in the environment and curl is not installed, the agent registration can still complete successfully. 
Review the agent service logs at /opt/dpsapps/agentsvc/logs to verify that the registration is successful. If the registration is successful, the status of the agent host indicates Registered in the PowerProtect Data Manager UI.
2.	If the ping command is not blocked in the environment, the agent registration might not complete successfully because a network connection cannot be started. If this occurs, complete the following steps to troubleshoot the issue:

If agent registration fails with this error message, complete the following operation:

1. Use any network packet tracing tool to trace the packets from the agent system to PowerProtect Data Manager.
2. Start the packet tracing between the source IP of the agent system and the destination IP of PowerProtect Data Manager.
3. Start the network traffic between the agent system and PowerProtect Data Manager.

   Wait 10 to 15 seconds.

4. Analyze the captured packets.
5. Look for SYN and SYN_ACK packets to see if a 3-way handshake is being performed.

   Determine whether the source agent or the destination PowerProtect Data Manager is blocking the connection.

   If network traffic is blocked, contact your network security team to resolve the port communication issue.

PowerProtect agent service operations

To troubleshoot PowerProtect agent service operations, you can check the PowerProtect agent service log file OpAgentSvc-<timestamp>.log, which is created in <agent_service_installation_location>/logs on AIX or Linux. To modify the log level and retention of temporary files, you can modify specific parameter settings in the config.yml file.

To modify the log level and retention of temporary files, you can perform the following steps:

1. Stop the agent service.
2. Open the config.yml file in an editor.
3. Modify the log-level settings in the following parameters, as required:
   • DEBUG
   • INFO
   • WARNING
   • ERROR
   • CRITICAL
   NOTE:These parameters are listed in order of decreasing number of messages in the debug information output. The default log-level is INFO.
4. To retain the temporary files, set the keepTempFiles parameter to True in the config.yml file.
   NOTE:The agent service and application agent communicate through the temporary files, which are typically deleted after use but can be useful for troubleshooting purposes. Do not leave the keepTempFiles parameter set to True permanently, or the temporary files can use excessive space on the file system.
5. Start the agent service.

Uninstalling the PowerProtect agent service results in "Segmentation fault(coredump)" error on AIX platforms

After updating the application agent from version 19.12 to 19.13, uninstalling the PowerProtect agent service results in the following error:

/usr/sbin/rpm_share [470]: 19005466 Segmentation fault(coredump)

This behavior does not impact reinstallation of the PowerProtect agent service.

The core dump error is due to a faulty rpm.rte file with version 4.13.0.3. For more information, see update rpm.rte to the latest version to resolve rpm errors and core dumps.

To resolve this issue, either update the rpm.rte file to the latest version or update the AIX Technology Level (TL) version. The following article provides more information about TL versions:

Fileset information for: rpm.rte

Deletion compliance for SLA validation job marked as "Failed" in PowerProtect Data Manager UI

Due to a cancel request and create-new-copy deletion request occurring simultaneously, deletion compliance for an SLA validation job is incorrectly shown as "Failed" in the UI, even though the copy deletion has succeeded. This issue can occur if replication copies exist, but the corresponding backup copy has been deleted. In this case, the retention time of the replication copy is greater than the retention time of the backup copy. Once the replication copies expire or are manually deleted, it triggers the catalog deletion request, which marks the copy as "Deleted."

Application agent update failure in PowerProtect Data Manager UI on Linux

On a Linux OS that uses /etc/init.d instead of systemd to start the OS services, an update of a previous application agent version in the PowerProtect Data Manager UI might fail with the following error message:

Upgrade operation failed with error 'tuple' object has no attribute 'lower', errorcode: .

As a workaround, update the application agent by using the appropriate update commands and command-line procedure as described in this guide.

Configuration failure after application agent update when database is unresponsive

After an update of the application agent and PowerProtect Data Manager to release 19.13, while the Oracle database is in a shut down or unresponsive state, asset discovery by PowerProtect Data Manager does not discover the database. As a result, a configuration operation fails, such as the creation or movement of a protection policy.

As a workaround, complete the following steps:

1. Ensure that the Oracle database is in a responsive state.
2. Run a full asset discovery manually or wait until the next auto-discovery is completed.
3. Perform the configuration operation.

Asset duplication issue after downgrade

When you downgrade the Oracle RMAN agent to an earlier version and register the agent with the same PowerProtect Data Manager server, the downgrade creates duplicate asset entries in the PowerProtect Data Manager UI.

In the PowerProtect Data Manager UI, you can view the duplicate asset entries by selecting Infrastructure > Assets and then clicking the Oracle tab.

The downgrade of the Oracle RMAN agent is not supported.

Issue with hostnames used for discoveries after a hostname change

After the Oracle RMAN hostname is changed, a scheduled discovery might run with the new hostname and a manual discovery might run with the old hostname. The PowerProtect Data Manager UI displays the different hostnames that are used for the scheduled and manual discovery operations.

As a workaround, complete the procedure in Enable the Oracle RMAN agent after Oracle RMAN hostname change.

Incorrect SID or database name in rename.config file

After you change the SID of a stand-alone database or the database name of a RAC database, you can continue using an existing asset from an existing protection policy with PowerProtect Data Manager. You need to provide the previous SID of the stand-alone database or previous database name of the RAC database in the rename.config file. If the SID or database name in the rename.config file is incorrect, the following error appears during a backup:

Some of the assets have the new natural ID status. Please reconfigure the PLC.

Use the following steps to resolve the issue:

1. Make any required corrections of the SIDs or database names in the rename.config file. Enable the use of an existing asset in a modified environment provides more details about updating the rename.config file to enable the continued use of an existing asset.

2. Reconfigure the PowerProtect Data Manager protection policy:

   1. In the PowerProtect Data Manager UI, select Protection > Protection Policies.

   2. On the Protection Policies page, select the existing protection policy in the list and click Set Lockbox, which starts the configuration.

      NOTE:Use of Set Lockbox is required only when a previous SID or database name was incorrectly specified and then later corrected.

Failed migration of RAC_BACKUP_NODE_IP parameter setting

When the RAC_BACKUP_NODE_IP parameter in the configuration file is set to an FQDN (rather than IP address), an update of the PowerProtect Data Manager fails to migrate this parameter setting to the preferred node setting on the Assets page in the PowerProtect Data Manager UI.

The preferred node setting in an Oracle RAC environment is only applicable for a centralized protection policy, not for a self-service protection policy.

As a workaround, ensure that the RAC_BACKUP_NODE_IP parameter in the configuration file is set to the required IP address prior to updating the PowerProtect Data Manager from a previous version. Then the PowerProtect Data Manager update automatically migrates the RAC_BACKUP_NODE_IP parameter setting to the preferred node setting on the Assets page in the PowerProtect Data Manager UI.

Assign a preferred node in Oracle RAC environment provides details on the preferred node setting in an Oracle RAC environment.

Issue on AIX with ddbmcon process not exiting

On AIX, due to an Oracle OCI library issue, the ddbmcon process does not exit after completing all operations, which can cause different types of symptoms. The following symptoms can occur:

1. The ddbmcon process has a child process.
2. The process ID is 1 for the parent process of the ddbmcon process.
3. The process ID is not 1 for the parent process of the ddbmcon process and the debug log shows that the ddbmcon process finished all operations. However, the ddbmcon process does not exit.

These symptoms are caused by the Oracle bug 34372439, "OCI APPLICATION LOADING DLLS DYNAMICALLY HANGS".

As a workaround, contact Oracle to obtain the latest patch.

Edit Retention button is disabled for asset copies

On the Infrastructure > Assets page > Asset <asset name> page, the Edit Retention button is disabled.

The Edit Retention button is enabled only when the copy status of the asset source is Available. In detail, if deleting a copy of a protected asset fails, the Copy Status will be updated to another state from Available, and the Edit Retention button is disabled.

Workarounds are:

• Edit the retention period of an asset copy if its status is Available.
• Do not try deleting an asset copy before its retention period.

Stand-alone database is incorrectly discovered as RAC database with Oracle RMAN agent pre-19.14

With an Oracle RMAN agent version earlier than 19.14, the PowerProtect Data Manager discovery of an Oracle stand-alone database might incorrectly create a RAC asset entry in the PowerProtect Data Manager UI.

When you update the Oracle RMAN agent to the latest version and register the agent with the same PowerProtect Data Manager server, the update creates a new STANDALONE asset entry in the PowerProtect Data Manager UI for the stand-alone database. To view the duplicate asset entries in the PowerProtect Data Manager UI, select Infrastructure > Assets and then select the Oracle tab.

The correct discovery of the stand-alone database occurs when the following conditions are met:

• When you run the ps -ef | grep lmon | grep <Oracle_SID> | grep -v grep command on the Oracle host, the following type of output is produced without errors:

  # ps -ef | grep lmon | grep test | grep -v grep
  oracle   1871582       1  0 Jun11 ?        00:00:37 ora_lmon_test

• When you set the ORACLE_HOME parameter as an Oracle user and run the srvctl status database -d <db_unique_name> command on the Oracle host, the following type of output is produced without errors:

  $ srvctl status database -d test
  Instance test is running on node blrv009b144

To protect the newly discovered database, perform the following steps.

1. Add the newly discovered stand-alone asset to the same protection policy or a newly created protection policy.
2. Perform a full backup of the Oracle database.

The backup copies created for the old asset are retained and usable for disaster recovery until they expire. It is recommended that you add the newly discovered asset to the same protection policy for performing an online database recovery. It is also recommended that you do not set the ENABLE_BROWNFIELD_DISCOVERY parameter to TRUE for the newly discovered asset.

Asset configuration job is queued when a policy configuration job fails to create a storage unit

An asset-level configuration job runs only after the corresponding policy configuration job creates the required storage unit for the protection policy. If the policy configuration job fails to create the storage unit, the asset configuration job is queued, waiting for the storage unit to be created. The asset configuration job remains queued for a maximum of 24 hours and is then canceled automatically.

You can monitor the status of policy configuration jobs and asset configuration jobs in the System Jobs window in the PowerProtect Data Manager UI.

As a workaround, try to restart the failed policy configuration job while the asset configuration job is queued. If the policy configuration job succeeds in creating the required storage unit, the asset configuration job restarts and runs to completion. Otherwise, if the policy configuration job cannot create the storage unit, the asset configuration job is canceled after being queued for 24 hours.

Oracle RAC passive nodes might not be detected during discovery

During the PowerProtect Data Manager automatic discovery of nodes in an Oracle RAC One Node or active/passive RAC environment, the passive nodes might not be detected. For example, when an Oracle database is in the mount state, the discovery of the passive nodes does not occur.

As a workaround, you can enable the scheduled or manual discovery in the Oracle active/passive RAC system by setting specific parameters in the Oracle RMAN agent configuration file $RMAN_AGENT_HOME/config/rman_agent.cfg. Set the following parameters in the [SID_name] section of the configuration file for each node of the cluster (both active and passive nodes):

• Set the IS_RAC parameter to TRUE.
• Set the CLUSTER_TYPE parameter to RACOneNode.
• Set the RAC_NODE_IP<n> and RAC_NODE_FQDN_<n> parameters to the IP address and FQDN value, respectively, of each node in the cluster.

For example, the Oracle active/passive RAC system has the database orcl with the cluster nodes blrv009b144.blr.lab.emc.com and blrv009b145.blr.lab.emc.com with instances orcl_1 and orcl_2. In this case, the following parameters must be set in the configuration file $RMAN_AGENT_HOME/config/rman_agent.cfg:

[SID_orcl_1]
IS_RAC = TRUE
CLUSTER_TYPE = RACOneNode
RAC_NODE_IP_1 = 10.125.33.144
RAC_NODE_FQDN_1 = blrv009b144.blr.lab.emc.com
RAC_NODE_IP_2 = 10.125.33.145
RAC_NODE_FQDN_2 = blrv009b145.blr.lab.emc.com

[SID_orcl_2]
IS_RAC = TRUE
CLUSTER_TYPE = RACOneNode
RAC_NODE_IP_1 = 10.125.33.144
RAC_NODE_FQDN_1 = blrv009b144.blr.lab.emc.com
RAC_NODE_IP_2 = 10.125.33.145
RAC_NODE_FQDN_2 = blrv009b145.blr.lab.emc.com

Alert might be generated from Oracle RAC passive node

In an Oracle RAC One Node or active/passive RAC environment, the database is not running from the passive nodes, which is expected. However, if an /etc/oratab entry exists for the instance, then the Oracle RMAN agent tries to connect to the database and generates error. This error is sent to PowerProtect Data Manager, which generates a critical alert.

To suppress the generated alerts, perform either of the following actions:

• Comment out or remove the Oracle RAC One Node instance entry in the /etc/oratab file for the passive nodes.
• In the [GENERAL] section of the configuration file $RMAN_AGENT_HOME/config/rman_agent.cfg, set the IGNORE_ORATAB_INACTIVE_SID parameter to TRUE.

Manual discovery of Oracle client might fail with error message

When you perform a manual or on-demand discovery of the Oracle client, the discovery operation might fail with the following type of error message:

ImportError: Could not load /opt/dpsapps/agentsvc/lib/libddstat.so.

This error can be caused by stale application agent directories, such as stale File System agent directories, which are located on the Oracle client host.

As a workaround, ensure that all the agents and the agent service on the Oracle client host are the same version. Check for and remove any stale application agent directories, and then restart the agent service.

Oracle Incremental Merge operation might fail after BoostFS RPM update

After an update of the Boost RPM package from a previous version to the latest version, an Oracle Incremental Merge configuration or backup operation might fail with the following error message:

Unable to complete BoostFS operation.

As a workaround, use any of the following alternative procedures to reset the lockbox or migrate the lockbox contents:

• Delete the older lockbox files from the boostfs directory and then reset the lockbox:

  1. Run the following command to delete the older lockbox files:

     rm -f  /opt/emc/boostfs/lockbox/boostfs*_

  2. In the PowerProtect Data Manager UI, select Protection > Protection Policies, and then select the policy in the list and click Set Lockbox.

• With customized locations of lockbox files, run the following command to migrate the lockbox files manually:

  NOTE:Run this command after you update the Boost RPM package.

  <lockbox_directory>/bin/boostfs migrate -l <lockbox_pathname>

  For example:

  /opt/emc/boostfs/bin/boostfs lockbox migrate -l /opt/emc/boostfs/lockbox/boostfs.lockbox

• To enable the automatic migration of lockbox files, edit the /opt/emc/boostfs/etc/boostfs.conf file and specify the lockbox-path parameter setting under [global] within the file.

  NOTE:Enable the automatic migration before you update the Boost RPM package.

  For example:

  lockbox-path=/opt/emc/boostfs/lockbox/boostfs_oracle.lockbox