PowerProtect DP Series appliance and IDPA: Apache Log4j CVE-2021-44228

DSA-2021-285: Dell EMC Integrated Data Protection Appliance (PowerProtect DP Series) Security Update for Apache Log4j Remote Code Execution Vulnerability (CVE-2021-44228).
​​

Impact Matrix

 

Dell EMC PowerProtect DP Series Appliance       Integrated Data Protection Appliance	Version Effected?
Component	v2.3.x	v2.4.x	v2.5	v2.6.x	v2.7.0
vCenter (Hypervisor Manager)	Yes	Yes	Yes	Yes	Yes
Data Domain (Protection Storage)	No	No	No	No	Yes
Avamar (Backup Server/Protection Software)	No	No	No	No	No
Data Protection Central (System Manager)	Yes	Yes	Yes	Yes	Yes
Data Protection Search (Search)	Yes	Yes	Yes	Yes	Yes
Cloud Disaster Recovery	No	No	No	No	Yes
Appliance Configuration Manager (ACM)	No	No	No	No	Yes
Data Protection Advisor (DPA/Reporting and Analytics)	Yes	Yes	Yes	Yes	Yes

Permanent Fix

Upgrade to PowerProtect DP Series appliance and IDPA version 2.7.2

• PowerProtect DP Series appliance and IDPA version 2.7.2 release is available on the Dell EMC Support Site.

Goal

This workaround is to remediate CVE-2021-44228 Apache Log4j Remote Code Execution  using the "LORD" automation tool (LOg4J Remediation for Dell) for PowerProtect DP Series Appliance and Integrated Data Protection Appliance (IDPA).

Watch this video on execution of "LORD" automation tool:



Note: Estimated time to run through these steps can be approximately 20 minutes. Reach out to Dell Support at any time for further assistance.

Note:

• This Knowledge base article contains workaround remediation steps for versions 2.3.x, 2.4.x, 2.5, 2.6.x, and 2.7.0.
• Do not use this KB article for any other PowerProtect DP Series or IDPA version.
• This workaround article remediates only CVE-2021-44228.
• Dell EMC Engineering has released PowerProtect DP Series-IDPA version 2.7.1 that fixes CVE-2021-44228 on all components.
• If a user upgrades to a non-remediated version of PowerProtect DP Series appliance or IDPA, then the workaround steps must be re-applied.
• Be logged into Dell.com/support in order to see the attached files and tools.
• Cloud DR component Remediation is not included in the automation tool at this time. If Cloud DR component is deployed, contact Dell EMC Support for assistance to resolve.

Impact/Risks

This tool may disable http and https on the Data Domain in version 2.7.0 to secure that system from CVE-2021-44228, disabling the UI. See below for further information.

There is no functional impact on the appliance. Undo changes on ACM VM before PowerProtect DP Series Appliance or IDPA upgrades to avoid any impact on upgrades. 
 

Pre-requisites:

• Download the latest version of LORD tool from this article (attached files).
• Extract the zip file. 

Running the tool:

Welcome to CVE-2021-44228 Patching Tool.

• This utility assists you in patching CVE-2021-44228 for several Dell products including but not limited to IDPA, PPDM, and NetWorker.
• Special Note: The tool automates remediation steps for all internal components.
• Following remediation and validation checks are also run.
• Advice from Apache regarding CVE-2021-45105 continues to evolve, with new vulnerabilities being assigned new CVE reference IDs. As these new CVEs are discovered, Dell Technologies' Engineering teams will clarify impact and remediation steps where necessary.  
• When these are available, this tool will be updated to include these new steps.

Steps:

1. Copy the "lord_vX" tool onto the ACM into "/tmp" directory using file transfer software like WinSCP, etc.

2. Open SSH to the Appliance Configuration Manager(ACM) server and the login as 'root' user.

3. Run the following command to provide executable permissions:

4. Run the following command to execute the LORD tool:

5. Follow the prompts.

acm5800:/tmp # ./tmp/lord_v9
sh: /tmp/_MEIFMNULP/libreadline.so.6: no version information available (required by sh)
2021-12-23 20:54:22,399 [INFO]  SESSION STARTED
2021-12-23 20:54:22,400 [INFO]  Logging everything to : lord.log
Session Start Time : 2021-12-23 20:54
-------------------------------------------------
-------------------------------------------------
            PowerProtect CVE-2021-44228 Patcher 5.0
      Developer : Pankaj Pande(p.pande@dell.com)
           Release : 21 Dec 2021
-------------------------------------------------
Welcome to CVE-2021-44228 Patching Tool.
This utility will assist you in patching CVE-2021-44228 for several Dell products including but not limited to IDPA, PPDM and NetWorker.
Special Note : The tool automates remediation steps for all internal components. Following remediation, validation checks are also run. Advice from Apache regarding CVE-2021-45105 continues to evolve, with new vulnerabilities being assigned new CVE reference id's.  As these new CVEs are discovered, Dell Technologies' Engineering teams will clarify impact and remediation steps where necessary.  When these are available, this tool will be updated to include these new steps
-------------------------------------------------
2021-12-23 20:54:22,403 [INFO]  Starting Build : 5.0
2021-12-23 20:54:22,403 [INFO]  Using ACM IP as : 10.10.10.99. If you would like to use a different IP, then use the -a flag and specify a different host IP or DNS name
Enter the ACM Password :
Are you using a common password(for all point products) which is same as ACM ? Enter 'y' or 'n' : 

2021-12-23 20:54:34,972 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 20:54:34,972 [INFO]  ///                  Main Menu                      ///
2021-12-23 20:54:34,972 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 20:54:34,972 [INFO]  -------------------------------------------------
Select how would you like to proceed :

1) Apply workaround for PowerProtect DP Series Appliance/IDPA [All components]
2) Apply workaround for vCenter
3) Apply workaround for Data Domain
4) Apply workaround for Avamar
5) Apply workaround for DPSearch
6) Apply workaround for ACM
7) Apply workaround for DPC
9) Exit the Program

Enter your choice number :

Select Option #1 to apply remediation steps on all components.

Important: When applying fix to Data Domain (Protection Storage), you will be prompted to disable the http and https (Only found in IDPA version 2.7.0).

• Selecting "No" will skip Data Domain remediation for now. Another option is to apply Minimum Disruptive Upgrade (MDU) as per KB: 194425

• If you would like to disable the UI, as per automated workflow, you can select "Yes"

Are you ready to disable the DD UI? If you say yes then you will see another prompt to either disable the UI completely or allow certain users to have access. Enter 'y/Yes/YES' or 'n/No/NO':Yes

• You can also choose to select which IP address or Host you would like to limit access to

Are you ready to disable the DD UI? If you say yes then you will see another prompt to either disable the UI completely or allow certain users to have access. Enter 'y/Yes/YES' or 'n/No/NO':y
Would you like to disable the UI completely? Enter 'y' or 'n':n
2021-12-23 21:15:48,365 [INFO]  Disabling GUI http and https access - User based
Enter the list of hostnames or IP-addresses that you would like to give access to(comman-seperated)(eg: 10.118.162.70,10.118.161.130) :

• Sample Output: 

2021-12-23 20:58:56,722 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 20:58:56,722 [INFO]  //        Performing Data Domain Patching            //
2021-12-23 20:58:56,722 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 20:58:56,722 [INFO]  ------------------------------------------------------
2021-12-23 20:58:56,722 [INFO]  Working on Data Domain Host : 10.60.9.51 for patching
2021-12-23 20:58:56,722 [INFO]  ------------------------------------------------------
2021-12-23 20:58:57,266 [INFO]  Found DD version as : 7.6.0.20-689174
2021-12-23 20:58:57,266 [INFO]  This version of Data Domain patching involves disabling the UI.
Are you ready to disable the DD UI? If you say yes then you will see another prompt to either disable the UI completely or allow certain users to have access. Enter 'y/Yes/YES' or 'n/No/NO':y
Would you like to disable the UI completely? Enter 'y' or 'n':y
2021-12-23 21:02:04,087 [INFO]  Disabling GUI http and https access - Completely
2021-12-23 21:02:05,919 [INFO]  HTTP Access:    disabled

2021-12-23 21:02:05,920 [INFO]  GUI http access has been disabled
2021-12-23 21:02:08,507 [INFO]  HTTPS Access:   disabled

2021-12-23 21:02:08,507 [INFO]  GUI https access has been disabled
2021-12-23 21:02:08,509 [INFO]  Data Domain patching completed

Once the remediation steps are done, LORD provides an Overall Status Review:

2021-12-23 21:03:23,782 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 21:03:23,782 [INFO]  //               OVERALL STATUS                   //
2021-12-23 21:03:23,782 [INFO]  ///////////////////////////////////////////////////////
2021-12-23 21:03:23,782 [INFO]  OVERALL STATUS
+---------------------+-------------------------------------------+
| Product             | Patching Status                           |
+---------------------+-------------------------------------------+
| Data Domain         | COMPLETED                                 |
| ACM                 | COMPLETED                                 |
| DPSearch_X.X.X.X    | COMPLETED                                 |
| DPCentral           | COMPLETED                                 |
| DPA                 | NOT_CONFIGURED_IN_IDPA                    |
| Avamar              | SKIPPED/NOT REQUIRED/VERIFICATION SUCCESS |
| vCenter             | SKIPPED/NOT REQUIRED/VERIFICATION SUCCESS |
+---------------------+-------------------------------------------+

Patching Status Definition:

• COMPLETED : This status denotes that the relevant component was vulnerable and has been patched.
• NOT_CONFIGURED_IN_IDPA: This status denotes that the relevant component is not deployed/configured. 
• SKIPPED/NOT REQUIRED/VERIFICATION SUCCESS: This status denotes that the relevant component was skipped as its not vulnerable to CVE-2021-44228, OR the CVE-2021-44228 does not impact the version of that relevant component, Or the relevant component has already been patched. 

Guidance on CVE-2021-45105

• Verification when run on System Manager component of PowerProtect DP Series Appliance[ DPC ] shows that it is vulnerable for CVE-2021-44228 but the impacted Log4j library is not mentioned or loaded in any service. Hence this is a false positive and may be safely ignored.
• Engineering teams are working to confirm whether other components in the IDPA are impacted by CVE-2021-45105.  
• If so, remediation steps will be released in line with Dell Technologies Security Office policy.
• For CDRA, see KB : https://www.dell.com/support/kbdoc/en-au/000194520/powerprotect-dp-series-appliance-and-idpa-apache-log4j-cve-2021-44228-remediation-for-versions-2-3-x-2-4-x-2-5-2-6-x-and-2-7-all-models

 

Subcomponent: Cloud Disaster Recovery

Note: Cloud Disaster Recovery (CDR) within PowerProtect DP Series Appliance/IDPA version 2.7.0 is vulnerable to this CVE-2021-44228 and CVE-2021-45046. 
CDR versions 19.5 and below (IDPA versions 2.6 and below) are not vulnerable and this Knowledge Base article section does not apply to customers on those versions. 

For CDRA – On premise virtual machine

1. Open SSH to CDRA VM using cdr user
2. Create cdra_log4jfix.sh in /tmp/ directory with following content: 

• Run the following command to create the cdra_log4jfix.sh script: vi /tmp/cdra_log4jfix.sh
• Press "I" key on the keyboard to enter Insert mode and copy the contents as provided below. 
• Press ESC, then :wq! to save the file.  

#! /bin/sh
cdr_backup()
{
  mkdir -p /tmp/cdr_backup
  cp /home/cdr/cdra/lib/cdra_main.jar /tmp/cdr_backup/cdra_main.jar.bak
  cp /home/cdr/cdra/resources/restore/restore_vm.jar /tmp/cdr_backup/restore_vm.jar.bak
}
update_executable()
{
  echo "Updating CDRA executable."
  sed -i 's/=CDRS/=CDRS -Dlog4j2.formatMsgNoLookups=true/g' /home/cdr/cdra/executable
}
update_restore_vm()
{
  echo "Updating restore_vm.jar."
  cd /home/cdr/cdra/resources/restore
  zip -q -d restore_vm.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  zip -q -d restore_vm.jar shadow/org/apache/logging/log4j/core/lookup/JndiLookup.class
}
update_cdra_main()
{
  echo "Updating cdra_main.jar."
  LOG4J_JAR_FILE_LOCATION=BOOT-INF/lib/log4j-core-2.13.2.jar
  echo "Stopping CDR service."
  sudo service cdra stop
  cd /home/cdr/cdra/lib/
  mkdir -p BOOT-INF/lib
  unzip -p cdra_main.jar $LOG4J_JAR_FILE_LOCATION > $LOG4J_JAR_FILE_LOCATION
  zip -q -d $LOG4J_JAR_FILE_LOCATION org/apache/logging/log4j/core/lookup/JndiLookup.class
  zip -u -0 -n *.jar cdra_main.jar $LOG4J_JAR_FILE_LOCATION
  rm -rf BOOT-INF
  echo "Starting CDR service. This may take a few minutes."
  sudo service cdra start
  for i in {1..10}
  do
    sleep 30
    echo "Checking CDR service status..."
    RESP_CODE=$(curl -kfsL -o /dev/null -w '%{http_code}' -X GET https://localhost/rest/cdr-version -H "accept: application/json")
    if [[ "$RESP_CODE" == 200 ]]; then
      echo "CDR service started successfully."
      return 0
    fi
  done
  echo "Failed to run CDR service. Please contact Dell Support."
  exit 1
}
main()
{
   CDR_VER=$(curl -s -X GET https://localhost/rest/cdr-version -H "accept: application/json" -k)
   echo "CDR version is : $CDR_VER"
   if [[ $CDR_VER =~ 19\.[6-9] ]]; then
      cdr_backup
      update_executable
      update_restore_vm
      update_cdra_main
    else
      echo "log4j workaround is required only for CDR versions between 19.6 and 19.9."
      exit 0
    fi
    rm -rf /tmp/cdr_backup
}
main

3. Run the following commands to execute the remediation script:

• dos2unix /tmp/cdra_log4jfix.sh
• chmod +x /tmp/cdra_log4jfix.sh
• sudo /tmp/cdra_log4jfix.sh

For CDRS – Deployed on cloud (AWS/AZURE/AWS GOV/AZURE GOV) 

Open a Service Request with Dell Support and see this article number 000194520 to apply the fix to the CDRS (Cloud Disaster Recovery).

Important: Some security scan tools when run on DPC(System Manager) and CDRA component of PowerProtect DP Series Appliance (IDPA) show that it is vulnerable for CVE-2021-44228 even after remediation but the impacted Log4j library is not mentioned or loaded in any service. Hence this is a false positive.

Subcomponent: Avamar Server (Protection Software) and Data Protection Extension

• Avamar Server (Protection Software/Backup Server) is not vulnerable to CVE-2021-44228 or CVE-2021-45046. These vulnerabilities are specific to the JNDI Lookup class  which only exists in the log4j-core jar file.  Avamar Server does not install the jog4j-core jar file. Upgrade to PowerProtect DP Series Appliance (IDPA) version 2.7.1 can be performed if customers would still like to update the version of log4j to 2.16. This update may prevent false positive notifications by security scanning tools.

• Cloud Director Data Protection Extension (if configured) is still vulnerable and the workaround steps mentioned in KB 194480 can be applied to Data Protection Extension components prior to version 19.4. 

• For 19.4 vCloud Director Data Protection Extension, we recommend applying the 19.4.0.214_HF.5 hotfix as described in the Remediation section. Patch details can be found on KB 194480. 

If you implement the workaround steps described in this section, and then upgrade the Data Protection Extension to a non-remediated version, then you must re-implement the workaround steps. 

PowerProtect DP4400, PowerProtect DP5300, PowerProtect DP5800, PowerProtect DP8300, PowerProtect DP8800, PowerProtect Data Protection Software, Integrated Data Protection Appliance Family, Integrated Data Protection Appliance Software , PowerProtect DP5900, PowerProtect DP8400, PowerProtect DP8900 ... View More about warranties View Less about warranties