VxRail: VxRail Manager Workaround to Remediate Apache Log4Shell Vulnerability (CVE-2021-44228, CVE-2021-45046, and CVE-2021-4104)

The Apache Software Foundation has published information about a critical Apache Log4j Library Remote Code Execution Vulnerability issue that is known as Log4Shell as per the GitHub Advisory Database (also detailed in CVE-2021-44228, CVE-2021-45046, and CVE-2021-4104). VxRail Manager is exposed to the issue outlined in the vulnerability.

Note: Two additional CVEs, CVE-2021-45046 and CVE-2021-4104, are reported indicating that the original recommendation to remediate the issue described in CVE-2021-44228 (Log4j 2.x) is not a complete fix.

For more on these CVEs, see the following articles:

• Apache Log4j Security Vulnerabilities
• CVE-2021-45046 (Log4j 2.15)
• CVE-2021-4104 (Log4j 1.2)

Note: The script in this article has been updated to version 1.1.2 which includes the remediations that are recommended for all three CVEs, CVE-2021-44228, CVE-2021-45046, and CVE-2021-4104.

An additional issue was discovered in the previous script which may result in an impacted file being restored on VxRail Manager from a system archive. This issue has also been addressed in this release.

If you used any previous scripts that were provided with this article, then download the latest script (1.1.2) and run it on VxRail Manager to ensure you have the complete fix.
 

Requirements and Scope

This scope of what is covered by the remediation steps in this article are:

• This article is applicable to VxRail Manager in VxRail 4.5.x, 4.7.x, and 7.0.x releases along with VxRail Manager in VCF 3.x and 4.x releases.
• The script and remediation steps provided remediate the vulnerability in VxRail Manager appliance VM only.
• Other components outside VxRail Manager such as vCenter Server Appliance (vCSA), NSX, and so forth must be mitigated separately, and are not in this script.
• Also, the script does not remediate any applications or services running inside VMs which may be exposed to the vulnerability. Dell EMC recommend all customers check with their application or software vendors for services running in VMs to ensure they are not impacted.

Links to impacted VMware products and potential workarounds are detailed in the following VMware VMSA article:

• VMware Security Advisory VMSA-2021-0028

VMware provides a script to automate the remediation in the vCenter Server Appliance in the following article:

• Python script to automate the workaround steps of VMSA-2021-0028 vulnerability on vCenter Server Appliance (87088)

Attached to this article is file fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip which contains the script for VxRail Manager only.
 

VxRail releases with fix included

This issue has is resolved in the following VxRail software releases:

• VxRail Package Software 7.0.320
• VxRail Appliance Software 4.7.541
• VxRail Appliance Software 4.5.471

It is recommended to upgrade to a VxRail software release which includes the fix.
The script is recommended for customers who are unable to upgrade immediately.

Note: If your VxRail 7.0.xxx cluster is managed by a customer managed vCenter, see the following article for additional considerations which may apply:

• Dell EMC VxRail: Upgrade external vCenter to 7.0 U3c or above fails at pre-check due to hosts are not upgraded first.

 

Remediation steps

The remediate against the issue perform the following steps:

1. Download the fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip file that is attached to this article.
2. Upload the .zip file to VxRail Manager using mystic user over SCP (WinSCP is an example of an SCP client that can be used).
3. Log in to VxRail Manager VM console or SSH using the mystic user. 
4. Change directory to where you uploaded the .zip file and extract it using the unzip command:

   mystic@vxrm:~> unzip fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip
   Archive:  fixlog4j-CVE-2021-44228-CVE-2021-45046-v-1-1-2.zip
     inflating: fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

5. Make the script executable using the chmod command:

   mystic@vxrm:~> chmod +x fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

6. Log in as the VxRail Manager root user using the su command:

   mystic@vxrm:~> su -
   Password:

7. Ensure you are in the same directory where you extracted the script package:

   vxrm:~ # cd /home/mystic
   vxrm:/home/mystic #

8. Run the script:

   vxrm:/home/mystic # ./fixlog4j-CVE-2021-44228-CVE-2021-45046.sh

   Example script output:

   Stop MARVIN and runjars service before patching the system
   
   /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar is affected by CVE-2021-44228 and CVE-2021-45046, need to apply patch
   patching /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
   Successfully patched /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
   
   /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar is affected by CVE-2021-44228 and CVE-2021-45046, need to apply patch
   patching /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
   Successfully patched /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
   
   To ensure there is no reload behavior, we need to pack the .war file as well.
   looks like /usr/lib/vmware-marvin/marvind/webapps/ROOT.war contains the bad log4j-core library WEB-INF/lib/log4j-core-2.13.0.jar
   Archive:  /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
     inflating: WEB-INF/lib/log4j-core-2.13.0.jar
   Patching WEB-INF/lib/log4j-core-2.13.0.jar in /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
   Repack /usr/lib/vmware-marvin/marvind/webapps/ROOT.war
   updating: WEB-INF/lib/log4j-core-2.13.0.jar (deflated 11%)
   Clean up the ROOT folder...
   
   Always apply a reboot of MARVIN and runjars services
   restart MARVIN
   MARVIN restart successfully
   restart runjars
   runjars restart successfully

    

Note: It may take anywhere from 5-10 minutes for all files to be remediated and for the VxRail Manager services to start.

Wait at least 10 minutes if you plan to perform any of the manual validation steps below.

There are several different versions of the lib4j-core library depending on the release of VxRail Manager.

The script has been designed to correctly remediate VxRail Manager regardless of the version of lib4j-core which is included with that version of VxRail Manager.

The above output from running the script may show different files being updated depending on the version of lib4j-core included.

Note: If you perform a VxRail Upgrade to another build of VxRail which does not contain a fix, then this script must be run a second time to reapply the remediation.
 

 NOTE: The full mitigation for VxRail requires both the vCenter Server Appliance (vCSA) workaround from VMware, and the remediation on VxRail Manager that is performed by this script to be implemented.

Links to VMware articles covering workarounds to their products and fixes is found in VxRail: Information on Log4Shell (CVE-2021-44228) and VxRail environments.
 

Validation Steps

In order to remediate the issue, the script removes the JndiLookup.class file from the lib4j-core-* jar files.

A jar file is a Java packaging format for including multiple class, metadata, and other java programs in a single file. It is similar in concept to a .zip file and is based on the .zip format. The script execution validates that each jar file has been successfully updated.

To perform a manual validation that the script has worked, you can check if the log4j-core-* jar files present on VxRail Manager still contain the impacted JndiLookup.class file. If it has worked, you should see no output to the below commands which confirms the impacted JndiLookup.class file is no longer present in the jar file.

Note: The issue with the JndiLookup.class file is fixed in log4j-core-2.17.1.jar and later versions so attempting to run the validation steps on these jar files shows the JndiLookup.class file is present, but the issue isn fixed. It is safe to ignore seeing that JndiLookup.class file in fixed versions of log4j-core.
 

Validation with automated command 

The following command can be run on VxRail Manager to scan for all log4j-core-xxxx.jar files present on VxRail Manager and check to see if they contain the impacted JndiLookup.class file:

vxrm:/home/mystic # for myfile in `find / -name log4j-core*jar -print |grep -v log4jbak`; do echo $myfile; unzip -l $myfile | grep JndiLookup.class; done

Sample output (updated system):

/mystic/connectors/eservice/lib/log4j-core-2.13.0.jar
/mystic/connectors/cluster/lib/log4j-core-2.13.0.jar
/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar

In the above example the JndiLookup.class file is not present in jar, thus the script worked, and verification is successful.

Below is Sample output of an impacted system which must be updated:

/mystic/connectors/eservice/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/cluster/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class
/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.3.jar
     2892  2020-05-10 12:08   org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example, the impacted JndiLookup.class file is still present in the log4j-core-2.13.3.jar jar file.

Below is Sample output of a system which has a fixed or updated log4j-core library where seeing the JndiLookup.class file can be ignored:

/usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/eservice/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class
/mystic/connectors/cluster/lib/log4j-core-2.17.1.jar
     3158  2021-12-27 17:30   org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example you can see the JndiLookup.class file in the output but the fix for the issue is in log4j-core-2.17.1.jar.

 

Validation with manual check of each file

To quickly identify any log4j-core-xxxx.jar files present on VxRail Manager, run the following command (this also formats the output into a usable command):

vxrm:/home/mystic # find / -name log4j-core*jar -print |grep -v log4jbak | awk '{print("unzip -l " $1 "|grep JndiLookup.class")}'

Sample output:

unzip -l /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
unzip -l /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
unzip -l /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar|grep JndiLookup.class

From the sample output above, run each command manually to see if they detect the impacted JndiLookup.class file:

vxrm:/home/mystic # unzip -l /mystic/connectors/eservice/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

vxrm:/home/mystic # unzip -l /mystic/connectors/cluster/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

vxrm:/home/mystic # unzip -l /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.0.jar|grep JndiLookup.class
vxrm:/home/mystic #

In the above example, the JndiLookup.class file is not present in jar, thus the script worked, and verification is successful.

Note: The jar filenames from the sample output above may differ depending on your version of VxRail Manager. Use the sample output that you receive from the above find command. 

An example of the output for a jar file which is still impacted and contains the impacted JndiLookup.class file:

vxrm:/home/mystic # unzip -l /mystic/connectors/cluster/lib/log4j-core-2.4.1.jar |grep JndiLookup.class
2576 2015-10-08 17:50 org/apache/logging/log4j/core/lookup/JndiLookup.class

In the above example, the impacted JndiLookup.class file is still present in the log4j-core-2.4.1.jar jar file.

 

 NOTE: Another reminder that full mitigation for VxRail requires both the vCenter Server Appliance (vCSA) workaround from VMware, and the remediation on VxRail Manager performed by this script to be implemented.