VCF on VxRail: Install NSX VIB Manually if Installation Fails on ESXi during the VCF Upgrade

It is advised to follow the Broadcom article ESXi host in Not Ready state after major ESXi version upgrade (External Link)

If the above article does not fix the Issue, proceed with the steps shown below.

VIB:
A VIB is an ESXi software package. VMware and its partners package solutions, drivers, CIM providers, and applications that extend the ESXi platform as VIBs. VIBs are available in software depots. See VMware blog What is in a VIB? (External Link)

Image Profile: 
An image profile defines an ESXi image and consists of VIBs. An image profile always includes a base VIB, and might include more VIBs.

Software Depot:
A software depot is a collection of VIBs and image profiles. The software depot is a hierarchy of files and folders and can be available through an HTTP URL (online depot) or a ZIP file (offline depot).

A VIB has three parts:

• VIB: vSphere Installation BundleFile Archive - the main file - The file which gets deployed to the ESXi host.
• XML descriptor file: This has important info about requirements for installing the VIB (dependencies, compatibility, reboot necessary?)
• Signature File: Signature which verifies the level of trust (Integrity, Information about the creator and verification that has been done).

There are two methods for getting the NSX VIBs for manual installation. See Broadcom article Downloading NSX VIB bundles for ESXi hosts from NSX Manager Appliances (External Link) 

Method 1: 
Download the VIBs from the Broadcom Support Portal.

• Leverage the following Broadcom article on how to Download Broadcom Products. (External Link)
• Find the NSX VIB bundle by searching for NSX Kernel Module for VMware ESXi x.x and download the VIB bundle for the current ESXi version. 
  • The VIB bundle is specific to the NSX version, not the ESXi version. If an ESXi host is running 7.0 P09, the VIB bundle NSX Kernel Module for VMware ESXi 7.0 should be downloaded.
• Once the VIB bundle is downloaded, the following process can be used to manually install the kernel modules on an ESXi host:
  • NSX-T Data Center (External Link)
  • VMware NSX (External Link)

Method 2:
Generate the VIB bundle directly from an NSX Manager:

1. SSH as root into any NSX Manager
2. cd into the /repository directory and run ll to list the NSX versions.

Example Screenshot:

3. cd into the current NSX version directory, followed by /HostComponents and run another ll.

Example Screenshot:

4. cd into the esxXX directory (depending on the current ESXi version) and run ll to list the VIBs for the specific ESXi version. 

In this example, the ESXi hosts are running 7.0 P09.

Example Screenshot:

5. Run pwd to output the current path. Copy the path and use it to create the URL in the next step. 

Example Screenshot:

6. Build the URL and paste into any Web Browser to download the NSX VIB bundle. 

7. Download the VIB bundle directly onto the ESXi host by running the below wget command:

wget --no-check-certificate https://<manager_IP/FQDN>/repository/3.2.4.0.0.23653566/HostComponents/esx70/nsx-lcp-3.2.4.0.0.23653567-esx70.zip

8. Once completed, follow the standard instructions to install VIBs manually on an ESXi host:

• NSX-T Data Center (External Link)
• VMware NSX (External Link)

• Place the Host into Maintenance mode before Installing the VIB. To enter maintenance mode through CLI, run the command:

vim-cmd /hostsvc/maintenance_mode_enter

• To Exit out of maintenance mode, run the command:

vimsh -n -e /hostsvc/maintenance_mode_exit

• Alternatively, you can also run the command for maintenance mode:

esxcli system maintenanceMode set –enable true
esxcli system maintenanceMode set –enable false

• To check if the Host is in Maintenance Mode, run the command on the ESXi Host by logging in as a root user:

vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode OR vimsh -n -e /hostsvc/hostsummary | grep inMaintenanceMode

• To confirm there are no VMs running on the ESXi host, run the command:

esxcli vm process list

• Manually install the NSX VIBs on the ESXi host:

• Log in to the host as root or as a user with administrative privileges. Go to the /tmp directory:

[root@host:~]: cd /tmp

• Download and copy the nsx-lcp file into the /tmp directory. Run the install command:

[root@host:/tmp]: esxcli software vib install -d /tmp/nsx-lcp-<release>.zip
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMware_bootbank_nsx-adf_<release>, VMware_bootbank_nsx-aggservice_<release>, VMware_bootbank_nsx-cli-libs_<release>, VMware_bootbank_nsx-common-libs_<release>, VMware_bootbank_nsx-context-mux_<release>, VMware_bootbank_nsx-esx-datapath_<release>, VMware_bootbank_nsx-exporter_<release>, VMware_bootbank_nsx-host_<release>, VMware_bootbank_nsx-metrics-libs_<release>, VMware_bootbank_nsx-mpa_<release>, VMware_bootbank_nsx-nestdb-libs_<release>, VMware_bootbank_nsx-nestdb_<release>, VMware_bootbank_nsx-netcpa_<release>, VMware_bootbank_nsx-netopa_<release>, VMware_bootbank_nsx-opsagent_<release>, VMware_bootbank_nsx-platform-client_<release>, VMware_bootbank_nsx-profiling-libs_<release>, VMware_bootbank_nsx-proxy_<release>, VMware_bootbank_nsx-python-gevent_<release>, VMware_bootbank_nsx-python-greenlet_<release>, VMware_bootbank_nsx-python-logging_<release>, VMware_bootbank_nsx-python-protobuf_<release>, VMware_bootbank_nsx-rpc-libs_<release>, VMware_bootbank_nsx-sfhc_<release>, VMware_bootbank_nsx-shared-libs_<release>, VMware_bootbank_nsx-upm-libs_<release>, VMware_bootbank_nsx-vdpi_<release>, VMware_bootbank_nsxcli_<release>, VMware_bootbank_vsipfwlib_<release>
   VIBs Removed: 
   VIBs Skipped:

Refer to Broadcom documentation Host Fails to Install as Transport Node (External Link)

As a result of adding an ESXi host to the NSX fabric, the following VIBs get installed on the host.

nsx-adf                (Automated Diagnostics Framework) Collects and analyzes performance data to produce both local (at host) and central (across datacenter) diagnoses of performance issues.
nsx-aggservice         Provides host-side libraries for NSX aggregation service. NSX aggregation service is a service that runs in the management-plane nodes and fetches runtime state from NSX components.
nsx-cfgagent           Provides communication between the central control plane and hypervisors. Receives logical networking state from the central control plane and programs this state in the data plane.
nsx-cli-libs           Provides the  NSX CLI on hypervisor hosts.
nsx-common-libs        Provide some utilities classes such as AES, SHA-1, UUID, bitmap, and others.
nsx-context-mux        Provides NSX Guest Introspection relay functionality. Allows VMware Tools guest agents to relay guest context to inhouse and registered third-party partner appliances.
nsx-esx-datapath       Provides NSX data plane packet processing functionality.
nsx-exporter           Provides host agents that report runtime state to the aggregation service running in the management plane.
nsx-host               Provides metadata for the VIB bundle that is installed on the host.
nsx-metrics-libs       Provides metric utility classes for collecting daemon metrics.
nsx-mpa                Provides communication between NSX Manager and hypervisor hosts.
nsx-nestdb             NestDB is a database that stores NSX configurations related to the host (desired/runtime state, etc).
nsx-opsagent           Communicates operations agent executions (transport node realization, Link Layer Discovery Protocol - LLDP,traceflow, packet capture, etc.) with the management plane.
nsx-netcpa             Provides communication required by the different components.
nsx-platform-client    Provides a common CLI execution agent, for centralized CLI and audit log collecting.
nsx-profiling-libs     Provides the functionality of profiling based on gpeftool which used for daemon process profiling.
nsx-proxy              Provides the only northbound contact point agent, which talks to the central control plane and management plane.
nsx-python-gevent      Contains Python Gevent.
nsx-python-greenlet    Contains Python Greenlet library (third party libraries).
nsx-python-logging     Contains the Python logs.
nsx-python-protobuf    Provides Python bindings for protocol buffers.
nsx-rpc-libs           This library provides nsx-rpc functionality.
nsx-sfhc               Service fabric host component (SFHC). Provides a host agent for managing the lifecycle of the hypervisor as a fabric host in the management plane's inventory. This provides a channel for operations such as NSX upgrade and uninstall and monitoring of NSX modules on hypervisors.
nsx-shared-libs        Contains the shared NSX libraries.
nsx-upm-libs           Provides unified profile management functionality for flattening client-side configuration and avoiding duplicate data transmission.
nsx-vdpi               Provides Deep Packet Inspection capabilities for NSX Distributed Firewall.
vsipfwlib              Provides distributed firewall functionality.
nsxcli                 Provides the NSX CLI on hypervisor hosts.

• Verify the VIBs installed:

esxcli software vib list | grep -E 'nsx|vsip' 

or 

esxcli software vib list | grep <yyyy-mm-dd>

More Broadcom articles to review on this topic:

• esxcli software VIB commands to update an ESXi 6.x or 7.x host

Patching ESXi host using Command Line (External Link)

• NSX Troubleshooting