Secure Connect Gateway Virtual Edition (Virtual Appliance/SCG-VE): Unable to register Secure Connect Gateway

Registering Secure Connect Gateway fails with the following error even though the local system can connect to the Dell Enterprise servers through ports 443 and 8443.

We cannot connect to the Dell backend. Please check and verify your network configuration, ports 8443 and 443 are required and must be open to esrs3-core.emc.com,esrs3-coredr.emc.com.

connectivityreport.log:

2023-02-12 07:12:41,863 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , IOException ConnectTimeoutException
2023-02-12 07:12:41,863 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , Exception - Connect to esrs3-core.emc.com:8443 [esrs3-core.emc.com/128.221.236.246] failed: Read timed out
2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy false
2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy()
2023-02-12 07:12:41,872 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , IOException ConnectTimeoutException
2023-02-12 07:12:41,872 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , Exception - Connect to esrs3-coredr.emc.com:8443 [esrs3-coredr.emc.com/168.159.224.236] failed: Read timed out
2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy false
2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy()
2023-02-12 07:12:52,068 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , IOException ConnectTimeoutException
2023-02-12 07:12:52,069 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , Exception - Connect to esrs3-core.emc.com:8443 [esrs3-core.emc.com/128.221.236.246] failed: Read timed out
2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy false
2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Call populateConnectivityBean: host esrs3-core.emc.com port: 8443
2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy()
2023-02-12 07:12:52,087 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , IOException ConnectTimeoutException
2023-02-12 07:12:52,088 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , Exception - Connect to esrs3-coredr.emc.com:8443 [esrs3-coredr.emc.com/168.159.224.236] failed: Read timed out
2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy false
2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Call populateConnectivityBean: host esrs3-coredr.emc.com port: 8443
2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy()
2023-02-12 07:12:54,485 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , IOException SSLHandshakeException
2023-02-12 07:12:54,485 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure
2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server.
2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy true
2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ]  avgCount: 1 timeTaken: 2397 host: esrs3-coredr.emc.com 
2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:54,486 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy()
2023-02-12 07:12:54,503 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , IOException SSLHandshakeException
2023-02-12 07:12:54,503 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure
2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server.
2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy true
2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ]  avgCount: 1 timeTaken: 2434 host: esrs3-core.emc.com 
2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 
2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy()
2023-02-12 07:12:56,838 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , IOException SSLHandshakeException
2023-02-12 07:12:56,838 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server.
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy true
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ]  avgCount: 2 timeTaken: 4750 host: esrs3-coredr.emc.com 
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Call populateConnectivityBean: host esrs3-coredr.emc.com port: 443
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Time taken: 4750
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] avgPingTime: 2375.0
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Latch count is   1
2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] ----------------------------------------
2023-02-12 07:12:56,945 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , IOException SSLHandshakeException
2023-02-12 07:12:56,946 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure
2023-02-12 07:12:56,946 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server.

The network traces captured by the Tcpdump command indicates that SCG-VE could connect the Dell Enterprise servers through ports 443 and 8443.

Secure Connect Gateway uses 1500 MTU size by default to connect to the Dell Enterprise servers. However, the customer's Internet provider does not allow the 1500 MTU size.

Perform the below commands directly on the SCG-VE to change the MTU size to 1454.
Changing the MTU size of eth0 on the SCG-VE does not resolve the issue since one of the dockers named "esrsde-app" on the SCG-VE connects to the Dell backend.

Check the current MTU size of sae-srs-bridge:

ifconfig sae-srs-bridge

# ifconfig sae-srs-bridge
sae-srs-b Link encap:Ethernet  HWaddr 02:42:25:62:85:66
          inet addr:172.18.0.1  Bcast:172.18.0.7  Mask:255.255.255.248
          inet6 addr: fe80::42:25ff:fe62:8566/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2472 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2528 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:12309976 (11.7 Mb)  TX bytes:289066 (282.2 Kb)

Take a Snapshot of the VM before any changes are performed as detailed below:

docker network disconnect sae-srs-bridge saede-app
docker network disconnect sae-srs-bridge esrsde-app
docker network disconnect sae-srs-bridge esrsalert-app
docker network rm sae-srs-bridge
docker network create --driver bridge --subnet 172.18.0.1/29 --opt com.docker.network.bridge.name=sae-srs-bridge --opt com.docker.network.driver.mtu=1454 sae-srs-bridge
docker network connect sae-srs-bridge esrsde-app
docker network connect sae-srs-bridge saede-app
docker network connect sae-srs-bridge esrsalert-app

Verify:

ifconfig sae-srs-bridge

# ifconfig sae-srs-bridge
sae-srs-b Link encap:Ethernet  HWaddr 02:42:DE:D8:AB:D2
          inet addr:172.18.0.1  Bcast:172.18.0.7  Mask:255.255.255.248
          inet6 addr: fe80::42:deff:fed8:abd2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1454  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:746 (746.0 b)

FYI:
If unable to access the UI after the above change, then either REBOOT the VM and wait for 4 minutes for containers to initialize. Or restart the service systemctl restart esrsve.service.