User has changed/updated VPlex management server IP address(either cluster-1 or/both cluster-2) or cluster-witness IP address
Issue Description:
Configuring 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server fails with below error message:
VPlexcli:/> configuration cw-vpn-configure -i xx.xx.xx.xx --force
Please enter the IP address of the remote cluster management server that will be included in the 3-way VPN setup: yy.yy.yy.yy
Verifying the VPN status between the management servers...
IPSEC is UP
Remote Management Server at IP Address yy.yy.yy.yy is reachable
Remote Internal Gateway addresses are reachable
Verifying the VPN status between the management server and the cluster witness server...
IPSEC is not UP
Cluster Witness Server at IP Address 128.221.254.3 is not reachable
Error during Cluster Witness VPN Establishment: IPSEC configuration failed: Cannot proceed to configure the Cluster Witness Server IPSec configuration.
It is possible that a 3-way VPN has already been established and you have given a wrong Cluster Witness Server public IP address.
Please re-run the cluster witness server vpn configuration with the right public IP address.
Resetting the Cluster Witness VPN configuration
Resetting the Cluster Witness Server VPN configuration
.
.
.
<./truncated>
There could be 2(two) scenario's during which configuration of 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server can fail as follows:
1) User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server.
AND/OR,
2) User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server.
Please go through below scenario details and resolution step's in-order to resolve this issue:
Scenario 1: User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server
NOTE: In below example's, user has changed VPlex management server IP address of both cluster-1 and cluster-2.
1) Please gather correct IP-address's assigned to VPlex management server (both cluster-1 and cluster-2) as follows:
Cluster-1:
VPlexcli:/> vpn status
Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14N.NNN.N.NNN is reachable Remote Internal Gateway addresses are reachable . </truncated>
Cluster-2:
VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14M.MMM.M.MMM is reachable Remote Internal Gateway addresses are reachable . </truncated>
2) Please perform SSH to the cluster-witness server using its public IP address :
2.a) To find the public IP-address of cluster-witness server, run the following VPlexcli command:
Example:
VPlexcli:/> ll /cluster-witness/
/cluster-witness:
Attributes:
Name Value
------------------ -------------
admin-state unknown
private-ip-address 128.221.254.3
public-ip-address XX.XX.XX.XX <<< Cluster-Witness server public IP-address
2.b) Please perform SSH to the public IP-address of cluster-witness obtained from step 1.a as follows:
service@ManagementServer:~> ssh <cluster-witness-public-IP-address>
Example:
service@ManagementServer:~> ssh xx.xx.xx.xx >> cluster-witness-public-IP-address
Warning: Permanently added 'xx.xx.xx.xx' (ECDSA) to the list of known hosts.
Last login: Mon Jun 06 15:33:14 2016 from xx.xx.xx.xx
service@ClusterWitness:~>
3) Cat the IPsec.config file and search for VPlex management server cluster-1 and cluster-2 IP address s as follows:
Note: Before carrying out to step(3) confirm the actual IP address of VPlex management server's from vpn status output using step(1). Once this information is gathered, please compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not.
Example:
service@ClusterWitness:~> cat /etc/ipsec.conf
# Add connections here.
# Setup a tunnel between the management servers and the Cluster Witness Server
# "left" means local, "right" means remote.
# Connection between Cluster Witness Server and Management Server
conn witness-cluster2
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.254.3/32
leftcert=hostCert.pem
right=15Y.YYY.Y.YYY <<========== Old/incorrect IP address of VPlex management server-2
rightsubnet=128.221.252.64/27,128.221.253.64/27
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes256-sha256
auto=start
# Connection between Cluster Witness Server and Management Server
conn witness-cluster1
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.254.3/32
leftcert=hostCert.pem
right=15X.XXX.X.XXX <<========== Old/incorrect IP address of VPlex management server-1
rightsubnet=128.221.252.32/27,128.221.253.32/27
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes256-sha256
auto=start
4) As from the above example, we found that IPsec.config file of cluster-witness is still carrying
old IP-address of VPlex management server-1 and cluster-2. Therefore, please edit the file IPsec.config of cluster-witness server using vi editor to update the correct IP-address of VPlex management server-1 and cluster-2.
NOTE: Place the correct IP address of VPlex cluster-1 and cluster-2 after the equals sign with no space between the equal sign and IP address, save and exit the file.
Example:
service@ClusterWitness:~> vi /etc/ipsec.conf
# Add connections here.
# Setup a tunnel between the management servers and the Cluster Witness Server
# "left" means local, "right" means remote.
# Connection between Cluster Witness Server and Management Server
conn witness-cluster1
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.254.3/32
leftcert=hostCert.pem
right=14M.MMM.M.MMM <<========== Add/update the correct IP address of VPlex cluster-1.
rightsubnet=128.221.252.32/27,128.221.253.32/27
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes256-sha256
auto=start
# Connection between Cluster Witness Server and Management Server
conn witness-cluster2
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.254.3/32
leftcert=hostCert.pem
right=14N.NNN.N.NNN <<========== Add/update the correct IP address of VPlex cluster-2.
rightsubnet=128.221.252.64/27,128.221.253.64/27
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes256-sha256
auto=start
5) Restart IPSEC service in both cluster-witness server and VPlex management server s(both cluster-1 and cluster-2) as follows:
service@ClusterWitness:~> sudo /usr/sbin/ipsec restart
service@ManagementServer:~> sudo /usr/sbin/ipsec restart
5.a) Verify the status of IPsec service as follows:
service@ClusterWitness:~> sudo /usr/sbin/ipsec status
service@ManagementServer:~> sudo /usr/sbin/ipsec status
6) Re-perform the below command to re-configure 3-way VPN connection between VPlex management server's and cluster-witness server as follows:
Example:
VPlexcli in Cluster-1:
VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> force
Then from VPlexcli in Cluster-2
VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> force
7) Verify the status of cluster-witness following step-1(both cluster-1 and cluster-2) as follows:
Example:
VPlexcli:/> ll /cluster-witness/* /cluster-witness/components: Name ID Admin State Operational State Mgmt Connectivity ----------------- -- ----------- ------------------- ----------------- cluster-1 1 enabled in-contact ok cluster-2 2 enabled in-contact ok server - enabled clusters-in-contact ok
8) Check VPN connectivity using command vpn status(both cluster-1 and cluster-2) as follows:
Example:
Cluster-1:
VPlexcli:/> vpn status
Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14N.NNN.N.NNN is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable
Cluster-2:
VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14M.MMM.M.MMM is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable
Scenario 2: User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server.
9) Verify the correct cluster-witness public IP-address by following the below command:
VPlexcli:/> ll /cluster-witness/** /cluster-witness: Attributes: Name Value ------------------ ------------- admin-state enabled private-ip-address 128.221.254.3 public-ip-address xx.xx.xx.65 <<< Cluster-Witness server public IP-address Contexts: Name Description ---------- -------------------------- components Cluster Witness Components
10) From VPlex management server(either cluster-1 or cluster-2) cat the file ipsec.conf as follows:
Note: Before carrying out to step(10) confirm the actual IP address of VPlex management server's from vpn status output using step(9). Once this information is gathered, please compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not.
Example:
service@Managementserver:~> cat /etc/ipsec.conf >> Cluster-1
# Add connections here.
# Setup a tunnel between the management servers and their networks
# "left" means local, "right" means remote.
# Connection between Cluster Witness Server and Management Server
conn net-witness
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.252.64/27,128.221.253.64/27
leftcert=hostCert.pem
right=xx.xx.xx.45 <<========== Old/incorrect IP address of cluster-witness
rightsubnet=128.221.254.3/32
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN CWS, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes128-sha1
auto=start
# Connection between Management Server 1 and Management Server 2
conn net-net
type=tunnel
keyexchange=ikev2
mobike=no
reauth=no
left=%defaultroute
leftsubnet=128.221.252.64/27,128.221.253.64/27
leftcert=hostCert.pem
right=14N.NNN.N.NNN <<========== IP address of remote management server
rightsubnet=128.221.252.32/27,128.221.253.32/27
rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN:CKxxxxxxxxxxxx, E=support@emc.com"
ike=3des-sha256-modp2048
esp=aes256-sha256
auto=start
11) As from the above example, we found that IPsec.config file of VPlex management server (either cluster-1 or/both cluster-2) is still carrying
old IP-address of cluster-witness server. Therefore, please edit the file IPsec.config using vi editor to update the correct IP-address of cluster-witness server.
NOTE: Place the correct IP address of Cluster-witness server on affected VPlex management server after the equals sign with no space between the equal sign and IP address, save and exit the file.
Repeat step's from 4 to 8 from scenario 1 in-order to resolve this issue.