RM Snapshot Invalidation takes forever - Any suggestions

Question

We are running a replication utilizing snapshots. There are 13 luns associated with the system we are replicating. In the log, in takes 5-7 minutes to invalidate each snapshot. We run round-robin with 2 replica's. When it takes 5-7 minutes per lun snapshot, that add's over an hour to the time it takes the job to complete. The job right now is taking 1.5 hours to finish. I am not mounting the snapshot anywhere, just keeping it for an online backup if needed to roll-back. Any idea's? I've attached a snippet from the log below showing what I mean:

INFO: 2007 10 19 18:59:10 Removed unused storage clariion_APM0004120****_nts433-0347-4434.
INFO: 2007 10 19 19:05:05 Removed unused storage clariion_APM0004120****_nts433-0402-4434.
INFO: 2007 10 19 19:09:24 Removed unused storage clariion_APM0004120****_nts433-0505-4434.
INFO: 2007 10 19 19:14:00 Removed unused storage clariion_APM0004120****_nts433-0702-4434.
INFO: 2007 10 19 19:17:56 Removed unused storage clariion_APM0004120****_nts433-0802-4434.
INFO: 2007 10 19 19:22:05 Removed unused storage clariion_APM0004120****_nts433-0911-4434.
INFO: 2007 10 19 19:25:53 Removed unused storage clariion_APM0004120****_nts433-1024-4434.
INFO: 2007 10 19 19:30:19 Removed unused storage clariion_APM0004120****_nts433-1049-4434.
INFO: 2007 10 19 19:35:34 Removed unused storage clariion_APM0004120****_nts433-1153-4434.
INFO: 2007 10 19 19:39:30 Removed unused storage clariion_APM0004120****_nts433-2113-4434.
OVERALL PROGRESS: 2 percent complete
INFO: 2007 10 19 19:44:31 Removed unused storage clariion_APM0004120****_nts433-2157-4434.
INFO: 2007 10 19 19:49:02 Removed unused storage clariion_APM0004120****_nts433-2200-4434.
PHASE: ExpireReplicaSinglePhase

JamesBEMC · Answer

Hi ChadAny chance you have 2 RM Servers managing the same Clariion array?I would first check to see how long it takes for a Navi command to come back from the array - ie SP latency.Run the following command for each sp to the Clariion from the production host 10 times.navicli -h (Clariion sp ip) getagent.Time how long each command takes to come back. If each command is taking more than 10 seconds, I recommend you check the network switch port settings which the Clariion is connected to. The Clariion NICs are set to auto-neg full-duplex and so the network switch ports should also be set the same. If the switch ports are forced-set to say 100-FD - then you would experience what you are seeing.Hope it helpsJames.

ChadHarter · Answer

We only have 1 RM server. There is some Navi command latency, which I did look into. I was a network engineer in a prior life, and have checked that possibility out. It is set to Auto on both sides, like it should be. One thing I did notice is that on both SP's and even another Clariion SP's, I see network errors on the ports. I have swapped out patch cables, verified configuration, even changed switches, and these errors follow. There is something not right on the EMC network stack on Clariion arrays as I should never see network errors. These are the only hosts on that switch that show these errors. I have to believe there is something not right on the EMC Lan stack as I see this on 2 different arrays, both SP ports. Navicli latency shouldn't affect a 1 command that removes a snapshot though, but yet it takes 5 minutes to do this.

JamesBEMC · Answer

You could try to manually do what RM is attempting to do
by using Navicli to create a snapshot, activate it and stop/delete it.

From a single run of the job, you can check the RM client debug logs for the syntax of each command.

If this is slow in itself, I highly recommend opening a call with the Clariion group to see if there is some sort of performance issue going on with the SPs.

Cheers
James.

Replication Manager

RM Snapshot Invalidation takes forever - Any suggestions

Was this post helpful?