ScaleIO 2.0.0.3 : Unexpected MDM_DATA_DEGRADED events (rc=BUSY)
Summary: ScaleIO 2.0.0.3 : Unexpected MDM_DATA_DEGRADED events (rc=BUSY)
Symptoms
Issue Description
Unexpected MDM_DATA_DEGRADED events are logged, without SDS disconnections or other related events.
Note: This KB explains about a specific issue (see Symptoms section). But there can be other causes that lead to the same or similar issues.
Scenario
Possibly under high workload.
Symptoms
MDM_DATA_DEGRADED in MDM's event.
No SDS_DISCONNECTED or SDS_RECONNECTED events.
No related entry in cmatrix log / no SDS found guilty at the same time.
MDM trc shows that the Comb goes degraded, instead of a request from SDS (TGT_ID is the one that moved the state without requesting MDM and getting permission):
05/11 14:01:33.040717 bf4deb8:multiHeadMgr_HandleTgtUpdateCombState:02256: Tgt: #####TGT_ID##### CombId: ##COMBID## raidTgt: #####TGTID###### ProtType: SECONDARY State: 0x10 primaryTgtGenNum: 14 tgtCombCmdGenNum: 3 05/11 14:01:33.040754 bf4deb8:multiHeadRow_MoveState_Inner:02954: [multiHeadRow_InitiateMoveNorm2Degraded:2402]: MultiHead: #MH_ID## Row: 730 NORMAL->DEGRADED (NORM2DEG_SEC_TGT_FAILED)
SDS (TGT_ID) trc shows complains about another SDS (TGTID), on the same comb as above:
05/11 14:01:30.978178 c8ec2eb8:raidComb_WaitForRemoteResponse:05520: (combId=##COMBID##) write to tgtId=#####TGTID###### failed (rc=BUSY) 05/11 14:01:32.040480 c8ec2eb8:raidComb_WaitForRemoteResponse:05520: (combId=##COMBID##) write to tgtId=#####TGTID###### failed (rc=BUSY) 05/11 14:01:33.040399 c8ec2eb8:raidComb_WaitForRemoteResponse:05520: (combId=##COMBID##) write to tgtId=#####TGTID###### failed (rc=BUSY) 05/11 14:01:33.040414 c8ec2eb8:raidComb_ShouldTryAgain:06079: (combId=##COMBID##) allowed attempts or total time exhausted, retry is not allowed. num attempts: 3, total time: 0 millis 05/11 14:01:33.040417 c8ec2eb8:raidComb_UpdateWriteError:01497: Write error comb ##COMBID##, raid state 0x1, local rc SUCCESS, remote rc BUSY 05/11 14:01:33.040420 c8ec2eb8:raidComb_SetCombState:01057: Set comb raid state ##COMBID##, raid state 0x1 => 0x10 (dit) 05/11 14:01:33.874635 c7d52eb8:contCmd_StartMigration:02768: CombId ##COMBID## - Start rc = SUCCESS
Impact
Data degraded but it should recover very soon.
Cause
Resolution
Workaround
N/A. This is usually just temporary network issue.
Impacted versions
2.0.0.3 and below.
Fixed in version
2.0.0.4
2.0.1.0