Dell ObjectScale 1.3 Administration Guide

Place a node into temporary maintenance mode (ObjectScale on OpenShift)

Use this manual process to place a healthy node into temporary maintenance mode (TMM). Use this process for ObjectScale instances on a Red Hat OpenShift cluster.

1. Apply a taint to the node to be placed into temporary maintenance mode:

   kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule

2. Verify that the PHASE of the cluster now displays Maintenance.

   kubectl get ecs-cluster -n <NAMESPACE>

   NAME          PHASE         READY COMPONENTS   S3 ENDPOINT         MGMT API
   ecs-cluster   Maintenance   22/23              10.236.228.53:443   10.236.228.52:4443

   The ObjectScale Portal shows the object store status as Maintenance.

3. Once the taint has been applied to a node, the ObjectScale Operator creates a TMM service procedure. Retrieve the list of service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure name:

   kubectl get serviceprocedures -n <NAMESPACE>

   NAME                                               AGE
   recovery-ecs-cluster-bk-bookie-0-18369f2d          32m
   recovery-ecs-cluster-bk-bookie-1-426c0578          26h
   recovery-ecs-cluster-influxdb-0-abee6329           34m
   recovery-ecs-cluster-influxdb-2-1d470d3e           33m
   recovery-ecs-cluster-zookeeper-4-c01cff83          33m
   recovery-objectscale-manager-influxdb-0-42e4e0a9   46m
   tmm-a4a9b606-4126-4914-b18c-27337e841f63           15m

   NOTE:To obtain details about a service procedure, including its status, use:

   kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

   NOTE:Do not delete the service procedure while it is running.
4. Monitor the status of the service procedure with the following command:

    while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.managedFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

   The service procedure transitions through various phases as it progresses. The Reason value for the TMM service procedure should progress from NotStarted, In Progress, PostCheck, and finally to Success. TMM should enter Waiting until the taint is removed, at which point it goes to Success.

5. Verify that only pods controlled by a DaemonSet remain running.

   You may see other non-ObjectScale pods running on the node, such as metallb, OpenShift pods.

   kubectl get pods --all-namespaces -o custom-columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].kind,Started:status.startTime | grep <NODE_NAME>

   baremetal-csi-node-56jb2        worker2.ocp4.cmo.com   DaemonSet       2021-02-19T16:11:24Z
   speaker-sg4ws                   worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:36:00Z
   tuned-pwphr                     worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   dns-default-prbvg               worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   node-ca-tpgwk                   worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   machine-config-daemon-nkdbx     worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   node-exporter-wnltt             worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   multus-m7pgs                    worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   network-metrics-daemon-xp68j    worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   ovs-gm962                       worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z
   sdn-fcrkh                       worker2.ocp4.cmo.com   DaemonSet       2021-02-11T14:13:05Z

6. Also, any pods previously running on the TMM node that belong to a StatefulSet enter the Pending state. These are pods that have a persistent volume claim (PVC) that is bound to a persistent volume (PV) on the node in TMM.

   There may also be stateless pods (that is ReplicaSet) in the Pending state if the pods cannot be relocated due to pod anti-affinity rules. This is expected behavior.

   kubectl get pods --all-namespaces=true -o custom-columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].kind,Status:status.phase,Started:status.startTime | grep Pending

   decks-support-store-0           <none>      StatefulSet   Pending     <none>
   objectscale-iam-atlas-1         <none>      StatefulSet   Pending     <none>
   objs-mgr-rsyslog-4              <none>      StatefulSet   Pending     <none>

7. Verify the ObjectScale Portal UI shows that the node has entered TMM by reviewing the Monitoring > Alerts tab.