Dell ObjectScale 1.3 Administration Guide

ObjectScale maintenance mode overview

Taints and toleration are used to coordinate maintenance modes. A user can take a single node down for maintenance in order to repair a faulted hardware component. Or, a user might do a sequential rolling maintenance mode to perform a software upgrade on all nodes. In the rolling MM case one node enters MM, finishes the maintenance operation, and is exited from MM and the next node is put into MM.

ObjectScale on OpenShift supports two types of maintenance modes; Temporary Maintenance Mode (TMM) and Permanent Maintenance Mode (PMM). The ObjectScale Software Bundle supports Temporary Maintenance Mode (TMM) and Node Remove maintenance modes.

Two different types of taints trigger these maintenance modes. ObjectScale Operator reacts to these taints to handle the pods on the node. ObjectScale Operator only handles the pods with a special label, and only ObjectScale pods have the label.

In TMM, all ObjectScale and object store stateful sets and Stateless pods are rescheduled to an available spare node, if available. Pods controlled by a stateful set are not rescheduled to other nodes during TMM. Daemon sets remain in Running status on the node under TMM. Finally, the ObjectScale Software Bundle stateful CMO pods are rescheduled to an available spare node, by deleting the PVC and pod. Once the TMM taint is removed from the node, the pending stateful pods previously running on the node under TMM return to Running status on the original node.

In PMM, the ObjectScale Operator deletes all the pods and PVCs on the node.

Placing multiple nodes into maintenance mode

ObjectScale supports placing multiple nodes into maintenance mode simultaneously, depending on the size and the state of the object store and ObjectScale. An object store configured with the recommended number of SS replicas can place up to two nodes into maintenance mode. When a third node attempts to enter maintenance mode, it is either rejected from entering maintenance mode or placed into a queue until another node exits maintenance mode. See alert messages within ObjectScale Portal for details on status of nodes in or rejected from maintenance mode.

ObjectScale recommends that:

• Object stores with a 12+4 EC scheme should have at least 10 SS replicas

Temporary maintenance mode

Temporary maintenance mode (TMM) is a service procedure used to place a node into maintenance mode for maintenance activities at the node level, such as software upgrades. While the node is in TMM, all user data remains accessible (read and write).

During TMM, all object stores move to maintenance phase until the node is taken out of TMM. Once taken out of TMM, the object store status returns to Available.

For ObjectScale instances deployed on a Red Hat OpenShift cluster, manually place a node into TMM by placing a taint on the node.

For the ObjectScale Software Bundle Cluster, you must enter TMM by making the node unschedulable and then manually cordon the node within the CMO Platform. This step is not automatic and initiated by the cluster administrator similar to original step used to enter TMM.

When TMM occurs, ObjectScale Operator reacts to the node taint and undertakes the following actions.

• Once placed into TMM, all stateless pods (ReplicaSet pods) are relocated to other available nodes in the cluster automatically.
• Pods controlled by a DaemonSet may continue to run on the node while in TMM. This includes some CSI Bare-Metal, Kubernetes, and platform-related pods continue running on the node under TMM.
• ObjectScale and object store pods that have a persistent volume claim (PVC) which is bound to a persistent volume (PV) on the node in TMM remain in the Pending state until the node is taken out of TMM. Pods controlled by a stateful set are not rescheduled to other nodes.

Permanent maintenance mode

Permanent maintenance mode (PMM) is a service procedure that is used to place a node into maintenance mode permanently for node removal. Permanent removal should only occur while the cluster is healthy. When you place a taint on the node to go into PMM, then the ObjectScale Operator handles the taint and moves resources to an available node.

Maintenance activities for ObjectScale Software Bundle

For ObjectScale Software Bundle, placing nodes into maintenance modes and performing node maintenance activities, such as adding or removing a node from the cluster, requires additional CMO Platform API calls be made after entering the node into an ObjectScale maintenance mode.

Using the ObjectScale maintenance modes and/or the CMO Platform APIs allows you to perform these node activities on the ObjectScale software bundle clusters.

• Place a node in an ObjectScale software bundle cluster into temporary maintenance mode
• Add a new node to the ObjectScale software bundle cluster
• Remove a node from the ObjectScale software bundle cluster
• Prepare a node for node hardware or software maintenance
• Replace a node within the ObjectScale software bundle cluster