Follow this Node Removal service procedure to remove a node from the ObjectScale Software Bundle cluster.
Prerequisites
Ensure that the ObjectScale Software Bundle is equipped with a spare node with enough drives for the pods. This spare node receives the contents of the node that is placed into permanent maintenance mode.
Steps
The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used to request cluster management tasks, like service procedures.
Collect the keycloak account information from the secret:
Once the taint has been applied to a node, the ObjectScale Operator creates a PMM service procedure. Retrieve the list of service procedures and locate the PMM service procedure with
pmm- prefixed to the service procedure name.
kubectl get serviceprocedures
NOTE:To obtain details about a service procedure, including its status, use:
NOTE:Do not delete the service procedure while it is running.
Monitor the status of the service procedure with the following command:
while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.managedFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done
The service procedure transitions through various phases as it progresses. The Reason value for the PMM service procedure should progress from
NotStarted,
In Progress,
PostCheck, and finally to
Success. A reason of
Success indicates that the service procedure has completed without error, and the node is now in PMM.
Once the PMM service procedure is successful, place the node into maintenance mode within the CMO Platform within the ObjectScale Software Bundle.
NOTE:If the
remove_os_packages parameter is set to
true, the OS packages are removed from the node. This precludes the user from adding the node back to the cluster without reinstalling those OS packages.
Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even though the scale down happens successfully.