Dell ObjectScale 1.3 Administration Guide

Remove a node (ObjectScale Software Bundle)

Follow this Node Removal service procedure to remove a node from the ObjectScale Software Bundle cluster.

1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for cluster management tasks.

   The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used to request cluster management tasks, like service procedures.

   1. Collect the keycloak account information from the secret:

      export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq -r '.data["keycloak-username"]' | base64 --decode)
      export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq -r '.data["keycloak-password"]' | base64 --decode)
      export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq -r '.data["keycloak-realm"]' | base64 --decode)
      export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq -r '.data["keycloak-client"]' | base64 --decode)
      export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

   2. Set an environment variable for the access token:

      export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode 'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

   kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Apply a taint to the node to be removed:

   kubectl taint node <NODE_NAME> node.dell.com/drain=drain:NoSchedule

4. View the status of the PMM service procedure.
   Once the taint has been applied to a node, the ObjectScale Operator creates a PMM service procedure. Retrieve the list of service procedures and locate the PMM service procedure with pmm- prefixed to the service procedure name.

   kubectl get serviceprocedures

   NOTE:To obtain details about a service procedure, including its status, use:

   kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

   NOTE:Do not delete the service procedure while it is running.
5. Monitor the status of the service procedure with the following command:

   while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.managedFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

   The service procedure transitions through various phases as it progresses. The Reason value for the PMM service procedure should progress from NotStarted, In Progress, PostCheck, and finally to Success. A reason of Success indicates that the service procedure has completed without error, and the node is now in PMM.

6. Once the PMM service procedure is successful, place the node into maintenance mode within the CMO Platform within the ObjectScale Software Bundle.

   kubectl cordon <NODE_NAME>

7. Safely evict all your pods from the node:

   kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

8. Verify the status of the drained node:

   kubectl get node <NODE_NAME>

   NAME         STATUS                     ROLES    AGE     VERSION
   hostname6   Ready,SchedulingDisabled   <none>   6d19h   v1.24.7+rke2r1

9. Verify that all CMO component pods, except the DaemonSet-managed Pods, have been rescheduled to the other nodes.

   kubectl get pod -n cmo -o wide | grep <NODE_NAME>

10. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle.
    Place this JSON payload in the node where you are going to perform the scale down of the node.

    {
      "hosts":  [{
        "hostname": "<NODE_HOSTNAME>"
      }],
      "remove_os_packages": "true"
    }

    NOTE:If the remove_os_packages parameter is set to true, the OS packages are removed from the node. This precludes the user from adding the node back to the cluster without reinstalling those OS packages.

    For example:

    {
        "worker": [{
            "hostname": "hostname6",
        }],
        "remove_os_packages": "true" 
    }

11. Scale down the node using the CMO Platform Manager scale down API.
    NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even though the scale down happens successfully.

    curl --header "Content-Type: application/json" --header "Authorization: Bearer $TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/clusters/nodes -v -k | json_pp

    For example:

    ......
    {
       "created_at" : "2023-04-15T11:35:35Z",
       "completed_tasks" : 0,
       "total_tasks" : 273,
       "recap" : {
          "hosts" : {}
       },
       "id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
       "link" : {
          "href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
          "rel" : "self"
       },
       "logs" : "",
       "state" : "created",
       "updated_at" : "2023-04-15T11:35:36Z",
       "playbook_id" : "remove-node"
    }

12. Collect the "id" value from the returned output. You will use this value in the next step.
    For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
13. After performing the scale down API, check the status of the operation through the API below:
    NOTE:The CMO Platform Manager TOKEN may expire, and be refreshed by running:

    export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode 'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

    curl --header "Content-Type: application/json" --header "Authorization: Bearer $TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

    When the operation is finished, the operation "state" is marked as "complete".

    NOTE:In certain situations, the status may show as Failed when the failure node was removed successfully. Check the node status.
14. Confirm that the node has been removed from the node list.

    kubectl get node