Draining Kubernetes Nodes

danger

This is unsafe if user has set multiPodPerHost true and replication-factor of namespace is less than the pods scheduled per node.

You can drain a Kubernetes node with the kubectl drain command.

Specific manual steps are required if AerospikeCluster uses a local volume provisioner in storage.

To drain the Kubernetes node:

kubectl drain <node-name>
...
pod/aerocluster-1-0 evicted
...

Check the pods.

Example output:

$ kubectl get pods -n aerospike
NAME          READY   STATUS      RESTARTS   AGE
aerocluster-1-0                                                   0/1     Pending     0          2m28s
aerocluster-2-0                                                   1/1     Running     0          6m3s

Delete the PVCs of the pods belonging to above k8s node.

 % kubectl delete pvc ns-aerocluster-1-0 -n aerospike
   persistentvolumeclaim "ns-aerocluster-1-0" deleted

Delete the above pending pods.

 % kubectl delete pod aerocluster-1-0 -n aerospike
   pod "aerocluster-1-0" deleted

Pending pod will get rescheduled on another node.

Check the pods.

Output:

$ kubectl get pods -n aerospike
NAME          READY   STATUS      RESTARTS   AGE
aerocluster-1-0                                                   1/1     Running     0          10m
aerocluster-2-0                                                   1/1     Running     0          16m