Node Maintenance

Use the Node Affinity feature in the Kubernetes Operator to migrate Aerospike pods out of Kubernetes nodes. This is useful for doing maintenance work such as upgrading the Kubernetes version on the nodes.

With Network-attached storage

Setting a scheduling policy like affinity, taint & tolerations, and nodeSelectors can help migrate the pods to a different node pool, and the current node pool can be brought down. Set RollingUpdateBatchSize to expedite this process by migrating pods in a batch.

For example, you can set the following nodeAffinity in the podSpec section of the CR file. AKO performs a rolling restart of the cluster and migrates the pods based on the scheduling policies.

Following nodeAffinity ensures that pods are migrated to a node-pool named upgrade-pool. AKO restarts the pods and move them to the nodes with the node label cloud.google.com/gke-nodepool: upgrade-pool.

  podSpec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                    - upgrade-pool

With Local attached storage:

When Kubernetes pods use local storage, they are unable to move to different Aerospike cluster nodes because of volume affinity. This prevents a rolling restart with a different scheduling policy from working.

However, you can use the K8sNodeBlockList feature to migrate the pods to other Kubernetes nodes in case of local storage.

K8sNodeBlockList

K8sNodeBlockList is an array of Kubernetes node names that you can provide in the CR. AKO reads this CR and safely migrates the pods out of those Kubernetes nodes. Therefore, if you want to bring some Kubernetes nodes down for maintenance or upgrade purposes, you can use this feature to safely migrate the pods out of the given Kubernetes nodes.

1. When you specify any Kubernetes nodes in the K8sNodeBlockList fields of the CR and apply your changes, AKO starts migrating the pods out of the given Kubernetes nodes by setting the pod’s affinity.
2. If the pods are using the network-attached storage, AKO migrates the pods out of their Kubernetes nodes without any issues.
3. If the pods are using the local-attached storage, you must specify those local storage classes in the spec.Storage.LocalStorageClasses field of the CR. AKO uses this field to delete the corresponding local volumes so that the pods can be easily migrated out of the Kubernetes nodes.

This Operation also considers RollingUpdateBatchSize. Therefore, a batch of pods is migrated in one go, and the migration process is expedited.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike
spec:
  k8sNodeBlockList:
    - gke-test-default-pool-b6f71594-1w85
    - gke-test-default-pool-b6f71594-9vm2

  size: 4

  image: aerospike/aerospike-server-enterprise:7.2.0.1

  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
      - id: 2

...