Scaling Aerospike on Kubernetes

You can scale Aerospike on Kubernetes horizontally (adjusting the number of pods in your cluster) or vertically (adjusting the resources available to them). Additionally, you can scale Aerospike namespace storage by adding a new rack and deleting the old one as a workaround due to StatefulSet PVC limitations.

See Horizontal Scaling to scale the number of pods.
See Vertical Scaling to scale the CPU/memory resources allocated to the pod containers.
See Aerospike Namespace Storage Scaling to scale aerospike namespace storage.
See Kubernetes Cluster autoscaling to scale the nodes in the k8s cluster.

Horizontal scaling

The Custom Resource (CR) file controls the number of pods (nodes) in a rack. When you change the cluster size in the CR file, Aerospike Kubernetes Operator (AKO) adds pods to the rack following the rack order defined in the CR.

AKO distributes the nodes equally across all racks. If any pods remain after equal distribution, they are distributed following the rack order.

Example: A cluster of two racks and five pods.

After equal pod distribution, both racks have two pods, with one left over as a remainder.
The remaining pod goes to Rack 1, resulting in Rack 1 having three pods and Rack 2 having two pods.
If the cluster size is scaled up to six pods, a new pod is added to Rack 2.
Scaling down follows the rack order and removes pods with the goal of equal distribution.
- In this example of two racks and six pods, scaling down to four pods results in two racks with two pods each.
- The third pod (third replica) on Rack 1 goes down first, followed by the third pod on Rack 2.

Horizontal scaling CR parameters

For this example, the cluster is deployed using a CR file named aerospike-cluster.yaml.

Change the spec.size field in the CR file to scale the cluster up or down to the specified number of pods.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike
spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:8.0.0.2
  .
  .

Use kubectl to apply the change.
Terminal window
```
kubectl apply -f aerospike-cluster.yaml
```

Check the pods.

Output:

$ kubectl get pods -n aerospike
NAME          READY   STATUS      RESTARTS   AGE
aerocluster-0-0     1/1     Running     0          3m6s
aerocluster-0-1     1/1     Running     0          3m6s

Batch scale-down

You can scale down multiple pods within the same rack with a single scaling command by configuring scaleDownBatchSize in the CR file. This parameter is a percentage or absolute number of rack pods that the AKO scales down simultaneously when the cluster size is decreased.

Vertical scaling

Vertical scaling refers to adjusting the compute resources, such as CPU and memory, allocated to existing pod containers in a Kubernetes cluster. This can be useful if applications experience variable workloads that require more or less computing power at different times, such as peak and off-peak traffic times requiring changes in the amount of memory.

The spec.podSpec.aerospikeContainer.resources parameter in the CR file governs the amount of compute resources (CPU or memory) available to each Aerospike pod container. Modifying this parameter causes a rolling restart of all pods with updated resources.

Vertical scaling CR parameters

For this example, the cluster is deployed using a CR file named aerospike-cluster.yaml.

Change the spec.podSpec.aerospikeContainer.resources field in the CR file to scale the pod container resources.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike
spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:8.0.0.2
  podSpec:
    aerospikeContainer:
      resources:
        requests:
          memory: 2Gi
          cpu: 200m
        limits:
          memory: 10Gi
          cpu: 5000m
  .
  .

Use kubectl to apply the change. This will cause a rolling restart of the pods. Eventually, pods will be running with the updated resources.
Terminal window
```
kubectl apply -f aerospike-cluster.yaml
```

Check the pods.

Output:

$ kubectl get pods -n aerospike
NAME          READY   STATUS      RESTARTS   AGE
aerocluster-0-0     1/1     Running     0          3m6s
aerocluster-0-1     1/1     Running     0          3m6s

Aerospike Namespace Storage Scaling

AKO uses Kubernetes StatefulSets for deploying Aerospike clusters. StatefulSets use PersistentVolumeClaims for providing persistent storage. A PersistentVolumeClaim in a StatefulSet cannot be resized after creation, which prevents AKO from providing a simple solution for Aerospike namespace storage scaling.

Storage Scaling using Rack Awareness feature

We recommend using the Aerospike Rack Awareness feature as a workaround to perform Aerospike namespace storage scaling. Instead of modifying the storage size of an existing rack, AKO allows you to replace the rack in a single step, creating a new rack with updated storage while removing the old one. The AKO seamlessly migrates data from the old rack to the new rack and removes the old rack automatically.

Example: Expanding Namespace Storage

In this example, the cluster is deployed using aerospike-cluster.yaml. The storage scaling process involves replacing an existing rack (id: 1) with a new rack (id: 2) that has increased storage.

The existing rack has two PersistentVolumes:

workdir (1Gi): Used as the work directory in the Aerospike server pod.
ns (3Gi): Used by the Aerospike namespace test.

The goal is to increase the ns volume size from 3Gi to 8Gi. Because Kubernetes does not allow resizing StatefulSet PVCs after creation, you must create a new rack with id: 2 in the CR file with an updated storage configuration, then delete the old rack with id: 1.

Rack-configured storage before scaling

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:8.0.0.2

  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        zone: us-central1-b
        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /dev/sdf
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Block
                  size: 3Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/sdf
.
.
.

Rack-configured storage after scaling

The second volume is increased from 3Gi to 8Gi.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:8.0.0.2

  rackConfig:
    namespaces:
      - test
    racks:
      # Added new rack with id: 2 and removed the old rack with id: 1
      - id: 2
        zone: us-central1-b
        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /dev/sdf
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Block
                  size: 8Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/sdf
...

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

This creates a new rack(id: 2) and an updated storage section. AKO removes the old rack after verifying that the Aerospike server has migrated the old data to the new rack.

Check the pods with the get pods command:

> $ kubectl get pods -n aerospike

> NAME                READY   STATUS          RESTARTS   AGE
> aerocluster-2-0     1/1     Running         0          3m6s
> aerocluster-2-1     1/1     Running         0          3m6s
> aerocluster-1-1     1/1     Terminating     0          30s

Kubernetes Cluster autoscaling

Kubernetes Cluster Autoscaler automatically scales up the Kubernetes cluster when resources are insufficient for the workload, and scales down the cluster when nodes are underused for an extended period of time. See the documentation hosted on GitHub for more details.

Karpenter is an open-source node lifecycle management project built for Kubernetes, with features that fit into cloud providers workflow. See the Karpenter documentation for more details.

If Aerospike pods only have in-memory and dynamic network-attached storage, both autoscalers scale up and down by adjusting the number of Kubernetes nodes and shifting the load automatically to prevent data loss.

Kubernetes Cluster autoscaling with local volumes

The primary challenge with autoscalers and local storage provisioners is ensuring the availability of local disks during Kubernetes node startup after the autoscaler scales up the node. When using local volumes, the ability to successfully autoscale depends on the underlying storage provisioner: SIGs Storage Local Static Provisioner provided by Kubernetes-sigs, or OpenEBS.

The Kubernetes cluster autoscaler cannot add a new node when the underlying storage uses the Kubernetes static local storage provisioner. Scale up in this case must be done manually.

When scaling up using Karpenter, you can configure OpenEBS to automatically initialize local storage on newly-provisioned nodes by specifying the appropriate commands for volume setup.

This ensures that the local volumes required by OpenEBS are prepared automatically as soon as a node joins the cluster. If you prefer using a local storage provisioner, you can apply a DaemonSet that runs these same pvcreate and vgcreate commands on every new node. See Google’s documentation for automatic bootstrapping on GKE for an example of using DaemonSets to bootstrap.

Neither autoscaler can scale down the nodes if any pod is running with local storage attached.