Skip to content
K8s Operator ( latest 4.1.2 )
/Manage AKO/Configure AKO/Scaling

Scaling Aerospike on Kubernetes

Each Kubernetes pod runs a single Aerospike Database instance, which is equivalent to an Aerospike node in non-Kubernetes deployments.

You can scale Aerospike on Kubernetes horizontally by adjusting the number of pods in your cluster, or vertically by adjusting the resources available to them. You can also scale Aerospike namespace storage by adding a new rack and deleting the old one. This is a workaround due to StatefulSet PersistentVolumeClaim limitations.

Horizontal scaling

Scaling horizontally means adding or removing Kubernetes pods from the cluster.

The Custom Resource (CR) file controls the number of pods in a rack. When you change the cluster size in the CR file, Aerospike Kubernetes Operator (AKO) adds pods to the rack following the rack order defined in the CR.

AKO distributes the pods equally across all racks. If any pods remain after equal distribution, they are distributed following the rack order.

In a cluster of two racks and five Kubernetes pods:

  • After equal pod distribution, both racks have two pods, with one left over as a remainder.
  • The remaining pod goes to Rack 1, resulting in Rack 1 having three pods and Rack 2 having two pods.
  • If the cluster size is scaled up to six pods, a new pod is added to Rack 2.
  • Scaling down follows the rack order and removes pods with the goal of equal distribution.
    • In this example of two racks and six pods, scaling down to four pods results in two racks with two pods each.
    • The third pod on Rack 1 goes down first, followed by the third pod on Rack 2.

Scale a cluster horizontally

For this example, the cluster is deployed using a CR file named aerospike-cluster.yaml.

  1. Change the spec.size field in the CR file to scale the cluster up or down to the specified number of pods.

    apiVersion: asdb.aerospike.com/v1
    kind: AerospikeCluster
    metadata:
      name: aerocluster
      namespace: aerospike
    spec:
      size: 2
      image: aerospike/aerospike-server-enterprise:8.1.0.0

  2. Use kubectl to apply the change.

    Terminal window
    kubectl apply -f aerospike-cluster.yaml

  3. Check the pods.

    Terminal
    $ kubectl get pods -n aerospike
    Expected output
    NAME               READY   STATUS    RESTARTS   AGE
    aerocluster-0-0    1/1     Running   0          3m6s
    aerocluster-0-1    1/1     Running   0          3m6s

Batch scale-down

You can scale down multiple pods within the same rack with a single scaling command by configuring scaleDownBatchSize in the CR file. This parameter is a percentage or absolute number of rack pods that AKO scales down simultaneously when the cluster size is decreased.

Vertical scaling

Vertical scaling in Kubernetes adjusts the CPU and memory resources allocated to a Pod’s containers. This ensures variable workloads, such as traffic peaks, always have the necessary resources without wasting capacity.

The spec.podSpec.aerospikeContainer.resources parameter in the CR file governs the amount of compute resources (CPU or memory) available to each pod’s Aerospike container. Modifying this parameter causes a rolling restart of all pods with updated resources.

Scale a cluster vertically

For this example, the cluster is deployed using a CR file named aerospike-cluster.yaml.

  1. Change the spec.podSpec.aerospikeContainer.resources field in the CR file to scale each pod’s Aerospike container resources.

    apiVersion: asdb.aerospike.com/v1
    kind: AerospikeCluster
    metadata:
      name: aerocluster
      namespace: aerospike
    spec:
      size: 2
      image: aerospike/aerospike-server-enterprise:8.1.0.0
      podSpec:
        aerospikeContainer:
          resources:
            requests:
              memory: 2Gi
              cpu: 200m
            limits:
              memory: 10Gi
              cpu: 5000m

  2. Use kubectl to apply the change. This causes a rolling restart of the pods, after which the pods run with the updated resources.

    Terminal window
    kubectl apply -f aerospike-cluster.yaml

  3. Check the pods.

    Terminal
    $ kubectl get pods -n aerospike
    Expected output
    NAME               READY   STATUS    RESTARTS   AGE
    aerocluster-0-0    1/1     Running   0          3m6s
    aerocluster-0-1    1/1     Running   0          3m6s

Aerospike namespace storage scaling

AKO uses Kubernetes StatefulSets for deploying Aerospike clusters. StatefulSets use PersistentVolumeClaims (PVCs) for providing persistent storage. A PVC in a StatefulSet cannot be resized after creation, which prevents a simple solution for Aerospike namespace storage scaling.

Storage scaling using Rack Awareness

We recommend using the Aerospike Rack Awareness feature as a workaround to perform Aerospike namespace storage scaling. Instead of modifying the storage size of an existing rack, use AKO to replace the rack in a single step, creating a new rack with updated storage while removing the old one. AKO seamlessly migrates data from the old rack to the new rack and removes the old rack automatically.

Example: expanding namespace storage

In this example, the cluster is deployed using aerospike-cluster.yaml. The storage scaling process involves replacing an existing rack (id: 1) with a new rack (id: 2) that has increased storage.

The existing rack has two PersistentVolumes:

  • workdir (1Gi): Used as the Aerospike Database server work directory.
  • ns (3Gi): Used by the Aerospike namespace test.

The goal is to increase the ns volume size from 3Gi to 8Gi. Because Kubernetes does not allow resizing StatefulSet PVCs after creation, you must create a new rack with id: 2 in the CR file with an updated storage configuration, then delete the old rack with id: 1.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike


spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:8.1.0.0


  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        zone: us-central1-b
        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /dev/sdf
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Block
                  size: 3Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret


  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/sdf

Rack-configured storage after scaling

The second volume is increased from 3Gi to 8Gi.

Save and exit the CR file, then use kubectl to apply the change.

Terminal window
kubectl apply -f aerospike-cluster.yaml

This creates a new rack(id: 2) and an updated storage section. AKO removes the old rack after verifying that the Aerospike server has migrated the old data to the new rack.

Check the pods with the get pods command:

Terminal
$ kubectl get pods -n aerospike
Expected output
NAME               READY   STATUS    RESTARTS   AGE
aerocluster-2-0     1/1     Running         0          3m6s
aerocluster-2-1     1/1     Running         0          3m6s
aerocluster-1-1     1/1     Terminating     0          30s

Kubernetes cluster autoscaling

A Kubernetes node is a physical or virtual machine in the Kubernetes cluster infrastructure that runs pods. This is distinct from an Aerospike node, which is the Aerospike Database instance running inside a pod.

Kubernetes Cluster Autoscaler automatically scales up the Kubernetes cluster when resources are insufficient for the workload, and scales down the cluster when Kubernetes nodes are underused for an extended period of time. See the documentation hosted on GitHub for more details.

Karpenter is an open-source node lifecycle management project built for Kubernetes, with features that fit into cloud providers workflow. See the Karpenter documentation for more details.

If Aerospike cluster pods only have in-memory and dynamic network-attached storage, both autoscalers scale up and down by adjusting the number of Kubernetes nodes and shifting the load automatically to prevent data loss.

Kubernetes cluster autoscaling with local volumes

The primary challenge with autoscalers and local storage provisioners is ensuring the availability of local disks during Kubernetes node startup after the autoscaler scales up the node. When using local volumes, the ability to successfully autoscale depends on the underlying storage provisioner: Kubernetes SIGs Storage Local Static Provisioner, or OpenEBS.

The Kubernetes cluster autoscaler cannot add a new node when the underlying storage uses the Kubernetes static local storage provisioner. Scale up in this case must be done manually.

When scaling up using Karpenter, you can configure OpenEBS to automatically initialize local storage on newly-provisioned nodes by specifying the appropriate commands for volume setup.

This ensures that the local volumes required by OpenEBS are prepared automatically as soon as a node joins the cluster. If you prefer using a local storage provisioner, you can apply a DaemonSet that runs these same pvcreate and vgcreate commands on every new node. See Google’s documentation for automatic bootstrapping on GKE for an example of using DaemonSets to bootstrap.

Neither autoscaler can scale down the nodes if any pod is running with local storage attached.

Feedback

Was this page helpful?

What type of feedback are you giving?

What would you like us to know?

+Capture screenshot

Can we reach out to you?