Skip to main content
Loading
Version: Operator 3.3.0

Scaling Namespace Storage for Aerospike on Kubernetes

Scaling namespace storage (vertical scaling) can be a complex topic. The Operator uses Kubernetes StatefulSet for deploying Aerospike clusters. StatefulSet uses PersistentVolumeClaim for providing storage. Currently a PersistentVolumeClaim cannot be updated under these circumstances, which prevents the Operator from providing a simple solution for vertical scaling.

Scale rack-configured storage

We recommend using the Aerospike Rack Awareness feature as a workaround to perform vertical scaling. Instead of changing the size of the existing rack, you create a new, larger rack and delete the old, smaller rack. The Operator automatically transfers the data. In this process, you leave the aerospikeConfig section unmodified and update the storage in the rackConfig section.

Example 1: Rack-configured storage before scaling

For this example, we assume that the cluster is deployed with the custom resource (CR) file aerospike-cluster.yaml. We start with one rack with an id of 1 as the target to replace with a new, larger rack. The cluster currently has two persistentVolume configurations, one with a size of 1Gi and the other with a size of 3Gi.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
name: aerocluster
namespace: aerospike

spec:
size: 2
image: aerospike/aerospike-server-enterprise:7.1.0.0

rackConfig:
namespaces:
- test
racks:
- id: 1
zone: us-central1-b
storage:
filesystemVolumePolicy:
cascadeDelete: true
initMethod: deleteFiles
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: ssd
volumeMode: Filesystem
size: 1Gi
- name: ns
aerospike:
path: /dev/sdf
source:
persistentVolume:
storageClass: ssd
volumeMode: Block
size: 3Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret

aerospikeConfig:
service:
feature-key-file: /etc/aerospike/secret/features.conf
security: {}
namespaces:
- name: test
replication-factor: 2
storage-engine:
type: device
devices:
- /dev/sdf
.
.
.

To resize /dev/sdf for namespace test, create a new rack inside rackConfig with an updated storage section and delete the old rack from the CR file.

Example 2: Rack-configured storage after scaling

In this example, the second volume is increased from 3Gi to 8Gi.

You can create the new rack in the same physical rack if there is enough space. Use the existing zone/region to hold the new storage and old storage together.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
name: aerocluster
namespace: aerospike

spec:
size: 2
image: aerospike/aerospike-server-enterprise:7.1.0.0

rackConfig:
namespaces:
- test
racks:
# Added new rack with id: 2 and removed the old rack with id: 1
- id: 2
zone: us-central1-b
storage:
filesystemVolumePolicy:
cascadeDelete: true
initMethod: deleteFiles
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: ssd
volumeMode: Filesystem
size: 1Gi
- name: ns
aerospike:
path: /dev/sdf
source:
persistentVolume:
storageClass: ssd
volumeMode: Block
size: 8Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret

aerospikeConfig:
service:
feature-key-file: /etc/aerospike/secret/features.conf
security: {}
namespaces:
- name: test
replication-factor: 2
storage-engine:
type: device
devices:
- /dev/sdf
...

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

This creates a new rack with id: 2 and an updated storage section. The Operator removes the old rack after verifying that the Aerospike server has migrated the old data to the new rack.

Check the pods with the get pods command:

> $ kubectl get pods -n aerospike

> NAME READY STATUS RESTARTS AGE
> aerocluster-2-0 1/1 Running 0 3m6s
> aerocluster-2-1 1/1 Running 0 3m6s
> aerocluster-1-1 1/1 Terminating 0 30s

Scale global storage configurations

In some Kubernetes deployments, you specify all storage as PersistentVolumeClaims and have nothing specified by default in the rackConfig section. This configuration works for normal usage, but it prevents the Aerospike Operator from scaling global storage due to Kubernetes limitations. To scale storage when you have configured your deployment only for global storage, first copy your global storage configuration into the rackConfig section, then make the changes to the CR file as described in the previous section under "Scale rack-configured storage."

Example 1: Global storage before scaling

Example 1 represents the "before" state, prior to scaling. It uses global storage, with example volumes workdir and test in the volumes section under the global storage parameter. Some sections of this CR appear truncated to save space.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
name: aerocluster
namespace: aerospike
spec:
size: 3
image: aerospike/aerospike-server-enterprise:6.3.0.2
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 1Gi
- name: test
aerospike:
path: /aerospike/dev/xvdf_test
source:
persistentVolume:
storageClass: openebs-device
volumeMode: Block
size: 60Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
podSpec:
aerospikeInitContainer:
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "1"
memory: 1Gi
metadata:
annotations:
example: annotation
multiPodPerHost: false
sidecars:
- name: exporter
image: aerospike/aerospike-prometheus-exporter:latest
ports:
- containerPort: 9145
name: exporter
env:
- name: "AS_AUTH_USER"
value: "admin"
- name: "AS_AUTH_PASSWORD"
value: "admin123"
- name: "AS_AUTH_MODE"
value: "internal"
rackConfig:
namespaces:
- test
- bar
- foo
racks:
- id: 1
- id: 2
- id: 3
aerospikeConfig:
service:
feature-key-file: /etc/aerospike/secret/features.conf
security: {}
network:
...
namespaces:
- name: test
memory-size: 2000000000
replication-factor: 2
nsup-period: 120
storage-engine:
type: device
devices:
- /aerospike/dev/xvdf_test

To scale the cluster, copy the volumes over to new racks and label them with unique IDs.

Example 2: Global storage after scaling

Example 2 shows a configuration representing the state of the same cluster after scaling. In this example, the volumes are copied over to three racks, labeled with IDs 11, 12, and 13.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
name: aerocluster
namespace: aerospike

spec:
size: 3
image: aerospike/aerospike-server-enterprise:6.3.0.2
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 1Gi
- name: test
aerospike:
path: /aerospike/dev/xvdf_test
source:
persistentVolume:
storageClass: openebs-device
volumeMode: Block
size: 60Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
podSpec:
aerospikeInitContainer:
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "1"
memory: 1Gi
metadata:
annotations:
example: annotation
multiPodPerHost: false
sidecars:
- name: exporter
image: aerospike/aerospike-prometheus-exporter:latest
ports:
- containerPort: 9145
name: exporter
env:
- name: "AS_AUTH_USER"
value: "admin"
- name: "AS_AUTH_PASSWORD"
value: "admin123"
- name: "AS_AUTH_MODE"
value: "internal"

rackConfig:
namespaces:
- test
- bar
- foo
racks:
- id: 11
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 1Gi
- name: test
aerospike:
path: /aerospike/dev/xvdf_test
source:
persistentVolume:
storageClass: openebs-device
volumeMode: Block
size: 60Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
- id: 12
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 1Gi
- name: test
aerospike:
path: /aerospike/dev/xvdf_test
source:
persistentVolume:
storageClass: openebs-device
volumeMode: Block
size: 60Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
- id: 13
storage:
cleanupThreads: 10
filesystemVolumePolicy:
initMethod: deleteFiles
cascadeDelete: true
blockVolumePolicy:
cascadeDelete: true
initMethod: dd
volumes:
- name: workdir
aerospike:
path: /opt/aerospike
source:
persistentVolume:
storageClass: gp2
volumeMode: Filesystem
size: 1Gi
- name: test
aerospike:
path: /aerospike/dev/xvdf_test
source:
persistentVolume:
storageClass: openebs-device
volumeMode: Block
size: 60Gi
- name: aerospike-config-secret
source:
secret:
secretName: aerospike-secret
aerospike:
path: /etc/aerospike/secret
aerospikeConfig:
service:
...
security: {}
network:
...
namespaces:
- name: test
memory-size: 2000000000
replication-factor: 2
nsup-period: 120
storage-engine:
type: device
devices:
- /aerospike/dev/xvdf_test
- /aerospike/dev/xvdf_test_2
- name: bar
memory-size: 2000000000
replication-factor: 2
nsup-period: 120
storage-engine:
type: device
devices:
- /aerospike/dev/xvdf_bar
- /aerospike/dev/xvdf_bar_2
- name: foo
memory-size: 2000000000
replication-factor: 2
nsup-period: 120
storage-engine:
type: device
devices:
- /aerospike/dev/xvdf_foo
- /aerospike/dev/xvdf_foo_2
- /aerospike/dev/xvdf_foo_3
- /aerospike/dev/xvdf_foo_4