Rack Awareness for Aerospike on Kubernetes

Aerospike’s rack awareness feature makes copies of your database and stores them across multiple racks or zones. The primary copy and its replica are stored in separate hardware failure groups, meaning that in case of a hardware failure on the primary partition copy, Kubernetes will be able to dynamically appoint resources to the backups to avoid serious downtime. This feature is available in Aerospike Database Enterprise Edition 4.0 and later.

Rack awareness also allows database clients to connect to database servers in the closest rack or zone. This can result in lower latency, increased stability, and significantly reduced traffic charges by limiting cross-availability-zone traffic.

For more information, see the documentation on Aerospike Rack Awareness.

Add rack awareness

This example adds rack awareness to an existing Aerospike cluster custom resource (CR) file.

It adds a rackConfig section outside the aerospikeConfig section specifying two racks in the test namespace: one in the us-central1-b zone and the other in the uscentral1-a zone.

  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        zone: us-central1-b
        aerospikeConfig:
          service:
            proto-fd-max: 18000
        storage:
          filesystemVolumePolicy:
            initMethod: deleteFiles
            cascadeDelete: true
          blockVolumePolicy:
            cascadeDelete: true
          volumes:
            - storageClass: ssd
              path: /opt/aerospike
              volumeMode: filesystem
              sizeInGB: 1
            - path: /opt/aerospike/data
              storageClass: ssd
              volumeMode: filesystem
              sizeInGB: 3
      - id: 2
        zone: us-central1-a
        aerospikeConfig:
          service:
            proto-fd-max: 16000
  .
  .
  .
  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 1
        storage-engine:
          type: memory
          files:
            - /opt/aerospike/data/test.dat
          filesize: 2000000000
      - name: testMem
        replication-factor: 1
        storage-engine:
          type: memory
          data-size: 1073741824

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

For the full CR file, see the example rack-enabled cluster CR.

This and other example CRs are stored in the main Aerospike Kubernetes Operator repository.

Cluster node distribution

Cluster nodes are distributed across racks as evenly as possible. The cluster size is divided by the number of racks to determine the number of nodes per rack. Any remainder nodes are distributed one-by-one across racks starting from the first rack.

For example, in a setup with 10 nodes spread across 4 racks, the topology is:

• Rack 1: 3 nodes
• Rack 2: 3 nodes
• Rack 3: 2 nodes
• Rack 4: 2 nodes

In this example, two nodes were assigned per rack up through Rack 4. Afterward, the two remainder nodes were added to Rack 1 and Rack 2 in sequence.

Add a new rack

To add a new rack, add a new section beginning with id to the CR file under the rackConfig.racks section.

  rackConfig:
    namespaces:
      - test
    racks:
      .
      .
      .
      - id: 3
        zone: us-central1-c

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

The Aerospike Kubernetes Operator redistributes cluster nodes across racks whenever the cluster size or the number of racks changes. If you add a rack without increasing the cluster size, the nodes are redistributed, reducing the number of nodes on each existing rack to fill the newly created rack.

Set rack-level storage and aerospikeConfig

Aerospike's rack awareness lets you set local storage and aerospikeConfig options. If you provide local storage for a rack, the rack uses this storage. Otherwise, it uses common global storage.

In the following example, the aerospikeConfig section under racks is a patch used for the rack and merged with the common global aerospikeConfig at the root of spec of the CR file (not shown).

  rackConfig:
    namespaces:
      - test
    racks:
      - id: 1
        zone: us-central1-b

        aerospikeConfig:
          service:
            proto-fd-max: 18000

        storage:
          filesystemVolumePolicy:
            cascadeDelete: true
            initMethod: deleteFiles
          volumes:
            - name: workdir
              aerospike:
                path: /opt/aerospike
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 1Gi
            - name: ns
              aerospike:
                path: /opt/aerospike/data
              source:
                persistentVolume:
                  storageClass: ssd
                  volumeMode: Filesystem
                  size: 3Gi
            - name: aerospike-config-secret
              source:
                secret:
                  secretName: aerospike-secret
              aerospike:
                path: /etc/aerospike/secret

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

Merge aerospikeConfig

A rack's local aerospikeConfig patch is merged with the common global base aerospikeConfig based on the following rules:

• New elements from the patch configMap are added to the base configMap.
• A base element is replaced with a new patch element if:
  • The element value type is changed.
  • The element value is a primitive type and updated.
  • The element value is a primitive list type and updated.
  • The element key is storage-engine and the storage-engine type has been changed (storage-engine can be of device, file, or memory type).
• If the element is of map type, patch and base elements is recursively merged.
• If the elements are list of maps, new list elements in the patch list is appended to the base list and corresponding entries are merged using the same merge algorithm.
  • The order of elements in the base list is maintained.
  • Corresponding entries are found by matching the special name key in maps.
  • This list of maps is actually a map of maps.
  • Main map keys are added in sub-map with key as name to convert a map of maps to a list of maps.

As an example, here is the original rack-local aerospikeConfig and common global aerospikeConfig:

  rackConfig:
    racks:
        aerospikeConfig:
          service:
            proto-fd-max: 18000
          namespaces:
            - name: test
              storage-engine:
                type: device
                devices:
                  - /dev/nvme0n2 /dev/sdf2
            - name: bar
              storage-engine:
                type: memory
                data-size: 1073741824
.
.
.
  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/nvme0n1 /dev/sdf
      - name: bar
        replication-factor: 2
        storage-engine:
          type: device
          devices:
            - /dev/nvme0n10 /dev/sdf10

After merging the rack-local aerospikeConfig with the common global aerospikeConfig:

  aerospikeConfig:
    service:
      proto-fd-max: 18000
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    namespaces:
      - name: test
        replication-factor: 2
        # storage-engine type is not changed, so it is merged recursively
        storage-engine:
          type: device
          devices:
            - /dev/nvme0n2 /dev/sdf2
      - name: bar
        replication-factor: 2
        # storage-engine type is changed, so it is replaced
        storage-engine:
          type: memory
          data-size: 1073741824

Save and exit the CR file, then use kubectl to apply the change.

kubectl apply -f aerospike-cluster.yaml

Remove a rack

To remove a rack, delete that rack's section from the rackConfig.racks section, then use kubectl to apply the new configuration.

The Aerospike Kubernetes Operator scales down the desired rack to size 0. It removes one node at a time from the rack, then removes the entire rack. If you are removing a rack without decreasing cluster size, other racks are scaled up as new nodes get added to the existing racks to match the total number before the rack was removed.

Simultaneously add and remove racks

If the Operator has to scale up some racks and scale down other racks in a single call, the Operator always scales up first and scales down second. As a result, for a short time during the procedure, the actual cluster size may be larger than the desired cluster size.