# Rack awareness for Aerospike on Kubernetes Aerospike’s rack awareness feature makes copies of your database and stores them across multiple racks or zones. The primary copy and its replica are stored in separate hardware failure groups, meaning that in case of a hardware failure on the primary partition copy, Kubernetes can dynamically appoint resources to the backups to avoid serious downtime. This feature is available in Aerospike Database Enterprise Edition 4.0 and later. Rack awareness also lets database clients connect to database servers in the closest rack or zone. This can result in lower latency, increased stability, and significantly reduced traffic charges by limiting cross-availability-zone traffic. For more information, [see the documentation on Aerospike rack awareness](https://aerospike.com/docs/database/learn/architecture/clustering/rack-aware). ## Add rack awareness This example adds rack awareness to an existing Aerospike cluster custom resource (CR) file. It adds a `rackConfig` section outside the `aerospikeConfig` section specifying two racks in the `test` namespace: one in the `us-central1-b` zone and the other in the `us-central1-a` zone. ```yaml rackConfig: namespaces: - test racks: - id: 1 zone: us-central1-b aerospikeConfig: service: proto-fd-max: 18000 storage: filesystemVolumePolicy: initMethod: deleteFiles cascadeDelete: true blockVolumePolicy: cascadeDelete: true volumes: - storageClass: ssd path: /opt/aerospike volumeMode: filesystem sizeInGB: 1 - path: /opt/aerospike/data storageClass: ssd volumeMode: filesystem sizeInGB: 3 - id: 2 zone: us-central1-a aerospikeConfig: service: proto-fd-max: 16000 ... aerospikeConfig: service: feature-key-file: /etc/aerospike/secret/features.conf security: {} namespaces: - name: test replication-factor: 1 storage-engine: type: memory files: - /opt/aerospike/data/test.dat filesize: 2000000000 - name: testMem replication-factor: 1 storage-engine: type: memory data-size: 1073741824 ``` After making changes, save and exit the CR file, then use `kubectl` to apply the change. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` For the full CR file, see the [example rack-enabled cluster CR](https://github.com/aerospike/aerospike-kubernetes-operator/blob/v4.1.2/config/samples/rack_enabled_cluster_cr.yaml). This and other example CRs are available in [the main Aerospike Kubernetes Operator repository](https://github.com/aerospike/aerospike-kubernetes-operator/tree/v4.1.2/config/samples). ## Cluster pod distribution Cluster pods are distributed across racks as evenly as possible. The cluster size is divided by the number of racks to determine the number of pods per rack. Any remainder pods are distributed one-by-one across racks starting from the first rack. For example, in a setup with 10 pods spread across 4 racks, the topology is: - Rack 1: 3 pods - Rack 2: 3 pods - Rack 3: 2 pods - Rack 4: 2 pods In this example, two pods were assigned per rack up through rack 4. Afterward, the two remainder pods were added to rack 1 and rack 2 in sequence. ## Add a new rack 1. Add a new section beginning with `id` to the CR file under the `rackConfig.racks` section. ```yaml rackConfig: namespaces: - test racks: ... - id: 3 zone: us-central1-c ``` 2. Save and exit the CR file, then use `kubectl` to apply the change. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` The Aerospike Kubernetes Operator redistributes cluster pods across racks whenever the cluster size or the number of racks changes. If you add a rack without increasing the cluster size, the pods are redistributed, reducing the number of pods on each existing rack to fill the newly created rack. ## Set rack-level storage and aerospikeConfig Aerospike’s rack awareness lets you set local storage and aerospikeConfig options. If you provide local storage for a rack, the rack uses this storage. Otherwise, it uses common global storage. In the following example, the `aerospikeConfig` section under `racks` is a patch used for the rack and merged with the common global `aerospikeConfig` at the root of spec of the CR file (not shown). ```yaml rackConfig: namespaces: - test racks: - id: 1 zone: us-central1-b aerospikeConfig: service: proto-fd-max: 18000 storage: filesystemVolumePolicy: cascadeDelete: true initMethod: deleteFiles volumes: - name: workdir aerospike: path: /opt/aerospike source: persistentVolume: storageClass: ssd volumeMode: Filesystem size: 1Gi - name: ns aerospike: path: /opt/aerospike/data source: persistentVolume: storageClass: ssd volumeMode: Filesystem size: 3Gi - name: aerospike-config-secret source: secret: secretName: aerospike-secret aerospike: path: /etc/aerospike/secret ``` After making changes, save and exit the CR file, then use `kubectl` to apply the change. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` ## Merge aerospikeConfig A rack’s local aerospikeConfig patch is merged with the common global base aerospikeConfig based on the following rules: - New elements from the patch configMap are added to the base configMap. - A base element is replaced with a new patch element if: - The element value type is changed. - The element value is a primitive type and updated. - The element value is a primitive list type and updated. - The element key is `storage-engine` and the storage-engine type has been changed (storage-engine can be of `device`, `file`, or `memory` type). - If the element is of map type, patch and base elements are recursively merged. - If the elements are list of maps, new list elements in the patch list is appended to the base list and corresponding entries are merged using the same merge algorithm. - The order of elements in the base list is maintained. - Corresponding entries are found by matching the special `name` key in maps. - This list of maps is actually a map of maps. - Main map keys are added in sub-map with key as `name` to convert a map of maps to a list of maps. As an example, here is the original rack-local aerospikeConfig and common global aerospikeConfig: ```yaml rackConfig: racks: aerospikeConfig: service: proto-fd-max: 18000 namespaces: - name: test storage-engine: type: device devices: - /dev/nvme0n2 /dev/sdf2 - name: bar storage-engine: type: memory data-size: 1073741824 ... aerospikeConfig: service: feature-key-file: /etc/aerospike/secret/features.conf security: {} namespaces: - name: test replication-factor: 2 storage-engine: type: device devices: - /dev/nvme0n1 /dev/sdf - name: bar replication-factor: 2 storage-engine: type: device devices: - /dev/nvme0n10 /dev/sdf10 ``` After merging the rack-local aerospikeConfig with the common global aerospikeConfig: ```yaml aerospikeConfig: service: proto-fd-max: 18000 feature-key-file: /etc/aerospike/secret/features.conf security: {} namespaces: - name: test replication-factor: 2 # storage-engine type is not changed, so it is merged recursively storage-engine: type: device devices: - /dev/nvme0n2 /dev/sdf2 - name: bar replication-factor: 2 # storage-engine type is changed, so it is replaced storage-engine: type: memory data-size: 1073741824 ``` After making changes, save and exit the CR file, then use `kubectl` to apply the change. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` ## Remove a rack 1. Delete that rack’s section from the `rackConfig.racks` section. 2. Save and exit the CR file, then use `kubectl` to apply the change. Terminal window ```shell kubectl apply -f aerospike-cluster.yaml ``` The Aerospike Kubernetes Operator scales down the desired rack to size 0. It removes one pod at a time from the rack, then removes the entire rack. If you are removing a rack without decreasing cluster size, other racks are scaled up as new pods get added to the existing racks to match the total number before the rack was removed. ## Simultaneously add and remove racks If AKO has to scale up some racks and scale down other racks in a single call, AKO always scales up first and scales down second. As a result, for a short time during the procedure, the actual cluster size may be larger than the desired cluster size.