Configuration Settings for an Aerospike Cluster on Kubernetes

Aerospike cluster configuration settings are in the Aerospike cluster Custom Resource (CR) file. The Operator reads this file to make changes to the cluster based on any changes in the file.

The Operator Custom Resource Definition (CRD) specifies the CR that the Operator uses to manage the cluster.

Example CR

The following example is a basic CR to get you started. It creates a simple cluster with no storage, using data-in-memory (DIM).

For a more realistic real-world example, we recommend the example rack-enabled cluster CR.

These and other example CRs are stored in the main Aerospike Kubernetes Operator repository.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike
spec:
  size: 2
  image: aerospike/aerospike-server-enterprise:6.4.0.0

  podSpec:
    multiPodPerHost: true

  storage:
    filesystemVolumePolicy:
      cascadeDelete: true
      initMethod: deleteFiles
    volumes:
      - name: workdir
        source:
          persistentVolume:
            storageClass: ssd
            volumeMode: Filesystem
            size: 3Gi
        aerospike:
          path: /opt/aerospike
      - name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

  aerospikeAccessControl:
    users:
      - name: admin
        secretName: auth-secret
        roles:
          - sys-admin
          - user-admin

  aerospikeConfig:
    service:
      feature-key-file: /etc/aerospike/secret/features.conf
    security: {}
    network:
      service:
        port: 3000
      fabric:
        port: 3001
      heartbeat:
        port: 3002
    namespaces:
      - name: test
        memory-size: 3000000000
        replication-factor: 2
        storage-engine:
          type: memory

Configuration

The initial part of the CR selects the CRD and the namespace to use for the Aerospike cluster.

apiVersion: asdb.aerospike.com/v1
kind: AerospikeCluster
metadata:
  name: aerocluster
  namespace: aerospike

Spec

The spec section defines the cluster's configurations.

Field	Required	Type	Default	Description
size Dynamic	Yes	Integer		The size/number of Aerospike node pods to run for this cluster.
image Dynamic Rolling restart	Yes	String		The official Aerospike Enterprise Server docker image to use for the node in the cluster.
podSpec Dynamic Rolling restart	No	Structure		Configures the Kubernetes pod running Aerospike server. See Pod Spec for details.
storage Dynamic	No	Structure		Required for persistent namespaces and for Aerospike work directory, unless the validation policy skips validating persistence of the work directory. See Storage for details.
aerospikeNetworkPolicy Dynamic Rolling restart	No	Structure		Configures IP and port types used for access and fabric. See Network Policy for details.
aerospikeAccessControl Dynamic	No	Structure		Required if Aerospike security is enabled. See Access Control for details.
aerospikeConfig Dynamic Rolling restart	Yes	configMap		A freeform configMap confirming to the configuration schema for the deployed Aerospike server version. See Aerospike Config for details.
rackConfig Dynamic	No	Structure		Configures the operator to deploy rack aware Aerospike clusters. Pods will be deployed in given racks based on given configuration. See Rack Config for details.
operatorClientCert Dynamic	With TLS	Structure		Configures the TLS certs used by the operator when connecting to the Aerospike cluster. See Operator Client Certs for details.
validationPolicy Dynamic	No	Structure		Configures the custom resource validation. See Validation Policy for details.
seedsFinderServices Dynamic	No	Structure		Creates additional Kubernetes services that allow clients to discover Aerospike cluster nodes. See Seeds Finder Services for details.
rosterNodeBlockList Dynamic	No	List of Strings		Configures a list of nodeIDs which are blocked from roster in a strong consistency setup.

|

Operator Client Cert

The operator needs to connect as an Aerospike client to perform management asinfo calls. If the Aerospike service is configured to use TLS for clients, you need to specify the certificates the client uses to make these calls.

Field	Required	Type	Default	Description
tlsClientName	No	String		If specified, this name will be added to tls-authenticate-client list by the operator so that the Aerospike servers will always allow it.
secretCertSource	*Yes	Structure		The operator certs should be read from a secret. See Secret Cert Source for details.
certPathInOperator	*Yes	Structure		The operator certs should be read from files mounted into the operator container. See Cert Path In Operator Source for details.

* Either secretCertSource or certPathInOperator must be specified, but not both.

Secret Cert Source

Specifies that the certificate should be read from a Kubernetes Secret.

Field	Required	Type	Default	Description
caCertsSource	No	Structure		Structure containing CA certificates secret object info.
secretName	Yes	String		The name of the secret.
secretNamespace	No	String	Kubernetes namespace where Aerospike cluster is deployed	The namespace containing the secret.
caCertsFilename	No	String		The name of the secret key /file containing the CA certificate.
clientCertFilename	No	String		The name of the secret key /file containing the operator's certificate.
clientKeyFilename	No	String		The name of the secret key /file containing the operator's secret key.

CA Certs Source

Field	Required	Type	Default	Description
secretName	Yes	String		The name of the secret.
secretNamespace	No	String	Kubernetes namespace where Aerospike cluster is deployed	The namespace containing the secret.

Cert Path In Operator Source

Specifies that the operator's certificate files are mounted onto the operator's containers.

Field	Required	Type	Default	Description
caCertsPath	No	String		The path to the file or directory containing the CA certificates.
clientCertPath	No	String		The path to the file containing the operator's certificate.
clientKeyPath	No	String		The path to the file containing the operator's secret key.

Pod Spec

Configures the Kubernetes pod running Aerospike server. Sidecar containers for monitoring or running connectors can be added to each Aerospike pod.

Field	Required	Type	Default	Description
metadata	No	Structure		Kubernetes Annotations and Labels to add to pods. See metadata for details
multiPodPerHost	No	Boolean	False	Indicates if the operator should run multiple pods per Kubernetes cluster host.
affinity	No	Kubernetes Pod Affinity		Kubernetes Affinity rules for pod placement. These rules will be merged with affinity rules generated by the operator.
tolerations	No	Kubernetes Pod Tolerations		Kubernetes Toleration for Aerospike pod placement
nodeSelector	No	Map from string to string		Node selector constraints for the Aerospike pods.
hostNetwork	No	Boolean	False	Enables host networking for the pod. To enable hostNetwork, multiPodPerHost must be false.
dnsPolicy	No	Kubernetes DNSPolicy		DnsPolicy to use. If hostNetwork is true and policy is not specified, it defaults to ClusterFirstWithHostNet
aerospikeContainer	No	Structure		Configures the aerospike-server container created by operator. See Aerospike Container for details.
aerospikeInitContainer	No	Structure		Configures the aerospike-init container created by operator. See Aerospike Init Container for details.
sidecars	No	List of Container structures		List of side containers to run along with the main Aerospike server container. Volume mounts are not supported. See storage for attaching volumes to sidecars.
initContainers	No	List of Container structures		List of init containers before running Aerospike and the sidecar containers.
securityContext	No	Structure PodSecurityContext		SecurityContext holds pod-level security attributes and common container settings
imagePullSecrets	No	List of TypedLocalObjectReference		ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec.

See Monitoring for details on configuring monitoring sidecars or Aerospike containers.

Metadata

Kubernetes annotations and labels to add to the pods.

Field	Required	Type	Default	Description
annotations	No	Map from annotation name to its value		Kubernetes Annotations
labels	No	Map from label name to its value		Kubernetes Labels

Aerospike Container

Configures the aerospike-server container created by the operator.

Field	Required	Type	Default	Description
securityContext	No	Structure SecurityContext		Kubernetes security context for the Aerospike container.
resources	No	Structure ResourceRequirements		Configures resource requirements and limits like CPU or memory for the Aerospike container.

Aerospike Init Container

Configures the aerospike-init container created by the operator.

Field	Required	Type	Default	Description
imageRegistry	No	String	docker.io	Name of image registry for Aerospike init container image.
securityContext	No	Structure SecurityContext		Kubernetes security context for the Aerospike init container.
resources	No	Structure ResourceRequirements		Configures resources requirements and limits like CPU or memory for the Aerospike init container.

Storage

The storage section configures persistent volume devices to provision and attach to the containers.

This section is required by default for persisting the Aerospike work directory. Store the work directory on persistent storage to ensure pod restarts do not reset Aerospike server metadata files.

This section is also required if Aerospike namespaces require persistent storage.

Field	Required	Type	Default	Description
filesystemVolumePolicy Dynamic	No	Structure		Volume policy for filesystem volumes
blockVolumePolicy Dynamic	No	Structure		Volume policy for block volumes
cleanupThreads Dynamic	No	Integer	1	Defines the maximum number of disk cleanup threads (dd or blkdiscard) per init container that are used to wipe or init disk volumes.
Volumes Dynamic	No	List of Volumes Structures		List of Volumes to attach to Aerospike pods. Persistent storage volumes cannot be added or removed dynamically.

Volume Policy

Specifies persistent volume policy to determine how new volumes are initialized.

Field	Required	Type	Default	Description
initMethod Dynamic	No	Enum	none	Controls how the volumes are initialized when the persistent volume is attached the first time to a pod. Valid values are none, dd, blkdiscard, deleteFiles.
wipeMethod Dynamic	No	Enum	dd for block volumes and deleteFiles for filesystem volumes	Controls how the volumes are wiped for special upgrades/downgrades. Valid values are dd, blkdiscard, deleteFiles.
cascadeDelete Dynamic	No	Boolean	false	CascadeDelete determines if the persistent volumes are deleted after the pods these volumes bind to are terminated and removed from the cluster.

note

blkdiscard only works for devices that support TRIM. For AWS please refer to the storage volumes guide to check TRIM support. If TRIM is not supported please use the slower dd to initialize/wipe your devices. For other devices please verify the support for TRIM command with your vendor.

InitMethod

For filesystem volumes, initMethod can be none or deleteFiles. For block volumes, initMethod can be none, dd or blkdiscard.

WipeMethod

For filesystem volumes, wipeMethod can be only deleteFiles. For block volumes, wipeMethod can be dd or blkdiscard.

caution

When using blkdiscard ensure that the drive deterministically returns zeros after TRIM (RZAT). Check with your hardware provider or cloud provider for this information. If you are unsure, it is safer to wipe the device using dd as the wipe method.

Volume

Describes volumes to be created and attached to the init containers and main Aerospike container, as well as other sidecars.

Field	Required	Type	Default	Description
name	Yes	String		A name identifying this volume. The name must be unique among all created volumes.
source	Yes	Structure		Specifies the source for this volume. See Volume Source for details.
aerospike	No	Structure		Specifies how this volume is attached to the main Aerospike server container. See AerospikeServerVolumeAttachment for details.
sidecars	No	Structure		Attaches this volume to other sidecar containers. See Volume Attachment for details.
initContainers	No	Structure		Attaches this volume to other init containers. See Volume Attachment for details.
initMethod Dynamic	No	Enum	none	Controls how this volume is initialized when the persistent volume is attached the first time to a pod. Valid values are none, dd, blkdiscard, deleteFiles.
cascadeDelete Dynamic	No	Boolean	false	CascadeDelete determines if the persistent volume is deleted after the pod this volume binds to is terminated and removed from the cluster.

For filesystem volumes, initMethod can be none or deleteFiles. For block volumes, initMethod can be none, dd or blkdiscard.

note

blkdiscard only works for devices that support TRIM. For AWS please refer to the storage volumes guide to check TRIM support. If TRIM is not supported please use the slower dd to initialize your devices. For other devices please verify the support for TRIM command.

Aerospike Server Volume Attachment

Specifies attaching a volume to the main Aerospike server container.

Field	Required	Type	Default	Description
path	Yes	String		The path to attach the volume to the Aerospike Container.

Volume Attachment

Specifies attaching a volume to a sidecar container.

Field	Required	Type	Default	Description
containerName	Yes	String		The name of the container to attach this volume to.
path	Yes	String		The path to attach the volume to the Aerospike Container.

Volume Source

A Volume source specifies the source for an attached volume. Volumes are created using a volume source that can be one of the following:

• Persistent volume
• EmptyDir
• Secret
• ConfigMap

Persistent Volume

Specifies a persistent volume to claim and attach to Aerospike pods.

Field	Required	Type	Default	Description
storageClass	Yes	String		The name of the storage class to use.
volumeMode	Yes	Enum - Filesystem, Block		Specifies the mode this volume should be created with. Filesystem creates a pre-formatted filesystem. Block mode creates a raw device.
size	Yes	String Quantity		The device size to be provisioned. For example, 5Gi.
accessModes	No	List of PersistentVolumeAccessMode		Allowed access modes to the volume. For example, ReadWriteOnce, ReadOnlyMany, ReadWriteMany.
selector	No	LabelSelector		Only the volumes whose labels match the selector will be claimed.
initMethod Dynamic	No	Enum	none	Controls how this volume is initialized when the persistent volume is attached the first time to a pod. Valid values are none, dd, blkdiscard, deleteFiles.
cascadeDelete Dynamic	No	Boolean	false	CascadeDelete determines if the persistent volume is deleted after the pod this volume binds to is terminated and removed from the cluster.

EmptyDir

These are volumes to be used as temporary working disk space. See the official documentation for options.

      - name: tempFiles
        aerospike:
          path: /opt/aerospike/temp
        source:
          emptyDir: {}

Secret

A Kubernetes Secret can be mounted as a volume. You can use a Secret for mounting the Aerospike feature-key file (features.conf) or sensitive files like TLS certificates and credentials onto the containers.

- name: aerospike-config-secret
        source:
          secret:
            secretName: aerospike-secret
        aerospike:
          path: /etc/aerospike/secret

ConfigMap

A Kubernetes ConfigMap can be mounted as a volume. ConfigMaps hold non-confidential data like configuration for applications running in your custom init containers or sidecars.

- name: app-config
    source:
      configMap:
        # Provide the name of the ConfigMap you want to mount.
        name: app-config
        # An optional array of keys from the ConfigMap to create as files
        items:
        - key: "game.properties"
          path: "game.properties"
        - key: "user-interface.properties"
          path: "user-interface.properties"

Validation Policy

This section configures the policy for validating the cluster CR.

Field	Required	Type	Default	Description
skipWorkDirValidate Dynamic	No	Boolean	false	If true, skips validating that the Aerospike work directory is stored on a persistent volume.
skipXdrDlogFileValidate Dynamic	No	Boolean	false	If true, skips validating that the XDR digest log is stored on a persistent volume.

Network Policy

This section configures IP and port types used for access, alternate access, TLS access, TLS alternate access, fabric and TLS fabric endpoints on the Aerospike cluster.

Five types of endpoint configurations are supported.

• pod uses the Kubernetes pod IP and Aerospike port that works to connect from other pods in the same Kubernetes cluster.
• hostInternal uses the Kubernetes cluster node's host IP and a mapped Aerospike port that works to connect from the VPC or internal network used by the Kubernetes cluster.
• hostExternal uses the Kubernetes cluster node's host external/public IP and a mapped Aerospike port that works to connect from the external network.
• configuredIP uses the IP address configured by the user in the Kubernetes node labels, and a mapped Aerospike port, that is used to connect from the internal/external network. Label "aerospike.com/configured-access-address" in the Kubernetes node is used as accessAddress, and label "aerospike.com/configured-alternate-access-address" in the Kubernetes node is used as alternateAccessAddress.
• customInterface uses the IP address assigned to the custom interfaces attached to the pod and Aerospike port, that is used to connect from the same custom interfaces network. These custom interfaces are given separately as a list of NetworkAttachmentDefinition CRs and also, must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod.

Field	Required	Type	Default	Description
access Dynamic Rolling restart	No	Enum [pod, hostInternal, hostExternal, configuredIP, customInterface]	hostInternal	Configures Aerospike access endpoint.
customAccessNetworkNames Dynamic Rolling restart	No	List of Strings		CustomAccessNetworkNames is the list of the pod's network interfaces used for Aerospike access address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' access type.
alternateAccess Dynamic Rolling restart	No	Enum [pod, hostInternal, hostExternal, configuredIP, customInterface]	hostExternal	Configures Aerospike alternate access endpoint.
customAlternateAccessNetworkNames Dynamic Rolling restart	No	List of Strings		CustomAlternateAccessNetworkNames is the list of the pod's network interfaces used for Aerospike alternate access address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' alternateAccess type.
tlsAccess Dynamic Rolling restart	No	Enum [pod, hostInternal, hostExternal, configuredIP, customInterface]	hostInternal	Configures Aerospike TLS access endpoint.
customTLSAccessNetworkNames Dynamic Rolling restart	No	List of Strings		CustomTLSAccessNetworkNames is the list of the pod's network interfaces used for Aerospike TLS access address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' tlsAccess type.
tlsAlternateAccess Dynamic Rolling restart	No	Enum [pod, hostInternal, hostExternal, configuredIP, customInterface]	hostExternal	Configures Aerospike TLS alternate endpoint.
customTLSAlternateAccessNetworkNames Dynamic Rolling restart	No	List of Strings		CustomTLSAlternateAccessNetworkNames is the list of the pod's network interfaces used for Aerospike TLS alternate access address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' tlsAlternateAccess type.
fabric	No	Enum [customInterface]		Configures Aerospike fabric endpoint.
customFabricNetworkNames	No	List of Strings		CustomFabricNetworkNames is the list of the pod's network interfaces used for Aerospike fabric address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' fabric type.
tlsFabric	No	Enum [customInterface]		Configures Aerospike TLS fabric endpoint.
customTLSFabricNetworkNames	No	List of Strings		CustomTLSFabricNetworkNames is the list of the pod's network interfaces used for Aerospike TLS fabric address. Each element in the list is specified with a namespace and the name of a NetworkAttachmentDefinition, separated by a forward slash (/). These elements must be defined in the pod annotation k8s.v1.cni.cncf.io/networks in order to assign network interfaces to the pod. Required with 'customInterface' tlsFabric type.

Aerospike Access Control

Provides Aerospike access control configuration for the Aerospike cluster.

Field	Required	Type	Default	Description
roles Dynamic	No	List of Structures		A list of Role structures with an entry for each role.
users Dynamic	No	List of Structures		A list of User structures with an entry for each user. Required if Aerospike security is enabled.

If the Aerospike cluster has security enabled, an entry for the "admin" user having at least "sys-admin" and "user-admin" roles is mandatory.

Aerospike Role

Configures roles for the Aerospike cluster.

Field	Required	Type	Default	Description
name	Yes	Strings		The name of this role.
privileges Dynamic	Yes	List of Strings		The privileges to grant this role.
whitelist Dynamic	No	List of Strings		Allowlist of host address allowed for this role.
readQuota Dynamic	No	Integer		Specifies permitted rate of read records for current role (the value is in RPS)
writeQuota Dynamic	No	Integer		Specifies permitted rate of writes for current role (the value is in RPS)

Aerospike User

Configures users for the Aerospike cluster.

Field	Required	Type	Default	Description
name	Yes	Strings		The name of this user.
secretName Dynamic	Yes	String		The name of the secret containing this user's password.
roles Dynamic	Yes	List of Strings		The roles to grant to this user.

Aerospike Config Secret

Configures the name of the secret to use and the mount path to mount the secret files on the container.

Field	Required	Type	Default	Description
secretName Dynamic	Yes	String		The name of the secret.
mountPath Dynamic	Yes	String		The path where the secret files will be mounted in the container.

Aerospike Config

The YAML form of Aerospike server configuration. See Aerospike Configuration for details.

Rack Config

Configures the operator to deploy a rack-aware Aerospike cluster. Pods are deployed in given racks based on the given configuration.

Field	Required	Type	Default	Description
namespaces Dynamic Rolling restart	No	List of Strings		List of Aerospike namespaces for which rack feature will be enabled.
racks Dynamic	Yes	List of structures		List of racks.
rollingUpdateBatchSize Dynamic	No	Integer or String		Configures the number or percentage of rack pods that will be restarted simultaneously during rolling restart. This value can be an integer, like 5, or a string, like 25%.

See Rack awareness for details.

Rack

Specifies single rack config

Field	Required	Type	Default	Description
id	Yes	Integer		Identifier for the rack.
zone	No	String		Cloud provider specific Zone name for setting rack affinity. Rack pods will be deployed to the given cloud Zone.
region	No	String		Cloud provider specific Region name for setting rack affinity. Rack pods will be deployed to the given cloud Region.
rackLabel	No	String		Rack label for setting rack affinity. Rack pods will be deployed in k8s nodes having rack label aerospike.com/rack-label: <rack-label>.
nodeName	No	String		Kubernetes node name for setting rack affinity. Rack pods will be deployed on the given node.
aerospikeConfig Dynamic Rolling restart	No	Structure		This local AerospikeConfig is a patch, which will be merged recursively with common global AerospikeConfig and will be used for this Rack. See merge AerospikeConfig. If this AerospikeConfig is not given, then global AerospikeConfig will be used.
storage Dynamic Rolling restart	No	Structure		This local Storage specify persistent storage to use for the pods in this rack. If this Storage is not given then global Storage will be used.
podSpec Dynamic Rolling restart	No	Structure		Pod overrides for this rack. See Rack Pod Overrides for details.

Rack Pod Overrides

Provides rack-specific overrides to the Pod spec.

The following overrides are supported.

Field	Required	Type	Default	Description
affinity	No	Kubernetes Pod Affinity		Kubernetes Affinity rules for pod placement. These rules will be merged with affinity rules generated by the operator.
tolerations	No	Kubernetes Pod Tolerations		Kubernetes Toleration for Aerospike pod placement.
nodeSelector	No	Map from string to string		Node selector constraints for the Aerospike pods.

Seeds Finder Services

Creates additional Kubernetes service that allow clients to discover Aerospike cluster nodes.

Field	Required	Type	Default	Description
loadBalancer	No	Structure		Creates a load balancer service that allows Aerospike clients to discover Aerospike cluster nodes. See #load balancer Service for details.

Load Balancer Service

Creates a load balancer service which lets Aerospike clients discover Aerospike cluster nodes.

Field	Required	Type	Default	Description
externalTrafficPolicy	No	Enum - Local, Cluster		External Traffic Policy Type string. See ServiceExternalTrafficPolicyType for details.
annotations	No	Map from annotation name to its value		Kubernetes Annotations for the load balancer.
port	No	Integer		Exposed port on load balancer. If not specified targetPort is used.