Monitor Aerospike clusters on Kubernetes

This page describes how to use Prometheus and Grafana to monitor Aerospike clusters.

Overview

To monitor Aerospike clusters, expose the Aerospike Database metrics and make them readable by Prometheus. Use the Aerospike Monitoring Stack or the Prometheus Operator to monitor and set alerts for Aerospike clusters deployed by Aerospike Kubernetes Operator (AKO).

Expose metrics for Prometheus

Expose the metrics for Prometheus with the Aerospike Prometheus Exporter (APE), which runs as a sidecar container in the same Kubernetes pod as your Aerospike Database container.

You can configure the APE to connect to your Aerospike Database cluster using either plaintext authentication or a Kubernetes secret. The process for using a secret is longer, but you should always enable security in a production environment.

Plain text credentials

For testing purposes, you can pass your credentials to the cluster in plain text. The following example modifies the podSpec section of your Aerospike cluster’s Custom Resource (CR) file with a sidecars section. Replace the values for AS_AUTH_USER and AS_AUTH_PASSWORD with your actual credentials.

spec:
  ...
  podSpec:
    multiPodPerHost: true
    sidecars:
      - name: aerospike-prometheus-exporter
        image: aerospike/aerospike-prometheus-exporter:latest
        env:
          # Replace with your credentials
          - name: "AS_AUTH_USER"
            value: "exporter"
          - name: "AS_AUTH_PASSWORD"
            value: "exporter123"
          - name: "AS_AUTH_MODE"
            value: "internal"
        ports:
          - containerPort: 9145
            name: exporter

Kubernetes secret

A Kubernetes secret securely passes the location of a credentials file to the Aerospike Prometheus Exporter without revealing it in your Aerospike Database cluster Custom Resource (CR) file.

1. Download the ape.toml configuration file from the official Aerospike Prometheus Exporter repository: ape.toml.

2. Modify ape.toml to include file-based security credentials.

   [Aerospike]
   ...
   # database user
   user = "admin"
   
   # database password
   password = "file:/var/secret/password"
   
   # authentication mode: internal (server authentication) [default], external (such as LDAP), pki.
   auth_mode = "internal"
   ...

3. Create a Kubernetes ConfigMap from the ape.toml file. A ConfigMap maps the configuration file in a way that a Kubernetes container, such as the APE sidecar container, can read it dynamically without it being included in the container itself.

   kubectl create configmap sidecar-config --from-file=ape.toml

4. Enable secure authentication for the Aerospike Prometheus Exporter sidecar using a Kubernetes secret.

   Use kubectl to create the secret from your credentials.

   kubectl create secret generic auth-secret --from-literal=password=your_password

5. Update the cluster Custom Resource (CR) specification.

   To make ape.toml and the secret accessible in the APE container, add the following volume mounts in the Cluster CR specification:

   Volume for the APE ConfigMap:

   - name: config-volume
     sidecars:
       - containerName: aerospike-prometheus-exporter
         path: /var/aerospike-prometheus-exporter
     source:
       configMap:
         name: sidecar-config

   Volume for the authentication secret:

   - name: aerospike-auth-secret
     sidecars:
       - containerName: aerospike-prometheus-exporter
         path: /var/secret
     source:
       secret:
         secretName: auth-secret

6. Configure the Prometheus Exporter sidecar.

   In the aerospike-prometheus-exporter sidecar container definition, specify the path to ape.toml as an argument. The configuration in the following example ensures that the ape.toml file is available at /var/aerospike-prometheus-exporter/ape.toml. The password is securely stored in /var/secret/password and is accessible to the APE.

   Example podSpec:

   podSpec:
      multiPodPerHost: true
      sidecars:
        - name: aerospike-prometheus-exporter
          image: aerospike/aerospike-prometheus-exporter:latest
          args:
            - -config=/var/aerospike-prometheus-exporter/ape.toml
          ports:
            - containerPort: 9145
              name: exporter

Scrape metrics with Prometheus

To collect metrics from the APE, configure Prometheus to scrape the /metrics endpoint exposed by the exporter. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack.

Scrape with the Prometheus Operator

1. Follow the directions in GitHub at Install Prometheus Operator to install the kube-prometheus-stack, which includes the Prometheus Operator and Grafana. The Prometheus Operator uses a PodMonitor resource to scrape the exporter endpoints.

2. Create a file named pod-monitor.yaml with the following content.

   apiVersion: monitoring.coreos.com/v1
   kind: PodMonitor
   metadata:
     name: aerospike-cluster-pod-monitor
     namespace: aerospike
     labels:
       release: prometheus-operator
   spec:
     selector:
       matchLabels:
         app: aerospike-cluster
     namespaceSelector:
       matchNames:
         - default
         - aerospike
     podMetricsEndpoints:
       - port: exporter
         path: /metrics
         interval: 30s

3. Apply the PodMonitor resource.

   kubectl apply -f pod-monitor.yaml

Scrape with the AKO monitoring stack

The AKO repository, includes monitoring configurations in the config/ directory. Apply them with kubectl.

1. Apply the monitoring stack:

   kubectl apply -k config/monitoring

2. To configure alerts, create Prometheus rule YAML files in the aerospike-kubernetes-operator/config/monitoring/prometheus/config/alert-rules directory. Aerospike provides predefined Prometheus alert rules in the Aerospike Monitoring GitHub repository.

Grafana dashboards

To visualize the metrics, import pre-built Grafana dashboards from the Aerospike Monitoring GitHub repository or from Grafana Labs.