--- title: "Monitor Aerospike clusters on Kubernetes" description: "Monitor Aerospike clusters on Kubernetes using Prometheus, Grafana, and the Aerospike Prometheus Exporter." --- # Monitor Aerospike clusters on Kubernetes > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to use Prometheus and Grafana to monitor Aerospike clusters. ## Overview To monitor Aerospike clusters, expose the Aerospike Database metrics and make them readable by Prometheus. Use the [Aerospike Monitoring Stack](https://aerospike.com/docs/database/observe/monitor/components) or the Prometheus Operator to monitor and set alerts for Aerospike clusters deployed by Aerospike Kubernetes Operator (AKO). ## Expose metrics for Prometheus Expose the metrics for Prometheus with the [Aerospike Prometheus Exporter](https://github.com/aerospike/aerospike-prometheus-exporter) (APE), which runs as a sidecar container in the same Kubernetes pod as your Aerospike Database container. You can configure the APE to connect to your Aerospike Database cluster using either plaintext authentication or a Kubernetes secret. The process for using a secret is longer, but you should always enable security in a production environment. - [Plain text credentials](#tab-panel-3402) - [Kubernetes secret](#tab-panel-3403) For testing purposes, you can pass your credentials to the cluster in plain text. The following example modifies the `podSpec` section of your Aerospike cluster’s Custom Resource (CR) file with a `sidecars` section. Replace the values for `AS_AUTH_USER` and `AS_AUTH_PASSWORD` with your actual credentials. ```yaml spec: ... podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: aerospike/aerospike-prometheus-exporter:latest env: # Replace with your credentials - name: "AS_AUTH_USER" value: "exporter" - name: "AS_AUTH_PASSWORD" value: "exporter123" - name: "AS_AUTH_MODE" value: "internal" ports: - containerPort: 9145 name: exporter ``` A Kubernetes secret securely passes the location of a credentials file to the Aerospike Prometheus Exporter without revealing it in your Aerospike Database cluster Custom Resource (CR) file. 1. Download the `ape.toml` configuration file from the official Aerospike Prometheus Exporter repository: [ape.toml](https://github.com/aerospike/aerospike-prometheus-exporter/blob/master/configs/ape.toml). 2. Modify `ape.toml` to include file-based security credentials. ```toml [Aerospike] ... # database user user = "admin" # database password password = "file:/var/secret/password" # authentication mode: internal (server authentication) [default], external (such as LDAP), pki. auth_mode = "internal" ... ``` 3. Create a Kubernetes [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the `ape.toml` file. A ConfigMap maps the configuration file in a way that a Kubernetes container, such as the APE sidecar container, can read it dynamically without it being included in the container itself. Terminal window ```bash kubectl create configmap sidecar-config --from-file=ape.toml ``` 4. Enable secure authentication for the Aerospike Prometheus Exporter sidecar using a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/). Use `kubectl` to create the secret from your credentials. Terminal window ```bash kubectl create secret generic auth-secret --from-literal=password=your_password ``` 5. Update the cluster Custom Resource (CR) specification. To make `ape.toml` and the secret accessible in the APE container, add the following volume mounts in the Cluster CR specification: Volume for the APE ConfigMap: ```plaintext - name: config-volume sidecars: - containerName: aerospike-prometheus-exporter path: /var/aerospike-prometheus-exporter source: configMap: name: sidecar-config ``` Volume for the authentication secret: ```plaintext - name: aerospike-auth-secret sidecars: - containerName: aerospike-prometheus-exporter path: /var/secret source: secret: secretName: auth-secret ``` 6. Configure the Prometheus Exporter sidecar. In the `aerospike-prometheus-exporter` sidecar container definition, specify the path to `ape.toml` as an argument. The configuration in the following example ensures that the `ape.toml` file is available at `/var/aerospike-prometheus-exporter/ape.toml`. The password is securely stored in `/var/secret/password` and is accessible to the APE. Example `podSpec`: ```plaintext podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: aerospike/aerospike-prometheus-exporter:latest args: - -config=/var/aerospike-prometheus-exporter/ape.toml ports: - containerPort: 9145 name: exporter ``` ## Scrape metrics with Prometheus To collect metrics from the APE, configure Prometheus to scrape the `/metrics` endpoint exposed by the exporter. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack. ### Scrape with the Prometheus Operator 1. Follow the directions in GitHub at [Install Prometheus Operator](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install the `kube-prometheus-stack`, which includes the Prometheus Operator and Grafana. The [Prometheus Operator](https://github.com/prometheus-operator/kube-prometheus) uses a PodMonitor resource to scrape the exporter endpoints. 2. Create a file named `pod-monitor.yaml` with the following content. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: aerospike-cluster-pod-monitor namespace: aerospike labels: release: prometheus-operator spec: selector: matchLabels: app: aerospike-cluster namespaceSelector: matchNames: - default - aerospike podMetricsEndpoints: - port: exporter path: /metrics interval: 30s ``` 3. Apply the PodMonitor resource. ```plaintext kubectl apply -f pod-monitor.yaml ``` ### Scrape with the AKO monitoring stack The [AKO repository](https://github.com/aerospike/aerospike-kubernetes-operator), includes monitoring configurations in the `config/` directory. Apply them with `kubectl`. 1. Apply the monitoring stack: ```plaintext kubectl apply -k config/monitoring ``` 2. To configure alerts, create Prometheus rule YAML files in the `aerospike-kubernetes-operator/config/monitoring/prometheus/config/alert-rules` directory. Aerospike provides predefined Prometheus alert rules in the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/prometheus). ### Grafana dashboards To visualize the metrics, import pre-built Grafana dashboards from the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/grafana/dashboards) or from [Grafana Labs](https://grafana.com/orgs/aerospike/dashboards).