--- title: "Monitor Aerospike clusters on Kubernetes" description: "Monitor Aerospike clusters on Kubernetes using Prometheus, Grafana, and the Aerospike Prometheus Exporter." --- # Monitor Aerospike clusters on Kubernetes > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. This page describes how to use Prometheus and Grafana to monitor Aerospike clusters. To monitor Aerospike clusters, expose the Aerospike Database metrics and make them readable by Prometheus. Use the [Aerospike Monitoring Stack](https://aerospike.com/docs/database/observe/monitor/components) or the Prometheus Operator to monitor and set alerts for Aerospike clusters deployed by Aerospike Kubernetes Operator (AKO). ## Expose metrics for Prometheus Expose the metrics for Prometheus with the [Aerospike Prometheus Exporter](https://github.com/aerospike/aerospike-prometheus-exporter) (APE), which runs as a sidecar container in the same Kubernetes pod as your Aerospike Database container. You can configure the APE to connect to your Aerospike Database cluster using either plaintext authentication or a Kubernetes secret. The process for using a secret is longer, but you should always enable security in a production environment. - [Plain text credentials](#tab-panel-3480) - [Kubernetes secret](#tab-panel-3481) For testing purposes, you can pass your credentials to the cluster in plain text. The following example modifies the `podSpec` section of your Aerospike cluster’s CR file with a `sidecars` section. Replace `AS_AUTH_USER`, `AS_AUTH_PASSWORD`, and `AEROSPIKE_PROMETHEUS_EXPORTER_IMAGE` with values for your deployment. Use a tag from the [Aerospike Prometheus Exporter image repository](https://hub.docker.com/r/aerospike/aerospike-prometheus-exporter) (for example, `aerospike/aerospike-prometheus-exporter:latest`). ```yaml spec: ... podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: AEROSPIKE_PROMETHEUS_EXPORTER_IMAGE env: # Replace with your credentials - name: "AS_AUTH_USER" value: "exporter" - name: "AS_AUTH_PASSWORD" value: "exporter123" - name: "AS_AUTH_MODE" value: "internal" ports: - containerPort: 9145 name: exporter ``` A Kubernetes secret securely passes the location of a credentials file to the Aerospike Prometheus Exporter without revealing it in your Aerospike Database cluster CR file. 1. Download the `ape.toml` configuration file from the official Aerospike Prometheus Exporter repository: [ape.toml](https://github.com/aerospike/aerospike-prometheus-exporter/blob/master/configs/ape.toml). 2. Modify `ape.toml` to include file-based security credentials. ```toml [Aerospike] ... # database user user = "admin" # database password password = "file:/var/secret/password" # authentication mode: internal (server authentication) [default], external (such as LDAP), pki. auth_mode = "internal" ... ``` 3. Create a Kubernetes [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the `ape.toml` file. A ConfigMap maps the configuration file in a way that a Kubernetes container, such as the APE sidecar container, can read it dynamically without it being included in the container itself. Terminal window ```bash kubectl create configmap sidecar-config --from-file=ape.toml ``` 4. Enable secure authentication for the Aerospike Prometheus Exporter sidecar using a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/). Use `kubectl` to create the secret from your credentials. Terminal window ```bash kubectl create secret generic auth-secret --from-literal=password=your_password ``` 5. Update the cluster CR file. To make `ape.toml` and the secret accessible in the APE container, add the following volumes to `spec.storage.volumes` in the AerospikeCluster CR: ```yaml spec: storage: volumes: - name: config-volume sidecars: - containerName: aerospike-prometheus-exporter path: /var/aerospike-prometheus-exporter source: configMap: name: sidecar-config - name: aerospike-auth-secret sidecars: - containerName: aerospike-prometheus-exporter path: /var/secret source: secret: secretName: auth-secret ``` 6. Configure the Prometheus Exporter sidecar. In the `aerospike-prometheus-exporter` sidecar container definition, specify the path to `ape.toml` as an argument. The configuration in the following example ensures that the `ape.toml` file is available at `/var/aerospike-prometheus-exporter/ape.toml`. The password is securely stored in `/var/secret/password` and is accessible to the APE. Example `spec.podSpec`. Replace `AEROSPIKE_PROMETHEUS_EXPORTER_IMAGE` with a tag from the [Aerospike Prometheus Exporter image repository](https://hub.docker.com/r/aerospike/aerospike-prometheus-exporter) (for example, `aerospike/aerospike-prometheus-exporter:latest`): ```yaml spec: podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: AEROSPIKE_PROMETHEUS_EXPORTER_IMAGE args: - -config=/var/aerospike-prometheus-exporter/ape.toml ports: - containerPort: 9145 name: exporter ``` ## Scrape metrics with Prometheus To collect metrics from the APE, configure Prometheus to scrape the `/metrics` endpoint exposed by the exporter. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack. ### Scrape with the Prometheus Operator 1. Follow the directions in GitHub at [Install Prometheus Operator](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install the `kube-prometheus-stack`, which includes the Prometheus Operator and Grafana. The [Prometheus Operator](https://github.com/prometheus-operator/kube-prometheus) uses a PodMonitor resource to scrape the exporter endpoints. 2. Create a file named `pod-monitor.yaml` with the following content. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: aerospike-cluster-pod-monitor namespace: aerospike labels: release: prometheus-operator spec: selector: matchLabels: app: aerospike-cluster namespaceSelector: matchNames: - default - aerospike podMetricsEndpoints: - port: exporter path: /metrics interval: 30s ``` 3. Apply the PodMonitor resource. Terminal window ```bash kubectl apply -f pod-monitor.yaml ``` ### Scrape with the AKO monitoring stack The [AKO repository](https://github.com/aerospike/aerospike-kubernetes-operator) includes monitoring configurations in the `config/` directory. Apply them with `kubectl`. 1. Apply the monitoring stack: Run this command from the root of a local checkout of the `aerospike-kubernetes-operator` repository. Terminal window ```bash kubectl apply -k config/monitoring ``` 2. To configure alerts, create Prometheus rule YAML files in the `aerospike-kubernetes-operator/config/monitoring/prometheus/config/alert-rules` directory. Aerospike provides predefined Prometheus alert rules in the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/prometheus). ### Grafana dashboards To visualize the metrics, import pre-built Grafana dashboards from the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/grafana/dashboards) or from [Grafana Labs](https://grafana.com/orgs/aerospike/dashboards).