# Monitor Aerospike clusters on Kubernetes This page describes how to use Prometheus and Grafana to monitor Aerospike clusters. To monitor Aerospike clusters, expose the Aerospike Database metrics and make them readable by Prometheus. Use the [Aerospike Monitoring Stack](https://aerospike.com/docs/database/observe/monitor/components) or the Prometheus Operator to monitor and set alerts for Aerospike clusters deployed by Aerospike Kubernetes Operator (AKO). ## Expose metrics for Prometheus Expose the metrics for Prometheus with the [Aerospike Prometheus Exporter](https://github.com/aerospike/aerospike-prometheus-exporter) (APE), which runs as a sidecar container in the same Kubernetes pod as your Aerospike Database container. You can configure the APE to connect to your Aerospike Database cluster using either plaintext authentication or a Kubernetes secret. The process for using a secret is longer, but you should always enable security in a production environment. - [Plain text credentials](#tab-panel-3069) - [Kubernetes secret](#tab-panel-3070) For testing purposes, you can pass your credentials to the cluster in plain text. The following example modifies the `podSpec` section of your Aerospike cluster’s Custom Resource (CR) file with a `sidecars` section. Replace the values for `AS_AUTH_USER` and `AS_AUTH_PASSWORD` with your actual credentials. ```yaml spec: ... podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: aerospike/aerospike-prometheus-exporter:latest env: # Replace with your credentials - name: "AS_AUTH_USER" value: "exporter" - name: "AS_AUTH_PASSWORD" value: "exporter123" - name: "AS_AUTH_MODE" value: "internal" ports: - containerPort: 9145 name: exporter ``` A Kubernetes secret securely passes the location of a credentials file to the Aerospike Prometheus Exporter without revealing it in your Aerospike Database cluster Custom Resource (CR) file. 1. Download the `ape.toml` configuration file from the official Aerospike Prometheus Exporter repository: [ape.toml](https://github.com/aerospike/aerospike-prometheus-exporter/blob/master/configs/ape.toml). 2. Modify `ape.toml` to include file-based security credentials. ```toml [Aerospike] ... # database user user = "admin" # database password password = "file:/var/secret/password" # authentication mode: internal (server authentication) [default], external (such as LDAP), pki. auth_mode = "internal" ... ``` 3. Create a Kubernetes [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the `ape.toml` file. A ConfigMap maps the configuration file in a way that a Kubernetes container, such as the APE sidecar container, can read it dynamically without it being included in the container itself. Terminal window ```bash kubectl create configmap sidecar-config --from-file=ape.toml ``` 4. Enable secure authentication for the Aerospike Prometheus Exporter sidecar using a [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/). Use `kubectl` to create the secret from your credentials. Terminal window ```bash kubectl create secret generic auth-secret --from-literal=password=your_password ``` 5. Update the cluster Custom Resource (CR) specification. To make `ape.toml` and the secret accessible in the APE container, add the following volume mounts in the Cluster CR specification: Volume for the APE ConfigMap: ```plaintext - name: config-volume sidecars: - containerName: aerospike-prometheus-exporter path: /var/aerospike-prometheus-exporter source: configMap: name: sidecar-config ``` Volume for the authentication secret: ```plaintext - name: aerospike-auth-secret sidecars: - containerName: aerospike-prometheus-exporter path: /var/secret source: secret: secretName: auth-secret ``` 6. Configure the Prometheus Exporter sidecar. In the `aerospike-prometheus-exporter` sidecar container definition, specify the path to `ape.toml` as an argument. The configuration in the following example ensures that the `ape.toml` file is available at `/var/aerospike-prometheus-exporter/ape.toml`. The password is securely stored in `/var/secret/password` and is accessible to the APE. Example `podSpec`: ```plaintext podSpec: multiPodPerHost: true sidecars: - name: aerospike-prometheus-exporter image: aerospike/aerospike-prometheus-exporter:latest args: - -config=/var/aerospike-prometheus-exporter/ape.toml ports: - containerPort: 9145 name: exporter ``` ## Scrape metrics with Prometheus To collect metrics from the APE, configure Prometheus to scrape the `/metrics` endpoint exposed by the exporter. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack. ### Scrape with the Prometheus Operator 1. Follow the directions in GitHub at [Install Prometheus Operator](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install the `kube-prometheus-stack`, which includes the Prometheus Operator and Grafana. The [Prometheus Operator](https://github.com/prometheus-operator/kube-prometheus) uses a PodMonitor resource to scrape the exporter endpoints. 2. Create a file named `pod-monitor.yaml` with the following content. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: aerospike-cluster-pod-monitor namespace: aerospike labels: release: prometheus-operator spec: selector: matchLabels: app: aerospike-cluster namespaceSelector: matchNames: - default - aerospike podMetricsEndpoints: - port: exporter path: /metrics interval: 30s ``` 3. Apply the PodMonitor resource. Terminal window ```bash kubectl apply -f pod-monitor.yaml ``` ### Scrape with the AKO monitoring stack The [AKO repository](https://github.com/aerospike/aerospike-kubernetes-operator), includes monitoring configurations in the `config/` directory. Apply them with `kubectl`. 1. Apply the monitoring stack: Terminal window ```bash kubectl apply -k config/monitoring ``` 2. To configure alerts, create Prometheus rule YAML files in the `aerospike-kubernetes-operator/config/monitoring/prometheus/config/alert-rules` directory. Aerospike provides predefined Prometheus alert rules in the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/prometheus). ### Grafana dashboards To visualize the metrics, import pre-built Grafana dashboards from the [Aerospike Monitoring GitHub repository](https://github.com/aerospike/aerospike-monitoring/tree/master/config/grafana/dashboards) or from [Grafana Labs](https://grafana.com/orgs/aerospike/dashboards).