# Monitor Aerospike Kubernetes Operator This page describes how to monitor Aerospike Kubernetes Operator. ## Overview Aerospike Kubernetes Operator (AKO) exposes [metrics](https://book.kubebuilder.io/reference/metrics-reference.html) on the `/metrics` endpoint. This endpoint is by default secure (HTTPS) and requires TLS certificates for secure communication. If no TLS certificates are provided, self-signed certificates are generated. Prometheus must have the required permissions to scrape these metrics. ## Configure system to scrape AKO metrics To collect metrics, configure Prometheus to scrape the `/metrics` endpoint. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack. ### Using Prometheus Operator See [Install Prometheus Operator](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install the `kube-prometheus-stack`, which includes the Prometheus Operator and Grafana. You can create the ServiceMonitor resource using any one of the following methods based on your AKO installation mode: - [OLM](#tab-panel-1054) - [Helm](#tab-panel-1055) AKO uses a Kubernetes Service labeled `control-plane: controller-manager` to make its metrics available for scraping. The [Prometheus Operator](https://github.com/prometheus-operator/kube-prometheus) uses `ServiceMonitor` resources to learn which pods to scrape for endpoints and how they should be scraped. Create a `ServiceMonitor` resource to configure Prometheus to find this Kubernetes Service and monitor it at regular intervals to scrape the AKO `/metrics` endpoint. 1. Create a file named `service-monitor.yaml`. - **With custom TLS certificates (Recommended for production)** ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: control-plane: controller-manager name: aerospike-operator-controller-manager-metrics-monitor spec: endpoints: - path: /metrics interval: 15s # Scraping interval port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: serverName: aerospike-operator-controller-manager-metrics-service.operators.svc # Naming Format: ..svc insecureSkipVerify: false ca: secret: name: metrics-server-cert key: ca.crt cert: secret: name: metrics-server-cert key: tls.crt keySecret: name: metrics-server-cert key: tls.key selector: matchLabels: control-plane: controller-manager # AKO metric service label ``` - **With TLS certificate validation disabled (insecureSkipVerify: true) for testing environment** ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: control-plane: controller-manager name: aerospike-operator-controller-manager-metrics-monitor spec: endpoints: - path: /metrics interval: 15s # Scraping interval port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: insecureSkipVerify: true # Disables TLS certificate verification. Not recommended for production. selector: matchLabels: control-plane: controller-manager # AKO metric service label ``` AKO uses a Kubernetes Service labeled `chart: aerospike-kubernetes-operator` to make its metrics available for scraping. The [Prometheus Operator](https://github.com/prometheus-operator/kube-prometheus) uses `ServiceMonitor` resources to learn which pods to scrape for endpoints and how they should be scraped. Create a `ServiceMonitor` resource to configure Prometheus to find this Kubernetes Service and monitor it at regular intervals to scrape the AKO `/metrics` endpoint. 1. Create a file named `service-monitor.yaml`. - **With custom TLS certificates (Recommended for production)** ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: chart: aerospike-kubernetes-operator name: aerospike-operator-controller-manager-metrics-monitor spec: endpoints: - path: /metrics interval: 15s # Scraping interval port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: serverName: aerospike-operator-controller-manager-metrics-service.aerospike.svc # Naming Format: ..svc insecureSkipVerify: false ca: secret: name: metrics-server-cert key: ca.crt cert: secret: name: metrics-server-cert key: tls.crt keySecret: name: metrics-server-cert key: tls.key selector: matchLabels: chart: aerospike-kubernetes-operator # AKO metric service label ``` - **With TLS certificate validation disabled (insecureSkipVerify: true) for testing environment** ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: chart: aerospike-kubernetes-operator name: aerospike-operator-controller-manager-metrics-monitor spec: endpoints: - path: /metrics interval: 15s # Scraping interval port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: insecureSkipVerify: true # Disables TLS certificate verification. Not recommended for production. selector: matchLabels: chart: aerospike-kubernetes-operator # AKO metric service label ``` 2. Verify that the Prometheus deployment is configured to select the AKO `ServiceMonitor` using `serviceMonitorSelector`. Ensure that the `serviceMonitorSelector` includes the labels used in the `ServiceMonitor`. ```plaintext kubectl -n PROMETHEUS_NAMESPACE get prometheus PROMETHEUS_CR_NAME -o yaml ``` 3. Apply the `ServiceMonitor` resource: ```plaintext kubectl apply -f service-monitor.yaml -n AKO_NAMESPACE ``` AKO exposes two types of Prometheus metrics: 1. **K8s controller runtime metrics** – These are the default metrics exposed by the controller-runtime framework. See the full list here: [Kubebuilder Metrics](https://book.kubebuilder.io/reference/metrics-reference.html). 2. **AKO custom metrics** – Additional metrics exposed by AKO to provide more insights. | Metric Name | Description | | --- | --- | | `aerospike_ako_aerospikecluster_phase` | Phase of AerospikeCluster CRs. (InProgress, Error or Completed) | ### Using Aerospike Kubernetes Operator Monitoring Stack If you have cloned the [AKO repository](https://github.com/aerospike/aerospike-kubernetes-operator), you can apply the monitoring configurations in one step with `kubectl`: ```plaintext kubectl apply -k config/monitoring ``` ### Grafana dashboards To visualize metrics exposed by AKO, import pre-built Grafana dashboards from [Grafana Dashboard](https://grafana.com/grafana/dashboards/23455-aerospike-kubernetes-operator-view/).