Skip to main content
Loading
Version: Operator 3.3.1

Monitoring Aerospike Kubernetes Operator

The Aerospike Kubernetes Operator (AKO) runs a metrics server along with the controller inside the same pod. All the metrics are exposed on the /metrics endpoint of the pod. This endpoint is protected behind a kube-rbac-proxy, so any monitoring service like Prometheus must have the required permissions to scrape the metrics.

Exporting Metrics for Prometheus:

We recommend installing Prometheus in Kubernetes with Prometheus Operator.

For Prometheus Operator

  1. Verify that the Prometheus deployment installed via Prometheus Operator has the required RBAC for the /metrics endpoint.
  2. If not, add the following configuration lines to your YAML file to create a ClusterRole and attach it to the required Prometheus ServiceAccount using ClusterRoleBinding.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metrics-reader
rules:
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
  1. Create a ServiceMonitor resource so that Prometheus scrapes /metrics endpoint. A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource. In this example, the configurations are in a file called service-monitor.yaml.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
control-plane: controller-manager
name: aerospike-operator-controller-manager-metrics-monitor
spec:
endpoints:
- path: /metrics
interval: 15s
port: https
scheme: https
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
control-plane: controller-manager

The label selector.matchLabels points to the aerospike-operator-controller-manager-metrics-service service which exposes AKO metrics. This service gets created as part of AKO installation.

note

Verify that the Prometheus deployment is configured to select AKO ServiceMonitor using serviceMonitorSelector.

kubectl -n <Prometheus-namespace> get prometheus <Prometheus-cr-name> -o yaml

Use kubectl to apply the resource definition and create a ServiceMonitor resource.

kubectl apply -f service-monitor.yaml -n <AKO-namespace>

Once ServiceMonitor is created, Prometheus Operator updates the required scraping configurations for the Prometheus deployment. Prometheus starts scraping AKO metrics every 15 seconds. These metrics are displayed on the Prometheus Dashboard.

For the list of metrics exposed by AKO, see Metrics

For Non-Operator based Prometheus

If Prometheus is deployed using direct manifest files or a Helm chart, update its scraping configuration in the Prometheus configuration file, prometheus.yml

Add the following scrape job definition in the scrape_configs section of prometheus.yml for Prometheus to start scraping AKO metrics.

  - job_name: "aerospike-kubernetes-operator"
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
separator: ;
regex: (controller-manager);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: https
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints