Monitoring Aerospike Kubernetes Operator

The Aerospike Kubernetes Operator (AKO) runs a metrics server along with the controller inside the same pod. All the metrics are exposed on /metrics endpoint of the pod. This endpoint is protected behind a kube-rbac-proxy, so any monitoring service like Prometheus must have the required RBAC to scrape the metrics.

Exporting Metrics for Prometheus:

We recommend installing Prometheus in Kubernetes with Prometheus Operator.

For Prometheus Operator

1. Verify that the Prometheus deployment installed via Prometheus Operator has the required RBAC for the /metrics endpoint. If not, use the following configuration to create ClusterRole and attach it to the required Prometheus ServiceAccount using ClusterRoleBinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

2. Create a ServiceMonitor resource so that Prometheus scrapes /metrics endpoint. A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: controller-manager
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true
  selector:
    matchLabels:
      control-plane: controller-manager

The label selector.matchLabels points to the aerospike-operator-controller-manager-metrics-service service which exposes AKO metrics. This service gets created as part of AKO installation.

note

Verify that the Prometheus deployment is configured to select AKO ServiceMonitor using serviceMonitorSelector.

Use kubectl to create ServiceMonitor resource.

kubectl apply -f service-monitor.yaml -n <AKO-namespace>

Once ServiceMonitor is created, Prometheus Operator updates required scrapping config for the Prometheus deployment and Prometheus starts scrapping AKO metrics every 15 seconds. These metrics are displayed on the Prometheus Dashboard.

For the list of metrics exposed by AKO, see Metrics

For Non-Operator based Prometheus

For Prometheus which is deployed using direct manifest files or helm chart, its scrapping configuration must be updated in the Prometheus configuration file, prometheus.yml

Add the following scrape job definition in the scrape_configs section of Prometheus configuration file for Prometheus to start scrapping AKO metrics.

  - job_name: "aerospike-kubernetes-operator"
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: https
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
        separator: ;
        regex: (controller-manager);true
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        separator: ;
        regex: https
        replacement: $1
        action: keep
    kubernetes_sd_configs:
      - role: endpoints