Monitoring Aerospike Kubernetes Operator

The Aerospike Kubernetes Operator (AKO) runs a metrics server along with the controller inside the same pod. All the metrics are exposed on the /metrics endpoint of the pod. This endpoint is protected behind a kube-rbac-proxy, so any monitoring service like Prometheus must have the required permissions to scrape the metrics.

Exporting Metrics for Prometheus:

We recommend installing Prometheus in Kubernetes with Prometheus Operator.

For Prometheus Operator

1. Verify that the Prometheus deployment installed via Prometheus Operator has the required RBAC for the /metrics endpoint.
2. If not, add the following configuration lines to your YAML file to create a ClusterRole and attach it to the required Prometheus ServiceAccount using ClusterRoleBinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

3. Create a ServiceMonitor resource so that Prometheus scrapes /metrics endpoint. A ServiceMonitor defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource. In this example, the configurations are in a file called service-monitor.yaml.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: controller-manager
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true
  selector:
    matchLabels:
      control-plane: controller-manager

The label selector.matchLabels points to the aerospike-operator-controller-manager-metrics-service service which exposes AKO metrics. This service gets created as part of AKO installation.

note

Verify that the Prometheus deployment is configured to select AKO ServiceMonitor using serviceMonitorSelector.

kubectl -n <Prometheus-namespace> get prometheus <Prometheus-cr-name> -o yaml

Use kubectl to apply the resource definition and create a ServiceMonitor resource.

kubectl apply -f service-monitor.yaml -n <AKO-namespace>

Once ServiceMonitor is created, Prometheus Operator updates the required scraping configurations for the Prometheus deployment. Prometheus starts scraping AKO metrics every 15 seconds. These metrics are displayed on the Prometheus Dashboard.

For the list of metrics exposed by AKO, see Metrics

For Non-Operator based Prometheus

If Prometheus is deployed using direct manifest files or a Helm chart, update its scraping configuration in the Prometheus configuration file, prometheus.yml

Add the following scrape job definition in the scrape_configs section of prometheus.yml for Prometheus to start scraping AKO metrics.

  - job_name: "aerospike-kubernetes-operator"
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: https
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
        separator: ;
        regex: (controller-manager);true
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        separator: ;
        regex: https
        replacement: $1
        action: keep
    kubernetes_sd_configs:
      - role: endpoints