Monitoring Aerospike Kubernetes Operator
The Aerospike Kubernetes Operator (AKO) runs a metrics server along with the controller inside the same pod.
All the metrics are exposed on the /metrics
endpoint of the pod.
This endpoint is protected behind a kube-rbac-proxy
, so any monitoring service like Prometheus must have the required permissions to scrape the metrics.
Exporting Metrics for Prometheus:
We recommend installing Prometheus in Kubernetes with Prometheus Operator.
For Prometheus Operator
- Verify that the Prometheus deployment installed via Prometheus Operator has the required RBAC for the
/metrics
endpoint. - If not, add the following configuration lines to your YAML file to create a ClusterRole and attach it to the required Prometheus ServiceAccount using ClusterRoleBinding.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metrics-reader
rules:
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
- Create a
ServiceMonitor
resource so that Prometheus scrapes/metrics
endpoint. AServiceMonitor
defines a set of targets for Prometheus to monitor and scrape. Prometheus Operator abstracts away the implementation details of configuring Kubernetes service discovery and scrapes targets using this ServiceMonitor resource. In this example, the configurations are in a file calledservice-monitor.yaml
.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
control-plane: controller-manager
name: aerospike-operator-controller-manager-metrics-monitor
spec:
endpoints:
- path: /metrics
interval: 15s
port: https
scheme: https
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
control-plane: controller-manager
The label selector.matchLabels
points to the aerospike-operator-controller-manager-metrics-service
service, which exposes AKO metrics.
This service is created as part of AKO installation.
Verify that the Prometheus deployment is configured to select AKO ServiceMonitor using serviceMonitorSelector
.
kubectl -n <Prometheus-namespace> get prometheus <Prometheus-cr-name> -o yaml
use kubectl
to apply the resource definition and create a ServiceMonitor resource.
kubectl apply -f service-monitor.yaml -n <AKO-namespace>
Once ServiceMonitor is created, Prometheus Operator updates the required scraping configurations for the Prometheus deployment. Prometheus scrapes AKO metrics every 15 seconds. These metrics are displayed on the Prometheus Dashboard.
For the list of metrics exposed by AKO, see Metrics
For Non-Operator based Prometheus
If Prometheus is deployed using direct manifest files or a Helm chart, update its scraping configuration in the Prometheus configuration file, prometheus.yml
Add the following scrape job definition in the scrape_configs
section of prometheus.yml
for Prometheus to start scraping AKO metrics.
- job_name: "aerospike-kubernetes-operator"
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_control_plane, __meta_kubernetes_service_labelpresent_control_plane]
separator: ;
regex: (controller-manager);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: https
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints