Prometheus Integration
Prometheus integration is enabled by default for all HTTP source (outbound) connectors, and is enabled for source connectors compatible with Aerospike 5.0 and later when the manage subsection in the service section is configured in the connector's configuration file. The REST endpoint to integrate with Prometheus is manage/rest/v1/prometheus.
For an HTTP source connector, the service port and manage port are the same.
Prometheus Configuration for an Aerospike Connector
The scrape_configs section of Prometheus configuration has to point to the Aerospike Connector's manage REST API endpoint.
...
scrape_configs:
- job_name: aerospike-connect
metrics_path: /manage/rest/v1/prometheus
static_configs:
targets: # List of Manage Endpoints of all Aerospike Connectors exposing Prometheus metrics.
- outbound-connector-1:8081
- outbound-connector-2:8081
...
Aerospike Connector's Metrics available in Prometheus
Connector exposes several metrics. We have only highlighted some of the important ones here.
Counters
A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
| Name | Label | Description | From Connector version |
|---|---|---|---|
| aerospike_kafka_outbound_connections | {status=active} | Count of active XDR connections to the outbound connector. | 4.0.0 |
| aerospike_kafka_outbound_connections | {status=closed} | Count of closed XDR connections to the outbound connector. | 4.0.0 |
| aerospike_kafka_outbound_requests | {status=error} | Count of requests that could not be written to the external Kafka system. | 3.0.0 |
| aerospike_kafka_outbound_requests | {status=parse-error} | Count of XDR records which could not be parsed. | 3.0.0 |
| aerospike_kafka_outbound_requests | {status=skipped} | Count of XDR records which were skipped and not written to the external Kafka system. | 3.0.0 |
| aerospike_kafka_outbound_requests | {status=success} | Count of records written to the external Kafka system. | 3.0.0 |
| aerospike_kafka_outbound_requests | {status=queued} | Current count of records being executed by the outbound connector. | 4.0.0 |
| aerospike_kafka_outbound_kafka_producers | {status=active} | Count of active Kafka producers. | 4.0.0 |
| aerospike_kafka_outbound_kafka_producers | {status=closed} | Count of closed Kafka producers. | 4.0.0 |
Gauge
A gauge always returns the current value of the metric and can increase or decrease.
| Name | Description | From Connector version |
|---|---|---|
| aerospike_kafka_outbound_requests_lag | The lag between update of a record in Aerospike and it reaching the Connector measured in milliseconds. Calculated as aerospike_record.lut - current_time. Its sampled every second based on the XDR record being processed at that time, it is approximate and not exact. | 4.0.0 |
Meter
Meters measure the rate of events in the connector. It provides the total count of the metric, mean value and one-minute, five-minute, and fifteen-minute moving average values.
| Name | Description | From Connector version |
|---|---|---|
| aerospike_kafka_outbound_ordering_error | Count of records rejected to maintain an ordering guarantee of records having the same Aerospike key. XDR retries such a records. | 5.0.0 |
Histograms
Histograms calculate the distribution of a value. It provides the total count of the metric, maximum value, minimum value, mean value, and values at various quantiles (99%, 95%, etc).
| Name | Description | From Connector version |
|---|---|---|
| aerospike_kafka_outbound_record_payload_size | Size of records received by the connector in Bytes. | 4.0.0 |
Timers
Timers time the processing of tasks. A timer metric aggregates timing durations and provides duration statistics, plus throughput statistics. It provides the total count of the metric, maximum time, minimum time, mean time, times at various quantiles (99%, 95%, etc), and mean throughput value and one-minute, five-minute, and fifteen-minute moving average throughput values.
| Name | Description | From Connector version |
|---|---|---|
| aerospike_kafka_outbound_record_ack_queue | Time spent by a processed XDR record in the ack-queue waiting to be acked to Aerospike XDR source. | 4.0.0 |
| aerospike_kafka_outbound_record_dispatch | Time spent dispatching an XDR record to the external Kafka system. | 4.0.0 |
| aerospike_kafka_outbound_record_parsing | Time spent parsing an XDR record. | 4.0.0 |
| aerospike_kafka_outbound_requests_total | Total time a XDR record has spent in the connector. | 3.0.0 |
Grafana Integration
Grafana dashboards can be integrated with Prometheus.
Grafana dashboards are packaged as JSON and can be imported.
The file path for the Aerospike Kafka source (outbound) connector is:
/opt/aerospike-kafka-outbound/grafana/aerospike-kafka-outbound-instances.json, /opt/aerospike-kafka-outbound/grafana/aerospike-kafka-outbound-summary.json
A screenshot of the Grafana dashboard querying Prometheus for Aerospike Outbound metrics data.
