ABS Monitoring

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

Aerospike Backup Service (ABS) exposes system metrics that Prometheus can scrape.

Prometheus configuration

ABS exposes metrics directly on its HTTP port, so you don’t need a dedicated Prometheus exporter. By default, metrics are available at http://<ABS_HOST>:8080/metrics. You can change the port with the service.http.port parameter.

The following example shows a standalone Prometheus configuration for scraping ABS metrics:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'aerospike-backup-service'
    static_configs:
      - targets: ['abs-service:8080']

Replace abs-service:8080 with your ABS host and port.

Grafana dashboard

A pre-built Grafana dashboard is available for visualizing ABS metrics. The dashboard includes panels for backup success and failure rates, backup duration, and restore operations.

Metrics

ABS includes the following application metrics:

Name	Description	Labels
`aerospike_backup_service_runs_total`	Successful full backup runs counter
`aerospike_backup_service_incremental_runs_total`	Successful incremental backup runs counter
`aerospike_backup_service_skip_total`	Full backup skip counter
`aerospike_backup_service_incremental_skip_total`	Incremental backup skip counter
`aerospike_backup_service_failure_total`	Full backup failure counter
`aerospike_backup_service_incremental_failure_total`	Incremental backup failure counter
`aerospike_backup_service_duration_millis`	Full backup duration in milliseconds
`aerospike_backup_service_incremental_duration_millis`	Incremental backup duration in milliseconds
`aerospike_backup_service_backup_progress_pct`	Progress of backup processes in percentage	routine, type
`aerospike_backup_service_restore_progress_pct`	Progress of restore processes in percentage	label

Backup skip outcomes

Under some conditions, ABS skips performing a scheduled or triggered backup.

Common reasons include:

A full backup of the same routine is already running.
An incremental backup of the same routine is already running.
A full backup is running, so ABS skips the overlapping incremental backup.
A full backup is scheduled at the same time as the incremental backup.
No full backup has completed successfully yet for the routine.

ABS increments aerospike_backup_service_skip_total or aerospike_backup_service_incremental_skip_total when a scheduled or ad-hoc backup is skipped. The backup does not run, and no backup artifacts are written to the backup destination.

ABS logs a message such as full backup skipped or incremental backup skipped with an error field describing the reason.

Skipped attempts are not queued. The next scheduled backup run tries again. See What happens when a backup doesn’t finish before another starts? for scheduling overlap rules.

Incremental backups with no data changes

When an incremental backup runs but no records changed since the previous backup, ABS still completes successfully. Prometheus increments aerospike_backup_service_incremental_runs_total for the run.

Because no records were backed up for a namespace, ABS does not write metadata.yaml for that namespace for that run. In a routine with multiple namespaces, ABS still writes metadata.yaml for any namespace that did have changes. The run does not create a new incremental backup artifact in the backup destination for namespaces with no changes.

Use aerospike_backup_service_incremental_runs_total to confirm that scheduled incremental jobs executed. Use storage listings to track the backup baseline when incremental runs are frequently empty.

Example PromQL queries

Monitor and alert on backup performance with the following queries in Grafana panels or the Prometheus expression browser.

Number of successful full backups:
Terminal window
```
aerospike_backup_service_runs_total
```
Number of successful incremental backups:
Terminal window
```
aerospike_backup_service_incremental_runs_total
```
Number of failed full backups:
Terminal window
```
aerospike_backup_service_failure_total
```
Number of skipped full backup attempts:
Terminal window
```
aerospike_backup_service_skip_total
```
Full backup duration in milliseconds:
Terminal window
```
aerospike_backup_service_duration_millis
```
Backup progress for a running job:
Terminal window
```
aerospike_backup_service_backup_progress_pct
```

Example Prometheus alerts

Integrate ABS metrics into your Prometheus alerting pipeline to stay informed of job failures or service latencies.

Detect backup job failures recorded within the last 15 minutes with this alert.

- alert: BackupJobFailureDetected
expr: increase(aerospike_backup_service_failure_total[15m]) > 0 or increase(aerospike_backup_service_incremental_failure_total[15m]) > 0
for: 0m
labels:
  severity: warning
annotations:
  summary: "Backup job failure detected"
  description: "A backup failure was detected in the last 15 minutes."

Process and Go runtime metrics

ABS also exposes standard process and Go runtime metrics on /metrics. Use them to detect resource saturation and runtime behavior changes before backup failures occur.

Metric	Description	What to watch
`process_cpu_seconds_total`	Cumulative CPU time (seconds) across all cores.	`rate(process_cpu_seconds_total[5m]) * 100` gives CPU percent per core. A sustained value above 100 means ABS uses more than one core on average.
`process_resident_memory_bytes`	Resident set size (RSS) in bytes.	Keep below container memory limits and watch for sustained growth during large backup or restore windows.
`process_open_fds`	Open file descriptors held by ABS.	Compare with `process_max_fds`; a sustained high ratio indicates descriptor pressure and possible `"too many open files"` errors.
`process_max_fds`	Hard limit for open file descriptors.	Track `process_open_fds / process_max_fds` and alert when the ratio approaches 1.0 for multiple scrape intervals.
`go_goroutines`	Active Go goroutines.	In a stable workload the count stays bounded. A monotonic increase usually indicates stalled background tasks or leaked goroutines.
`go_memstats_heap_alloc_bytes`	Go heap bytes allocated for live objects.	Compare with `process_resident_memory_bytes` to separate Go heap growth from non-heap memory pressure.

Prometheus query examples:

rate(process_cpu_seconds_total[5m]) * 100
- The process_cpu_seconds_total counter tracks total CPU time since the process started. This query turns that running total into a per-second average over the last 5 minutes, then multiplies by 100 to get a percentage. In this scale, 100 means one full CPU core is busy and 200 means two cores. If the result stays above your expected core budget for several minutes, compare the spike with aerospike_backup_service_backup_progress_pct to identify which routine is running. To reduce CPU usage, stagger routine schedules so fewer backups overlap, or increase the CPU resources allocated to the ABS instance.
process_resident_memory_bytes / 1024 / 1024 / 1024
- Converts the resident memory (RSS) of the ABS process from bytes to gibibytes, which is easier to compare against any memory limits. A good practice is to alert at roughly 80% of the memory limit. If memory keeps growing, increase the container or host memory limit, or reduce the number of backup routines that run concurrently.

For details about these collectors in the Prometheus Go client, see the collectors package documentation.

Endpoints

Name	Description
`/metrics`	Exposes metrics for Prometheus to check performance of the backup service.
`/health`	Allows monitoring systems to check the service health.
`/ready`	Checks whether the service is able to handle requests.
`/version`	Returns the application version.
`/api-docs`	Serves the API documentation in Swagger UI format.

See the official Kubernetes documentation and Prometheus documentation for more information.