--- title: "ABS Monitoring" description: "Monitor Aerospike Backup Service (ABS) using Prometheus metrics, Grafana dashboards, PromQL queries, and alerts." --- # ABS Monitoring > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. Aerospike Backup Service (ABS) exposes system metrics that [Prometheus](https://prometheus.io/) can scrape. ## Prometheus configuration ABS exposes metrics directly on its HTTP port, so you don’t need a dedicated Prometheus exporter. By default, metrics are available at `http://:8080/metrics`. You can change the port with the [`service.http.port`](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/config#service.http.port) parameter. The following example shows a standalone Prometheus configuration for scraping ABS metrics: /etc/prometheus/prometheus.yml ```yaml global: scrape_interval: 15s scrape_configs: - job_name: 'aerospike-backup-service' static_configs: - targets: ['abs-service:8080'] ``` Replace `abs-service:8080` with your ABS host and port. ::: note If you also monitor Aerospike Database servers with [Aerospike Prometheus Exporter](https://aerospike.com/docs/database/observe/monitor/components/), keep the ABS scrape configuration as a separate `job_name` to avoid confusion and make it easier to manage alerts and dashboards for each service independently. ::: ## Grafana dashboard A pre-built [Grafana dashboard](https://grafana.com/grafana/dashboards/21375-aerospike-backup-service/) is available for visualizing ABS metrics. The dashboard includes panels for backup success and failure rates, backup duration, and restore operations. ## Metrics ABS includes the following application metrics: | Name | Description | Labels | | --- | --- | --- | | `aerospike_backup_service_runs_total` | Successful full backup runs counter | | | `aerospike_backup_service_incremental_runs_total` | Successful incremental backup runs counter | | | `aerospike_backup_service_skip_total` | Full backup skip counter | | | `aerospike_backup_service_incremental_skip_total` | Incremental backup skip counter | | | `aerospike_backup_service_failure_total` | Full backup failure counter | | | `aerospike_backup_service_incremental_failure_total` | Incremental backup failure counter | | | `aerospike_backup_service_duration_millis` | Full backup duration in milliseconds | | | `aerospike_backup_service_incremental_duration_millis` | Incremental backup duration in milliseconds | | | `aerospike_backup_service_backup_progress_pct` | Progress of backup processes in percentage | routine, type | | `aerospike_backup_service_restore_progress_pct` | Progress of restore processes in percentage | label | ### Backup skip outcomes Under some conditions, ABS skips performing a scheduled or triggered backup. Common reasons include: - A full backup of the same routine is already running. - An incremental backup of the same routine is already running and [`concurrent-incremental`](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/config#backup-policies.POLICY_NAME.concurrent-incremental) is `false`. - A full backup is running and [`concurrent-incremental`](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/config#backup-policies.POLICY_NAME.concurrent-incremental) is `false`. - A full backup is scheduled at the same time as the incremental backup and [`concurrent-incremental`](https://aerospike.com/docs/database/tools/backup-and-restore/backup-service/config#backup-policies.POLICY_NAME.concurrent-incremental) is `false`. - No full backup has completed successfully yet for the routine. ABS increments `aerospike_backup_service_skip_total` or `aerospike_backup_service_incremental_skip_total` when a scheduled or ad-hoc backup is skipped. The backup does not run, and no backup artifacts are written to the backup destination. ABS logs a message such as `full backup skipped` or `incremental backup skipped` with an `error` field describing the reason. Skipped attempts are not queued. The next scheduled backup run tries again. See [What happens when a backup doesn’t finish before another starts?](https://aerospike.com/docs/database/tools/backup-and-restore/overview#what-happens-when-a-backup-doesnt-finish-before-another-starts) for scheduling overlap rules. ::: note A skip is not the same as a successful incremental backup that finds no changed data. Skipped backups never start scanning the cluster. ::: ### Incremental backups with no data changes When an incremental backup runs but no records changed since the previous backup, ABS still completes successfully. Prometheus increments `aerospike_backup_service_incremental_runs_total` for the run. Because no records were backed up for a namespace, ABS does not write `metadata.yaml` for that namespace for that run. In a routine with multiple namespaces, ABS still writes `metadata.yaml` for any namespace that did have changes. The run does not create a new incremental backup artifact in the backup destination for namespaces with no changes. Use `aerospike_backup_service_incremental_runs_total` to confirm that scheduled incremental jobs executed. Use storage listings to track the backup baseline when incremental runs are frequently empty. ## Example PromQL queries Monitor and alert on backup performance with the following queries in Grafana panels or the Prometheus expression browser. - Number of successful full backups: Terminal window ```bash aerospike_backup_service_runs_total ``` - Number of successful incremental backups: Terminal window ```bash aerospike_backup_service_incremental_runs_total ``` - Number of failed full backups: Terminal window ```bash aerospike_backup_service_failure_total ``` - Number of skipped full backup attempts: Terminal window ```bash aerospike_backup_service_skip_total ``` - Full backup duration in milliseconds: Terminal window ```bash aerospike_backup_service_duration_millis ``` - Backup progress for a running job: Terminal window ```bash aerospike_backup_service_backup_progress_pct ``` ## Example Prometheus alerts Integrate ABS metrics into your Prometheus alerting pipeline to stay informed of job failures or service latencies. - Detect backup job failures recorded within the last 15 minutes with this alert. ```yaml - alert: BackupJobFailureDetected expr: increase(aerospike_backup_service_failure_total[15m]) > 0 or increase(aerospike_backup_service_incremental_failure_total[15m]) > 0 for: 0m labels: severity: warning annotations: summary: "Backup job failure detected" description: "A backup failure was detected in the last 15 minutes." ``` ## Process and Go runtime metrics ABS also exposes standard process and Go runtime metrics on `/metrics`. Use them to detect resource saturation and runtime behavior changes before backup failures occur. | Metric | Description | What to watch | | --- | --- | --- | | `process_cpu_seconds_total` | Cumulative CPU time (seconds) across all cores. | `rate(process_cpu_seconds_total[5m]) * 100` gives CPU percent per core. A sustained value above 100 means ABS uses more than one core on average. | | `process_resident_memory_bytes` | Resident set size (RSS) in bytes. | Keep below container memory limits and watch for sustained growth during large backup or restore windows. | | `process_open_fds` | Open file descriptors held by ABS. | Compare with `process_max_fds`; a sustained high ratio indicates descriptor pressure and possible `"too many open files"` errors. | | `process_max_fds` | Hard limit for open file descriptors. | Track `process_open_fds / process_max_fds` and alert when the ratio approaches 1.0 for multiple scrape intervals. | | `go_goroutines` | Active Go goroutines. | In a stable workload the count stays bounded. A monotonic increase usually indicates stalled background tasks or leaked goroutines. | | `go_memstats_heap_alloc_bytes` | Go heap bytes allocated for live objects. | Compare with `process_resident_memory_bytes` to separate Go heap growth from non-heap memory pressure. | Prometheus query examples: - `rate(process_cpu_seconds_total[5m]) * 100` - The `process_cpu_seconds_total` counter tracks total CPU time since the process started. This query turns that running total into a per-second average over the last 5 minutes, then multiplies by 100 to get a percentage. In this scale, 100 means one full CPU core is busy and 200 means two cores. If the result stays above your expected core budget for several minutes, compare the spike with `aerospike_backup_service_backup_progress_pct` to identify which routine is running. To reduce CPU usage, stagger routine schedules so fewer backups overlap, or increase the CPU resources allocated to the ABS instance. - `process_resident_memory_bytes / 1024 / 1024 / 1024` - Converts the resident memory (RSS) of the ABS process from bytes to gibibytes, which is easier to compare against any memory limits. A good practice is to alert at roughly 80% of the memory limit. If memory keeps growing, increase the container or host memory limit, or reduce the number of backup routines that run concurrently. For details about these collectors in the Prometheus Go client, see the [collectors package documentation](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/collectors). ## Endpoints | Name | Description | | --- | --- | | `/metrics` | Exposes metrics for Prometheus to check performance of the backup service. | | `/health` | Allows monitoring systems to check the service health. | | `/ready` | Checks whether the service is able to handle requests. | | `/version` | Returns the application version. | | `/api-docs` | Serves the API documentation in Swagger UI format. | See the official [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) and [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/getting_started/) for more information.