Monitor Aerospike Backup Service

Aerospike Backup Service (ABS) exposes Prometheus metrics on its listen port (HTTP or HTTPS). Monitoring ABS means combining four signals: metrics for history and alerts, REST job status for the live job, logs for skip and failure reasons, and /health plus /ready for process liveness.

This page is the SRE chapter of the ABS manual. For how to run a first Grafana walkthrough locally, see Quickstart.

Overview

Signal	What it answers	Where
Prometheus metrics	Did backups and restores succeed over time? How long did they take?	/metrics on the listen port
REST job status	What is running right now?	GET /v1/backups/currentBackup/{routine}, GET /v1/restore/status/{jobId}
Logs	Why was a job skipped, canceled, or failed?	journalctl, container logs
Probes	Is the ABS process up?	/health, /ready

/health and /ready return HTTP 200 if the process is up. They do not verify storage or the Database cluster.

By default, metrics are available at http://<ABS_HOST>:8080/metrics. Change the port and API prefix with service.http.port and service.http.context-path. If only HTTPS is enabled, scrape https://<ABS_HOST>:8443/metrics (or the port in service.https.port). See service.

What to monitor first

Start with these golden signals. Add more after the first alerts are quiet.

Question	Metric or probe	Alert first?
Is a scheduled full backup overdue?	aerospike_backup_service_last_successful_backup_timestamp{type="full"}	Yes
Did a backup fail?	aerospike_backup_service_backup_events_total{outcome="failure"}	Yes
Did a restore fail?	aerospike_backup_service_restore_events_total{outcome="failure"}	Yes
Are jobs being skipped in a storm?	aerospike_backup_service_backup_events_total{outcome="skip"}	Optional
Is a job stuck in progress?	aerospike_backup_service_backup_in_progress, aerospike_backup_service_backup_progress_pct	After duration budget
How long do finished jobs take?	aerospike_backup_service_backup_duration_seconds	Capacity planning
Is a restore running?	aerospike_backup_service_restore_in_progress	Dashboard
Is the host saturated?	process and Go runtime metrics	Capacity, not correctness

The canonical progress metric is aerospike_backup_service_backup_progress_pct (gauge). pipeline is a REST currentBackup field, not a Prometheus metric. See Performance tuning.

Scrape ABS with Prometheus

ABS exposes metrics directly on its listen port. You do not need a dedicated Prometheus exporter.

/etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'aerospike-backup-service'
    static_configs:
      - targets: ['abs-service:8080']

Replace abs-service:8080 with your ABS host and port. For HTTPS-only, set scheme: https on the scrape job and use the HTTPS port.

Note

If you also monitor Aerospike Database servers with Aerospike Prometheus Exporter, keep the ABS scrape configuration as a separate job_name. That avoids mixing series and makes alerts easier to manage.

Kubernetes probes

/health and /ready are process liveness only. They do not check storage or the Database cluster.

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

If HTTP is disabled, probe the HTTPS port and set scheme: HTTPS:

livenessProbe:
  httpGet:
    path: /health
    port: 8443
    scheme: HTTPS

A healthy process returns HTTP 200 and the body Ok.

Grafana dashboard

A pre-built Grafana dashboard (ID 21375) visualizes ABS metrics.

Typical panels:

• Backup success, failure, skip, retry, and cancel event rates
• Backup duration for full and incremental jobs
• Backup progress percentage while a job runs
• Restore events and restore-in-progress
• Time since last successful backup

Import dashboard 21375, point it at the Prometheus data source that scrapes ABS, and set the time range to cover recent jobs. For a local walkthrough, see Quickstart.

Backup and restore job model

Backup counters use labels routine, type (full or incremental), and outcome. Restore counters use outcome. Histograms and duration series record finished runs only.

Outcome	Meaning
success	The job finished. For incremental backups, this includes runs that found no changed records.
failure	The job started and failed.
canceled	A user canceled the job, or the process shut down during the job.
retry	A retryable attempt was recorded.
skip	ABS did not start the job. No backup artifacts were written. Duration is not recorded.

Backup cancellation outcomes

aerospike_backup_service_backup_events_total{outcome="canceled"} increases when a backup is canceled.

• If a user explicitly cancels a backup (see Cancel all jobs for a backup routine), this increase is expected. Disabling a routine stops future scheduled runs but does not cancel a backup that is already running.
• If the service shuts down gracefully during a running backup, logs can report the backup as canceled, but Prometheus may miss the final increment if it cannot scrape before shutdown completes.

Backup skip outcomes

Under some conditions, ABS skips a scheduled or triggered backup.

Common reasons include:

• A full backup of the same routine is already running.
• An incremental backup of the same routine is already running and concurrent-incremental is false.
• A full backup is running and concurrent-incremental is false.
• A full backup is scheduled at the same time as the incremental backup and concurrent-incremental is false.
• No full backup has completed successfully yet for the routine.

ABS increments aerospike_backup_service_backup_events_total{outcome="skip"} when a scheduled or ad-hoc backup is skipped. The backup does not run, no backup artifacts are written, and aerospike_backup_service_backup_duration_seconds does not record a duration for that attempt.

ABS logs a message such as full backup skipped or incremental backup skipped with an error field describing the reason.

Skipped attempts are not queued. The next scheduled backup run tries again. See What happens when a backup doesn’t finish before another starts?.

Note

A skip is not the same as a successful incremental backup that finds no changed data. Skipped backups never start querying the cluster.

Incremental backups with no data changes

When an incremental backup runs but no records changed since the previous backup, ABS still completes successfully. Prometheus records aerospike_backup_service_backup_events_total{outcome="success"} for the run.

Because no records were backed up for a namespace, ABS does not write metadata.yaml for that namespace for that run. In a routine with multiple namespaces, ABS still writes metadata.yaml for any namespace that did have changes. The run does not create a new incremental backup artifact in the backup destination for namespaces with no changes.

Use aerospike_backup_service_backup_events_total{outcome="success"} to confirm that scheduled incremental jobs executed. Use aerospike_backup_service_last_successful_backup_timestamp{type="full"} or storage listings to track the backup baseline when incremental runs are frequently empty.

Application metrics

Name	Type	Labels	When the series moves
aerospike_backup_service_backup_duration_seconds	histogram	routine, type	A backup finishes (success, failure, or cancel that records duration)
aerospike_backup_service_backup_events_total	counter	routine, type, outcome	A job succeeds, fails, is canceled, retries, or is skipped
aerospike_backup_service_backup_in_progress	gauge	routine, type	A backup is running
aerospike_backup_service_backup_progress_pct	gauge	routine, type	Progress of a running backup, in percent
aerospike_backup_service_last_successful_backup_timestamp	gauge	routine, type	Unix timestamp of the last successful backup
aerospike_backup_service_restore_events_total	counter	outcome	A restore job finishes (success, failure, canceled)
aerospike_backup_service_restore_in_progress	gauge	(none)	A restore is running

PromQL by question

Prefer increase() or rate() for counters. A bare sum(..._total) grows forever and is a weak dashboard query.

Is a full backup overdue?

time() - aerospike_backup_service_last_successful_backup_timestamp{routine="daily-ns1", type="full"}

How many backup failures in the last 15 minutes?

increase(aerospike_backup_service_backup_events_total{outcome="failure"}[15m])

What is the skip rate?

sum by (routine, type) (rate(aerospike_backup_service_backup_events_total{outcome="skip"}[15m]))

Are retries increasing?

sum by (routine) (increase(aerospike_backup_service_backup_events_total{outcome="retry"}[15m]))

Is a backup stuck in progress?

aerospike_backup_service_backup_in_progress == 1

Pair with aerospike_backup_service_backup_progress_pct and job duration from logs or currentBackup.

Did restores fail?

increase(aerospike_backup_service_restore_events_total{outcome="failure"}[15m])

Average backup duration (finished jobs):

rate(aerospike_backup_service_backup_duration_seconds_sum[5m])
/
rate(aerospike_backup_service_backup_duration_seconds_count[5m])

Empty incremental successes may not advance the incremental timestamp series. See Incremental backups with no data changes.

Example alerts

• Detect backup job failures recorded within the last 15 minutes.

  - alert: BackupJobFailureDetected
    expr: increase(aerospike_backup_service_backup_events_total{outcome="failure"}[15m]) > 0
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "Backup job failure detected"
      description: "A backup failure was detected in the last 15 minutes for routine {{ $labels.routine }}."

• Detect a restore failure.

  - alert: RestoreJobFailureDetected
    expr: increase(aerospike_backup_service_restore_events_total{outcome="failure"}[15m]) > 0
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "Restore job failure detected"
      description: "A restore failure was detected in the last 15 minutes."

• Detect a full backup that is older than 24 hours. Filter to type="full" so empty incrementals do not hide a missing full backup.

  - alert: BackupTooOld
    expr: time() - aerospike_backup_service_last_successful_backup_timestamp{routine="daily-ns1", type="full"} > 86400
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "Full backup is older than 24 hours"
      description: "The last successful full backup for routine daily-ns1 was more than 24 hours ago."

• Optional: skip storm (many skips, which often means overlap or a missing first full backup).

  - alert: BackupSkipStorm
    expr: sum by (routine) (increase(aerospike_backup_service_backup_events_total{outcome="skip"}[15m])) > 5
    for: 0m
    labels:
      severity: warning
    annotations:
      summary: "Backup skip storm"
      description: "Routine {{ $labels.routine }} skipped more than five backups in 15 minutes."

Investigate with logs and the REST API

When a backup is missed, stuck, or failed, use metrics first, then logs, then REST.

1. Confirm the outcome in Prometheus (failure, skip, canceled, or a stale last_successful_backup_timestamp).

2. Read ABS logs for backup skipped, query errors, or storage errors:

   journalctl -u aerospike-backup-service -n 200 --no-pager

   In Docker: docker logs aerospike-backup-service.

   ABS does not log HTTP request bodies. Literal secrets are redacted in structured logs. Use metrics and the job APIs for forensics. See Secure ABS.

3. Check the live job:

   curl -s BASE_URL/v1/backups/currentBackup/ROUTINE_NAME
   curl -s BASE_URL/v1/restore/status/JOB_ID

For request and response bodies, see API usage examples.

Process and Go runtime metrics

ABS also exposes standard process and Go runtime metrics on /metrics. Use them to detect resource saturation, not backup correctness.

Metric	Description	What to watch
process_cpu_seconds_total	Cumulative CPU time (seconds) across all cores	rate(process_cpu_seconds_total[5m]) * 100 is CPU percent per core. Sustained values above your core budget mean overlap or undersized hosts.
process_resident_memory_bytes	Resident set size (RSS) in bytes	Keep below container memory limits. Watch for growth during large backup or restore windows.
process_open_fds	Open file descriptors	Compare with process_max_fds. A high ratio can cause "too many open files".
go_goroutines	Active goroutines	A monotonic increase usually indicates stalled tasks or leaked goroutines.
go_memstats_heap_alloc_bytes	Go heap bytes for live objects	Compare with RSS to separate Go heap from non-heap pressure.

Example queries:

• rate(process_cpu_seconds_total[5m]) * 100: 100 means one full CPU core. Compare spikes with aerospike_backup_service_backup_progress_pct to see which routine is running.
• process_resident_memory_bytes / 1024 / 1024 / 1024: RSS in GiB. Alert around 80% of the memory limit.

For collector details, see the Prometheus Go collectors documentation.

HTTP endpoints

Name	Description
/metrics	Prometheus metrics.
/health	Process liveness. HTTP 200 and Ok if the process is up. Does not verify storage or the Database cluster.
/ready	Process readiness. HTTP 200 if the process can handle requests. Does not verify storage or the Database cluster.
/version	Application version, commit hash, and build time.
/api-docs	Swagger UI for the REST API.

See the Kubernetes probe documentation and Prometheus documentation.

Deprecated metrics

The following metrics are deprecated as of ABS 3.2.0. Use aerospike_backup_service_backup_events_total, aerospike_backup_service_backup_duration_seconds, and aerospike_backup_service_last_successful_backup_timestamp instead.

Click to show deprecated metrics

Name	Description	Replacement
aerospike_backup_service_runs_total	Successful backup runs counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_incremental_runs_total	Successful incremental backup runs counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_skip_total	Full backup skip counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_incremental_skip_total	Incremental backup skip counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_failure_total	Full backup failure counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_incremental_failure_total	Incremental backup failure counter	aerospike_backup_service_backup_events_total
aerospike_backup_service_duration_millis	Full backup duration in milliseconds	aerospike_backup_service_backup_duration_seconds
aerospike_backup_service_incremental_duration_millis	Incremental backup duration in milliseconds	aerospike_backup_service_backup_duration_seconds