Metrics

Secret Agent publishes metrics to a Prometheus endpoint and to the log ticker.

Prometheus configuration

To expose a Prometheus metrics endpoint, configure the HTTP or HTTPS service in the Secret Agent configuration file:

service:
  http:
    endpoint: 0.0.0.0:8080
    metrics:
      prometheus: {}

With the default base path, the Prometheus endpoint is at /manage/rest/v1/prometheus (full URL for HTTP: http://localhost:8080/manage/rest/v1/prometheus). You can set a custom base path with service.http.metrics.prometheus.url-base-path or service.https.metrics.prometheus.url-base-path. See Configure HTTP and HTTPS for validation rules and examples.

Secret Agent exposes metrics for:

Listeners (TCP, UDS)
Secret managers (AWS, GCP)

All metrics include a module label whose value identifies the module emitting the metric.

Listener metrics

Metric name	Description	Modules	Type
`aerospike_sa_connections_active`	Number of active connections	TCP, UDS	Gauge
`aerospike_sa_connections_open`	Number of opened connections	TCP, UDS	Counter
`aerospike_sa_connections_closed`	Number of closed connections	TCP, UDS	Counter
`aerospike_sa_connections_err`	Number of connection errors	TCP, UDS	Counter
`aerospike_sa_read_err`	Number of read errors on connections	TCP, UDS	Counter
`aerospike_sa_write_err`	Number of write errors on connections	TCP, UDS	Counter
`aerospike_sa_tls_err`	Number of TLS handshake errors	TCP, UDS	Counter
`aerospike_sa_parse_err`	Number of request parsing errors	TCP, UDS	Counter

Secret manager metrics

Metric name	Description	Modules	Type
`aerospike_sa_success`	Number of successful fetch requests	AWS, GCP	Counter
`aerospike_sa_fetch_latency_p50`	p50 latency for fetch requests	AWS, GCP	Gauge
`aerospike_sa_fetch_latency_p95`	p95 latency for fetch requests	AWS, GCP	Gauge
`aerospike_sa_resource_err`	Number of errors from invalid resource names in requests	AWS, GCP	Counter
`aerospike_sa_fetch_err`	Number of errors fetching secrets from the external secret manager	AWS, GCP	Counter
`aerospike_sa_key_not_found_err`	Number of errors where the requested key is not found in the secret	AWS	Counter
`aerospike_sa_session_err`	Number of errors creating a client session to the external secret manager	AWS, GCP	Counter
`aerospike_sa_role_err`	Number of errors assuming a role (AWS) or impersonating (GCP)	AWS, GCP	Counter

Custom labels

You can add custom labels to Prometheus metrics. Define labels as key-value pairs in the metrics context:

service:
  http:
    endpoint: 0.0.0.0:8080
    metrics:
      prometheus:
        labels:
          label1: val1
          label2: val2

Log ticker

In addition to Prometheus metrics, Secret Agent prints metrics to the log every 10 seconds.

The log ticker format is:

<module name>: <metric1> <val1>, <metric2> <val2>, ...

Example log ticker output:

[INFO] 2023/09/09 11:22:34 metrics.go:65: tcp: connections_active 1, connections_opened 2, connections_closed 1, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: uds: connections_active 0, connections_opened 0, connections_closed 0, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: aws: success 2, fetch_latency_p50 38, fetch_latency_p95 108, resource_err 0, fetch_err 0, key_not_found_err 0, session_err 0, role_err 0