Metrics

Prometheus

Configuration

To configure a Prometheus metrics endpoint, you need to configure the HTTP endpoint in the Secret Agent configuration file.

Sample Secret Agent configuration file to configure the HTTP endpoint:

service:
  http:
    endpoint: 0.0.0.0:8080

Secret Agent uses the configured HTTP service to expose Prometheus endpoint using /manage/rest/v1/prometheus. The full URL is similar to http://localhost:8080/manage/rest/v1/prometheus

Exposed metrics

Secret Agent exposes metrics for

Listeners like TCP, UDS.
Secret Managers like AWS, GCP.

All metrics have the label module which value represents the module emitting the metric.

Metrics for listeners

Metrics Name	Description	Modules	Type
`aerospike_sa_connections_active`	Number of active connections	TCP, UDS	Gauge
`aerospike_sa_connections_open`	Number of opened connections	TCP, UDS	Counter
`aerospike_sa_connections_closed`	Number of closed connections	TCP, UDS	Counter
`aerospike_sa_connections_err`	Number of connection errors	TCP, UDS	Counter
`aerospike_sa_read_err`	Number of errors while reading from connections	TCP, UDS	Counter
`aerospike_sa_write_err`	Number of errors while writing to connections	TCP, UDS	Counter
`aerospike_sa_tls_err`	Number of errors during the tls handshake for connections	TCP, UDS	Counter
`aerospike_sa_parse_err`	Number of errors while parsing the requests	TCP, UDS	Counter

Metrics for secret managers

Metrics Name	Description	Modules	Type
`aerospike_sa_success`	Number of successful fetch requests	AWS, GCP	Counter
`aerospike_sa_fetch_latency_p50`	p50 latency for fetch requests	AWS, GCP	Gauge
`aerospike_sa_fetch_latency_p95`	p95 latency for fetch requests	AWS, GCP	Gauge
`aerospike_sa_resource_err`	Number of errors due to bad resource name in the request	AWS, GCP	Counter
`aerospike_sa_fetch_err`	Number of errors while fetching secrets from external secret managers	AWS, GCP	Counter
`aerospike_sa_key_not_found_err`	Number of errors where the requested key is not found in the fetched key-value pairs	AWS	Counter
`aerospike_sa_session_err`	Number of errors while creating a client session to external secret managers	AWS, GCP	Counter
`aerospike_sa_role_err`	Number of errors while assuming role (AWS)/impersonating (GCP)	AWS, GCP	Counter

Custom labels

You can configure custom labels for Prometheus metrics as well. Labels can be provided as key-value pairs in the metrics context.

Sample Secret Agent configuration file section to configure custom labels:

...
http:
    endpoint: 0.0.0.0:8080
    metrics:
      prometheus:
        labels:
          label1: val1
          label2: val2
...

Log ticker

Along with Prometheus metrics, Secret Agent also prints metrics in the log file every 10 seconds.

The log ticker format is:

<module name>: <metric1> <val1>, <metric2> <val2>, ...

Sample ticker log:

[INFO] 2023/09/09 11:22:34 metrics.go:65: tcp: connections_active 1, connections_opened 2, connections_closed 1, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: uds: connections_active 0, connections_opened 0, connections_closed 0, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:34 metrics.go:65: aws: success 2, fetch_latency_p50 38, fetch_latency_p95 108, resource_err 0, fetch_err 0, key_not_found_err 0, session_err 0, role_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: tcp: connections_active 1, connections_opened 2, connections_closed 1, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: uds: connections_active 0, connections_opened 0, connections_closed 0, connections_err 0, read_err 0, write_err 0, tls_err 0, parse_err 0
[INFO] 2023/09/09 11:22:44 metrics.go:65: aws: success 2, fetch_latency_p50 38, fetch_latency_p95 108, resource_err 0, fetch_err 0, key_not_found_err 0, session_err 0, role_err 0