Metrics

C client 6.6.0 provides two levels of metrics: as_cluster_stats and Extended.

`as_cluster_stats`

as_cluster_stats is an on-demand snapshot of a cluster’s thread and connection usage. To obtain as_cluster_stats on an active Aerospike instance:

as_cluster_stats stats;
aerospike_stats(as, &stats);

as_cluster_stats contains the following properties:

nodes: Metrics for each node.
- sync: Sync connections.
  - in_use: Active connections from connection pool(s) currently executing commands.
  - in_pool: Initialized connections in connection pool(s) that are not currently active.
  - opened: Total number of node connections opened since node was started.
  - closed: Total number of node connections closed since node was started.
- async: Async connections.
  - in_use: Active connections from connection pool(s) currently executing commands.
  - in_pool: Initialized connections in connection pool(s) that are not currently active.
  - opened: Total number of node connections opened since node was started.
  - closed: Total number of node connections closed since node was started.
- error_count: Command error count since node was started. If the error is retryable, multiple errors per command may occur.
- timeout_count: Command timeout count since node was started. If the timeout is retryable (ie socket_timeout), multiple timeouts per command may occur.
nodes_size: Count of how many nodes are in the cluster.
event_loops: Metrics for each async event loop.
- process_size: Approximate number of commands actively being processed on the event loop.
- queue_size: Approximate number of commands stored on this event loop’s delay queue that have not been started yet.
event_loops_size: Count of how many event loops there are.
thread_pool_queued_tasks: Number of active threads executing sync batch/scan/query commands.
retry_count: Count of command retries since the client was started.

Extended metrics

To use extended metrics, you must explicitly notify the client to track latency and command counts for every node. There is a performance penalty when extended metrics is enabled. To enable:

as_metrics_policy mp;
as_metrics_policy_init(&mp);
as_metrics_policy_set_report_dir(&mp, "/var/log/aerospike/metrics");
mp.interval = 600; // Write metrics snapshot approximately every 600 seconds.
as_status status = aerospike_enable_metrics(as, &err, &mp);

To disable:

as_status status = aerospike_disable_metrics(as, &err);

The as_metrics_policy fields are:

metrics_listeners: Listener that handles metrics notification events. The default listener implementation writes the metrics snapshot to a file. The listener can also be overridden to send the metrics snapshot directly to OpenTelemetry.

The following is a list of metrics_listeners fields:
- enable_listener: Called when metrics have been enabled for the cluster.
- snapshot_listener: Called when a metrics snapshot has been requested for the given cluster.
- node_close_listener: Called when a node is dropped from the cluster.
- disable_listener: Called when metrics have been disabled for the cluster.
- udata: Used to store user defined data relating to the metrics collection, such as a pointer to the output file.
report_dir: Directory path to write metrics log files for listeners that write logs.
report_size_limit: Metrics file size soft limit, in bytes, for listeners that write logs. When report_size_limit is reached or exceeded, the current metrics file is closed and a new metrics file is created with a new timestamp. If report_size_limit is set to 0, the metrics file size is unbounded and the file is only closed when aerospike_disable_metrics() or close() is called. Defaults to 0.
interval: Number of cluster tend iterations between metrics notification events. One tend iteration is defined as config.tender_interval (default: 1 second) plus the time to tend all nodes. Defaults to 30.
latency_columns: Number of elapsed time range buckets in latency histograms. Defaults to 7.
latency_shift: Power of 2 multiple between each range bucket in latency histograms starting at column 3. The bucket units are in milliseconds. The first 2 buckets are <=1ms and >1ms. Examples:

// latency_columns=5 latency_shift=3
<=1ms >1ms >8ms >64ms >512ms
// latency_columns=7 latency_shift=2
<=1ms >1ms >4ms >16ms >64ms >256ms >1024ms

The default extended metrics file includes all as_cluster_stats fields plus the following:

cluster_name: Cluster name.
cpu: Current CPU usage percentage of the client process.
mem: Current memory usage of the client process.
tran_count: Count of commands since client was started.
delay_queue_timeout_count: Count, since client was started, of async commands that timed out in the delay queue before the command was processed.
nodes: Metrics for each node.
- node_name: Node name.
- address: Node IP address.
- port: Node port.
- latency: Latency buckets for the following types:
  - conn: Connection creation latency.
  - write: Single record write commands.
  - read: Single record read commands.
  - batch: Batch read/write commands.
  - query: Scan/Query commands.

Extended metrics file format: <reportDir>/metrics-yyyyMMddHHmmss.log

Extended metrics file example:

2023-08-03 17:56:45.444 header(1) cluster[name,cpu,mem,invalidNodeCount,tranCount,retryCount,delayQueueTimeoutCount,eventloop[],node[]] eventloop[processSize,queueSize] node[name,address,port,syncConn,asyncConn,errors,timeouts,latency[]] conn[inUse,inPool,opened,closed] latency(5,3)[type[l1,l2,l3...]]
2023-08-03 17:57:45.472 cluster[,0,29539536,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]
2023-08-03 17:58:45.476 cluster[,0,29539536,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]
2023-08-03 17:59:45.483 cluster[,0,29539536,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]
...