Skip to main content
Visit booth 3171 at Google Cloud Next to see how to unlock real-time decisions at scaleMore info

Metrics

The Java client provides two levels of metrics: ClusterStats and Extended.

ClusterStats​

ClusterStats is an on-demand snapshot of a cluster’s thread and connection usage. To obtain ClusterStats on an active AerospikeClient instance:

ClusterStats cs = client.getClusterStats();
System.out.println(cs);

ClusterStats contains the following properties:

  • nodes: Metrics for each node.

    • sync: Sync connections.

      • inUse: Active connections from connection pool(s) currently executing commands.
      • inPool: Initialized connections in connection pool(s) that are not currently active.
      • opened: Total number of node connections opened since node was started.
      • closed: Total number of node connections closed since node was started.

    • async: Async connections.

      • inUse: Active connections from connection pool(s) currently executing commands.
      • inPool: Initialized connections in connection pool(s) that are not currently active.
      • opened: Total number of node connections opened since node was started.
      • closed: Total number of node connections closed since node was started.

    • errorCount: Command error count since node was started. If the error is retryable, multiple errors per command may occur.

    • timeoutCount: Command timeout count since node was started. If the timeout is retryable (ie socketTimeout), multiple timeouts per command may occur.

  • eventLoops: Metrics for each async event loop.

    • processSize: Approximate number of commands actively being processed on the event loop.
    • queueSize: Approximate number of commands stored on this event loop's delay queue that have not been started yet.

  • threadsInUse: Number of active threads executing sync batch/scan/query commands.

  • recoverQueueSize Number of connections residing in sync connection recover queue. This queue is only used when the timeoutDelay policy is greater than zero and a timeout occurs on a sync command.

  • invalidNodeCount: Count of add node failures since the client was started. If greater than zero, some peer nodes are not accessible by the client.

  • retryCount: Count of command retries since the client was started.

Extended Metrics​

To use extended metrics, you must explicitly notify the client to track latency and command counts for every node. There is a performance penalty when extended metrics is enabled. To enable:

MetricsPolicy mp = new MetricsPolicy();
mp.reportDir = "/var/log/metrics";
mp.interval = 600;  // Write metrics snapshot approximately every 600 seconds.
client.enableMetrics(mp);

To disable:

client.disableMetrics();

The MetricsPolicy fields are:

  • listener: Listener that handles metrics notification events. The default listener implementation writes the metrics snapshot to a file. The listener can also be overridden to send the metrics snapshot directly to OpenTelemetry.

  • reportDir: Directory path to write metrics log files for listeners that write logs.

  • reportSizeLimit: Metrics file size soft limit in bytes for listeners that write logs. When reportSizeLimit is reached or exceeded, the current metrics file is closed and a new metrics file is created with a new timestamp. If reportSizeLimit is set to 0, the metrics file size is unbounded and the file is only closed when disableMetrics() or close() is called. Defaults to 0.

  • interval: Number of cluster tend iterations between metrics notification events. One tend iteration is defined as ClientPolicy.tendInterval (default: 1 second) plus the time to tend all nodes. Defaults to 30.

  • latencyColumns: Number of elapsed time range buckets in latency histograms. Default to 7.

  • latencyShift: Power of 2 multiple between each range bucket in latency histograms starting at column 3. The bucket units are in milliseconds. The first 2 buckets are <=1ms and >1ms. Examples:

// latencyColumns=5 latencyShift=3
<=1ms >1ms >8ms >64ms >512ms
// latencyColumns=7 latencyShift=2
<=1ms >1ms >4ms >16ms >64ms >256ms >1024ms

The default extended metrics file includes all ClusterStats fields plus the following:

  • clusterName: Cluster name.

  • cpu: Current CPU usage percentage of the client process.

  • mem: Current memory usage of the client process.

  • tranCount: Count of commands since client was started.

  • delayQueueTimeoutCount: Count since client was started of async commands that timed out in the delay queue before the command was processed.

  • nodes: Metrics for each node.

    • nodeName: Node name.

    • address: Node IP address.

    • port: Node port.

    • latency: Latency buckets for the following types:

      • conn: Connection creation latency.

      • write: Single record write commands.

      • read: Single record read commands.

      • batch: Batch read/write commands.

      • query: Scan/Query commands.

Extended metrics file format: <reportDir>/metrics-yyyyMMddHHmmss.log

Extended metrics file example:

2023-08-03 17:56:45.444 header(1) cluster[name,cpu,mem,threadsInUse,recoverQueueSize,invalidNodeCount,tranCount,retryCount,delayQueueTimeoutCount,eventloop[],node[]] eventloop[processSize,queueSize] node[name,address,port,syncConn,asyncConn,errors,timeouts,latency[]] conn[inUse,inPool,opened,closed] latency(5,3)[type[l1,l2,l3...]]
2023-08-03 17:57:45.472 cluster[,0,29539536,0,0,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]
2023-08-03 17:58:45.476 cluster[,0,29539536,0,0,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]
2023-08-03 17:59:45.483 cluster[,0,29539536,0,0,0,86,0,0,[],[[BB9BF3DDF290C00,172.16.70.243,3000,0,1,2,0,0,0,0,0,0,0,[conn[0,0,0,0,0],write[6,1,0,0,0],read[14,0,0,0,0],batch[6,3,0,0,0],query[0,0,0,0,0]]],[BCDBF3DDF290C00,172.16.70.243,3020,0,1,2,0,0,0,0,0,2,0,[conn[1,0,0,0,0],write[13,1,0,0,0],read[3,0,0,0,0],batch[9,0,0,0,0],query[0,0,0,0,0]]],[BC3BF3DDF290C00,172.16.70.243,3010,0,1,2,0,0,0,0,0,0,0,[conn[1,0,0,0,0],write[7,1,0,0,0],read[27,0,0,0,0],batch[10,0,0,0,0],query[0,0,0,0,0]]]]]