Management and Metrics API

The Management and Metrics API is enabled by default for all HTTP outbound connectors, and is enabled for outbound connectors compatible with Aerospike 5.0+ and JMS Inbound 2.0+, when the manage subsection in the service section is configured in the connector's configuration file.

note

For an HTTP outbound connector, the service port and manage port are the same.

GET manage/rest/v1/metrics

Returns JSON of all the metrics of the connector.

info

Connector exposes several metrics. We have only highlighted some of the important ones here.

Counters

A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.

Counters for the outbound connector

Name	Description	Note
connections-active	Count of active XDR connections to the outbound connector.	Database 5.0.0 and later.
connections-closed	Count of closed XDR connections to the outbound connector.	Database 5.0.0 and later.
requests-errors	Count of requests that could not be written to the external system, such as Kafka, JMS, Pulsar, etc. Includes all errors: permanent (as skip and parse errors) as well as retry-able (as timeout or most other backend returned error).
records-inprocess	Count of number of records currently being processed by the Outbound connector	Database 5.0 and later.
requests-parse-error	Count of XDR records which could not be parsed. Parse errors are permanent and are not retried. There will be no traces of those failures on the backend system.
requests-skipped	Count of XDR records which were skipped and not written to the external system. Skip errors are permanent and will not be retried. There will be no traces of those failures on the backend system.
requests-success	Count of records written to the external system.
jms-connections-active	Count of active JMS connections.	Only for Aerospike JMS Outbound.
jms-connections-closed	Count of closed JMS connections.	Only for Aerospike JMS Outbound.
kafka-producers-active	Count of active Kafka producers.	Only for Aerospike Kafka Outbound.
kafka-producers-closed	Count of closed Kafka producers.	Only for Aerospike Kafka Outbound.
pubsub-publishers-active	Count of active Google Pub/Sub publishers.	Only for Aerospike Pub/Sub Outbound.
pubsub-publishers-closed	Count of closed Google Pub/Sub publishers.	Only for Aerospike Pub/Sub Outbound.
pulsar-producers-active	Count of active Pulsar producers.	Only for Aerospike Pulsar Outbound.
pulsar-producers-closed	Count of closed Pulsar producers.	Only for Aerospike Pulsar Outbound.

Counters for the inbound connector

Name	Description	Note
jms-connections-active	Count of an active connections to the JMS broker.	Only for JMS Inbound 2.0+
jms-connections-closed	Count of a closed connections to the JMS broker.	Only for JMS Inbound 2.0+
`<jms-destination-type>`:`<jms-destination-name>`-consumers-active	Count of an active consumers for the given JMS destination.	Only for JMS Inbound 2.0+
`<jms-destination-type>`:`<jms-destination-name>`-errors-aerospike	Count of message handling errors which are neither parsing nor conversion errors.	Only for JMS Inbound 2.0+
`<jms-destination-type>`:`<jms-destination-name>`-errors-conversion	Count of errors which occurred while converting JMS message to an Aerospike message.	Only for JMS Inbound 2.0+
`<jms-destination-type>`:`<jms-destination-name>`-errors-parsing	Count of errors which occurred while parsing an incoming JMS message.	Only for JMS Inbound 2.0+
`<jms-destination-type>`:`<jms-destination-name>`-success	Count of successfully processed incoming JMS message.	Only for JMS Inbound 2.0+

where

the jms-destination-type is one of the valid JMS destinations: queue or topic
the jms-destination-name is a name given to that destination. For e.g. queue:apple_queue-consumers-active or topic:apple_topic-consumers-active.

Timers

Timers time the processing of tasks. A timer metric aggregates timing durations and provides duration statistics, plus throughput statistics. It provides the total count of the metric, maximum time, minimum time, mean time, times at various quantiles (99%, 95%, etc), and mean throughput value and one-minute, five-minute, and fifteen-minute moving average throughput values.

Name	Description	Note
record-ack-queue	Time spent by a processed XDR record in the ack-queue waiting to be acked to Aerospike XDR source.	Database 5.0 and later.
record-dispatch	Time spent dispatching an XDR record to the external system being used, such as JMS, Kafka, Pulsar, etc.	Database 5.0 and later.
record-parsing	Time spent parsing an XDR record.	Database 5.0 and later.
requests-total	Total time a XDR record has spent in the connector.	Database 5.0 and later.

Histograms

Histograms calculate the distribution of a value. It provides the total count of the metric, maximum value, minimum value, mean value, and values at various quantiles (99%, 95%, etc).

Name	Description	Note
record-payload-size	Size of records received by the connector in Bytes.	Database 5.0 and later.

Meters

Meters measure the rate of events in the connector. It provides the total count of the metric, mean value and one-minute, five-minute, and fifteen-minute moving average values.

Name	Description	Note
connections	The rate of XDR connections to the connector.	Deprecated in favor of `connections-active` and `connections-closed`. Database 5.0 and later.

Gauge

A gauge metric is an instantaneous reading of a particular value such as queue's depth.

Name	Description	Note
records-process	Number of records currently being processed by the outbound connector	Database 5.0 and later.
requests-lag	The lag between update of a record in Aerospike and it reaching the Connector. Output in milliseconds. Calculated as `aerospike_record.lut - current_time`. It's sampled every second based on the XDR record being processed at that time, it is approximate and not exact.	Database 5.6 and later.
jvm.attribute.uptime	The uptime of the Java virtual machine in milliseconds.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.buffers.direct.capacity	The maximum amount of a direct buffer which can be used by the connector in bytes.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.buffers.direct.count	The number of direct buffers currently allocated by the connector.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.buffers.direct.used	The size of a direct buffer used by the connector in bytes.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.files.max	The maximum number of file descriptors.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.files.open	The number of open file descriptors.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.gc.`<gc-name>`-Old-Generation.count	The total number of collections that have occurred for the `Old Generation`.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.gc.`<gc-name>`-Old-Generation.time	The approximate accumulated collection elapsed time in milliseconds for the `Old Generation` GC.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.gc.`<gc-name>`-Young-Generation.count	The total number of collections that have occurred for the `Young Generation`.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.gc.`<gc-name>`-Young-Generation.time	The approximate accumulated collection elapsed time in milliseconds for the `Young Generation` GC.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.memory.heap.max	The maximum amount of `heap` memory in bytes that can be used by the Java Virtual Machine.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.memory.heap.used	The amount of used `heap` memory in bytes.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.memory.non-heap.used	The amount of used `non-heap` memory in bytes.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.memory.total.max	The maximum amount of `heap` and `non-heap` memory in bytes that can be used by the Java Virtual Machine.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.memory.total.used	The amount of used `heap` and `non-heap` memory in bytes.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.os.processCpuLoad	The "recent cpu usage" for the Java Virtual Machine process.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.os.systemCpuLoad	The "recent cpu usage" for the whole system.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.
jvm.threads.total.count	The sum of `runnable`, `terminated`, `waiting` and `timed_waiting` threads.	Only available in Aerospike JMS Inbound 2.0+ and Aerospike ESP Outbound 2.0+.

where

the gc-name is the name of the GC used by the JVM. For e.g. G1.

Example response for the outbound connectors

{
  "counters": {
    "connections-active": {
      "count": 10
    },
    "connections-closed": {
      "count": 48
    },
    "records-inprocess": {
      "count": 0
    },
    "requests-error": {
      "count": 0
    },
    "requests-parse-error": {
      "count": 0
    },
    "requests-skipped": {
      "count": 0
    },
    "requests-success": {
      "count": 498
    }
  },
  "timers": {
    "record-ack-queue": {
      "count": 498,
      "max": 2051.907832,
      "mean": 1163.2236708686287,
      "min": 0.16272399999999998,
      "p50": 1220.575885,
      "p75": 1923.971672,
      "p95": 1996.4796489999999,
      "p98": 2038.0708619999998,
      "p99": 2048.850305,
      "p999": 2051.907832,
      "stddev": 664.955034643945,
      "m15_rate": 0.4961578086196813,
      "m1_rate": 1.6175162336487592,
      "m5_rate": 1.196781946695958,
      "mean_rate": 4.064848866969468,
      "duration_units": "milliseconds",
      "rate_units": "calls/second"
    },
    "record-dispatch": {
      "count": 498,
      "max": 4553.8197629999995,
      "mean": 2715.295325838721,
      "min": 2500.492921,
      "p50": 2600.6392769999998,
      "p75": 2643.216744,
      "p95": 3476.529067,
      "p98": 3813.2396289999997,
      "p99": 3822.618782,
      "p999": 4553.8197629999995,
      "stddev": 330.19447171921576,
      "m15_rate": 0.49407536855720546,
      "m1_rate": 1.518561623864363,
      "m5_rate": 1.181774249073358,
      "mean_rate": 4.064825124292382,
      "duration_units": "milliseconds",
      "rate_units": "calls/second"
    },
    "record-parsing": {
      "count": 498,
      "max": 57.421721999999995,
      "mean": 1.4433150983267966,
      "min": 0.16609,
      "p50": 0.33832799999999996,
      "p75": 0.395851,
      "p95": 1.013417,
      "p98": 17.077586,
      "p99": 56.833085,
      "p999": 57.421721999999995,
      "stddev": 7.137755816474087,
      "m15_rate": 0.49377156430405933,
      "m1_rate": 1.5041252749568248,
      "m5_rate": 1.179584797165551,
      "mean_rate": 4.064802337516862,
      "duration_units": "milliseconds",
      "rate_units": "calls/second"
    },
    "requests-total": {
      "count": 498,
      "max": 4731.708243,
      "mean": 4033.1591903548174,
      "min": 2776.8367359999997,
      "p50": 4004.654642,
      "p75": 4680.3988739999995,
      "p95": 4727.156959,
      "p98": 4729.64192,
      "p99": 4730.605834,
      "p999": 4731.708243,
      "stddev": 603.790753932718,
      "m15_rate": 0.4961578086196813,
      "m1_rate": 1.6175162336487592,
      "m5_rate": 1.196781946695958,
      "mean_rate": 4.06477401807955,
      "duration_units": "milliseconds",
      "rate_units": "calls/second"
    }
  },
  "gauges": {
    "requests-lag": {
      "value": 6486
    }
  },
  "histograms": {
    "record-size": {
      "count": 506,
      "max": 100,
      "mean": 99.99999999999916,
      "min": 100,
      "p50": 100,
      "p75": 100,
      "p95": 100,
      "p98": 100,
      "p99": 100,
      "p999": 100,
      "stddev": 8.384404281969126e-13
    }
  }
}

Example response for the JMS Inbound connector

{
  "counters": {
    "jms-connections-active": {
      "count": 4
    },
    "jms-connections-closed": {
      "count": 0
    },
    "queue:apple_queue-consumers-active": {
      "count": 1
    },
    "queue:apple_queue-errors-aerospike": {
      "count": 0
    },
    "queue:apple_queue-errors-conversion": {
      "count": 0
    },
    "queue:apple_queue-errors-parsing": {
      "count": 0
    },
    "queue:apple_queue-success": {
      "count": 100000
    }
  },
  "gauges": {
    "jvm.attribute.uptime": {
      "value": 3349603
    },
    "jvm.buffers.direct.capacity": {
      "value": 115744668
    },
    "jvm.buffers.direct.count": {
      "value": 87
    },
    "jvm.buffers.direct.used": {
      "value": 115744670
    },
    "jvm.files.max": {
      "value": 1048576
    },
    "jvm.files.open": {
      "value": 325
    },
    "jvm.gc.G1-Old-Generation.count": {
      "value": 0
    },
    "jvm.gc.G1-Old-Generation.time": {
      "value": 0
    },
    "jvm.gc.G1-Young-Generation.count": {
      "value": 22
    },
    "jvm.gc.G1-Young-Generation.time": {
      "value": 296
    },
    "jvm.memory.heap.max": {
      "value": 8323596288
    },
    "jvm.memory.heap.used": {
      "value": 55260800
    },
    "jvm.memory.non-heap.used": {
      "value": 95822072
    },
    "jvm.memory.total.max": {
      "value": 8323596287
    },
    "jvm.memory.total.used": {
      "value": 151091952
    },
    "jvm.os.processCpuLoad": {
      "value": 0.008506616257088847
    },
    "jvm.os.systemCpuLoad": {
      "value": 0.08026440037771483
    },
    "jvm.threads.total.count": {
      "value": 43
    }
  },
  "meters": {}
}

GET manage/rest/v1/logging

Returns the logging configuration of the connect server.

Example response

{
  "file": "/var/log/aerospike-xdr-proxy/aerospike-xdr-proxy.log",
  "max-history": 30,
  "levels": {
    "io.netty": "info",
    "ROOT": "info",
    "org.reflections": "error",
    "org.xnio": "error",
    "org.jboss": "off",
    "com.aerospike.connect.outbound.server.manage.app.ManageExceptionMapper": "info",
    "org.glassfish": "info",
    "io.undertow": "info"
  },
  "rolling-file-pattern": "%d{yyyy-MM-dd}",
  "enable-console-logging": true,
  "log-pattern": "%date{yyyy-MM-dd HH:mm:ss.SSS z__TZ__} %-5level %logger{0} - %msg%n%ex",
  "ticker-interval": 1
}

POST manage/rest/v1/logging/{loggerName}

Set the log level for a logger identified by name.

Example

curl -X POST --data "ERROR" -H 'Content-Type: text/plain' http://localhost:8902/manage/rest/v1/logging/io.netty

GET manage/rest/v1/metrics/errors

Returns the errors encountered within the current window of metric logging. Current window is the ticker-interval value set in the logging config section.

Example

{
  "counters": {
    "com.aerospike.client.AerospikeException$Connection - Error -8: Failed to connect to host(s): \n 192.168.1.131 32858 Error -8: java.net.SocketException: Connection reset\n": {
      "count": 2
    }
  }
}

GET manage/rest/v1/metrics​

Counters​

Counters for the outbound connector​

Counters for the inbound connector​

Timers​

Histograms​

Meters​

Gauge​

Example response for the outbound connectors​

Example response for the JMS Inbound connector​

GET manage/rest/v1/logging​

Example response​

POST manage/rest/v1/logging/{loggerName}​

Example​

GET manage/rest/v1/metrics/errors​

Example​

GET manage/rest/v1/metrics

Counters

Counters for the outbound connector

Counters for the inbound connector

Timers

Histograms

Meters

Gauge

Example response for the outbound connectors

Example response for the JMS Inbound connector

GET manage/rest/v1/logging

Example response

POST manage/rest/v1/logging/{loggerName}

Example

GET manage/rest/v1/metrics/errors

Example