--- title: "Database Log Reference" description: "Comprehensive Aerospike Database log reference covering all message types, severity levels, and troubleshooting content." --- # Database Log Reference > For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt) > > All documentation pages available in markdown. See [Configuring logs](https://aerospike.com/docs/database/manage/logging/logs) for more information on configuring server logs. ## Arenax #### `could not allocate arena stage` `WARNING` Message: ```bash could not allocate xxxxxxxxxx-byte arena stage xxx: No space left on device ``` Description: Indicates for the `index-type flash` configuration that the mount points have run out of space. You may need to delete the arena files manually and run `fsck` on the disk partitions. Context: arenax Introduced: \- Removed: \- --- ## As #### `allowing fill-migrations after delay` `INFO` Message: ```bash allowing x fill-migrations after y seconds delay ``` Description: This message appears after a recluster event, if a value is set for the [`migrate-fill-delay`](https://aerospike.com/docs/database/reference/config#service__migrate-fill-delay) and if the recluster event has caused fill-migrations to be scheduled. Context: as Introduced: 4.3.1 Removed: \- --- #### `finished clean shutdown` `INFO` Message: ```bash finished clean shutdown - exiting ``` Description: This message is the last one of a sequence of messages logged during Aerospike server shutdown. The message signifies that Aerospike was shut down with “trusted” status which is a necessary condition for a subsequent fast restart of a namespace that is configured with storage-engine device. See [ASD shutdown process](https://support.aerospike.com/s/article/The-asd-process-cold-starts-after-a-dirty-exit-which-appears-to-be-clean). Context: as Introduced: 3.0.0 Removed: \- --- #### `waiting for storage` `INFO` Message: ```bash waiting for storage: 1569063 objects, 1819777 scanned ``` Description: **Objects** and **scanned** values diverge for various reasons, such as scanning records that were previously expired or expired while the system was down. Context: as Introduced: \- Removed: \- Occurs: When cold starting a node that has data on persistent storage, such as SSD. Parameters: objects Number of retained objects from storage device. --- scanned Number of objects that have been scanned on the storage device. --- ## Batch #### `abandoned batch after timeout` `WARNING` Message: ```bash abandoned batch from 11.22.33.44 with 23 transactions after 30000 ms. ``` Context: batch Introduced: 4.1.0 Removed: \- Occurs: When a batch-index transaction is abandoned due to one or more delays, where its total time exceeds the allowed threshold of either the client’s total timeout or 30 seconds, if the total timeout is not set by the client. Each occurrence also increments the [`batch_index_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__batch_index_error) statistic. Prior to 6.4, this stat increments when the total time exceeds twice the client’s total timeout or 30 seconds if the total timeout is not set by the client. Parameters: IP Address The client originating IP address for the transaction. --- Number of transactions The number of batch sub transactions in the impacted batch index transaction. --- Abandoned time The total time the batch index transaction had been running before being abandoned. --- #### `batch queues full` `WARNING` Message: ```bash failed to find active batch queue that is not full ``` Description: All batch index queues are greater than the [`batch-max-buffers-per-queue`](https://aerospike.com/docs/database/reference/config#service__batch-max-buffers-per-queue) limit, so batch requests are being rejected. Consult [Tuning Batches](https://aerospike.com/docs/database/manage/cluster/batches#tuning-batches). To save log space, this message is logged with a `(repeated: nnn)` prefix just once per [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) during periods of repetition. Context: batch Introduced: 4.1.0 Removed: \- --- ## Bin #### `bin-name quota full` `WARNING` Message: ```bash {NAMESPACE} bin-name quota full - can’t add new bin-name ``` Description: Max number of bins per namespace (65535) has been reached. Prior to Database 4.9.0: Max number of bins per namespace (32767) has been reached. See [How to clear up bin names when they exceed the limits](https://support.aerospike.com/s/article/How-to-clear-up-bin-names-when-they-exceed-the-limits). Context: bin Introduced: \- Removed: \- Parameters: NAMESPACE The namespace that has reached the maximum number of bins. --- ## Cf\_os #### `control group (cgroup) memory below 10 percent` `WARNING` Message: ```bash WARNING (cf_os): cgroup memory available kb is <= 10 percent of detected cgroup memory limit kb. enable the cgroup-mem-tracking config option to stop writes here ``` Description: Logged when the server detects a control group (cgroup) memory limit and less than 10% of that limit is available, even when [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is disabled. This warning alerts operators to enable `cgroup-mem-tracking` so that stop-writes calculations use the control group (cgroup) limit instead of host-level memory. Context: cf\_os Introduced: 8.1.2 Removed: \- Parameters: avail Available control group (cgroup) memory in kilobytes. --- limit Detected control group (cgroup) memory limit in kilobytes. --- #### `control group (cgroup) used bytes exceed limit` `WARNING` Message: ```bash WARNING (cf_os): used bytes > limit bytes ``` Description: Logged when the control group (cgroup) memory accounting reports usage exceeding the control group (cgroup) limit. This is an unexpected state that may indicate memory pressure or accounting lag in the control group (cgroup) controller. Context: cf\_os Introduced: 8.1.2 Removed: \- Parameters: used Current memory usage in bytes as reported by the control group (cgroup). --- limit Configured control group (cgroup) memory limit in bytes. --- #### `control group (cgroup) zero available memory` `WARNING` Message: ```bash WARNING (cf_os): zero available cg memory: used bytes == limit bytes ``` Description: Logged when the control group (cgroup) reports zero available memory (usage equals the limit). The node is at risk of an OOM kill. Investigate memory consumers and consider enabling [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) to trigger stop-writes before this state is reached. Context: cf\_os Introduced: 8.1.2 Removed: \- Parameters: used Current memory usage in bytes. --- limit Configured control group (cgroup) memory limit in bytes. --- ## Clustering #### `evicted from cluster by principal node` `WARNING` Message: ```bash evicted from cluster by principal node BB9030011AC4202 ``` Description: The paxos principal node has determined that this node is not a valid cluster member. See the log on the principal for more information. Context: clustering Introduced: \- Removed: \- Parameters: principal Node ID of the paxos principal node --- #### `ignoring node - exceeding maximum supported cluster size` `WARNING` Message: ```bash ignoring node: bb9030011ac4202 - exceeding maximum supported cluster size 1 ``` Description: The cluster has exceeded the `asdb-cluster-nodes-limit` in the `features.conf` file. Reduce the number of nodes attempting to cluster or install the correct license. Context: clustering Introduced: \- Removed: \- Parameters: node Node that is attempting to start cluster. --- #### `ignoring paxos accepted - node not in acceptor` `WARNING` Message: ```bash ignoring paxos accepted from node BB9030011AC4202 - it is not in acceptor list ``` Description: A paxos clustering message from another node was ignored. This can result from network stability issues. Context: clustering Introduced: \- Removed: \- Parameters: node Node sending the invalid paxos message --- #### `ignoring paxos accepted - invalid proposal id` `WARNING` Message: ```bash ignoring paxos accepted from node BB9030011AC4202 with invalid proposal id ``` Description: A paxos clustering message from another node was ignored. This can result from network stability issues. Context: clustering Introduced: \- Removed: \- Parameters: node Node sending the invalid paxos message --- #### `ignoring stale join request` `INFO` Message: ```bash ignoring stale join request from node BB9030011AC4202 - delay estimate 83108(ms) ``` Description: A paxos clustering message from another node was ignored. This can result from network stability issues causing delayed delivery of packets. Context: clustering Introduced: \- Removed: \- Parameters: node Node sending the invalid paxos message --- delay estimate Estimated delay of the message in milliseconds. --- ## Config #### `config: trailing garbage in features.conf` `CRITICAL` Message: ```bash CRITICAL (config): (features_ee.c:184) trailing garbage in /etc/aerospike/features.conf, line 21 ``` Description: The [feature-key-file](https://aerospike.com/docs/database/reference/config#service__feature-key-file) has been tampered with. Replace the file with the original provided by Aerospike. See [Aerospike fails to start due to corrupted features.conf file - trailing garbage error](https://support.aerospike.com/s/article/Aerospike-fails-to-start-due-to-corrupted-features-conf-file-trailing-garbage-error). Context: config Introduced: \- Removed: \- --- #### `CONFIG_CHECK failed - MESSAGE` `WARNING` Message: ```bash failed CONFIG_CHECK check - MESSAGE ``` Description: A configuration [best-practice](https://aerospike.com/docs/database/install/linux) was violated at startup. Context: config Introduced: 5.7.0 Removed: \- Parameters: CONFIG\_CHECK Name of the check that was violated, which could be one of the following: - [memory-size](https://aerospike.com/docs/database/install/linux) - [service-threads](https://aerospike.com/docs/database/install/linux) --- sMessage Description of how the best-practice was violated. --- #### `failed best-practices checks` `CRITICAL` Message: ```bash failed best-practices checks - see 'https://docs.aerospike.com/operations/install/linux/bestpractices' ``` Description: Indicates failed [best practices](https://aerospike.com/docs/database/install/linux). This message follows a set of warnings for each best practice that was violated. Becomes a WARNING when [enforce-best-practices](https://aerospike.com/docs/database/reference/config#service__enforce-best-practices) is set to `false`. Context: config Introduced: 5.7.0 Removed: \- --- #### `invalid feature key signature` `CRITICAL` Message: ```bash invalid feature key signature ``` Description: The key file has been modified and no longer matches the digital signature. The key file must be exactly the same as when it was downloaded. Seemingly harmless operations, like cutting and pasting the file’s contents, can change the file and invalidate it. Context: config Introduced: \- Removed: \- --- #### `tomb-raider-unmark-threads exceeds device count (config)` `WARNING` Message: ```bash WARNING (config): {} tomb-raider-unmark-threads () exceeds number of devices (), lowering to ``` Description: Emitted during startup or config post-processing for SSD or PMEM namespaces when [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads) is higher than the number of storage devices. The server lowers the value to the device count. Context: config Introduced: 8.1.2 Removed: \- Parameters: ns-name Namespace name. --- configured Value from configuration before clamping. --- actual Number of devices; this becomes the effective thread count. --- ## Deprecation #### `AS_BYTES_CSHARP deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_CSHARP is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_BYTES_ERLANG deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_ERLANG is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_BYTES_JAVA deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_JAVA is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_BYTES_PHP deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_PHP is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_BYTES_PYTHON deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_PYTHON is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_BYTES_RUBY deprecated - upgrade client` `INFO` Message: ```bash AS_BYTES_RUBY is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_CSHARP_BLOB deprecated - upgrade client` `INFO` Message: ```bash AS_PARTICLE_TYPE_CSHARP_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_ERLANG_BLOB deprecated` `INFO` Message: ```bash AS_PARTICLE_TYPE_ERLANG_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_JAVA_BLOB deprecated` `INFO` Message: ```bash AS_PARTICLE_TYPE_JAVA_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_PHP_BLOB is deprecated` `INFO` Message: ```bash AS_PARTICLE_TYPE_PHP_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_PYTHON_BLOB is deprecated` `INFO` Message: ```bash AS_PARTICLE_TYPE_PYTHON_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `AS_PARTICLE_TYPE_RUBY_BLOB deprecated` `INFO` Message: ```bash AS_PARTICLE_TYPE_RUBY_BLOB is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `cdt parameter protocol deprecated - upgrade client` `INFO` Message: ```bash cdt parameter protocol using 16 bit param count is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `little-endian compressed size deprecated` `INFO` Message: ```bash compressed size in little-endian format is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `config-get info deprecated - use get-config` `INFO` Message: ```bash config-get info command is deprecated - use get-config instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `config-set deprecated - use set-config` `INFO` Message: ```bash 'config-set' info command is deprecated - use 'set-config' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `info command deprecation (generic)` `INFO` Message: ```bash '' info command is deprecated - use '' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints a single notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `device_size expression deprecated - use record_size` `INFO` Message: ```bash 'device_size' expression is deprecated - use 'record_size' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `digests info command deprecated` `INFO` Message: ```bash 'digests' info command is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `features info command deprecated` `INFO` Message: ```bash 'features' info command is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `group deprecated` `INFO` Message: ```bash 'group' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `heartbeat.address deprecated` `INFO` Message: ```bash 'heartbeat.address' configuration parameter is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `sindex-create: indexdata deprecated` `INFO` Message: ```bash 'indexdata' parameter of 'sindex-create' info command is deprecated - use 'bin' and 'type' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `info summary deprecated` `INFO` Message: ```bash info summary is deprecated ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `mesh and mcast info deprecated - use get-config` `INFO` Message: ```bash 'mesh' and 'mcast' info commands are deprecated - use 'get-config' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `memory_size deprecated - use record_size` `INFO` Message: ```bash 'memory_size' expression is deprecated - use 'record_size' instead ``` Description: Deprecated. Use [`record_size`](https://aerospike.com/docs/develop/expressions/metadata#record_size) instead. Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `mesh and mcast info deprecated - use get-config` `INFO` Message: ```bash 'mesh' and 'mcast' info commands are deprecated - use 'get-config' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `heartbeat mode multicast deprecated` `INFO` Message: ```bash 'multicast' for heartbeat 'mode' configuration is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `name info command deprecated - use node` `INFO` Message: ```bash 'name' info command is deprecated - use node instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `objects info command deprecated - use statistics` `INFO` Message: ```bash 'objects' info command is deprecated - use statistics instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `partitions info command deprecated` `INFO` Message: ```bash 'partitions' info command is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `pidfile configuration parameter deprecated` `INFO` Message: ```bash 'pidfile' configuration parameter is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `proto-fd-idle-ms deprecated` `INFO` Message: ```bash 'proto-fd-idle-ms' configuration parameter is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `deprecated list option for bin names` `INFO` Message: ```bash queries using the deprecated list option for bin names is deprecated - upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `replicas-all deprecated - use replicas` `INFO` Message: ```bash 'replicas-all' info command is deprecated - use 'replicas' instead or upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `replicas-master info command deprecated` `INFO` Message: ```bash 'replicas-master' info command is deprecated - use 'replicas' instead or upgrade your client ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `service info command deprecated - use service-clear-std` `INFO` Message: ```bash 'service' info command is deprecated - use 'service-clear-std' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `services-alternate info command deprecated - use peers-clear-alt` `INFO` Message: ```bash 'services-alternate' info command is deprecated - use 'peers-clear-alt' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `services-alumni info command deprecated` `INFO` Message: ```bash 'services-alumni' info command is deprecated - use 'alumni-clear-std' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `services info command deprecated` `INFO` Message: ```bash 'services' info command is deprecated - use 'peers-clear-std' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `auto-pin configuration deprecated` `INFO` Message: ```bash Setting 'auto-pin' configuration parameter to 'adq' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) if used. --- #### `sindex info command deprecated` `INFO` Message: ```bash 'sindex' info command is deprecated - use 'sindex-list' instead ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `user is deprecated` `INFO` Message: ```bash 'user' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `vault-ca deprecated - see docs` `INFO` Message: ```bash 'vault-ca' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `vault-namespace deprecated` `INFO` Message: ```bash 'vault-namespace' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `vault-path deprecated` `INFO` Message: ```bash 'vault-path' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `vault-token-file deprecated` `INFO` Message: ```bash 'vault-token-file' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- #### `vault-url deprecated` `INFO` Message: ```bash 'vault-url' is deprecated - see https://aerospike.com/docs/database/release/8-1/ ``` Context: deprecation Introduced: \- Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval), if used. --- ## Drv\_mem #### `bad set-id` `CRITICAL` Message: ```bash bad set-id x ``` Description: Indicates an issue with the specified set ID, likely indicating a configuration or referencing error. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x set ID --- #### `bad size` `CRITICAL` Message: ```bash bad size x ``` Description: Indicates an issue with the specified size, likely outside acceptable parameters. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x size --- #### `bad void-time` `CRITICAL` Message: ```bash bad void-time x ``` Description: Indicates an issue with the specified void-time, likely indicating an error in expiration or deletion timing. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x void time --- #### `bad wblock-id` `CRITICAL` Message: ```bash bad wblock-id x ``` Description: Indicates an issue with the specified write block ID. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x write block ID --- #### `can't get set index from vmap` `CRITICAL` Message: ```bash can't get set index x from vmap ``` Description: Indicates a failure to retrieve the specified set index from the version map. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x set index --- #### `community edition called resume_devices()` `CRITICAL` Message: ```bash community edition called resume_devices() ``` Description: The Community Edition of Aerospike does not allow the resume\_devices() function. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `device encrypted - no encryption key file configured` `CRITICAL` Message: ```bash device encrypted but no encryption key file configured ``` Description: Indicates a device is encrypted but lacks configuration for an encryption key file. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `AP partition versions - strong-consistency configured` `CRITICAL` Message: ```bash device has AP partition versions but 'strong-consistency' is configured ``` Description: Indicates a configuration issue where AP partitions exist despite strong consistency being enabled. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `cp partition versions - strong-consistency not configured` `CRITICAL` Message: ```bash device has CP partition versions but 'strong-consistency' is not configured ``` Description: Indicates a configuration issue where CP partitions exist without strong consistency enabled. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `single-bin data no longer supported` `CRITICAL` Message: ```bash device has 'single-bin' data but 'single-bin' is no longer supported ``` Description: Indicates a configuration issue where the device contains data in a format no longer supported. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `device not encrypted but encryption key file configured` `CRITICAL` Message: ```bash device not encrypted but encryption key file x is configured ``` Description: Indicates a non-encrypted device has an encryption key file configured unnecessarily. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x path to encryption key file --- #### `devices smaller than memory stripes` `CRITICAL` Message: ```bash devices are smaller than memory stripes ``` Description: Device memory is less than the stripe size Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `device has old format - erase to upgrade` `CRITICAL` Message: ```bash drive-name: Aerospike device has old format - must erase device to upgrade ``` Description: Aerospike device is using an old format and needs to be erased for an upgrade. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `drive-name: bad device-id` `CRITICAL` Message: ```bash drive-name: bad device-id x ``` Description: Indicates an issue with the device ID on the Aerospike device. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x device ID --- #### `drive-name: bad n-devices` `CRITICAL` Message: ```bash drive-name: bad n-devices x ``` Description: Indicates an issue with the number of devices specified for the Aerospike device. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x number of devices --- #### `drive-name: bad pristine offset` `CRITICAL` Message: ```bash drive-name: bad pristine offset x ``` Description: Indicates an issue with the pristine offset on the Aerospike device. Pristine blocks are blocks the Aerospike server has never written to before. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x pristine offset --- #### `drive: can't change write-block-size` `CRITICAL` Message: ```bash drive-name: can't change write-block-size from x to y ``` Description: Indicates an issue changing the write block size on the Aerospike device. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x previous size --- y target size --- #### `drive-name: previous namespace changed` `CRITICAL` Message: ```bash drive-name: previous namespace x now y - check config or erase device ``` Description: Indicates a namespace name change on the Aerospike device and suggests checking configuration or erasing the device. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x previous namespace name --- y current namespace name --- #### `drive-name: random signature` `CRITICAL` Message: ```bash drive-name: random signature is 0 ``` Description: Indicates a problem where the random signature expected on the Aerospike device is 0. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `drive-name: unknown version` `CRITICAL` Message: ```bash drive-name: unknown version x ``` Description: Aerospike device is using an unknown version. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x version number --- #### `encryption key or algorithm mismatch` `CRITICAL` Message: ```bash encryption key or algorithm mismatch ``` Description: Indicates a mismatch between the encryption key or algorithm configured and what is required. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `memory stripe size differs from config` `CRITICAL` Message: ```bash existing memory stripe size differs from config ``` Description: Indicates a mismatch between the configured memory stripe size and the existing memory stripe size. To resolve this, configure [`data-size`](https://aerospike.com/docs/database/reference/config#namespace__data-size). Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `generation` `CRITICAL` Message: ```bash generation 0 ``` Description: Displays when a record is deleted and removed from the tree but not freed. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `stop-writes limit hit` `CRITICAL` Message: ```bash hit stop-writes limit before drive scan completed ``` Description: The stop-writes limit is a user-configurable limit to prevent more data being written than is available on the device. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `mprotect failed` `CRITICAL` Message: ```bash mprotect (address, length, protection) failed: x, (y) ``` Description: Indicates a failure in setting memory protection for a region of memory. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- y error description --- #### `can't add record to index` `CRITICAL` Message: ```bash {namespace} can't add record to index ``` Description: Indicates a failure to add a record to the index in the specified namespace. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `could not allocate shmem stripe` `CRITICAL` Message: ```bash {namespace} could not allocate x-byte shmem stripe ``` Description: Indicates a failure to allocate the specified amount of shared memory for a stripe in the namespace. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x number of bytes --- #### `namespace drive set - device counts differ` `CRITICAL` Message: ```bash namespace drive set with unmatched headers - devices x & y have different device counts ``` Description: Indicates a configuration issue where two devices in a namespace drive set have different device counts. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x device name --- y device name --- #### `namespace drive set unmatched headers` `CRITICAL` Message: ```bash namespace drive set with unmatched headers - devices x & y have different signatures ``` Description: Indicates a configuration issue where two devices in a namespace drive set have different signatures. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x device name --- y device name --- #### `namespace loaded: objects device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 device-pcts (20, 30, 25) ``` Description: Objects that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_mem Removed: \- --- #### `namespace loaded: objects tombstones device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 tombstones 77 device-pcts (20, 30, 25) ``` Description: Objects and tombstones that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_mem Removed: \- --- #### `no keys available for namespace` `CRITICAL` Message: ```bash {namespace}: no keys available for namespace ``` Description: Indicates a lack of available keys for the specified namespace, preventing operations. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `no keys available to create stripe` `CRITICAL` Message: ```bash {namespace} no keys available to create stripe ``` Description: Indicates a lack of available keys to create a stripe in the specified namespace. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `shmem stripe and device shadow mismatch` `CRITICAL` Message: ```bash {namespace} shmem stripe and device shadow-name have different device counts ``` Description: Indicates a device count mismatch between shared memory stripe and device in the specified namespace. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `shmem stripe and device shadow-name signature mismatch` `CRITICAL` Message: ```bash {namespace} shmem stripe and device shadow-name have different signatures ``` Description: Indicates a signature mismatch between shared memory stripe and device in the specified namespace. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `could not allocate shmem stripe` `CRITICAL` Message: ```bash {namespace_name}: could not allocate 17179869184-byte shmem stripe ``` Description: Aerospike server is attempting to pre-allocate in-memory data storage in shared memory, either as stripes (1/8th of `data-size`) or as a mirror of the storage-backed persistence (`filesize` or `device` size), and the `kernel.shmmax` or `kernel.shmall` is set too low. See [configuring namespace storage](https://aerospike.com/docs/database/manage/namespace/storage/config) Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `tomb raider unmark: detecting cenotaphs (memory engine)` `INFO` Message: ```bash INFO (drv_mem): {} tomb raider detecting cenotaphs using threads over devices ... ``` Description: Logged at the start of the unmark phase of a tomb raider cycle for the memory storage engine. `` is the number of threads in use; `` is the number of memory stripes (devices). See [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads). Context: drv\_mem Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- N Number of unmark threads for this phase. --- D Number of memory stripes being swept. --- #### `tomb-raider-unmark-threads exceeds memory stripe count (config)` `WARNING` Message: ```bash WARNING (drv_mem): {} tomb-raider-unmark-threads () exceeds number of memory stripes (), lowering to ``` Description: Emitted during storage initialization for the memory storage engine when tomb-raider-unmark-threads is higher than the number of memory stripes. The server lowers the value to the stripe count. Unlike SSD and PMEM engines, the memory engine performs this check at storage initialization because the stripe count is only known after initialization. Context: drv\_mem Introduced: 8.1.2 Removed: \- Parameters: ns-name Namespace name. --- configured Value from configuration before clamping. --- actual Number of memory stripes; this becomes the effective thread count. --- #### `random key generation failed` `CRITICAL` Message: ```bash random key generation failed ``` Description: Indicates a failure in generating a random key. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `not an Aerospike device but not erased` `CRITICAL` Message: ```bash DEVICE: not an Aerospike device but not erased - check config or erase device ``` Description: Warns that the device is not recognized as an Aerospike device but has not been erased. Context: drv\_mem Introduced: 7.0.0 Removed: \- --- #### `shadow-name: device failed open` `CRITICAL` Message: ```bash shadow-name: DEVICE FAILED open: errno x (y) ``` Description: Indicates a failure to open the specified device with an error number and description. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- y error description --- #### `shadow device write failed` `CRITICAL` Message: ```bash shadow-name: DEVICE FAILED write: errno x (y) ``` Description: Indicates a write failure on the specified device with an error number and description. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- y error description --- #### `shadow-name: read failed at all sizes` `CRITICAL` Message: ```bash shadow-name: read failed at all sizes from x to y bytes ``` Description: Indicates a failure to read the shadow-name at any size within the specified byte range. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x byte range start --- y byte range end --- #### `shadow-name: read failed` `CRITICAL` Message: ```bash shadow-name: read failed: errno x (y) ``` Description: Indicates a read failure on the shadow-name device, providing an error number and description. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- y error description --- #### `unable to open device` `CRITICAL` Message: ```bash unable to open device x: error y ``` Description: Displays the error code giving information about the reason the device could not be opened. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x device name --- y error number --- #### `unable to open file` `CRITICAL` Message: ```bash unable to open file x: error y ``` Description: Displays the error code giving information about the reason the file could not be opened. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x path to file --- y error number --- #### `unable to truncate file` `CRITICAL` Message: ```bash unable to truncate file: errorno x ``` Description: This error appears when a shadow file could not be sized properly. See [ftruncate()](https://man7.org/linux/man-pages/man3/ftruncate.3p.html) for more information. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x ftruncate() error code --- #### `load bins failed` `CRITICAL` Message: ```bash x - load bins failed ``` Description: Indicates a failure in loading bins for the specified process or identifier. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- #### `unpack bins failed` `CRITICAL` Message: ```bash x - unpack bins failed ``` Description: Indicates a failure in unpacking bins for the specified process or identifier. Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- #### `devices fresh during warm restart` `CRITICAL` Message: ```bash x found all y devices fresh during warm restart ``` Description: Shown when fresh headers were found during warm restart Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x error number --- y number of devices --- #### `size must be greater than header size` `CRITICAL` Message: ```bash x size y must be greater than header size z ``` Description: Shown when the file size is less than or equal to the drive header size. To resolve this, configure [`filesize`](https://aerospike.com/docs/database/reference/config#namespace__filesize). Context: drv\_mem Introduced: 7.0.0 Removed: \- Parameters: x Either “file” or “usable device” depending on whether files or devices are used for backing storage in the memory namespace. --- y file size --- z header size --- ## Drv\_pmem #### `get_key: failed pmem_read_record` `WARNING` Message: ```bash get_key: failed pmem_read_record() ``` Description: Aerospike was not able to read the record from storage, which may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure) for more information. Context: drv\_pmem Introduced: \- Removed: \- --- #### `load_bins: failed pmem_read_record()` `WARNING` Message: ```bash load_bins: failed pmem_read_record() ``` Description: Aerospike was not able to read the record from storage, which may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure?) for more information. Context: drv\_pmem Introduced: \- Removed: \- --- #### `load_n_bins: failed pmem_read_record()` `WARNING` Message: ```bash load_n_bins: failed pmem_read_record() ``` Description: Aerospike was not able to read the record from storage, which may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure?) for more information. Context: drv\_pmem Introduced: \- Removed: 5.1.0 --- #### `{namespace} loaded: device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 device-pcts (20, 30, 25) ``` Description: Objects that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_pmem Removed: \- --- #### `namespace loaded: objects tombstones device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 tombstones 77 device-pcts (20, 30, 25) ``` Description: Objects and tombstones that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_pmem Removed: \- --- #### `namespace - negative device bytes` `WARNING` Message: ```bash (namespace): (namespace.c:550) set-id 1 - negative device bytes! ``` Description: A statistic was not correctly initialized during a warm restart, and Aerospike server believes the PMEM storage has a negative size. This error may be printed once for every record held on the storage during the warm restart. This error poses no risk to the data stored on the device, and has been resolved in hotfix AER-6478, which is available in Aerospike Database 5.2.0.36, 5.3.0.26, 5.4.0.21, 5.5.0.19, 5.6.0.13, and later. Context: drv\_pmem Introduced: 4.9.0 Removed: \- --- #### `namespace out of space` `WARNING` Message: ```bash {namespace_name} out of space ``` Description: Indicates a shortage of free storage blocks. See [How do I recover from Available Percent Zero?](https://support.aerospike.com/s/article/How-do-I-recover-from-Available-Percent-Zero). To save log space, this message is logged with a `(repeated: nnn)` prefix just once per [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) during periods of repetition. Context: drv\_pmem Introduced: 5.2.0 Removed: \- Parameters: namespace Namespace being written to --- #### `{namespace_name} write rejected` `DETAIL` Message: ```bash {namespace_name} write: size 9437246 - rejecting 1142f0217ababf9fda5b1a4de66e6e8d4e51765e ``` Description: Most likely appearing as a result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). The record’s digest is the last item in the log entry. Context: drv\_pmem Introduced: 5.2.0 Removed: \- Parameters: namespace Namespace being written to --- size Total size of the record that was rejected --- #### `tomb raider unmark: detecting cenotaphs (PMEM)` `INFO` Message: ```bash INFO (drv_pmem): {} tomb raider detecting cenotaphs using threads over devices ... ``` Description: Logged at the start of the unmark phase of a tomb raider cycle on PMEM storage. `` is the number of threads in use; `` is the number of devices. See [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads). Context: drv\_pmem Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- N Number of unmark threads for this phase. --- D Number of devices being scanned. --- ## Drv\_ssd #### `arenax: too many chunks` `CRITICAL` Message: ```bash (arenax): (arenax_ee.c:98) too many chunks ``` Description: This error occurs when the primary index has been misconfigured [`partition-tree-sprigs`](https://aerospike.com/docs/database/reference/config#namespace__partition-tree-sprigs) for an all-flash namespace. To size all-flash installations adequately, you must account for the number of `partition-tree-sprigs` when you estimate the size of the index. Each sprig uses no more than a single 4 KiB chunk. Also, verify that all index entries are stored at the desired fill fraction, which defines the level to which a sprig is filled. This allows for some expansion without overfilling and consuming more than one chunk per sprig. See the [Capacity Planning Guide](https://aerospike.com/docs/database/manage/planning/capacity) for more information. Context: drv\_ssd Introduced: 4.2.0.2 Removed: \- --- #### `drv_ssd bad flat record` `WARNING` Message: ```bash bad flat record {digest} straddle-flush-boundary: {yes|no} end-marker: {good|bad} start {N} end {N} flush {N} ``` Description: Emitted during SSD cold start when a stored record fails the flat-format checks: bin data does not parse as valid. The log still reports whether the end marker region looks intact (`end-marker: good`) or not (`bad`). A good marker with corrupt middle bytes can indicate selective damage or a torn write, whereas a bad marker often goes with truncation or a more severely damaged write. As with the companion `bad end marker` case, `start`, `end`, and `flush` (the namespace [`flush-size`](https://aerospike.com/docs/database/reference/config#namespace__flush-size)) place the record on disk; `straddle-flush-boundary` calls out whether the span crosses a flush boundary, which is a common signature of a crash mid-flush where only the first portion of the record was persisted. Context: drv\_ssd Introduced: \- Removed: \- Occurs: SSD namespace [cold restart](https://aerospike.com/docs/database/manage/database/cold-start) while scanning persistent storage for records. Parameters: digest Identifier for the affected record. --- straddle-flush-boundary Whether the record spans a [`flush-size`](https://aerospike.com/docs/database/reference/config#namespace__flush-size) boundary—`yes` often aligns with a partial flush, `no` with other causes. --- end-marker Whether the end marker still looks valid despite the bad bin data. `good` indicates the tail was written but something earlier is wrong; `bad` indicates the marker is missing or invalid. --- start Byte offset where the record begins on the device. --- end Byte offset where the record ends on the device (inclusive). --- flush Configured flush size for the namespace—the unit used with `start` and `end` to interpret alignment to flush boundaries. --- #### `can't add record to index` `CRITICAL` Message: ```bash can't add record to index ``` Description: For the `index-type flash` configuration, indicates that the mount points have run out of space. You may need to delete the arena files manually and run `fsck` on the disk partitions. Context: drv\_ssd Introduced: \- Removed: \- --- #### `defrag_move_record: couldn't get swb` `WARNING` Message: ```bash defrag_move_record: couldn’t get swb ``` Description: Indicates the node has a shortage of free storage blocks. See [How do I recover from Available Percent Zero?](https://support.aerospike.com/s/article/How-do-I-recover-from-Available-Percent-Zero). Context: drv\_ssd Introduced: \- Removed: 5.7.0, 5.6.0.13, 5.4.0.21, 5.3.0.26, 5.2.0.36, 5.1.0.42 --- #### `swb buf valloc failed` `WARNING` Message: ```bash device /dev/sdb - swb buf valloc failed ``` Description: Indicates a shortage of memory. Make sure the nodes have enough memory. Context: drv\_ssd Introduced: \- Removed: \- Parameters: device Device that Aerospike was trying to read or write to at the time of the error. --- #### `device read complete: unique/replaced/older/expired/evicted/unowned/dropped/unparsable` `INFO` Message: ```bash device /dev/sdb: read complete: UNIQUE 20401681 (REPLACED 5619021) (OLDER 11905062) (EXPIRED 0) (EVICTED 0) (UNOWNED 0) (DROPPED 0) (UNPARSABLE 4) records ``` Context: drv\_ssd Introduced: 4.5.1.5 Removed: \- Parameters: device Name of the device for which the following stats apply. The stats are a summary of a cold start that read the entire device. --- UNIQUE Total number of unique records loaded from this device. --- REPLACED Number of records that replaced a version loaded earlier during the device scan (won the conflict resolution). --- OLDER Number of records that were skipped because a newer version was loaded earlier during the device scan (lost the conflict resolution). --- EXPIRED Number of records skipped because they were expired. --- EVICTED Number of records skipped because they were evicted. --- UNOWNED Number of records found in partitions no longer owned by this node. This parameter appears only in Database releases 7.1.0.19, 7.2.0.13, 8.0.0.11, 8.1.0.2, and later. --- DROPPED Number of records found in partitions currently owned by this node but are from previously dropped primary index trees. This parameter appears only in Database releases 7.1.0.19, 7.2.0.13, 8.0.0.11, 8.1.0.2, and later. --- UNPARSABLE Number of records skipped because they were unparsable. --- #### `read_complete: added expired` `INFO` Message: ```bash device /dev/sdb: read_complete: added 0 expired 0 ``` Context: drv\_ssd Introduced: 4.5.1.5 Removed: \- Parameters: device Name of the device for which the following stats apply. The stats are a summary of a cold start that read the entire device. --- added Total number of unique records loaded from this device. --- expired Number of records skipped because they were expired. --- #### `defrag: rblock generation mismatch` `WARNING` Message: ```bash device /dev/sdd defrag: rblock_id 952163461 generation mismatch (4:3) :0xc0b12ffb353e0385179f39c85edc7791264f11aa ``` Description: This can occur when the generation value for the index has been advanced while the generation value for the record on the drive has not advanced. This happened in some very old Aerospike Database versions. The error indicates that defrag found this discrepancy, and it should appear only once as the defrag process resolves the discrepancy. Context: drv\_ssd Introduced: \- Removed: \- Parameters: device Disk where this occurred. --- rblock\_id Identifier of the disk block being defragged. --- mismatch Generation of the record in the index and on disk, respectively. --- digest Digest of the record in question. --- #### `AP partition versions vs strong-consistency` `CRITICAL` Message: ```bash device has AP partition versions but 'strong-consistency' is configured ``` Description: A namespace has data that was written while it was in AP mode, but is being started in SC mode. This is not permitted. See [`strong-consistency`](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) for details. Context: drv\_ssd Introduced: 4.0.0 Removed: \- --- #### `device not encrypted but key file configured` `CRITICAL` Message: ```bash device not encrypted but encryption key file /etc/aerospike/key-256.dat is configured ``` Description: This happens when a block device has its header configured using `asd` without encryption enabled. This can happen if you: - zeroized the device. - started `asd` without enabling encryption in `aerospike.conf`. - took down `asd`. - restarted `asd` with encryption enabled in `aerospike.conf`. To resolve this and use encryption, zeroize again and reinitialize the drive: - Take down `asd`. - [Zeroize](https://aerospike.com/docs/database/manage/planning/ssd/manage) the device again. - Start `asd` with encryption enabled in `aerospike.conf`. Context: drv\_ssd Introduced: \- Removed: \- --- #### `bad device-id on nvme device` `CRITICAL` Message: ```bash /dev/nvme0n1p1: bad device-id 3192497567 ``` Description: Appears in cases of device corruption or if a device was not erased before starting the Aerospike service. See [SSD Initialization](https://aerospike.com/docs/database/manage/planning/ssd/manage) and [SSD Setup](https://aerospike.com/docs/database/manage/planning/ssd/setup) for details and recommended steps. In case of system log corruption, the device might need to be replaced. Context: drv\_ssd Introduced: 4.0.0 Removed: \- --- #### `device read failed - I/O error` `CRITICAL` Message: ```bash /dev/nvme0n1p1: read failed: errno 5 (Input/output error) ``` Description: Indicates an error during a system call to the storage device. Depending on the transaction path where this occurs, the database could abort if the integrity of the underlying data is not known (typically on write transactions). In case of corruption seen on system logs, the firmware version should be checked or the device might need to be replaced. Context: drv\_ssd Introduced: \- Removed: \- --- #### `init defrag profile` `INFO` Message: ```bash /dev/nvme3n1 init defrag profile: 3,1,1,2,0,0,1,0,3,1,0,0,0,0,1,1,2,0,0,1,1,1,0,0,0,0,2,1,3,4,2,1,1,2,2,3,1,4,3,3,3,4,2,1,3,3,2,0,4,2 ``` Description: At startup, the initial defragmentation profile for a device. This represents the number of blocks at each percentage all the way to the [`defrag-lwm-pct`](https://aerospike.com/docs/database/reference/config#namespace__defrag-lwm-pct). In this example, 50 buckets (default `defrag-lwm-pct` set to 50) are shown, including: 3 blocks that are less than 1% full, 1 that is between 1 and 2% full, all the way to 2 blocks that are between 49 and 50% full (last number). Blocks that are more than 50% full are not be eligible to be defragmented. Context: drv\_ssd Introduced: \- Removed: \- --- #### `device init wblocks pristine free-q defrag-q` `INFO` Message: ```bash /dev/nvme3n1 init wblocks: pristine-id 1464407 pristine 109155 free-q 591298, defrag-q 75 ``` Description: At startup, the status of unwritten blocks, free blocks, and blocks on the defrag queue. The sum of free-q and pristine blocks indicates the total space available for writing, which would match the number of free wblocks). Context: drv\_ssd Introduced: \- Removed: \- Parameters: pristine-id The ID of the first unwritten (pristine) block on the disk. --- pristine The number of completely unwritten (pristine) blocks on the disk. --- free-q The number of blocks that have been through the defragmentation process and are available to be re-written. --- defrag-q The number of blocks that are awaiting defragmentation on the defrag queue. --- #### `drv_ssd bad end marker` `WARNING` Message: ```bash bad end marker for {digest} straddle-flush-boundary: {yes|no} start {N} end {N} flush {N} ``` Description: On SSD cold start the server replays on-device records while rebuilding the primary index. This warning means bin data for the record could be read and parsed, but the record’s end marker did not match what the engine expected, so the entry is treated as invalid. The line includes byte offsets on the device (`start` through `end`, inclusive) and the namespace [`flush-size`](https://aerospike.com/docs/database/reference/config#namespace__flush-size) (`flush`), so you can see exactly how the record lines up with flush boundaries. When `straddle-flush-boundary` is `yes`, the record crosses a flush boundary; that pattern often fits a crash or power loss while only the leading portion of the write was persisted. When it is `no`, the problem is less likely to be a partial flush alone. Look for other corruption or hardware issues. In rare upgrade scenarios where only part of a device header was cleared before moving to Database 6.x, similar messages can appear; those are usually benign if newer versions of the same keys were written afterward. Context: drv\_ssd Introduced: \- Removed: \- Occurs: SSD namespace [cold restart](https://aerospike.com/docs/database/manage/database/cold-start) while scanning persistent storage for records. Parameters: digest Identifier for the affected record. --- straddle-flush-boundary Whether the record spans a [`flush-size`](https://aerospike.com/docs/database/reference/config#namespace__flush-size) boundary. `yes` often suggests a mid-flush failure with only a prefix durable; `no` points away from that specific failure mode. --- start Byte offset where the record begins on the device. --- end Byte offset where the record ends on the device (inclusive). --- flush Configured flush size for the namespace—the unit used with `start` and `end` to interpret alignment to flush boundaries. --- #### `shadow device too small` `CRITICAL` Message: ```bash (drv_ssd.c:3290) shadow device /dev/sdc is smaller than main device ``` Description: This can occur if the size of the shadow EBS device is not at least equal to the size of the direct-attached SSD. Otherwise, when you try to start your Aerospike cluster, it logs the critical error `shadow device is smaller than main device` and fails to start. Context: drv\_ssd Introduced: \- Removed: \- --- #### `encryption key or algorithm mismatch` `WARNING` Message: ```bash encryption key or algorithm mismatch ``` Description: At startup, this error indicates that the [`encryption key file`](https://aerospike.com/docs/database/reference/config#namespace__encryption-key-file) used previously to encrypt data on the storage device does not match the file currently provided. Context: drv\_ssd Introduced: \- Removed: \- --- #### `block extends over read size` `WARNING` Message: ```bash error: block extends over read size: foff 5242880 boff 1047552 blen 1392 ``` Description: The device likely has a bad sector. If this issue occurs frequently, replace the device. Context: drv\_ssd Introduced: \- Removed: \- Parameters: foff Offset of file containing malformed block. --- boff Offset of malformed block. --- blen Length of malformed block. --- #### `get_key: failed as_storage_record_read_ssd()` `WARNING` Message: ```bash get_key: failed as_storage_record_read_ssd() ``` Description: Symptom of having run out of storage space. Resolved by a cold start. Context: drv\_ssd Introduced: \- Removed: \- --- #### `get_key: failed ssd_read_record()` `WARNING` Message: ```bash get_key: failed ssd_read_record() ``` Description: Aerospike was not able to read the record from storage. This may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure) for more information. Context: drv\_ssd Introduced: \- Removed: \- --- #### `load_bins: failed ssd_read_record()` `WARNING` Message: ```bash load_bins: failed ssd_read_record() ``` Description: Aerospike was not able to read the record from storage. This may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure?) for more information. Context: drv\_ssd Introduced: \- Removed: \- --- #### `load_n_bins: failed ssd_read_record()` `WARNING` Message: ```bash load_n_bins: failed ssd_read_record() ``` Description: Aerospike was not able to read the record from storage. This may indicate a hardware failure. See [What is the expected behaviour when an Aerospike node experiences an SSD hardware failure?](https://support.aerospike.com/s/article/What-is-the-expected-behaviour-when-an-Aerospike-node-experiences-an-SSD-hardware-failure?) for more information. Context: drv\_ssd Introduced: \- Removed: 5.1.0 --- #### `metadata mismatch - removing digest` `WARNING` Message: ```bash metadata mismatch - removing ``` Description: This message indicates the change in a primary index bit and was introduced as a fix to AER-6335. The issue would occur for XDR-enabled namespaces upgrading from a 4.9 Database prior to 4.9.0.19. The workaround is to proceed with a rolling cold-restart. Context: drv\_ssd Introduced: 4.9.0.19 Removed: \- --- #### `namespace waiting for defrag` `INFO` Message: ```bash namespace NS waiting for defrag: 5 pct available, waiting for 10 ... ``` Description: The node is [stuck in a defrag loop at startup](https://aerospike.com/docs/database/advanced/troubleshoot/startup#stuck-in-defrag-loop-at-startup) where it cannot [defragment](https://support.aerospike.com/s/article/How-does-Defragmentation-work-in-Aerospike) enough to get [`device_free_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__device_free_pct) below [`defrag-startup-minimum`](https://aerospike.com/docs/database/reference/config#namespace__defrag-startup-minimum). Context: drv\_ssd Introduced: \- Removed: \- Parameters: namespace The namespace impacted. --- avail pct Current available percent. --- required pct Target available percent to reach in order to start up. --- #### `namespace loaded: device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 device-pcts (20, 30, 25) ``` Description: Objects that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_ssd Removed: \- --- #### `namespace loaded: objects/tombstones/device-pcts` `INFO` Message: ```bash {namespace} loaded: objects 12345 tombstones 77 device-pcts (20, 30, 25) ``` Description: Objects and tombstones that have been loaded during a cold start, and the percentage of each device that has been scanned. The device percentages are listed in the same order as the devices are defined in the config file. Context: drv\_ssd Removed: \- --- #### `namespace out of space` `WARNING` Message: ```bash {namespace} out of space ``` Description: Indicates a shortage of free storage blocks. See [How do I recover from Available Percent Zero?](https://support.aerospike.com/s/article/How-do-I-recover-from-Available-Percent-Zero). To save log space, this message is logged with a `(repeated: nnn)` prefix just once per [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) during periods of repetition. Context: drv\_ssd Introduced: 5.2.0 Removed: \- Parameters: {namespace} Namespace being written to. --- #### `defrag: drive totally full - waiting for wblocks` `WARNING` Message: ```bash {namespace_name} defrag: drive Drive1 totally full - waiting for vacated wblocks to be freed ``` Description: Indicates that the node has no free storage blocks. In this case, the defragmentation process waits until a free block is available. See \[How to Recover Contiguous Free Blocks\][How do I recover from Available Percent Zero?](https://support.aerospike.com/s/article/How-do-I-recover-from-Available-Percent-Zero) for more information. To save log space, this message is logged with a `(repeated: nnn)` prefix just once per [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) during periods of repetition. Context: drv\_ssd Introduced: 5.7.0, 5.6.0.13, 5.4.0.21, 5.3.0.26, 5.2.0.36, 5.1.0.42 Removed: \- Parameters: {namespace\_name} Affected namespace name. --- drive Affected storage device name. --- #### `namespace device shutdown not clean` `INFO` Message: ```bash {namespace_name} device /dev/sda prior shutdown not clean ``` Description: Indicates that the previous shutdown was not trusted. The node must perform a [cold start](https://aerospike.com/docs/database/manage/database/cold-start). Context: drv\_ssd Introduced: \- Removed: \- --- #### `namespace device stats: write-q defrag-q tomb-raider-read` `INFO` Message: ```bash {namespace_name} /dev/sda: used-bytes 296160983424 free-wblocks 885103 write-q 0 write (12659541,43.3) defrag-q 0 defrag-read (11936852,39.1) defrag-write (3586533,10.2) shadow-write-q 0 tomb-raider-read (13758,598.0) ``` Context: drv\_ssd Introduced: 3.10.0 Removed: \- Parameters: {namespace} Name of the namespace the device and stats belong to. --- /dev/sda Name of the device for which the following stats apply. --- used-bytes Number of bytes on this device that are in use. Corresponds to the [`storage-engine.device[ix].used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.used_bytes) statistic. --- free-wblocks The number of wblocks that are free (the `device_available_pct`). Corresponds to the [`storage-engine.device[ix].free_wblocks`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.free_wblocks) statistic. --- write-q Number of write buffers pending to be written to the SSD. When this reaches the [`max-write-cache`](https://aerospike.com/docs/database/reference/config#namespace__max-write-cache) configured value (default 64M), ‘device overload’ errors are returned and ‘queue too deep’ warnings are printed on the server log. Corresponds to the [`storage-engine.device[ix].write_q`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.write_q) statistic. --- write Total number of SSD write buffers written to this device since the Aerospike server started, including defragmentation, and the number of write buffers written per second. Corresponds to the [`storage-engine.device[ix].writes`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.writes) statistic. Does not include partial flushes at [`flush-max-ms`](https://aerospike.com/docs/database/reference/config#namespace__flush-max-ms). --- defrag-q Number of wblocks pending defragmentation. These are blocks that have fallen below the `defrag-lwm-pct` and are waiting to be read and have their relevant content recombined in a fresh streaming write buffer. The `defrag-sleep` setting controls the sleep period in between each block being read (default 1ms). Corresponds to the [`storage-engine.device[ix].defrag_q`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.defrag_q) statistic. --- defrag-read Total number of write blocks that have been sent to the defragmentation queue (defrag-q) and read by the defragmentation thread on this device, and the normalization to the average number of wblocks processed per second during the interval at which this message is logged. Usually, defrag-q is at 0 and wblocks are read as they are put on the defrag-q. In such cases, defrag-read represents the number of wblocks read by the defragmentation thread. Corresponds to the [`storage-engine.device[ix].defrag_reads`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.defrag_reads) statistic. --- defrag-write Total number of write blocks written by defragmentation on this device since the Aerospike server started, and the number of wblocks written per second (subset of write). Corresponds to the [`storage-engine.device[ix].defrag_writes`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.defrag_writes) statistic. --- shadow-write-q Number of write buffers pending to be written to the shadow device (only printed when a [shadow device](https://aerospike.com/docs/database/manage/namespace/storage/config/#setup-for-shadow-device) is configured). When this reaches the configured [`max-write-cache`](https://aerospike.com/docs/database/reference/config#namespace__max-write-cache) value (default 64M), ‘device overload’ errors are returned and `queue too deep` warnings are printed to the server log. Corresponds to the [`storage-engine.device[ix].shadow_write_q`](https://aerospike.com/docs/database/reference/metrics#namespace__storage-engine.device%5Bix%5D.shadow_write_q) statistic. --- tomb-raider-read Total number of blocks read by the tomb-raider in the current cycle, and the current number of wblocks read per second. Only printed when the tomb-raider is active. --- #### `durable delete queue too deep` `WARNING` Message: ```bash {namespace_name} durable delete fail: queue too deep: exceeds max 544 ``` Description: Indicates that the disks are not keeping up with the load placed upon them, although the disks themselves are not necessarily faulty or nearing end of life. See [Why do I see warning - queue too deep](https://support.aerospike.com/s/article/Warning-write-fail-queue-too-deep) for more information. Context: drv\_ssd Introduced: 5.7.0 Removed: \- --- #### `immigrate fail: queue too deep` `WARNING` Message: ```bash {namespace_name} immigrate fail: queue too deep: exceeds max 576 ``` Description: Indicates that the disks are not keeping up with the load placed upon them, although the disks themselves are not necessarily faulty or nearing end of life. See [Why do I see warning - queue too deep](https://support.aerospike.com/s/article/Warning-write-fail-queue-too-deep) for more information. Context: drv\_ssd Introduced: 5.7.0 Removed: \- --- #### `read_ssd: invalid rblock_id` `WARNING` Message: ```bash {namespace_name} read_ssd: invalid rblock_id :0x0c0008d663318a674a7bd379f6efd3bb1f55141d ``` Description: Indicates that a record with an invalid read-block is being read. This can happen if a node runs out of memory and swb cannot be allocated. It can also happen on some earlier database versions if a node runs out of [`device_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__device_available_pct) before checking upfront for the available free blocks. A cold start of the node should resolve the issue. Context: drv\_ssd Introduced: \- Removed: \- Parameters: NAMESPACE Namespace the record with invalid read-block resides in. --- rblock\_id Digest of the record with invalid read-block. --- #### `udf queue too deep` `WARNING` Message: ```bash {namespace_name} udf fail: queue too deep: exceeds max 512 ``` Description: All UDF writes fail by design. Context: drv\_ssd Introduced: 5.7.0 Removed: \- --- #### `write fail: queue too deep` `WARNING` Message: ```bash {namespace_name} write fail: queue too deep: exceeds max 512 ``` Description: Indicates that the disks are not keeping up with the load placed upon them, although the disks themselves are not necessarily faulty or nearing end of life. See [Why do I see warning - queue too deep](https://support.aerospike.com/s/article/Warning-write-fail-queue-too-deep) for more information. Context: drv\_ssd Introduced: 3.0.0 Removed: \- --- #### `write: rejecting record` `DETAIL` Message: ```bash {namespace_name} write: size 9437246 - rejecting 1142f0217ababf9fda5b1a4de66e6e8d4e51765e ``` Description: Most likely appears as a result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). The record’s digest is the last item in the log entry. Context: drv\_ssd Introduced: 5.2.0 Removed: \- Parameters: namespace Namespace being written to. --- size Total size of the record that was rejected. --- #### `tomb raider unmark: detecting cenotaphs (SSD)` `INFO` Message: ```bash INFO (drv_ssd): {} tomb raider detecting cenotaphs using threads over devices ... ``` Description: Logged at the start of the unmark phase of a tomb raider cycle on SSD storage. `` is the number of threads in use; `` is the number of devices. See [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads). Context: drv\_ssd Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- N Number of unmark threads for this phase. --- D Number of devices being scanned. --- #### `read: bad block magic` `WARNING` Message: ```bash read: bad block magic offset 303403269632 ``` Description: The SSD is corrupted and the expected value of a block does not match the actual value. This issue has three potential causes: - The names of raw devices were swapped on server reboot. For example, the storage pointed to by `/dev/sda` is now `/dev/sdb`. See [How to configure storage device to use the disk WWID?](https://support.aerospike.com/s/article/How-to-configure-storage-device-to-use-the-disk-WWID) to learn about configuring persistent device names. - The [`storage-engine`](https://aerospike.com/docs/database/manage/namespace/storage/config) configuration in `aerospike.conf` has changed to reorder devices. - Hardware failure and data corruption. Context: drv\_ssd Introduced: \- Removed: \- Parameters: offset Location in the device where the mismatch was noticed. --- #### `read failed - i/o error` `WARNING` Message: ```bash read failed: expected 512 got -1: fd 9900 data 0x7f277981b000 errno 5 ``` Description: Aerospike tried to read 512 bytes from the device and the read() call returned -1 (error) with errno 5, which is an I/O error. This indicates a potential hardware issue. Context: drv\_ssd Introduced: \- Removed: \- Parameters: expected Size of block requested from disk. --- fd File descriptor used for read. --- data Offset of block. --- #### `read_all: failed as_storage_record_read_ssd()` `INFO` Message: ```bash read_all: failed as_storage_record_read_ssd() ``` Description: Result of having run out of storage space. Resolved by a cold start. Context: drv\_ssd Introduced: \- Removed: \- --- #### `ssd_read: record has no block associated` `WARNING` Message: ```bash ssd_read: record b98946e3d616790 has no block associated, fail ``` Description: This is a result of having run out of space. The record was written to the index but not flushed to disk, creating this inconsistency. You can resolve this issue with a cold start. Context: drv\_ssd Introduced: \- Removed: \- Parameters: record The record’s hashed key. --- #### `write bins: couldn't get swb` `WARNING` Message: ```bash write bins: couldn’t get swb ``` Description: Indicates a shortage of free storage blocks. See [How do I recover from Available Percent Zero?](https://support.aerospike.com/s/article/How-do-I-recover-from-Available-Percent-Zero). Context: drv\_ssd Introduced: \- Removed: 5.2.0 --- #### `write rejected for being too large` `DETAIL` Message: ```bash write: size 9437246 - rejecting digest:0xd751c6d7eea87c82b3d6332467e8bc9a3c630e13 ``` Description: Appears with the `WARNING` message about `failed as_storage_record_write()` for exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). Context: drv\_ssd Introduced: 3.16.0 Removed: 5.2.0 Parameters: size Total size of the record that was rejected. --- digest Digest of the record that was rejected. --- ## Exchange #### `blocking client transactions in orphan state` `WARNING` Message: ```bash blocking client transactions in orphan state! ``` Description: The node is not currently part of any cluster and will not accept any transactions from clients that are connected to it, meaning clients cannot access any partitions on the node. Context: exchange Introduced: \- Removed: \- --- #### `error sending exchange data` `WARNING` Message: ```bash error sending exchange data ``` Description: Failure exchanging partition maps with another node because the node is down or disconnected from the network and unable to receive messages. Context: exchange Introduced: \- Removed: \- --- #### `received duplicate exchange data` `INFO` Message: ```bash received duplicate exchange data from node 783f4ac2fbb57e81 ``` Description: Another node has resent `exchange data` because it did not receive an acknowledgment from this node within half the [heartbeat interval](https://aerospike.com/docs/database/reference/config#network__interval). This is most likely due to a networking issue. Context: exchange Introduced: \- Removed: \- Parameters: node NodeID of the origin of the unacknowledged exchange data --- #### `duplicate ready to commit message` `INFO` Message: ```bash received duplicate ready to commit message from node 783f4ac2fbb57e81 ``` Description: A node has resent the `ready to commit` message because it did not receive an acknowledgment from this node within half the [heartbeat interval](https://aerospike.com/docs/database/reference/config#network__interval). This is most likely due to a networking issue between nodes in the cluster. Those `ready to commit` messages are sent by each node to the principal node when the `exchange data` has been completed on the node. The principal node also follows this pattern and sends itself such a `ready to commit` message. On a given node, the `exchange data` is done when the node has successfully sent its partition map to all the nodes in the cluster as well as received the partition map from each node in the cluster. This requires each node to ack back that it has received the `exchange data`. Only then would a node tell the principal that it is `ready to commit`. While the principal is waiting to receive the `ready to commit` from some nodes, other nodes would keep sending their `ready to commit` as they are ready and waiting. Therefore, the nodes for which such message is seen are the nodes that are ready and the other nodes would be the ones having potential issues completed their `exchange data`. Context: exchange Introduced: \- Removed: \- Parameters: node Node ID of the node that is continuing to send the ready to commit message as it is waiting for the principal to acknowledge. --- ## Exp #### `expression display null` `WARNING` Message: ```bash WARNING (exp): as_exp_display - exp is NULL ``` Description: Logged when the server attempts to display an expression that is NULL. Prior to Database 8.1.2, this was logged as “could not parse expressions”. The updated message more accurately describes the condition. Context: exp Introduced: 8.1.2 Removed: \- --- #### `build_count_sz invalid instruction` `WARNING` Message: ```bash WARNING (exp): build_count_sz - invalid instruction at offset ``` Description: An expression could not be compiled because an invalid instruction was encountered at the specified byte offset in the expression wire format. This typically indicates a client SDK version mismatch or a corrupt expression payload. Context: exp Introduced: 8.1.2 Removed: \- Parameters: offset Byte offset of the invalid instruction in the expression. --- #### `build_in_list invalid arg type` `WARNING` Message: ```bash WARNING (exp): build_in_list - error invalid arg type () ``` Description: An IN\_LIST expression could not be built because an argument has an invalid type. Verify that the expression arguments match the expected types for the IN\_LIST operation. Context: exp Introduced: 8.1.2 Removed: \- Parameters: code Internal error code. --- type Numeric type identifier of the invalid argument. --- name Human-readable name of the type. --- #### `build_cond - mismatched type at default condition` `WARNING` Message: ```bash build_cond - error 4 mismatched type (TYPE_NAME) expected type () at default condition ``` Description: Example: WARNING (exp): (exp.c:2168) build\_cond - error 4 mismatched type 5 (map) expected type 0 (nil) at default condition. In this example the type is map and the type is nil. [`cond`](https://aerospike.com/docs/develop/expressions/declare#cond) expression was illegally invoked by the client with mixed return types. The `cond` expression requires that all return types be the same with the exception of the [`unknown`](https://aerospike.com/docs/develop/expressions/declare#unknown) type. Context: exp Introduced: 5.6.0 Removed: \- --- #### `eval_in_list list expected` `WARNING` Message: ```bash WARNING (exp): eval_in_list - list expected, got ``` Description: At runtime, an IN\_LIST expression received a non-list value where a list was expected. Check the expression to ensure the argument resolves to a list type. Context: exp Introduced: 8.1.2 Removed: \- Parameters: type Numeric type identifier of the value that was received instead of a list. --- #### `predexp deprecated - use expressions API` `WARNING` Message: ```bash predexp deprecated - use new expressions API ``` Description: Indicates use of the deprecated [`Predicate Expressions`](https://aerospike.com/docs/develop/expressions) API, which was replaced in Database 5.2.0 by the [`Aerospike Expressions`](https://aerospike.com/docs/develop/expressions) API. The deprecated API is removed in Database 6.0.0. This warning is logged no more than once per log ticker cycle. Context: exp Introduced: 5.6.0 Removed: \- --- ## Fabric #### `error creating fabric published endpoint list` `CRITICAL` Message: ```bash error creating fabric published endpoint list ``` Description: This issue can occur in the case of a failure in network interface initialization. Check the system logs for time of issue and verify the fabric network interface was initialized properly. Context: fabric Introduced: \- Removed: \- --- #### `msg_read: could not deliver message` `INFO` Message: ```bash msg_read: could not deliver message type 1 ``` Description: Heartbeat message handler is not registered on the node. The message should disappear soon after the node joins the cluster. Otherwise, restart the `asd` service on the node. Context: fabric Introduced: \- Removed: \- Parameters: message type Internal code. --- #### `no IPv4 addresses configured for fabric` `WARNING` Message: ```bash no IPv4 addresses configured for fabric ``` Description: This issue can occur in the case of a failure in network interface initialization. Check the system logs for time of issue and verify the fabric network interface was initialized properly. Context: fabric Introduced: \- Removed: \- --- #### `node fds via_connect rw/ctrl/bulk/meta live q` `INFO` Message: ```bash node b5 fds {via_connect={rw=8 ctrl=1 bulk=2 meta=1} all=24} live 1 q {rw=0 ctrl=0 bulk=0 meta=0} ``` Description: Provides details on the number of fabric connections to each node in the cluster and pending outbound messages for each. Context: fabric Introduced: \- Removed: \- Occurs: This log line is generated when the info command `dump-fabric:verbose=true` is issued. Parameters: node The node to which this node is connected through fabric. --- fds rw= ctrl= bulk= meta= The number of file descriptors to the node, broken up by channel: - replica writes - includes proxies and duplicate resolution - control - for clustering - bulk - for migrations - meta - for system metadata (SMD) --- fds all= The total number of file descriptors to the node, typically double the sum of rw, ctrl, bulk and meta as each node establishes connections for each channel from each side. --- q rw= ctrl= bulk= meta= The number of pending outbound messages to the node, broken up by channel. --- #### `r_msg_sz exceeds fc->r_membuf` `WARNING` Message: ```bash r_msg_sz > sizeof(fc->r_membuf) 1048582 ``` Description: This is from the run\_fabric\_accept thread rather than the run\_fabric\_recv thread. This thread is responsible for accepting new connections. These messages are only expected to store the NodeID and the ChannelID, which would always be only a few bytes and never expected to be over 1MiB. This WARNING is likely to indicate non-Aerospike traffic on the fabric port. Verify that the fabric port (default 3001) is not exposed. Context: fabric Introduced: \- Removed: \- Parameters: r\_msg\_sz --- ## Flat #### `record too small` `WARNING` Message: ```bash record too small 0 ``` Description: Indicates that an inbound record for migration is corrupt. Appears with the `WARNING` message about `handle insert: got bad record` and is documented in [`Why Do I See Stalled Migrations And "record too small" Errors In The Log?`](https://support.aerospike.com/s/article/Why-Do-I-See-Stalled-Migrations-And-record-too-small-Errors-In-The-Log) Context: flat Introduced: \- Removed: \- --- ## Hardware #### `skew_monitor: hlc not in sync` `INFO` Message: ```bash (skew_monitor.c:685) node bb90adf81f64eb0 HLC not in sync - hlc 112495319903567872 self-hlc 112644426202057622 diff 2275181556 ``` Description: A strong consistency namespace can hit stop writes if [clock\_skew\_stop\_writes](https://aerospike.com/docs/database/reference/metrics#namespace__clock_skew_stop_writes) is in effect when [cluster\_clock\_skew\_ms](https://aerospike.com/docs/database/reference/metrics#node_stats__cluster_clock_skew_ms) is above the [cluster\_clock\_skew\_stop\_writes\_sec](https://aerospike.com/docs/database/reference/metrics#node_stats__cluster_clock_skew_stop_writes_sec) threshold. Context: hardware Introduced: 4.0.0 Removed: \- --- #### `failed to submit command to NVMe device` `WARNING` Message: ```bash failed to submit command to /dev/nvme0: x109 ``` Description: Indicates that the underlying NVMe device does not support the health information check. To suppress this message, set log level to critical for the hardware context. Context: hardware Introduced: 4.3.0 Removed: \- --- ## Hb #### `broken pipe` `INFO` Message: ```bash Broken pipe (under normal network conditions) ``` Description: This message appears before any namespace-specific shutdown messages in the normal shutdown sequence for mesh network nodes. It indicates that the node’s heartbeat is being stopped to prepare for prompt removal of the node from the cluster in case the shutdown process is lengthy and the node was not quiesced. Context: hb Introduced: 5.3.0 Removed: \- --- #### `socket without associated channel` `WARNING` Message: ```bash Found a socket 0x7f0979812460 without an associated channel. ``` Description: This warning occurs in rare cases where a network failure causes two error events on the same socket. No action is required. Context: hb Introduced: \- Removed: \- Parameters: socket HEX number identifying the socket within the OS. --- #### `timeout while connecting` `WARNING` Message: ```bash Timeout while connecting ``` Description: A peer node could not be reached on the heartbeat port. The node may be down or experiencing network issues. Context: hb Introduced: \- Removed: \- --- #### `closing mesh heartbeat sockets` `INFO` Message: ```bash closing mesh heartbeat sockets ``` Description: This message appears early in the normal shutdown sequence for mesh network nodes, before any namespace-specific shutdown messages. It indicates that the node’s heartbeat is being stopped so the node is promptly removed from the cluster in case the shutdown process is lengthy and the node was not quiesced. Context: hb Introduced: 5.3.0 Removed: \- --- #### `closing multicast heartbeat sockets` `INFO` Message: ```bash closing multicast heartbeat sockets ``` Description: This message appears early in the normal shutdown sequence for multicast network nodes, before any namespace-specific shutdown messages. It indicates that the node’s heartbeat is being stopped so the node is promptly removed from the cluster in case the shutdown process is lengthy and the node was not quiesced. Context: hb Introduced: 5.3.0 Removed: \- --- #### `could not create heartbeat connection to node` `WARNING` Message: ```bash could not create heartbeat connection to node - 10.219.136.101 {10.219.136.101:3012} ``` Description: This warning can indicate that a cluster node is down. It is also possible that IP addresses in the cluster have changed, which can occur following a node restart. If the indicated node is expected to be part of the cluster, [troubleshoot](https://aerospike.com/docs/database/advanced/troubleshoot) as normal. Otherwise, follow the [`tip-clear`](https://aerospike.com/docs/database/reference/info#tip-clear) and [`services-alumni-reset`](https://aerospike.com/docs/database/reference/info#services-alumni-reset) steps from the [node removal instructions](https://aerospike.com/docs/database/manage/cluster/remove-node/) to clear the errors. Context: hb Introduced: \- Removed: \- Occurs: When a node is expected to be part of the cluster and cannot be reached on the heartbeat port. Parameters: IP:port IP and heartbeat port of the host that could not be reached. --- #### `error allocating multicast/mesh recv buffer` `WARNING` Message: ```bash error allocating space for [multicast/mesh] recv buffer of size 1195725862 on fd 773 ``` Description: Indicates that non-Aerospike traffic may be sending data to the heartbeat port on this machine and some bits have been misinterpreted in the Aerospike protocol as a large buffer size. Such requests can lead to denial of service or disruption of cluster traffic if they are frequent. We recommend that you block access to the heartbeat port from outside the cluster. Context: hb Introduced: \- Removed: \- Parameters: size Size of the message received (or the interpretation of a size). --- fd File descriptor of the socket the non-Aerospike message was received on. --- #### `heartbeat TLS client handshake failed` `WARNING` Message: ```bash heartbeat TLS client handshake failed - 10.219.136.101 {10.219.136.101:3012} ``` Description: The other node could be reached, but there was an error in setting up the TLS connection. Check the other log messages near this one for details. Context: hb Introduced: \- Removed: \- Parameters: address:ports Address(es) of the peer where the handshake failed. --- #### `heartbeat TLS server handshake failed` `WARNING` Message: ```bash heartbeat TLS server handshake with 10.11.12.13:3012 failed ``` Description: The other node could be reached, but there was an error in setting up the TLS connection. Check the other log messages near this one for details. Context: hb Introduced: \- Removed: \- Parameters: address:port Address of the peer where the handshake failed. --- #### `ignoring delayed heartbeat - timestamp later than expected` `WARNING` Message: ```bash ignoring delayed heartbeat - expected timestamp less than 1483213248012 but was 1483213252345 from node: BB9020011AC4202 ``` Description: A received heartbeat message was not generated during the last [heartbeat interval](https://aerospike.com/docs/database/reference/config#network__interval), which may indicate clock skew across the cluster. Context: hb Introduced: \- Removed: \- Parameters: expected ts Latest timestamp expected (in milliseconds since the Aerospike Epoch of 2010-01-01 00:00:00) --- actual ts The actual timestamp in the message. --- node Source of the message. --- #### `ignoring message from node - different cluster name` `WARNING` Message: ```bash ignoring message from BB9030011AC4202 with different cluster name(dev_cluster) ``` Description: A node that is not part of the cluster is sending heartbeats. It is possible that the sending node has this node listed incorrectly as a seed node in the `network.heartbeat` stanza of its `aerospike.conf`. Context: hb Introduced: \- Removed: \- Parameters: node ID ID of the node sending heartbeats. --- cluster name The cluster name that the sending node is presenting, which should help in identifying the node. --- #### `mesh size recv failed: connection timed out` `WARNING` Message: ```bash mesh size recv failed fd 361: Connection timed out ``` Description: An incomplete heartbeat message was received because there was a timeout in the socket connection. Context: hb Introduced: \- Removed: \- Parameters: fd ID number of the file descriptor the message was received on. --- error message Error message returned from the OS. --- #### `mesh message send failed - broken pipe` `WARNING` Message: ```bash sending mesh message to BB9030011AC4202 on fd 361 failed : Broken pipe ``` Description: The node sent a heartbeat message to a peer node after the connection broke. The first parameter may be 0 rather than a node ID if the node has not been able to retrieve the remote node ID. Context: hb Introduced: \- Removed: \- Occurs: Under rare circumstances, particularly in large clusters after a cluster event, the server may continue to use a heartbeat connection that is no longer usable, which repeatedly generates the Broken pipe message. In this situation, the error occurs on the same file descriptor after the following error: sending mesh message to bb9030011ac4202 on fd 361 failed : Connection reset by peer Note: if the error keeps occurring on the same file descriptor, the node is not able to re-join the cluster and must be restarted. Parameters: node Node that should have received the message. --- fd ID number of the file descriptor the message was sent on. --- error message Error message returned from the OS. --- #### `sending mesh message failed - No route to` `WARNING` Message: ```bash sending mesh message to BB9030011AC4202 on fd 361 failed : No route to host ``` Description: Tried to send a heartbeat message to a peer, but the socket had some problem. Context: hb Introduced: \- Removed: \- Parameters: node Node that should have received the message. --- fd ID number of the file descriptor the message was sent on. --- error message Error message returned from the OS. --- #### `unable to parse heartbeat message` `WARNING` Message: ```bash unable to parse heartbeat message on fd 361 ``` Description: Received a malformed message on the heartbeat port, possibly due to a non-Aerospike process unintentionally or maliciously trying to connect on the port. Context: hb Introduced: \- Removed: \- Parameters: fd ID number of the file descriptor the message was received on. --- #### `updating mesh endpoint address` `INFO` Message: ```bash updating mesh endpoint address from {10.219.136.101:3002} to {10.219.136.101:3002,10.219.148.101:3002} ``` Description: This info message indicates that the local node connected to one IP but received two in return. This is common in situations where nodes have multiple NICs and an [address](https://aerospike.com/docs/database/reference/config#network__address) is not specified in the `network.heartbeat` context. Context: hb Introduced: \- Removed: \- Parameters: original address Address connected to initially. --- new addresses List of addresses advertised by the node connected to. --- ## Index #### `could not allocate arena stage` `WARNING` Message: ```bash could not allocate 1073741824-byte arena stage 13: Cannot allocate memory ``` Description: Aerospike Enterprise edition allocates memory in 1GiB arenas for the primary index. This message indicates that the process failed to allocate the 13th arena. A contiguous 1GiB of memory should be available. The process continues to serve `read` and `update` transactions, but all new writes fail. This message is always be followed by the following warning: _**WARNING (index): (index.c:737)(repeated:xxx)arenax alloc failed**_ Context: index Introduced: \- Removed: \- --- ## Info #### `node-id cluster-size cluster-name` `INFO` Message: ```bash NODE-ID bb97f1d46894206 CLUSTER-SIZE 12 CLUSTER-NAME myCluster ``` Context: info Introduced: 3.9.0 Removed: \- Parameters: NODE-ID The generated node ID, based on the MAC address and the service port. --- CLUSTER-SIZE Number of nodes recognized by this node as being in the cluster. --- CLUSTER-NAME Name of the cluster. Appears as `null` for unnamed clusters. --- #### `bad cgroup-mem-tracking value` `WARNING` Message: ```bash WARNING (info): bad cgroup-mem-tracking value '' - ignoring ``` Description: Logged when an invalid value is provided for the [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) configuration parameter via set-config. The value must be `true` or `false`. Context: info Introduced: 8.1.2 Removed: \- Parameters: value The invalid value that was provided. --- #### `batch-index: batches delays` `INFO` Message: ```bash batch-index: batches (234,0,0) delays 0 ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Displayed periodically, every 10 seconds by default, and only if `batch-index` transactions have been issued on this node. Message details are aggregated across all namespaces. Parameters: batches Number of batch-index jobs since the server started (Success, Error, Timed out). Success means all the sub-transactions for the batch-index job were dispatched successfully. The sub-transactions for the batch-index job could still error or time out individually, even if the parent batch-index job reported a success status. Also, an unsuccessful parent batch-index job could have some of its sub-transactions processed with any resulting status. No correlation can be made between the status of a parent batch-index job and and the statuses of its sub-transactions. Related metrics: - [`batch_index_complete`](https://aerospike.com/docs/database/reference/metrics#node_stats__batch_index_complete) - [`batch_index_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__batch_index_error) - [`batch_index_timeout`](https://aerospike.com/docs/database/reference/metrics#node_stats__batch_index_timeout) --- delays Number of times the job’s response buffer has been delayed by the sending process’s WOULDBLOCK to avoid overflowing the buffer. Related metric: - [`batch_index_delay`](https://aerospike.com/docs/database/reference/metrics#node_stats__batch_index_delay) --- #### `cgroup-mem-tracking already set` `INFO` Message: ```bash INFO (info): Value of cgroup-mem-tracking is already ``` Description: Logged when a set-config command attempts to set [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) to its current value. No change is applied. Context: info Introduced: 8.1.2 Removed: \- Parameters: value Current value of the parameter (`true` or `false`). --- #### `cgroup-mem-tracking changed from false` `INFO` Message: ```bash INFO (info): Changing value of cgroup-mem-tracking from false to ``` Description: Logged when [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is dynamically changed from `false` to a new value via set-config. Context: info Introduced: 8.1.2 Removed: \- Parameters: value New value (`true` or `false`). --- #### `cgroup-mem-tracking changed from true` `INFO` Message: ```bash INFO (info): Changing value of cgroup-mem-tracking from true to ``` Description: Logged when [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is dynamically changed from `true` to a new value via set-config. Context: info Introduced: 8.1.2 Removed: \- Parameters: value New value (`true` or `false`). --- #### `tomb-raider-unmark-threads changed dynamically` `INFO` Message: ```bash INFO (info): Changing value of tomb-raider-unmark-threads of ns from to ``` Description: Logged when [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads) is successfully updated at runtime via the info command. Context: info Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- old Previous value. --- new New value. --- #### `early-fail ticker` `INFO` Message: ```bash early-fail: demarshal 0 tsvc-client 1 tsvc-from-proxy 0 tsvc-from-proxy-batch-sub 0 ``` Context: info Introduced: 4.5.1 Removed: \- Occurs: Ticker log line. Prints rolled-up notification every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only when any transaction failed early on this node. Cumulative since `asd` start and aggregated across all namespaces. Single-digit counts are likely not problematic. Prior to Database 7.2.0, the log line read `early-fail: demarshal 0 tsvc-client 1 tsvc-from-proxy 0 tsvc-batch-sub 0 tsvc-from-proxy-batch-sub 0 tsvc-udf-sub 0 tsvc-ops-sub 0` Parameters: demarshal Failure during the demarshal phase of a transaction. Metric: [`demarshal_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__demarshal_error). --- tsvc-client Indicates a failure for client initiated transactions, before getting to the namespace part. This can be due to authentication failure at the socket level, a missing or bad namespace, or an initial partition imbalance caused by a node that just started and has not yet joined the cluster. The latter would result in an `unavailable` error back to the client. Metric: [`early_tsvc_client_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_client_error). --- tsvc-from-proxy Indicates a failure for proxied transactions, before getting to the namespace part. This can be due to authentication failure at the socket level, a missing or bad namespace, or an initial partition imbalance caused by a node that just started and has not yet joined the cluster. The latter would result in an `unavailable` error. Metric: [`early_tsvc_from_proxy_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_from_proxy_error). --- tsvc-batch-sub Part of a batch sub transaction. Metric: [`early_tsvc_batch_sub_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_batch_sub_error). --- tsvc-from-proxy-batch-sub Part of a proxied batch sub transaction. Metric: [`early_tsvc_from_proxy_batch_sub_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_from_proxy_batch_sub_error). --- tsvc-udf-sub Part of a UDF sub transaction. Metric: [`early_tsvc_udf_sub_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_udf_sub_error). --- tsvc-ops-sub Part of a scan/query background ops sub transaction. Metric: [`early_tsvc_ops_sub_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__early_tsvc_ops_sub_error). --- #### `fabric-bytes-per-second ticker` `INFO` Message: ```bash fabric-bytes-per-second: bulk (1525,7396) ctrl (33156,46738) meta (42,42) rw (128,128) ``` Context: info Introduced: 3.11.1.1 Removed: \- Occurs: Fabric traffic statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Parameters: bulk Current transmit and receive rate for fabric-channel-bulk. This channel is used for record migrations during rebalance. --- ctrl Current transmit and receive rate for fabric-channel-ctrl. This channel distributes cluster membership change events and partition migration control messages. --- meta Current transmit and receive rate for fabric-channel-meta. This channel distributes System Meta Data (SMD) after cluster change events. --- rw Current transmit and receive rate for fabric-channel-rw (**r**ead/**w**rite). This channel is used for replica writes, proxies, duplicate resolution, and various other intra-cluster record operations. --- #### `failed to read /proc/meminfo` `WARNING` Message: ```bash WARNING (info): failed to read /proc/meminfo ``` Description: The server could not read `/proc/meminfo` to obtain host memory statistics. This can occur in restricted container environments. When [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is enabled, the server falls back to control group (cgroup) memory files. Context: info Introduced: 8.1.2 Removed: \- --- #### `fds (sockets) ticker` `INFO` Message: ```bash fds: proto (38553,57711444,57672891) heartbeat (27,553,526) fabric (648,2686,2038) ``` Context: info Introduced: 3.9.0 Removed: \- Parameters: proto Client connections statistics, in order: - [client\_connections](https://aerospike.com/docs/database/reference/metrics#node_stats__client_connections) - [client\_connections\_opened](https://aerospike.com/docs/database/reference/metrics#node_stats__client_connections_opened) - [client\_connections\_closed](https://aerospike.com/docs/database/reference/metrics#node_stats__client_connections_closed) Include connections that are reaped after idle (reaped connections correspond to [reaped\_fds](https://aerospike.com/docs/database/reference/metrics#node_stats__reaped_fds)), properly shutdown by the client (initiated a proper socket close) or preliminary packet parsing errors like unexpected headers, most of these would have a WARNING in the logs. --- heartbeat Heartbeat connections statistics, in order: - [heartbeat\_connections](https://aerospike.com/docs/database/reference/metrics#node_stats__heartbeat_connections), - [heartbeat\_connections\_opened](https://aerospike.com/docs/database/reference/metrics#node_stats__heartbeat_connections_opened), - [heartbeat\_connections\_closed](https://aerospike.com/docs/database/reference/metrics#node_stats__heartbeat_connections_closed) --- fabric Fabric connections statistics, in order: - [fabric\_connections](https://aerospike.com/docs/database/reference/metrics#node_stats__fabric_connections) - [fabric\_connections\_opened](https://aerospike.com/docs/database/reference/metrics#node_stats__fabric_connections_opened) - [fabric\_connections\_closed](https://aerospike.com/docs/database/reference/metrics#node_stats__fabric_connections_closed) --- #### `heartbeat-received ticker` `INFO` Message: ```bash heartbeat-received: self 887075 : foreign 35456447 ``` Context: info Introduced: 3.9.0 Removed: \- Parameters: self Number of heartbeats the current node has received from itself (should be 0 for mesh). --- foreign Number of heartbeats the current node has received from all other nodes combined. --- #### `histogram dump ticker` `INFO` Message: ```bash histogram dump: {ns-name}-{hist-name} (1344911766 total) msec (00: 1262539302) (01: 0049561831) (02: 0013431778) (03: 0007273116) (04: 0004299011) (05: 0003086466) (06: 0002182478) (07: 0001854797) (08: 0000312272) (09: 0000370715) ``` Description: In the example, 05:0003086466 implies 3,086,466 data points took between 16 and 32 milliseconds. You can access additional histograms by enabling microbenchmarks or storage-benchmarks, statically or dynamically, in the service context of the configuration. See [Monitoring latencies](https://aerospike.com/docs/database/observe/latency/) for monitoring latencies and details about the histograms. Context: info Introduced: 3.9.0 Removed: \- Occurs: Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Parameters: histogram dump Name of the histogram to follow for the {ns-name} namespace --- total Number of data points represented by this histogram (since the server started) --- N Number of data points within units (msec or bytes) greater than 2(N-1) and less than 2N for N>0, between 0 and 1 for N=0 --- #### `in-progress ticker` `INFO` Message: ```bash in-progress: info-q 5 rw-hash 0 proxy-hash 0 tree-gc-q 0 long-queries 0 ``` Context: info Introduced: 3.9.0 Removed: \- Parameters: tsvc-q Removed in 4.7.0.2. Number of transactions siting in the transaction queue, waiting to be picked up by a transaction thread. Corresponds to the `tsvc_queue` statistic. --- info-q Number of transactions on the info transaction queue. Corresponds to the [`info_queue`](https://aerospike.com/docs/database/reference/metrics#node_stats__info_queue) statistic. --- nsup-delete-q Removed in 4.5.1. Number of records queued up for deletion by the nsup thread. --- rw-hash Number of transactions that are parked on the read write hash. This is used for transactions that have to be processed on a different node. For example, prole writes, or read duplicate resolutions (when requested through client policy). Corresponds to the [`rw_in_progress`](https://aerospike.com/docs/database/reference/metrics#node_stats__rw_in_progress) statistic. --- proxy-hash Number of transactions on the proxy hash waiting for transmission on the fabric. Corresponds to the [`proxy_in_progress`](https://aerospike.com/docs/database/reference/metrics#node_stats__proxy_in_progress) statistic. --- rec-refs Removed in 3.10. Number of references to a primary key. --- tree-gc-q Introduced in 3.10. This is the number of trees queued up, ready to be completely removed (partitions drop). Corresponds to the [`tree_gc_queue`](https://aerospike.com/docs/database/reference/metrics#node_stats__tree_gc_queue) statistic. --- long-queries Introduced in version 6.1.0. Number of long queries currently active. Corresponds to the [`long_queries_active`](https://aerospike.com/docs/database/reference/metrics#node_stats__long_queries_active) statistic. --- #### `namespace data-usage ticker` `INFO` Message: ```bash {} data-usage: used-bytes 1073741824000 avail-pct 20 ``` Context: info Introduced: 7.0.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. This is a break down of the device usage if a `storage-engine device` has been configured for the namespace. Parameters: used-bytes Number of bytes used on disk for {ns\_name} on the local node. --- avail-pct Minimum percentage of contiguous disk space in {ns\_name} on the local node across all devices. Corresponds to the [`device_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__device_available_pct) statistic. Appears for storage-engine device. --- cache-read-pct Percentage of reads from the post-write cache instead of disk. Only applicable when {ns\_name} is not configured for data in memory. Corresponds to the [`cache_read_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__cache_read_pct) statistic. --- #### `namespace index-flash-usage device ticker` `INFO` Message: ```bash {namespace_name} index-flash-usage: used-bytes 5502926848 used-pct 1 alloc-bytes 16384000 alloc-pct 92 ``` Context: info Introduced: 4.3.0.2 Removed: 7.0.0 Occurs: Displayed periodically for each namespace configured with ‘index-type flash’, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the device and stats belongs to. --- index-flash-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the primary index used by this namespace on this node. --- used-pct Percentage of the mount in use for the primary index used by this namespace on this node. --- alloc-bytes Total bytes allocated on the mount for the primary index used by this namespace on this node. This statistic represents entire 4KiB chunks that have at least one element in use. This statistic was introduced in 5.6. Corresponds to the [`index_flash_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_flash_alloc_bytes)statistic. --- alloc-pct Percentage of the mount allocated for the primary index used by this namespace on this node. This statistic represents entire 4KiB chunks that have at least one element in use. This statistic was introduced in 5.6. Corresponds to the [`index_flash_alloc_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__index_flash_alloc_pct) statistic. --- #### `namespace index-pmem-usage device ticker` `INFO` Message: ```bash {namespace_name} index-pmem-usage: used-bytes 5502926848 used-pct 1 ``` Context: info Introduced: 4.5.0 Removed: 7.0.0 Occurs: Displayed periodically for each namespace configured with ‘index-type pmem’, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the index and stats belongs to. --- index-pmem-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the primary index used by this namespace on this node. --- used-pct Percentage of the mount in use for the primary index used by this namespace on this node. --- #### `namespace index-usage ticker` `INFO` Message: ```bash {namespace_name} index-usage: used-bytes 5502926848 ``` Context: info Introduced: 7.0.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the device and stats belongs to. --- index-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the primary index used by this namespace on this node. --- used-pct Percentage of the mount in use for the primary index used by this namespace on this node. --- #### `namespace set-index-usage ticker` `INFO` Message: ```bash {namespace_name} set-index-usage: used-bytes 5502926848 used-pct 1 ``` Context: info Introduced: 7.0.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the index and stats belongs to. --- set-index-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the primary index used by this namespace on this node. --- used-pct Percentage of the mount in use for the primary index used by this namespace on this node. --- #### `sindex-pmem-usage` `INFO` Message: ```bash {namespace_name} sindex-pmem-usage: used-bytes 12345 used-pct 43 ``` Context: info Introduced: 6.3.0 Removed: 7.0.0 Occurs: Displayed periodically for each namespace configured with ‘sindex-type pmem’, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the index and stats belongs to. --- sindex-pmem-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the secondary indexes used by this namespace on this node. --- used-pct Percentage of the mount in use for the secondary indexes used by this namespace on this node. --- #### `namespace sindex-usage ticker` `INFO` Message: ```bash {namespace_name} sindex-usage: used-bytes 12345 used-pct 43 ``` Context: info Introduced: 7.0.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. Parameters: {ns\_name} Name of the namespace the index and stats belongs to. --- sindex-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the secondary indexes used by this namespace on this node. --- used-pct Percentage of the mount in use for the secondary indexes used by this namespace on this node. --- #### `tomb-raider-unmark-threads exceeds device count (dynamic)` `WARNING` Message: ```bash WARNING (info): {} tomb-raider-unmark-threads () exceeds number of devices (); will use threads ``` Description: Logged when a dynamic `set-config` requests more unmark threads than there are devices. The server uses the device count instead. Context: info Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- requested Value requested via the info command. --- actual Number of devices; effective thread count for the unmark phase. --- #### `tomb-raider-unmark-threads exceeds maximum` `WARNING` Message: ```bash WARNING (info): {} tomb-raider-unmark-threads exceeds maximum 128 ``` Description: Logged when a runtime change sets [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads) above the allowed maximum (128). Context: info Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- value Invalid value that was rejected. --- #### `tomb-raider-unmark-threads invalid (not non-negative integer)` `WARNING` Message: ```bash WARNING (info): {} tomb-raider-unmark-threads must be a non-negative integer ``` Description: Logged when [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads) is set to an invalid value at runtime (not a non-negative integer). Context: info Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- #### `namespace batch-sub stats ticker` `INFO` Message: ```bash {ns_name} batch-sub: tsvc (0,0) proxy (0,0,0) read (959,0,0,51,1) write (0,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0) ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Batch index transaction statistics for each namespace. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) only if batch index transactions hit this namespace on this node. No correlation can be made between the status of a parent batch-index job and the statuses of its sub-transactions. See the `batch-index` log entry for details. Parameters: tsvc Number of batch-index read sub transactions that failed in the transaction service (Error,Timed out). Corresponds to the [`batch_sub_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_tsvc_error) and [`batch_sub_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_tsvc_timeout) statistics. --- proxy Number of proxied batch-index read sub transactions (Success,Error,Timed out). Corresponds to the [`batch_sub_proxy_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_proxy_complete), [`batch_sub_proxy_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_proxy_error), and [`batch_sub_proxy_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_proxy_timeout) statistics. --- read Number of batch-index read sub transactions (Success,Error,Timed out,Not found,Filtered out). Corresponds to the [`batch_sub_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_read_success), [`batch_sub_read_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_read_error), [`batch_sub_read_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_read_timeout), [`batch_sub_read_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_read_not_found), and [`batch_sub_read_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_read_filtered_out) statistics. --- write Number of batch-index write sub transactions (Success,Error,Timed out,Filtered out). Corresponds to the [`batch_sub_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_write_success), [`batch_sub_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_write_error), [`batch_sub_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_write_timeout), and [`batch_sub_write_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_write_filtered_out) statistics. Displayed as of Database 6.0.0. --- delete Number of batch-index delete sub transactions (Success,Error,Timed out,Not found,Filtered out). Corresponds to the [`batch_sub_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_delete_success), [`batch_sub_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_delete_error), [`batch_sub_delete_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_delete_timeout), [`batch_sub_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_delete_not_found), and [`batch_sub_delete_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_delete_filtered_out) statistics. Displayed as of Database 6.0.0. --- udf Number of batch-index udf sub transactions (Complete,Error,Timed out,Filtered out). Corresponds to the [`batch_sub_udf_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_udf_complete), [`batch_sub_udf_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_udf_error), [`batch_sub_udf_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_udf_timeout), and [`batch_sub_udf_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_udf_filtered_out) statistics. Displayed as of Database 6.0.0. --- lang Number of batch-index lang sub transactions (Delete Success,Error,Read Success,Write Success). Corresponds to the [`batch_sub_lang_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_lang_delete_success), [`batch_sub_lang_error`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_lang_error), [`batch_sub_lang_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_lang_read_success), and [`batch_sub_lang_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__batch_sub_lang_write_success) statistics. Displayed as of Database 6.0.0. --- #### `namespace client requests ticker` `INFO` Message: ```bash {ns_name} client: tsvc (0,0) proxy (0,0,0) read (126,0,1,3,1) write (2886,0,23,2) delete (197,0,1,19,3) udf (35,0,1,4) lang (26,7,0,3) ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Basic client request statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval) only if client transactions hit this namespace on this node. The following values define various actions displayed in the logs: - S - success - C - complete, but success or failure is indeterminate. Proxy diverts and UDFs can successfully send a “FAILURE” response bin. - E - error - T - timed out - N - not found, which for reads and deletes is a result that wants to be distinguished from success but is not an error - F - result filtered out or action skipped by predexp - R,W,D - successful UDF read, write, or delete operation. Parameters: tsvc Failures in the transaction service, before attempting to handle the transaction (E,T). Also reported as the [`client_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_tsvc_error) and [`client_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_tsvc_timeout) statistics. --- proxy Client proxied transactions (C,E,T). This should only happen during migrations. Also reported as the [`client_proxy_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__client_proxy_complete), [`client_proxy_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_proxy_error), and [`client_proxy_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_proxy_timeout) statistics. --- read Client read transactions (S,E,T,N,F). Also reported as the [`client_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_read_success), [`client_read_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_read_error), [`client_read_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_read_timeout), [`client_read_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__client_read_not_found), and [`client_read_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__client_read_filtered_out) statistics. --- write Client write transactions (S,E,T,F). Also reported as the [`client_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_write_success), [`client_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_write_error), [`client_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_write_timeout), and [`client_write_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__client_write_filtered_out) statistics. --- delete Client delete transactions (S,E,T,N,F). Also reported as the [`client_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_delete_success), [`client_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_delete_error), [`client_delete_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_delete_timeout), [`client_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__client_delete_not_found), and [`client_delete_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__client_delete_filtered_out) statistics. --- udf Client UDF transactions (C,E,T,F). See the `lang` stat breakdown for the underlying operation statuses. Also reported as the [`client_udf_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__client_udf_complete), [`client_udf_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_udf_error), [`client_udf_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__client_udf_timeout), and [`client_udf_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__client_udf_filtered_out) statistics. --- lang Statistics for UDF operation statuses (R,W,D,E). Also reported as the [`client_lang_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_lang_read_success), [`client_lang_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_lang_write_success), [`client_lang_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__client_lang_delete_success), and [`client_lang_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_lang_error) statistics. --- #### `namespace data-usage ticker` `INFO` Message: ```bash {ns_name} data-usage: used-bytes 2054187648 avail-pct 92 ``` Context: info Introduced: 7.0.0 Removed: \- Occurs: If `storage-engine pmem` has been configured for the namespace, this is a break down of the pmem storage file usage. Displayed periodically for each namespace, every 10 seconds by default. Parameters: used-bytes Number of bytes used on pmem storage files for {ns\_name} on the local node. --- avail-pct Minimum percentage of contiguous pmem storage file space in {ns\_name} on the local node across all pmem storage files. Corresponds to the [`pmem_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__pmem_available_pct) statistic. --- #### `namespace device-usage ticker` `INFO` Message: ```bash {ns_name} device-usage: used-bytes 2054187648 avail-pct 92 cache-read-pct 12.35 ``` Context: info Introduced: 3.9.0 Removed: 7.0.0 Occurs: If `storage-engine device` has been configured for the namespace, this is a break down of the device usage. Displayed periodically for each namespace, every 10 seconds by default. Parameters: used-bytes Number of bytes used on disk for {ns\_name} on the local node. --- avail-pct Minimum percentage of contiguous disk space in {ns\_name} on the local node across all devices. Corresponds to the [`device_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__device_available_pct) statistic. --- cache-read-pct Percentage of reads from the post-write cache instead of disk. Only applicable when {ns\_name} is not configured for data in memory. Corresponds to the [`cache_read_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__cache_read_pct) statistic. --- #### `namespace dup-res ticker` `INFO` Message: ```bash {ns_name} dup-res: ask 1234 respond (10,4321) ``` Context: info Introduced: 5.5.0 Removed: \- Occurs: Statistics for transactions that are asking or handling duplicate resolution. Displayed periodically for each namespace, every 10 seconds by default. Parameters: ask Number of duplicate resolution requests made by the node to other individual nodes. Also reported as the [`dup_res_ask`](https://aerospike.com/docs/database/reference/metrics#namespace__dup_res_ask) statistic. --- respond Number of duplicate resolution requests handled by the node, broken up between transactions where a read was required and transactions where a read was not required. Also reported as the [`dup_res_respond_read`](https://aerospike.com/docs/database/reference/metrics#namespace__dup_res_respond_read) and [`dup_res_respond_no_read`](https://aerospike.com/docs/database/reference/metrics#namespace__dup_res_respond_no_read) statistics. --- #### `namespace from-proxy ticker` `INFO` Message: ```bash {ns_name} from-proxy: tsvc (0,0) read (105,0,1,7) write (2812,0,22,1) delete (188,0,1,16,2) udf (35,0,1,3) lang (26,7,0,3) ``` Context: info Introduced: 4.5.1 Removed: \- Occurs: Basic proxied transaction statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after proxied transactions hit this namespace on this node. The following values define various actions that are displayed in the logs: - S - success - C - complete, but success/failure indeterminate. UDFs can successfully send a “FAILURE” response bin. - E - error - T - timed out - N - not found, which for reads and deletes is a result that wants to be distinguished from success but is not an error - F - result filtered out or action skipped by predexp - R,W,D - successful UDF read, write, or delete operation. Parameters: tsvc Failures in the transaction service, before attempting to handle the transaction (E,T). Also reported as the following statistics: - [`from_proxy_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_tsvc_error) - [`from_proxy_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_tsvc_timeout) --- read Proxied read transactions (S,E,T,N,F). Also reported as the following statistics: - [`from_proxy_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_read_success), [`from_proxy_read_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_read_error) - [`from_proxy_read_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_read_timeout) - [`from_proxy_read_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_read_not_found) - [`from_proxy_read_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_read_filtered_out) statistics. --- write Proxied write transactions (S,E,T,F). Also reported as the following statistics: - [`from_proxy_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_write_success) - [`from_proxy_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_write_error) - [`from_proxy_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_write_timeout) - [`from_proxy_write_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_write_filtered_out) --- delete Proxied delete transactions (S,E,T,N,F). Also reported as the following statistics: - [`from_proxy_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_delete_success) - [`from_proxy_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_delete_error) - `from_proxy_delete_timeout`\](/database/reference/metrics#namespace\_\_from\_proxy\_delete\_timeout) - [`from_proxy_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_delete_not_found) - [`from_proxy_delete_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_delete_filtered_out) --- udf Proxied UDF transactions (C,E,T,F). See the `lang` stat breakdown for the underlying operation statuses. Also reported as the following statistics: - [`from_proxy_udf_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_udf_complete) - [`from_proxy_udf_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_udf_error) - [`from_proxy_udf_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_udf_timeout) - [`from_proxy_udf_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_udf_filtered_out) --- lang Statistics for proxied UDF operation statuses (R,W,D,E). Also reported as the following statistics: - [`from_proxy_lang_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_lang_read_success) - [`from_proxy_lang_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_lang_write_success) - `from_proxy_lang_delete_success`\](/database/reference/metrics#namespace\_\_from\_proxy\_lang\_delete\_success) - [`from_proxy_lang_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_lang_error) statistics. --- #### `namespace from-proxy-batch-sub stats ticker` `INFO` Message: ```bash {ns_name} from-proxy-batch-sub: tsvc (0,0) read (959,0,0,51,1) write (0,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0) ``` Context: info Introduced: 4.5.1 Removed: \- Occurs: Proxied batch index transaction statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after proxied batch index transactions hit this namespace on this node. No correlation can be made between the status of a parent batch-index job and the statuses of its sub-transactions. See the `batch-index` log entry for details. Parameters: tsvc Number of proxied batch-index read sub transactions that failed in the transaction service (Error,Timed out). Corresponds to the following statistics: - [`from_proxy_batch_sub_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_tsvc_error) - [`from_proxy_batch_sub_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_tsvc_timeout) --- read Number of proxied batch-index read sub transactions (Success,Error,Timed out,Not found). Corresponds to the following statistics: - [`from_proxy_batch_sub_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_read_success) - [`from_proxy_batch_sub_read_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_read_error) - [`from_proxy_batch_sub_read_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_read_timeout) - [`from_proxy_batch_sub_read_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_read_not_found) - [`from_proxy_batch_sub_read_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_read_filtered_out) --- write Number of proxied batch-index write sub transactions (Success,Error,Timed out,Filtered out). Corresponds to the following statistics - [`from_proxy_batch_sub_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_write_success) - [`from_proxy_batch_sub_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_write_error) - [`from_proxy_batch_sub_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_write_timeout) - [`from_proxy_batch_sub_write_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_write_filtered_out) Displayed as of Database 6.0.0. --- delete Number of proxied batch-index delete sub transactions (Success,Error,Timed out,Not found,Filtered out). Corresponds to the following statistics: - [`from_proxy_batch_sub_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_delete_success) - [`from_proxy_batch_sub_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_delete_error) - [`from_proxy_batch_sub_delete_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_delete_timeout) - [`from_proxy_batch_sub_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_delete_not_found) - [`from_proxy_batch_sub_delete_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_delete_filtered_out) Displayed as of Database 6.0.0. --- udf Number of proxied batch-index udf sub transactions (Complete,Error,Timed out,Filtered out). Corresponds to the following statistics: - [`from_proxy_batch_sub_udf_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_udf_complete) - [`from_proxy_batch_sub_udf_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_udf_error) - [`from_proxy_batch_sub_udf_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_udf_timeout) - [`from_proxy_batch_sub_udf_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_udf_filtered_out) Displayed as of Database 6.0.0. --- lang Number of proxied batch-index lang sub transactions (Delete Success,Error,Read Success,Write Success). Corresponds to the following statistics: - [`from_proxy_batch_sub_lang_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_lang_delete_success) - [`from_proxy_batch_sub_lang_error`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_lang_error) - [`from_proxy_batch_sub_lang_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_lang_read_success) - [`from_proxy_batch_sub_lang_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__from_proxy_batch_sub_lang_write_success) Displayed as of Database 6.0.0. --- #### `namespace index-usage ticker"` `INFO` Message: ```bash {ns_name} index-usage: used-bytes 3121300 used-pct 0.05 alloc-bytes 16777216 alloc-pct 0.50 ``` Context: info Introduced: 3.9.0 Removed: 7.0.0 Occurs: Index usage statistics for {ns\_name}. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). When ‘data-in-memory’ is false, ‘data-bytes’ is not included. ‘set-index-bytes’ is included as of Database 5.6. Parameters: used-bytes Total number of bytes used in memory for {ns\_name} on the local node. --- used-pct Percentage of bytes used in memory for {ns\_name} on the local node. --- alloc-pct Number of bytes holding secondary indexes in process memory for {ns\_name} on the local node. --- alloc-bytes Number of bytes holding data in process memory for {ns\_name} on the local node. Displayed only when ‘data-in-memory’ is set to true for {ns\_name}. --- #### `namespace memory-usage ticker` `INFO` Message: ```bash {ns_name} memory-usage: total-bytes 3121300 index-bytes 140544 set-index-bytes 70272 sindex-bytes 221544 data-bytes 2688940 used-pct 0.05 ``` Context: info Introduced: 3.9.0 Removed: 7.0.0 Occurs: Memory usage statistics for {ns\_name}. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). When ‘data-in-memory’ is false, ‘data-bytes’ is not included. ‘set-index-bytes’ is included as of Database 5.6. Parameters: total-bytes Total number of bytes used in memory for {ns\_name} on the local node. --- index-bytes Number of bytes holding the primary index in system memory for {ns\_name} on the local node. Displays 0 when index is not stored in RAM. --- set-index-bytes Number of bytes holding set indexes in process memory for {ns\_name} on the local node. Displayed as of Database 5.6.0. --- sindex-bytes Number of bytes holding secondary indexes in process memory for {ns\_name} on the local node. --- data-bytes Number of bytes holding data in process memory for {ns\_name} on the local node. Displayed only when ‘data-in-memory’ is set to true for {ns\_name}. --- used-pct Percentage of bytes used in memory for {ns\_name} on the local node. --- #### `namespace migrations ticker` `INFO` Message: ```bash {ns_name} migrations: remaining (654,289,254) active (1,1,0) complete-pct 88.49 ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Migration statistics for {ns\_name}. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). When migrations have completed, this line is reduced to `{ns_name} migrations - complete`. Parameters: {ns\_name} “ns\_name” is replaced by the name of a particular namespace. --- remaining Total number of transmit and receive partition migrations outstanding for this node as well as signals. Signals represents the number of signals to send to other nodes (non replica nodes) for partitions to drop. This log line changes to `complete` after migrations are completed on this node. --- active Number of transmit and receive partition migrations currently in progress, as well as active signals. --- complete-pct Percent of the total number of partition migrations scheduled for this rebalance that have already completed. --- #### `namespace objects ticker` `INFO` Message: ```bash {ns_name} objects: all 845922 master 281071 prole 564851 ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Object statistics for {ns\_name}. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Number of objects for this namespace on this node along with the master and prole breakdown. Parameters: {ns\_name} “ns\_name” is replaced by the name of a particular namespace. --- all Total number of objects for this namespace on this node (master and proles). --- master Number of master objects for this namespace on this node. --- prole Number of prole (replica) objects for this namespace on this node. --- #### `namespace ops-sub ticker` `INFO` Message: ```bash {ns_name} ops-sub: tsvc (0,0) write (2651,0,0,1) ``` Context: info Introduced: 4.7.0.2 Removed: \- Occurs: Scan/query ops sub-transactions statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after scan/query ops sub-transactions hit this namespace on this node. Parameters: tsvc Number of ops sub-transactions of scan/query background ops jobs that failed in the transaction service (Error,Timed out). Corresponds to the [`ops_sub_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_tsvc_error) and [`ops_sub_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_tsvc_timeout) statistics. --- write Number of ops sub-transactions of scan/query background ops jobs (Success,Error,Timed out,Filtered out). Corresponds to the [`ops_sub_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_write_success), [`ops_sub_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_write_error), [`ops_sub_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_write_timeout), and [`ops_sub_write_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__ops_sub_write_filtered_out) statistics. --- #### `namespace pi-query thread counts` `INFO` Message: ```bash {ns_name} pi-query: short-basic (4,0,0) long-basic (36,0,0) aggr (0,0,0) udf-bg (7,0,0), ops-bg (3,0,0) ``` Context: info Introduced: 6.0.0 Removed: \- Occurs: Primary index query (pi-query) transaction statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after pi-query transactions hit this namespace on this node. Parameters: short-basic Number of short primary index (pi)-query jobs since the server started (Success,Error,Aborted). Short queries are declared by the client, are unmonitored and typically run for a second or less. Corresponds to the [`pi_query_short_basic_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_short_basic_complete), [`pi_query_short_basic_error`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_short_basic_error), and [`pi_query_short_basic_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_short_basic_timeout) statistics. --- long-basic Number of long pi-query jobs since the server started (Success,Error,Aborted). Long queries are monitored and not time bounded. Corresponds to the [`pi_query_long_basic_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_long_basic_complete), [`pi_query_long_basic_error`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_long_basic_error), and [`pi_query__long_basic_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_long_basic_abort) statistics. --- aggr Number of pi-query aggregation jobs since the server started (Success,Error,Aborted). Corresponds to the [`pi_query_aggr_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_aggr_complete), [`pi_query_aggr_error`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_aggr_error), and [`pi_query_aggr_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_aggr_abort) statistics. --- udf-bg Number of pi-query background udf jobs since the server started (Success,Error,Aborted). Corresponds to the [`pi_query_udf_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_udf_bg_complete), [`pi_query_udf_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_udf_bg_error), and [`pi_query_udf_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_udf_bg_abort) statistics. --- ops-bg Number of pi-query background operations (ops) jobs since the server started (Success,Error,Aborted). Corresponds to the [`pi_query_ops_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_ops_bg_complete), [`pi_query_ops_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_ops_bg_error), and [`pi_query_ops_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__pi_query_ops_bg_abort) statistics. --- #### `pmem-usage: used-bytes avail-pct` `INFO` Message: ```bash {ns_name} pmem-usage: used-bytes 2054187648 avail-pct 92 ``` Context: info Introduced: 4.8.0 Removed: 7.0.0 Occurs: Break down of the pmem storage file usage. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only if `storage-engine pmem` has been configured for the namespace. Parameters: used-bytes Number of bytes used on pmem storage files for {ns\_name} on the local node. --- avail-pct Minimum percentage of contiguous pmem storage file space in {ns\_name} on the local node across all pmem storage files. Corresponds to the [`pmem_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__pmem_available_pct) statistic. --- #### `namespace query stat ticker` `INFO` Message: ```bash {ns_name} query: basic (210,0,0) aggr (0,0,0) udf-bg (0,0,0) ops-bg (0,0,0) ``` Context: info Introduced: 5.7.0 Removed: 6.0.0 Occurs: Secondary index query transactions statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after query transactions hit this namespace on this node. Parameters: basic Number of secondary index queries since the server started (Completed,Error,Abort). Corresponds to the [`query_basic_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__query_basic_complete), [`query_basic_error`](https://aerospike.com/docs/database/reference/metrics#namespace__query_basic_error) and [`query_basic_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__query_basic_abort) statistics. --- aggr Number of query aggregation jobs since the server started (Completed,Error,Abort). Corresponds to the [`query_aggr_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__query_aggr_complete), [`query_aggr_error`](https://aerospike.com/docs/database/reference/metrics#namespace__query_aggr_error) and [`query_aggr_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__query_aggr_abort) statistics. --- udf-bg Number of query background udf jobs since the server started (Completed,Error,Abort). Corresponds to the [`query_udf_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__query_udf_bg_complete), [`query_udf_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__query_udf_bg_error) and [`query_udf_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__query_udf_bg_abort) statistics. --- ops-bg Number of query background operations (ops) jobs since the server started (Success,Failure). Corresponds to the [`query_ops_bg_success`](https://aerospike.com/docs/database/reference/metrics#namespace__query_ops_bg_success) and [`query_ops_bg_failure`](https://aerospike.com/docs/database/reference/metrics#namespace__query_ops_bg_failure) statistics. --- #### `namespace query: basic aggr udf-bg ops-bg` `INFO` Message: ```bash {ns_name} query: basic (5,0) aggr (6,0) udf-bg (1,0) ops-bg (2,0) ``` Context: info Introduced: 3.9.0 Removed: 5.7.0 Occurs: Secondary index query transaction statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after query transactions hit this namespace on this node. Parameters: basic Number of secondary index queries since the server started (Success,Abort). Corresponds to the [`query_lookup_success`](https://aerospike.com/docs/database/reference/metrics#namespace__query_lookup_success) and [`query_lookup_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__query_lookup_abort) statistics. --- aggr Number of query aggregation jobs since the server started (Success,Abort). Corresponds to the [`query_agg_success`](https://aerospike.com/docs/database/reference/metrics#namespace__query_agg_success) and [`query_agg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__query_agg_abort) statistics. --- udf-bg Number of query background udf jobs since the server started (Success,Failure). Corresponds to the [`query_udf_bg_success`](https://aerospike.com/docs/database/reference/metrics#namespace__query_udf_bg_success) and [`query_udf_bg_failure`](https://aerospike.com/docs/database/reference/metrics#namespace__query_udf_bg_failure) statistics. --- ops-bg Number of query background operations (ops) jobs since the server started (Success,Failure). Corresponds to the [`query_ops_bg_success`](https://aerospike.com/docs/database/reference/metrics#namespace__query_ops_bg_success) and [`query_ops_bg_failure`](https://aerospike.com/docs/database/reference/metrics#namespace__query_ops_bg_failure) statistics. --- #### `namespace read-touch ticker` `INFO` Message: ```bash {ns_name} read-touch: tsvc (0,0) all-triggers (12345,0,3,45) 14 ``` Description: Statistics for the read-touch LRU behavior, described in [configuring namespace data retention](https://aerospike.com/docs/database/manage/namespace/retention) and controlled by [`default-read-touch-ttl-pct`](https://aerospike.com/docs/database/reference/config#namespace__default-read-touch-ttl-pct). Context: info Introduced: 7.1.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. The following values define various actions that are displayed in the logs: - S - success - E - error - T - timed out - K - skipped Parameters: tsvc Read-touch early timeouts (E, T). Also reported as the [`read_touch_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_tsvc_error) and [`read_touch_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_tsvc_timeout) statistics. --- all-triggers Read-touch transactions (S, E, T, K). Also reported as the [`read_touch_success`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_success),[`read_touch_error`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_error), [`read_touch_timeout`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_timeout), and [`read_touch_skip`](https://aerospike.com/docs/database/reference/metrics#node_stats__read_touch_skip) statistics. --- #### `namespace re-repl ticker` `INFO` Message: ```bash {ns_name} re-repl: tsvc (0,34) all-triggers (525,0,32) unreplicated-records 14 ``` Description: Statistics for transactions re-replicating, applicable to [`strong-consistency`](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) enabled namespaces. In strong consistency mode, write transactions failing replication are marked as unreplicated, and attempt to re-replicate one time immediately (despite returning an [in doubt](https://aerospike.com/blog/developers-understanding-aerospike-transactions/) timeout failure to the client), as well as on any subsequent transaction attempted on the record (read or write). Context: info Introduced: 4.0.0 Removed: \- Occurs: Displayed periodically for each namespace, every 10 seconds by default. The following values define various actions which are displayed in the logs: - S - success - E - error - T - timed out Parameters: tsvc Re-replication early timeouts (E, T). Also reported as the [`re_repl_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__re_repl_tsvc_error) and [`re_repl_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__re_repl_tsvc_timeout) statistics. --- all-triggers Re-replication transactions (S, E, T). Also reported as the [`re_repl_success`](https://aerospike.com/docs/database/reference/metrics#namespace__re_repl_success),[`re_repl_error`](https://aerospike.com/docs/database/reference/metrics#namespace__re_repl_error) and [`re_repl_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__re_repl_timeout) statistics. Starting with Database 6.3.0, `re_repl_timeout` only applies to timeouts during the actual replication. --- unreplicated-records Number of unreplicated records in the namespace. Displayed as of Database 5.7.0. Also reported as the [`unreplicated_records`](https://aerospike.com/docs/database/reference/metrics#namespace__unreplicated_records) statistic. --- #### `namespace retransmits ticker` `INFO` Message: ```bash {ns_name} retransmits: migration 1 dup-res (1,2,3,4,5,6,7,8,9,10) repl-ping (1,2) repl-write (1,2,3,4,5,6,7,8) ``` Context: info Introduced: 4.5.1 Removed: \- Occurs: Retransmit statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only if any retransmit has taken place. Retransmission happens when the server resends a message to another cluster node, after a prior attempt timed out. Parameters: migration Number of retransmits that occurred during migrations. Corresponds `migrate_record_retransmits` statistic. --- dup-res Groups the retransmission statistics related to duplicate-resolution. The numbered `dup-res` metrics correspond to the following: 1. `retransmit_all_read_dup_res` 2. `retransmit_all_write_dup_res` 3. `retransmit_all_delete_dup_res` 4. `retransmit_all_udf_dup_res` 5. `retransmit_all_batch_sub_read_dup_res` 6. `retransmit_all_batch_sub_write_dup_res` 7. `retransmit_all_batch_sub_delete_dup_res` 8. `retransmit_all_batch_sub_udf_dup_res` 9. `retransmit_udf_sub_dup_res` 10. `retransmit_ops_sub_dup_res` --- repl-ping Groups the retransmission statistics related to Linearized SC reads. The numbered `repl-ping` metrics correspond to the following: 1. `retransmit_all_read_repl_ping` 2. `retransmit_all_batch_sub_read_repl_ping` --- repl-write Groups the retransmission statistics related to replica writes. The numbered `repl-write` metrics correspond to the following: 1. `retransmit_all_write_repl_write` 2. `retransmit_all_delete_repl_write` 3. `retransmit_all_udf_repl_write` 4. `retransmit_all_batch_sub_write_repl_write` 5. `retransmit_all_batch_sub_delete_repl_write` 6. `retransmit_all_batch_sub_udf_repl_write` 7. `retransmit_udf_sub_repl_write` 8. `retransmit_ops_sub_repl_write` --- #### `namespace scan (pi query) ticker"` `INFO` Message: ```bash {ns_name} scan: basic (11,0,0) aggr (0,0,0) udf-bg (5,0,0), ops-bg (10,0,0) ``` Context: info Introduced: 3.9.0 Removed: 6.0.0 Occurs: Scan transactions statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after scan transactions hit this namespace on this node. Parameters: basic Number of scan jobs since the server started (Success,Error,Aborted). Corresponds to the [`scan_basic_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_basic_complete), [`scan_basic_error`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_basic_error), and [`scan_basic_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_basic_abort) statistics. --- aggr Number of scan aggregation jobs since the server started (Success,Error,Aborted). Corresponds to the [`scan_aggr_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_aggr_complete), [`scan_aggr_error`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_aggr_error), and [`scan_aggr_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_aggr_abort) statistics. --- udf-bg Number of scan background udf jobs since the server started (Success,Error,Aborted). Corresponds to the [`scan_udf_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_udf_bg_complete), [`scan_udf_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_udf_bg_error), and [`scan_udf_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_udf_bg_abort) statistics. --- ops-bg Number of scan background operations (ops) jobs since the server started (Success,Error,Aborted). Corresponds to the [`scan_ops_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_ops_bg_complete), [`scan_ops_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_ops_bg_error), and [`scan_ops_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__scan_ops_bg_abort) statistics. --- #### `sindex-flash-usage: used-bytes used-pct` `INFO` Message: ```bash {ns_name} sindex-flash-usage: used-bytes 12345 used-pct 43 ``` Context: info Introduced: 6.4.0 Removed: 7.0.0 Occurs: Secondary index flash usage (sindex-flash-usage) statistics. Ticker log line, prints rolled-up information every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). displayed for each namespace configured with ‘sindex-type flash’. Parameters: {ns\_name} Name of the namespace the index and stats belongs to. --- sindex-flash-usage Name for which the following stats apply. --- used-bytes Total bytes in use on the mount for the secondary indexes used by this namespace on this node. --- used-pct Percentage of the mount in use for the secondary indexes used by this namespace on this node. --- #### `namespace si-query ticker` `INFO` Message: ```bash {ns_name} si-query: short-basic (4,0,0) long-basic (26,0,0) aggr (0,0,0) udf-bg (7,0,0) ops-bg (3,0,0) ``` Context: info Introduced: 6.0.0 Removed: \- Occurs: Secondary index query (si-query) transactions statistics. Ticker log line, prints rolled-up information for each namespace every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after si-query transactions hit this namespace on this node. Parameters: short-basic Number of short secondary index queries since the server started (Completed,Error,Abort). Short queries are declared by the client, are unmonitored and typically run for a second or less. Corresponds to the [`si_query_short_basic_complete`](https://aerospike.com/docs/database/reference/metrics#sindex__si_query_short_basic_complete), [`si_query_short_basic_error`](https://aerospike.com/docs/database/reference/metrics#sindex__si_query_short_basic_error) and [`si_query_short_basic_timeout`](https://aerospike.com/docs/database/reference/metrics#sindex__si_query_short_basic_timeout) statistics. --- long-basic Number of long secondary index queries since the server started (Completed,Error,Abort). Long queries are monitored and not time bounded. Corresponds to the [`si_query_long_basic_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_long_basic_complete), [`si_query_long_basic_error`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_long_basic_error) and [`si_query_long_basic_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_long_basic_abort) statistics. --- aggr Number of si-query aggregation jobs since the server started (Completed,Error,Abort). Corresponds to the [`si_query_aggr_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_aggr_complete), [`si_query_aggr_error`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_aggr_error) and [`si_query_aggr_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_aggr_abort) statistics. --- udf-bg Number of si-query background udf jobs since the server started (Completed,Error,Abort). Corresponds to the [`si_query_udf_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_udf_bg_complete), [`si_query_udf_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_udf_bg_error) and [`si_query_udf_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_udf_bg_abort) statistics. --- ops-bg Number of si-query background operations (ops) jobs since the server started (Completed,Error,Abort). Corresponds to the [`si_query_ops_bg_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_ops_bg_complete), [`si_query_ops_bg_error`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_ops_bg_error), and [`si_query_ops_bg_abort`](https://aerospike.com/docs/database/reference/metrics#namespace__si_query_ops_bg_abort) statistics. --- #### `namespace special-errors ticker` `INFO` Message: ```bash {ns_name} special-errors: key-busy (1234, 40) record-too-big 5678 lost-conflict (256,32) ``` Context: info Introduced: 5.6.0 Removed: \- Occurs: Special errors statistics. Ticker log line, prints rolled-up information for each namespace every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only if any of those errors have occurred. Log ticket modified in Database 7.2.0, adding [`fail_xdr_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_xdr_key_busy) count as second number in the first parentheses. Parameters: key-busy Number of key busy errors. Corresponds to the [`fail_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_key_busy) and [`fail_xdr_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_xdr_key_busy) statistics. See[Hot Key Error code 14](https://support.aerospike.com/s/article/Why-does-my-client-return-Error-code-14-Hot-key). --- record-too-big Number of record too big errors. Corresponds to the [`fail_record_too_big`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_record_too_big) statistic. --- lost-conflict Composed of the following metrics: - [`fail_client_lost_conflict`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_client_lost_conflict)\- [`fail_xdr_lost_conflict`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_xdr_lost_conflict) --- #### `special-errors: key-busy record-too-big` `INFO` Message: ```bash {ns_name} special-errors: key-busy 1234 record-too-big 5678 ``` Context: info Introduced: 3.16.0.1 Removed: 5.6.0 Occurs: Special errors statistics. Ticker log line, prints rolled-up information for each namespace every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only if any of those errors have occurred. Parameters: key-busy Number of key busy errors. Corresponds to the [`fail_key_busy`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_key_busy) statistic. See [Hot Key Error code 14](https://support.aerospike.com/s/article/Why-does-my-client-return-Error-code-14-Hot-key). --- record-too-big Number of record too big errors. Corresponds to the [`fail_record_too_big`](https://aerospike.com/docs/database/reference/metrics#namespace__fail_record_too_big) statistic. --- #### `namespace tombstones ticker` `INFO` Message: ```bash {ns_name} tombstones: all 11252 xdr (11223,0) master 5501 prole 5751 non-replica 0 ``` Context: info Introduced: 5.5.0 Removed: \- Occurs: Number of tombstones for this namespace on this node along with the breakdown. Displayed periodically for each namespace, every 10 seconds by default. Parameters: {ns\_name} “ns\_name” is replaced by the name of a particular namespace. --- all Total number of tombstones for this namespace on this node. --- xdr Number of xdr tombstones and bin cemeteries - ([`xdr_tombstones`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_tombstones),[`xdr_bin_cemeteries`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_bin_cemeteries)). --- master Number of master tombstones for this namespace on this node. --- prole Number of prole (replica) tombstones for this namespace on this node. --- non-replica Number of non-replica tombstones for this namespace on this node. --- #### `namespace udf-sub ticker` `INFO` Message: ```bash {ns_name} udf-sub: tsvc (0,0) udf (2651,0,0,1) lang (52,2498,101,0) ``` Context: info Introduced: 3.9.0 Removed: \- Occurs: Scan/query UDF sub-transactions statistics. Ticker log line, prints rolled-up information for each namespace every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). Displayed only after query UDF sub-transactions hit this namespace on this node. Parameters: tsvc Number of udf sub transactions of scan/query background udf jobs that failed in the transaction service (Error,Timed out). Corresponds to the following statistics: - [`udf_sub_tsvc_error`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_tsvc_error) - [`udf_sub_tsvc_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_tsvc_timeout) --- udf Number of udf sub transactions of scan/query background udf jobs (Success,Error,Timed out,Filtered out). Corresponds to the following statistics: - [`udf_sub_udf_complete`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_udf_complete) - [`udf_sub_udf_error`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_udf_error) - [`udf_sub_udf_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_udf_timeout) - [`udf_sub_udf_filtered_out`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_udf_filtered_out) --- lang Different status counts for underlying udf operations for sub transactions of scan/query background udf jobs (Read,Write,Delete,Error). Corresponds to the following statistics: - [`udf_sub_lang_read_success`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_lang_read_success) - [`udf_sub_lang_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_lang_write_success) - [`udf_sub_lang_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_lang_delete_success) - [`udf_sub_lang_error`](https://aerospike.com/docs/database/reference/metrics#namespace__udf_sub_lang_error) --- #### `namespace xdr-client ticker` `INFO` Message: ```bash {ns_name} xdr-client: write (1543,0,15) delete (134,0,3,25) ``` Context: info Introduced: 3.16.0.1 Removed: \- Occurs: XDR client transactions statistics, displayed after an XDR client transactions hit this namespace on this node. Ticker log line, prints rolled-up information for each namespace receiving write transactions from an XDR client every [`ticker-interval`](https://aerospike.com/docs/database/reference/config#service__ticker-interval). The values on this line are a subset of the values displayed in the client statistics line just above in the log file. The following values define various actions which are displayed in the logs: - S - success - E - error - T - timed out - N - not found, which for reads and deletes is a result that wants to be distinguished from success but is not an error. Parameters: write XDR client write transactions (S,E,T). Also reported as the following statistics: - [`xdr_client_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_write_success) - [`xdr_client_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_write_error) - [`xdr_client_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_write_timeout) --- delete XDR client delete transactions (S,E,T,N). Also reported as the following statistics: - [`xdr_client_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_delete_success) - [`xdr_client_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_delete_error) - [`xdr_client_delete_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_delete_timeout) - [`xdr_client_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_client_delete_not_found) --- #### `xdr-dc dc2: lag throughput hot-keys` `INFO` Message: ```bash {ns_name} xdr-dc dc2: lag 12 throughput 710 bytes-shipped 212456900 in-queue 250563 in-progress 81150 complete (1002215,0,0,0) retries (0,0,23) recoveries (2048,0) hot-keys 4655 ``` Context: info Introduced: 6.0.0 Removed: \- Occurs: Displayed periodically for each combination of namespace and XDR destination cluster (DC), every 10 seconds by default. Parameters: {ns\_name} “ns\_name” is replaced by the name of a particular namespace. --- xdr-dc Name of the XDR destination cluster. --- lag See [`lag`](https://aerospike.com/docs/database/reference/metrics#xdr__lag). --- throughput See [`throughput`](https://aerospike.com/docs/database/reference/metrics#xdr__throughput). --- bytes-shipped See [`bytes_shipped`](https://aerospike.com/docs/database/reference/metrics#xdr__bytes_shipped). --- in-queue See [`in_queue`](https://aerospike.com/docs/database/reference/metrics#xdr__in_queue). --- in-progress See [`in_progress`](https://aerospike.com/docs/database/reference/metrics#xdr__in_progress). --- complete Composed of the following metrics: - [`success`](https://aerospike.com/docs/database/reference/metrics#xdr__success) - [`abandoned`](https://aerospike.com/docs/database/reference/metrics#xdr__abandoned) - [`not_found`](https://aerospike.com/docs/database/reference/metrics#xdr__not_found) - [`filtered_out`](https://aerospike.com/docs/database/reference/metrics#xdr__filtered_out) --- retries Composed of the following metrics: - [`retry_no_node`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_no_node) - [`retry_conn_reset`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_conn_reset) - [`retry_dest`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_dest) --- recoveries Composed of the following metrics: - [`recoveries`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries) - [`recoveries_pending`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries_pending) --- hot-keys See [`hot_keys`](https://aerospike.com/docs/database/reference/metrics#xdr__hot_keys). --- #### `namespace xdr-from-proxy ticker` `INFO` Message: ```bash {ns_name} xdr-from-proxy: write (743,0,11) delete (104,0,3,21) ``` Context: info Introduced: 4.5.1 Removed: \- Occurs: Proxied XDR transactions statistics. Displayed periodically, every 10 seconds by default, for each namespace receiving proxied write transactions from an XDR client. Displayed only after a proxied XDR transaction hits this namespace on this node. The values on this line are a **subset** of the values displayed in the from-proxy statistics line just above in the log file. The following values define various actions which are displayed in the logs - S - success - E - error - T - timed out - N - not found, which for reads and deletes is a result that wants to be distinguished from success but is not an error Parameters: write XDR client write transactions (S,E,T). Also reported as the [`xdr_from_proxy_write_success`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_write_success), [`xdr_from_proxy_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_write_error) and [`xdr_from_proxy_write_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_write_timeout) statistics. --- delete XDR client delete transactions (S,E,T,N). Also reported as the [`xdr_from_proxy_delete_success`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_delete_success), [`xdr_from_proxy_delete_error`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_delete_error), [`xdr_from_proxy_delete_timeout`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_delete_timeout), and [`xdr_from_proxy_delete_not_found`](https://aerospike.com/docs/database/reference/metrics#namespace__xdr_from_proxy_delete_not_found) statistics. --- #### `permitted: masking rule added` `INFO` Message: ```bash permitted | client: 127.0.0.1:54122 | authenticated user: bob | action: masking | detail: ns=test;set=users;bin=ssn;type=string;function=redact;value=*;position=-4 ``` Description: Audit log entry for successfully adding a masking rule. Logged when a user with the `masking-admin` privilege successfully adds or modifies a data masking rule. Context: audit Introduced: 8.1.1 Removed: \- Occurs: Occurs when a masking rule is successfully added by an authorized user. See [Data masking](https://aerospike.com/docs/database/learn/security/data-masking) and [Audit trail](https://aerospike.com/docs/database/manage/security/audit-trail) for details. Parameters: client The IP address and port of the client connection. --- authenticated user The authenticated user who added the masking rule. --- action The action type, which is `masking` for masking rule operations. --- detail The masking rule details including: - `ns` - The namespace - `set` - The set name - `bin` - The bin name - `type` - The data type (string, integer, bool, float) - `function` - The masking function (redact or constant) - Additional function parameters such as `value`, `position`, `length` --- #### `permitted: masking rule removed` `INFO` Message: ```bash permitted | client: 127.0.0.1:54122 | authenticated user: bob | action: masking | detail: ns=test;set=users;bin=ssn;type=string;function=remove ``` Description: Audit log entry for successfully removing a masking rule. Logged when a user with the `masking-admin` privilege successfully removes a data masking rule. Context: audit Introduced: 8.1.1 Removed: \- Occurs: Occurs when a masking rule is successfully removed by an authorized user. See [Data masking](https://aerospike.com/docs/database/learn/security/data-masking) and [Audit trail](https://aerospike.com/docs/database/manage/security/audit-trail) for details. Parameters: client The IP address and port of the client connection. --- authenticated user The authenticated user who removed the masking rule. --- action The action type, which is `masking` for masking rule operations. --- detail The masking rule details including: - `ns` - The namespace - `set` - The set name - `bin` - The bin name - `type` - The data type (string, integer, bool, float) - `function` - The value `remove` indicating the rule was removed --- #### `process resource usage ticker` `INFO` Message: ```bash process: cpu-pct 28 threads (8,67,46,46) heap-kbytes (71477,72292,118784) heap-efficiency-pct 60.2 ``` Context: info Introduced: 4.7.0.2 Removed: \- Parameters: cpu-pct Percent CPU time Aerospike was scheduled since previously reported. Corresponds to [`process_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__process_cpu_pct). --- threads Thread statistics, in order: ([`threads_joinable`](https://aerospike.com/docs/database/reference/metrics#node_stats__threads_joinable), [`threads_detached`](https://aerospike.com/docs/database/reference/metrics#node_stats__threads_detached), [`threads_pool_total`](https://aerospike.com/docs/database/reference/metrics#node_stats__threads_pool_total), [`threads_pool_active`](https://aerospike.com/docs/database/reference/metrics#node_stats__threads_pool_active)). Introduced in Database 5.6.0. --- heap-kbytes Heap statistics, in order: ([`heap_allocated_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_allocated_kbytes), [`heap_active_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_active_kbytes), [`heap_mapped_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_mapped_kbytes)). --- heap-efficiency-pct Indicates the jemalloc heap fragmentation. This represents the [`heap_allocated_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_allocated_kbytes) / [`heap_mapped_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_mapped_kbytes) ratio (prior to 5.7), or the [`heap_allocated_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_allocated_kbytes) / [`heap_active_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_active_kbytes) ratio (6.0 or later). A lower number indicates a higher fragmentation rate. Corresponds to the [`heap_efficiency_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_efficiency_pct) statistic. --- #### `query-threads-limit reached (ticker)` `WARNING` Message: ```bash WARNING (info): query-threads-limit reached - can't start new query ``` Description: Rate-limited ticker warning emitted when the configured `query-threads-limit` is reached and new queries cannot start. Prior to Database 8.1.2 this was logged as a per-query warning; it is now consolidated into the ticker to reduce log volume under sustained load. Per-job failure warnings (“failed to start”) are demoted to detail level. Context: info Introduced: 8.1.2 Removed: \- Parameters: limit The configured `query-threads-limit` value. --- #### `resolved mesh node hostname` `INFO` Message: ```bash resolved mesh node hostname aerospike-a-0.test-runner.test-runner to 172.22.22.3 ``` Description: Mesh node hostname resolves to IPv4 address. Context: info Introduced: 8.1.0 Removed: \- --- #### `resolved mesh node hostname` `INFO` Message: ```bash resolved mesh node hostname aerospike-a-0.test-runner.test-runner to fdae:22:22::3 ``` Description: Mesh node hostname resolves to IPv6 address. Context: info Introduced: 8.1.0 Removed: \- --- #### `role violation: unauthorized masking rule operation` `INFO` Message: ```bash role violation | client: 127.0.0.1:35488 | authenticated user: dave | action: masking | detail: namespace=test;set=users;bin=ssn;type=string;function=remove ``` Description: Audit log entry for an unauthorized attempt to modify a masking rule. Logged when a user without the `masking-admin` privilege attempts to add, modify, or remove a data masking rule. Context: audit Introduced: 8.1.1 Removed: \- Occurs: Occurs when a user without appropriate privileges attempts a masking rule operation. See [Data masking](https://aerospike.com/docs/database/learn/security/data-masking) and [Audit trail](https://aerospike.com/docs/database/manage/security/audit-trail) for details. Parameters: client The IP address and port of the client connection. --- authenticated user The authenticated user who attempted the unauthorized masking operation. --- action The action type, which is `masking` for masking rule operations. --- detail The masking rule details for the attempted operation including: - `namespace` - The namespace - `set` - The set name - `bin` - The bin name - `type` - The data type (string, integer, bool, float) - `function` - The masking function (redact, constant, remove) --- #### `sindex-create bin not supported for set index` `WARNING` Message: ```bash WARNING (info): sindex-create: bin parameter is not supported for set index ``` Description: The `bin` parameter was specified when creating a set index via [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create). Set indexes do not index bins; remove the `bin` parameter. Context: info Introduced: 8.1.2 Removed: \- --- #### `sindex-create context not supported for set index` `WARNING` Message: ```bash WARNING (info): sindex-create: context parameter is not supported for set index ``` Description: The `context` parameter was specified when creating a set index via [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create). Set indexes do not support CDT context; remove the `context` parameter. Context: info Introduced: 8.1.2 Removed: \- --- #### `sindex-create exp not supported for set index` `WARNING` Message: ```bash WARNING (info): sindex-create: exp parameter is not supported for set index ``` Description: The `exp` parameter was specified when creating a set index via [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create). Set indexes do not support expressions; remove the `exp` parameter. Context: info Introduced: 8.1.2 Removed: \- --- #### `sindex-create failed to get/create set` `WARNING` Message: ```bash WARNING (info): sindex-create: failed to get/create set '' in namespace '' ``` Description: The server could not find or create the specified set in the namespace when processing a [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create) command for a set index. Context: info Introduced: 8.1.2 Removed: \- Parameters: set Set name. --- ns Namespace name. --- #### `sindex-create indexdata not supported for set index` `WARNING` Message: ```bash WARNING (info): sindex-create: indexdata parameter is not supported for set index ``` Description: The deprecated `indexdata` parameter was specified when creating a set index via [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create). Set indexes do not use bin data; remove the `indexdata` parameter. Context: info Introduced: 8.1.2 Removed: \- --- #### `sindex-create namespace not found` `WARNING` Message: ```bash WARNING (info): sindex-create: namespace '' not found ``` Description: The namespace specified in the [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create) command does not exist on this node. Context: info Introduced: 8.1.2 Removed: \- Parameters: ns The namespace name that was not found. --- #### `sindex-create type not supported for set index` `WARNING` Message: ```bash WARNING (info): sindex-create: type parameter is not supported for set index ``` Description: The `type` parameter was specified when creating a set index via [`sindex-create`](https://aerospike.com/docs/database/reference/info#sindex-create). Set indexes do not require a bin type; remove the `type` parameter. Context: info Introduced: 8.1.2 Removed: \- --- #### `sindex-delete deleting in SMD` `INFO` Message: ```bash INFO (info): sindex-delete: deleting in SMD ``` Description: A set index or secondary index is being deleted and the deletion is being propagated through SMD (System Metadata) to all nodes in the cluster. Context: info Introduced: 8.1.2 Removed: \- Parameters: key The SMD key identifying the index being deleted. --- #### `starting Aerospike build` `INFO` Message: ```bash starting Aerospike EDITION Edition build BUILD os OS arch ARCH SHAS ``` Context: info Introduced: 8.1.0 Removed: \- Occurs: Logged at startup. SHAs are additional information to help Aerospike support quickly and unambiguously identify a build’s source. Parameters: Enterprise Edition (EE) Jul 10 2025 20:05:37 GMT: INFO (as): (as.c:306) starting Aerospike Enterprise Edition build 8.1.0.0 os debian13 arch x86\_64 sha 324013f ee-sha bb6b72c --- Federal Edition (FE) Jul 10 2025 20:05:37 GMT: INFO (as): (as.c:306) starting Aerospike Enterprise Federal build 8.1.0.0 os debian13 arch x86\_64 sha 324013f ee-sha bb6b72c fips-sha 107d078 --- Community Edition (CE) Jul 10 2025 20:02:11 GMT: INFO (as): (as.c:306) starting Aerospike Community Edition build 8.1.0.0 os debian13 arch x86\_64 sha 324013f --- #### `system cpu/memory stats ticker` `INFO` Message: ```bash system: total-cpu-pct 76 user-cpu-pct 44 kernel-cpu-pct 32 free-mem-kbytes 7462956 free-mem-pct 52 thp-mem-kbytes 4096 host-free-mem-kbytes 7462956 host-free-mem-pct 52 ``` Context: info Introduced: 4.7.0.2 Removed: \- Parameters: total-cpu-pct Percent of time the CPU spent servicing user-space or kernel space tasks. Corresponds to [`system_total_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_total_cpu_pct). --- user-cpu-pct Percent of time the CPUs spent servicing user-space tasks. Corresponds to [`system_user_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_user_cpu_pct). --- kernel-cpu-pct Percent of time the CPUs spent servicing kernel-space tasks. Corresponds to [`system_kernel_cpu_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_kernel_cpu_pct). --- free-mem-kbytes Amount of free RAM in kilobytes. When [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is enabled, reports control group (cgroup)-aware values. Corresponds to [`system_free_mem_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_free_mem_kbytes). --- free-mem-pct Percentage of all RAM free, rounded to nearest percent. When [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) is enabled, reports control group (cgroup)-aware values. Corresponds to [`system_free_mem_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_free_mem_pct). --- thp-mem-kbytes Amount of memory in use by the Transparent Huge Page mechanism, in kilobytes. Corresponds to [`system_thp_mem_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_thp_mem_kbytes). Displayed in 5.7 and later. --- host-free-mem-kbytes Amount of free RAM in kilobytes for the host. Always reports host-level memory regardless of [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking). Corresponds to [`host_free_mem_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__host_free_mem_kbytes). Displayed in Database 8.1.2 and later. --- host-free-mem-pct Percentage of all RAM free for the host, rounded to nearest percent. Always reports host-level memory regardless of [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking). Corresponds to [`host_free_mem_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__host_free_mem_pct). Displayed in Database 8.1.2 and later. --- #### `tomb-raider-unmark-threads enterprise-only` `WARNING` Message: ```bash WARNING (info): tomb-raider-unmark-threads is enterprise-only ``` Description: Logged on Community Edition when [`tomb-raider-unmark-threads`](https://aerospike.com/docs/database/reference/config#namespace__tomb-raider-unmark-threads) is set. This parameter applies only to Enterprise Edition. Context: info Introduced: 8.1.2 Removed: \- --- #### `xdr-dc: nodes latency` `INFO` Message: ```bash xdr-dc dc2: nodes 8 latency-ms 19 ``` Context: info Introduced: 6.0.0 Removed: \- Occurs: Displayed periodically for each XDR destination cluster (DC), every 10 seconds by default. Parameters: xdr-dc Name of the XDR destination cluster. --- nodes See [`nodes`](https://aerospike.com/docs/database/reference/metrics#xdr__nodes). --- latency-ms See [`latency_ms`](https://aerospike.com/docs/database/reference/metrics#xdr__latency_ms). --- ## Key busy #### `info-command detail: client command` `DETAIL` Message: ```bash DETAIL (info-command): (:) client command '' ``` Context: info-command Introduced: 7.1.0 Removed: \- Occurs: A detail logging context for logging all info protocol commands received by the server --- #### `key-busy detail` `DETAIL` Message: ```bash DETAIL (key-busy): (:) {} {digest} from ``` Context: key-busy Introduced: 6.4.0 Removed: \- Occurs: Describes the digest, transaction type, and transaction source of a possibly hot key. This message is issued whenever a transaction fails with a `KEY_BUSY` error (14). Parameters: Possible transaction types: - read - write - delete - udf - batch-sub-read - batch-sub-write - batch-sub-delete - batch-sub-udf --- Possible source types: - client ip address:port - proxy - bg-udf - bg-ops - re-repl See [Hot Key Error code 14](https://support.aerospike.com/s/article/Why-does-my-client-return-Error-code-14-Hot-key). --- ## Masking #### `masking - function 'nullify' for type 'integer' not found` `WARNING` Message: ```bash (d29e5fa1c53da18c) masking - function 'nullify' for type 'integer' not found ``` Description: Warning logged when a masking function is not available for the data type of a bin. This can occur when a masking rule is applied to a bin but the actual data type of the bin value does not support the configured masking function. Context: masking Introduced: 8.1.1 Removed: \- Occurs: Occurs when reading a record where the masking function does not support the bin’s data type. Review the masking rule configuration to ensure the function and type are compatible. See [Data masking](https://aerospike.com/docs/database/learn/security/data-masking) for details on supported functions and types. Parameters: digest The 160-bit digest of the record, in hex. --- ## Migrate #### `handle insert: dropping binless pickle` `WARNING` Message: ```bash handle insert: binless pickle, dropping {digest}:0x2b2c08f859bd4eb401982e038a2bdcae2b74c853 ``` Description: Attempted to migrate a record that had no bins, so didn’t insert it into the receiving partition. Context: migrate Introduced: \- Removed: \- Parameters: digest The 160-bit digest of the record, in hex. --- #### `migrate: handle insert: bad record` `WARNING` Message: ```bash migrate: handle insert: got bad record ``` Description: Inbound record for migration is corrupt. Appears with the `WARNING` message about `record too small 0` and is documented in the following [`Why Do I See Stalled Migrations And "record too small" Errors In The Log?`](https://support.aerospike.com/s/article/Why-Do-I-See-Stalled-Migrations-And-record-too-small-Errors-In-The-Log). Context: migrate Introduced: \- Removed: \- --- #### `migrate: handle insert: no record` `WARNING` Message: ```bash migrate: handle insert: got no record ``` Description: Appears on the destination (inbound) node with the warning message about `handle insert: got bad record.` Associated with the new `unreadable digest` warning for Database 6.0.0. Context: migrate Introduced: \- Removed: \- --- #### `migrate: record flatten failed` `WARNING` Message: ```bash migrate: record flatten failed ce8a775a68c93d49 ``` Description: The storage is full for this namespace, during the time of migration. Resolve by allocating more storage for namespace, or reducing volume of data in namespace. Context: migrate Introduced: \- Removed: \- Parameters: record Record identifier. --- #### `migrate: unreadable digest` `WARNING` Message: ```bash migrate: unreadable digest ``` Description: Appears on the source node when the record could not be read locally. Introduced with Database 6.0.0. Associated with the `handle insert: got no record` warning. Context: migrate Introduced: \- Removed: \- --- #### `missing acks from node` `WARNING` Message: ```bash missing acks from node BB9030011AC4202 ``` Description: This means that a particular migration thread has begun to throttle because it has 16MB or more of un-acknowledged records, and has remained in that state for at least 5 seconds. The message prints every 5 seconds until the outstanding acks drop below the threshold. This happens when migration has been pushed higher than your network or machine/disks can sustain. To mitigate, decrease [`migrate-threads`](https://aerospike.com/docs/database/reference/config#service__migrate-threads) down to a minimum of 1. This change does not take effect immediately; threads terminate as the partition they are handling completes migration. This warning could also be triggered if channel-bulk-recv-threads was reduced during ongoing migrations. Context: migrate Introduced: \- Removed: \- Parameters: node The destination node of the migrate thread. --- ## Mrt audit #### `monitor aborting mrt-id` `INFO` Message: ```bash {namespace} monitor aborting mrt-id 123456789abcdef0 ``` Description: Indicates when monitors kick in to abort transactions. Context: mrt-audit Introduced: 8.0.0 Removed: \- Parameters: mrt-id Transaction ID that was aborted. --- #### `monitor committing mrt-id` `INFO` Message: ```bash {namespace} monitor committing mrt-id 123456789abcdef0 ``` Description: Indicates when monitors kick in to commit transactions. Context: mrt-audit Introduced: 8.0.0 Removed: \- Parameters: mrt-id Transaction ID that was committed. --- ## Msg #### `malloc failure` `CRITICAL` Message: ```bash malloc ``` Description: Indicates a shortage of memory. Make sure nodes have enough memory. Context: msg Introduced: \- Removed: \- --- ## Namespace #### `set names limit reached` `WARNING` Message: ```bash at set names limit, can't add set ``` Description: This warning indicates that the maximum number of sets per namespace has been breached. A namespace can hold a maximum of 1023 sets prior to server 7 and 4095 sets since Database 7. Such an issue can cause migrations to get stuck and writes to fail. See [How to clear up set names when they exceed the limit](https://support.aerospike.com/s/article/How-to-clear-up-set-names-when-they-exceed-the-limit). Context: namespace Introduced: \- Removed: \- --- #### `can't add SET (at sets limit)` `WARNING` Message: ```bash can't add SET (at sets limit) ``` Description: Indicates that the maximum number of sets per namespace has been breached. A namespace can hold a maximum of 1023 sets prior to Database 7 and 4095 sets since Database 7. Such an issue can cause migrations to get stuck and writes to fail. See [How to clear up set names when they exceed the limit](https://support.aerospike.com/s/article/How-to-clear-up-set-names-when-they-exceed-the-limit). Context: namespace Introduced: \- Removed: \- Parameters: set Name of set that was being created in excess of the limit. --- #### `fail persistent memory delete` `CRITICAL` Message: ```bash fail persistent memory delete ``` Description: Indicates an issue deleting PMem memory on startup. Can be resultant from incorrect permissions on the `/mnt/pmem` directory. Context: namespace Introduced: \- Removed: \- --- #### `namespace mrt-finish ticker` `INFO` Message: ```bash {namespace} mrt-finish: verify-read (100,0,0) roll-forward (50,0,0) roll-back (1,0,0) ``` Description: Appears if any of the following stats are non-zero: - [mrt\_verify\_read\_success](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_verify_read_success) - [mrt\_verify\_read\_error](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_verify_read_error) - [mrt\_verify\_read\_timeout](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_verify_read_timeout) Context: namespace Introduced: 8.0.0 Removed: \- --- #### `namespace mrt-monitor-finish ticker"` `INFO` Message: ```bash {namespace} mrt-monitor-finish: roll-forward (0,0,0) roll-back (1,0,0) ``` Description: Appears if any of the following stats are non-zero: - [mrt\_roll\_forward\_success](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_forward_success) - [mrt\_roll\_forward\_error](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_forward_error) - [mrt\_roll\_forward\_timeout](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_forward_timeout) - [mrt\_roll\_back\_success](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_back_success) - [mrt\_roll\_back\_error](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_back_error) - [mrt\_roll\_back\_timeout](https://aerospike.com/docs/database/reference/metrics#namespace__mrt_roll_back_timeout) Context: namespace Introduced: 8.0.0 Removed: \- --- #### `namespace appeals ticker` `INFO` Message: ```bash {namespace_name} appeals: remaining-tx 0 active (0,19) ``` Description: Information about the number of partition appeals not yet sent and currently being sent. Partition appeals occur for namespaces operating under[`strong-consistency`](https://aerospike.com/docs/database/manage/namespace/consistency) mode when a node needs to validate the records it has when joining the cluster. Context: namespace Introduced: \- Removed: \- Parameters: remaining-tx Current value for [`appeals_tx_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_tx_remaining). --- active (active\_tx, active\_rx) A tuple based on [`appeals_tx_active`](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_tx_active) and [`appeals_rx_active`](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_rx_active). --- #### `namespace can't attach persistent memory base block` `WARNING` Message: ```bash ns can’t attach persistent memory base block: block does not exist ``` Description: A missing shared memory block. This typically happens when a node is rebooted, but shared memory blocks can be deleted for other reasons. The node must perform a cold start. Context: namespace Introduced: \- Removed: \- --- #### `no valid persistent memory blocks - cold start` `INFO` Message: ```bash {ns_name} found no valid persistent memory blocks, will cold start ``` Description: `treex` and `base` shared memory blocks are missing. This typically happens when a node is rebooted or after an ungraceful shutdown. The node must perform a [cold start](https://aerospike.com/docs/database/manage/database/cold-start). Context: namespace Introduced: \- Removed: \- --- #### `namespace persisted arena stages` `INFO` Message: ```bash {ns_name} persisted arena stages ``` Description: This message is one of a sequence of messages logged during Aerospike server shutdown of storage-engine device namespaces. The message signifies that the arena stages for the namespace have been persisted to storage. An unusual delay in the appearance of this message during shutdown might be due to [`index-type`](https://aerospike.com/docs/database/reference/config#namespace__index-type) configured to `pmem`. Context: namespace Introduced: 4.6.0.2 Removed: \- --- #### `namespace persisted tree roots` `INFO` Message: ```bash {ns_name} persisted tree roots ``` Description: This message is one of a sequence of messages logged during Aerospike server shutdown of storage-engine device namespaces. The message signifies that the namespace’s common partition index tree information has been persisted to storage. An unusual delay in the appearance of this message during shutdown might be due to a high number of [`partition-tree-sprigs`](https://aerospike.com/docs/database/reference/config#namespace__partition-tree-sprigs) configured for the namespace. Context: namespace Introduced: 4.6.0.2 Removed: \- --- #### `namespace persisted trusted base block` `INFO` Message: ```bash {ns_name} persisted trusted base block ``` Description: This message is one of a sequence of messages logged during Aerospike server shutdown of storage-engine device namespaces. The message signifies that the persistent memory base block for the namespace has been persisted to storage with “trusted” status. “trusted” status is a necessary condition for a subsequent fast restart of the namespace. Context: namespace Introduced: 4.6.0.2 Removed: \- --- ## Network #### `fabric_connection_process_readable - connection timed out` `WARNING` Message: ```bash fabric_connection_process_readable() recv_sz -1 msg_sz 0 errno 110 Connection timed out ``` Description: The warning message indicates that a fabric connection timed out. In Database 5.6.0 and later, the log line has the node-id to identify which node is having the fabric connection time out. Context: network Introduced: \- Removed: \- --- ## Nsup #### `failed to create evict-prep thread` `CRITICAL` Message: ```bash {NAMESPACE} failed to create evict-prep thread 5 ``` Description: Indicates a shortage of memory. Make sure nodes have enough memory. Context: nsup Introduced: \- Removed: \- Parameters: NAMESPACE The namespace nsup was looking at when memory ran low. --- thread Thread identifier. --- #### `hwm breached - nothing to evict` `WARNING` Message: ```bash {bigdata} hwm breached but nothing to evict ``` Description: The amount of data in memory ([high-water-memory-pct](https://aerospike.com/docs/database/reference/config#namespace__high-water-memory-pct)) or on disk ([high-water-disk-pct](https://aerospike.com/docs/database/reference/config#namespace__high-water-disk-pct), [mounts-high-water-pct](https://aerospike.com/docs/database/reference/config#namespace__mounts-high-water-pct)) has exceeded the limits set for the namespace, triggering evictions, but there is no data with a finite TTL to be evicted. Context: nsup Introduced: \- Removed: \- Parameters: ns Namespace where the high water mark was breached. --- #### `eviction limit breached - index/device usage` `WARNING` Message: ```bash {} breached eviction limit (), sys-memory pct:41, indexes-memory sz:0 (0 + 0 + 0), index-device sz:1073741824000 used-pct 88, data used-pct:58 ``` Context: nsup Introduced: \- Removed: \- Occurs: Checking memory or disk usage at start of nsup cycle and finding that the high-water mark has been breached Parameters: breached eviction limit `memory` or `disk` depending on which limit was breached --- memory Memory used in bytes (primary index + secondary indexes + data in memory), and the amount of the high-water mark (total available \* [`high-water-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__high-water-memory-pct). For Database 5.6.0 and later the set index memory used is also added. --- index-device Used space on device for `index-type flash` in bytes and the amount of the high-water mark (total available \* [`mounts-high-water-pct`](https://aerospike.com/docs/database/reference/config#namespace__mounts-high-water-pct) --- disk Disk space used in bytes and the amount of the high-water mark (total available \* [`high-water-disk-pct`](https://aerospike.com/docs/database/reference/config#namespace__high-water-disk-pct) --- #### `namespace stop-writes limit breached nsup ticker` `WARNING` Message: ```bash {} breached stop-writes limit (), sys-memory pct:96, indexes-memory sz:107374182400 (107374182400 + 0 + 0), data avail-pct:20 used-pct:58 ``` Context: nsup Introduced: \- Removed: \- Occurs: Upon checking memory at start of NSUP cycle and finding that a threshold triggering stop writes has been breached (either `memory` based on [`stop-writes-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-pct) or [`stop-writes-sys-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-sys-memory-pct), or `disk` based on [`min-avail-pct`](https://aerospike.com/docs/database/reference/config#namespace__min-avail-pct) or [`max-used-pct`](https://aerospike.com/docs/database/reference/config#namespace__max-used-pct)). Parameters: breached stop-writes limit `sys-memory`, `memory`, `device-avail-pct`, `device-used-size` or some combination depending on which threshold was breached --- sys-memory Total system memory used in bytes greater than the `stop-writes-sys-memory-pct` threshold, triggered when (100 - [`system_free_mem_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_free_mem_pct) >= [`stop-writes-sys-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-sys-memory-pct)) --- indexes-memory Memory used in bytes greater than the [`stop-writes-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-pct) threshold (total available \* [`stop-writes-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-pct)). Prior to Database 5.6.0, tracked under [`memory_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__memory_used_bytes) (primary index + secondary indexes + data in memory). Database 5.6.0 and later, tracked under [`memory_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__memory_used_bytes) (primary index + set index bytes + secondary indexes + data in memory). --- data avail-pct The available contiguous free space, tracked under [`device_available_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__device_available_pct) --- data used-pct The amount of used disk space, tracked as the ratio of the [`device_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__device_used_bytes) and [`device_total_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__device_total_bytes) metrics. --- #### `failed set evict-void-time` `WARNING` Message: ```bash {namespace} failed set evict-void-time 328967879 ``` Description: When [`high-water-disk-pct`](https://aerospike.com/docs/database/reference/config#namespace__high-water-disk-pct), [`high-water-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__high-water-memory-pct), or [`mounts-high-water-pct`](https://aerospike.com/docs/database/reference/config#namespace__mounts-high-water-pct) is breached on any node and [eviction](https://support.aerospike.com/s/article/FAQ---How-does-Aerospike-eviction-work) starts, the timestamp before which records need to be evicted is propagated to all nodes through the System Meta Data (SMD) mechanism, to ensure that no orphaned replicas are left anywhere in the cluster. If not all nodes have acknowledged the message within 5 seconds, this message is logged. Occasional instances of this message do not indicate a serious problem, as the timestamp is picked up on the next namespace supervisor (nsup) cycle. Context: nsup Introduced: \- Removed: \- Parameters: ns The namespace where evictions were triggered. --- evict void time The timestamp, in seconds since the Aerospike Epoch of 2010-01-01T00:00:00, before which records should be evicted. --- #### `would evict all eligible records - not evicting` `WARNING` Message: ```bash {namespace} would evict all 146897768 records eligible - not evicting! ``` Description: The namespace supervisor (nsup) configuration required it to evict all expirable records (records with a configured time-to-live or TTL). Most commonly seen when [`evict-tenths-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-tenths-pct) is set to 1000 or greater, but could also happen when the distribution of TTLs is extremely skewed. Context: nsup Introduced: \- Removed: \- Parameters: namespace Namespace where this happened. --- record count Total number of evictable records (that is, records with a TTL) in the namespace. --- #### `xdr-tomb-raid-start: threads` `INFO` Message: ```bash {} xdr-tomb-raid-start: threads ``` Description: XDR tomb raider has started for the namespace. This log entry appeared under the NSUP context prior to Database 7.1.0. Context: xdr Introduced: \- Removed: \- Occurs: Expected to run every [xdr-tomb-raider-period](https://aerospike.com/docs/database/reference/config#namespace__xdr-tomb-raider-period) seconds. Parameters: namespace The namespace for which the tombstones are being dropped --- n The number of threads configured for the XDR tomb raider --- #### `no records below eviction void-time` `WARNING` Message: ```bash {ns-name} no records below eviction void-time 329702784 - threshold bucket 9998, width 3154 sec, count 4000000 > target 20000 (0.5 pct) ``` Context: nsup Introduced: \- Removed: \- Occurs: Checking the eviction buckets from the bottom up, but no records are found until one bucket has enough to put the total over the limit for one eviction cycle. Increase the [`evict-tenths-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-tenths-pct) to increase the target number to evict per cycle or increase the [`evict-hist-buckets`](https://aerospike.com/docs/database/reference/config#namespace__evict-hist-buckets) in order to reduce the number of records per bucket and get it below the target. See [Evictions](https://aerospike.com/docs/database/manage/namespace/retention/#managing-evictions) and [How does Aerospike eviction work](https://support.aerospike.com/s/artdocumentation) for more information. Parameters: void-time Timestamp at the top of the bucket that breached the eviction limit --- threshold bucket Which bucket out of the [`evict-hist-buckets`](https://aerospike.com/docs/database/reference/config#namespace__evict-hist-buckets) breached the eviction limit --- width Width of the bucket in seconds --- count Total records found in this bucket --- target Number of records allowed for this eviction cycle and the [`evict-tenths-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-tenths-pct) used to calculate it --- #### `nsup deleted namespace - increase nsup threads?` `WARNING` Message: ```bash {ns-name} nsup deleted 1.56% of namespace - configure more nsup threads? ``` Context: nsup Introduced: 6.3.0 Removed: \- Occurs: When NSUP cycle takes more than 2 hours and deletes more than 1% of namespace. --- #### `namespace nsup-done nsup ticker` `INFO` Message: ```bash {ns-name} nsup-done: non-expirable 42162 expired (576066,922) evicted (24000935,259985) evict-ttl 134000 total-ms 155 ``` Context: nsup Introduced: 4.5.1 Removed: \- Occurs: Logged when the namespace supervisor (nsup) completes expiration or eviction processing for a namespace. See the most recently logged `nsup-start` entry for the namespace to determine which type of processing has just completed. Expiration processing never evicts records, but eviction processing can expire records. Parameters: non-expirable The number of records without a TTL. These records do not expire and are never eligible for eviction. --- expired Number of records removed due to expiration; total since the node started and total for the current nsup cycle. In this example, 922 records expired in the most recent nsup cycle, and 576066 records have expired since the node was last started. --- evicted Number of records evicted (early-expired); total since the node started and total for the current nsup cycle. In this example, nsup evicted 259985 records in the most recent cycle, and 24000935 records since the node was last started. --- evict-ttl The high-end expiration-time of evicted (early-expired) records (in seconds). --- total-ms Duration of the just completed nsup expiration or eviction processing cycle, in milliseconds. In this example, the processing cycle completed in 155 milliseconds --- #### `namespace nsup-start nsup ticker` `INFO` Message: ```bash {ns-name} nsup-start: evict-threads 1 evict-ttl 2745 evict-void-time (287665530,287665930) ``` Context: nsup Introduced: 4.6.0 Removed: \- Occurs: Logged when the namespace supervisor (nsup) begins eviction processing for a namespace. Parameters: evict-threads The number of threads to be used for the eviction processing cycle. --- evict-ttl The specified eviction depth for the namespace, expressed as a time to live threshold in seconds, below which any eligible records are evicted. --- evict-void-time The current effective eviction depth and the specified eviction depth for the namespace. Each is expressed as a void time, in seconds since 1 January 2010 UTC. --- #### `nsup-start: expire-threads` `INFO` Message: ```bash {ns-name} nsup-start: expire-threads 1 ``` Context: nsup Introduced: 4.6.0 Removed: \- Occurs: Logged when the namespace supervisor (nsup) begins expiration processing for a namespace. Parameters: expire-threads The number of threads to be used for the expiration processing cycle. Corresponds to the configured [`nsup-threads`](https://aerospike.com/docs/database/reference/config#namespace__nsup-threads). --- #### `sindex-gc completed` `INFO` Message: ```bash {ns-name} sindex-gc: Processed: 3133360101, found:365961945, deleted: 365952323: Total time: 62667962 ms ``` Context: nsup Introduced: 3.1.04.0 Removed: 4.6.0 Occurs: Secondary index (sindex) garbage collection cycle summary. Replaced by “sindex-gc-done” message in 4.6.0. Parameters: Processed Count of sindex entries that have been checked. Corresponds to the [`sindex_gc_objects_validated`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_objects_validated) statistic. --- found Count of sindex entries found eligible for garbage collection. Corresponds to the [`sindex_gc_garbage_found`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_garbage_found) statistic. --- deleted Count of sindex entries deleted through garbage collection (may be lower than above number if those entries got deleted while the garbage collector was running, for example through a competing `truncate` command). Corresponds to the [`sindex_gc_garbage_cleaned`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_garbage_cleaned) statistic. --- Total time Duration of a cycle of sindex garbage collection in milliseconds. --- #### `sindex-gc start` `INFO` Message: ```bash {ns-name} sindex-gc start ``` Context: nsup Introduced: 3.1.04.0 Removed: 4.6.0 Occurs: Starting secondary index (sindex) garbage collection. Replaced by “sindex-gc-start” message in 4.6.0. --- ## Os #### `OS_CHECK failed` `WARNING` Message: ```bash failed OS_CHECK check - MESSAGE ``` Description: Indicates that a linux [best-practices](https://aerospike.com/docs/database/install/linux) was violated at startup. Context: os Introduced: 5.7.0 Removed: \- Parameters: OS\_CHECK Name of the check which was violated, could be one of the following: - [min-free-kbytes](https://aerospike.com/docs/database/install/linux) - [swappiness](https://aerospike.com/docs/database/install/linux) - [thp-enabled](https://aerospike.com/docs/database/install/linux) - [thp-defrag](https://aerospike.com/docs/database/install/linux) - [zone-reclaim-mode](https://aerospike.com/docs/database/install/linux) --- sMessage Description of how the best-practice was violated. --- ## Particle #### `cdt_select_map invalid key list` `WARNING` Message: ```bash WARNING (particle): cdt_select_map() invalid key list ``` Description: A CDT map selection operation received an invalid key list. Verify that the keys passed to the map operation are well-formed and of the correct type. Context: particle Introduced: 8.1.2 Removed: \- Parameters: op The CDT operation code. --- #### `cdt_select_map key list not ordered` `WARNING` Message: ```bash WARNING (particle): cdt_select_map() key list not ordered ``` Description: A CDT map selection operation received a key list that is not in sorted order. Key lists must be ordered for MAP\_KEYS selection operations. Context: particle Introduced: 8.1.2 Removed: \- Parameters: op The CDT operation code. --- #### `cdt_select_stack_init AND invalid ctx type` `WARNING` Message: ```bash WARNING (particle): cdt_select_stack_init() AND invalid ctx type 0x at pair ``` Description: A CDT selection stack initialization encountered an AND predicate with an invalid context type at the specified pair index. Context: particle Introduced: 8.1.2 Removed: \- Parameters: type Hex context type value. --- pair Zero-based pair index in the selection predicate. --- #### `cdt_select_stack_init AND no preceding entry` `WARNING` Message: ```bash WARNING (particle): cdt_select_stack_init() AND at pair has no preceding entry ``` Description: A CDT selection stack initialization encountered an AND predicate without a preceding entry to combine with. Context: particle Introduced: 8.1.2 Removed: \- Parameters: pair Zero-based pair index in the selection predicate. --- #### `cdt_select_stack_init AND on EXP base` `WARNING` Message: ```bash WARNING (particle): cdt_select_stack_init() AND on EXP base at pair ``` Description: A CDT selection stack initialization encountered an AND predicate on an expression base at the specified pair index, which is not supported. Context: particle Introduced: 8.1.2 Removed: \- Parameters: pair Zero-based pair index in the selection predicate. --- #### `cdt_select_stack_init duplicate AND` `WARNING` Message: ```bash WARNING (particle): cdt_select_stack_init() duplicate AND at pair ``` Description: A CDT selection stack initialization encountered a duplicate AND predicate at the specified pair index. Context: particle Introduced: 8.1.2 Removed: \- Parameters: pair Zero-based pair index in the selection predicate. --- #### `parse_op - unable to build expression op` `WARNING` Message: ```bash parse_op - error 4 unable to build expression op ``` Description: This indicates that an attempt to execute an expression failed. Generic warning triggered by failure to parse the underlying expression encoding. Example: WARNING (particle): (expop.c:127) parse\_op - error 4 unable to build expression op Context: particle Introduced: 5.6.0 Removed: \- --- ## Partition #### `{ns_name} nodes quiesced` `INFO` Message: ```bash {ns_name} 2 of 5 nodes are quiesced ``` Description: Number of nodes quiesced in the cluster (or sub cluster). Context: partition Introduced: 4.3.1 Removed: \- Occurs: When the cluster changes (node addition, removal, or network splits) or when the cluster receives a ‘recluster’ info command. Parameters: nodes participating The number of nodes quiesced in this sub-cluster out of the total number of nodes observed in the sub-cluster. For [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) namespaces, it is full roster size instead of observed in the sub-cluster. --- #### `{ns_name} nodes participating - regime change` `INFO` Message: ```bash {ns_name} 5 of 6 nodes participating - regime 221 -> 223 ``` Description: Number of nodes participating in the cluster (or sub cluster) as well as the regime change. Context: partition Introduced: 4.0.0 Removed: \- Occurs: For [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) namespaces, when the cluster changes, because of node(s) leaving or joining the cluster (or network splits). Parameters: nodes participating The number of nodes participating in this cluster out of the total number of nodes for the full roster. --- regime This number increments every time there is a reclustering event. This is used in strong consistent namespace and is leveraged by the client libraries. For further details on regime, see [Strong consistency](https://aerospike.com/docs/database/learn/architecture/clustering/consistency-modes). --- #### `rebalanced - expected-migrations fresh-partitions` `INFO` Message: ```bash {ns_name} rebalanced: expected-migrations (1215,1224,1215) fresh-partitions 397 ``` Description: Number of partitions expected to migrate (transmitted, received, and signals) as well as other migration related statistics and fresh partition number. Context: partition Introduced: 3.13.0 Removed: \- Occurs: For non [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) namespaces, when the cluster changes, because of node addition, removal or network splits. Parameters: expected-migrations The number of partitions expected to migrate (Transmitted, Received, Signals) as part of this reclustering event. Those correspond to the [`migrate_tx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_tx_partitions_initial), [`migrate_rx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_rx_partitions_initial), and [`migrate_signals_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_signals_remaining) statistics respectively. --- fresh-partitions Number of partitions that are created fresh or empty because a number of nodes, greater than the replication factor, has left the cluster. --- #### `namespace rebalanced - unavailable partitions` `INFO` Message: ```bash {ns_name} rebalanced: regime 295 expected-migrations (826,826,826) expected-appeals 0 unavailable-partitions 425 ``` Description: Number of partitions expected to migrate (transmitted, received, and signals) as well as other migration related statistics and partition availability details. Context: partition Introduced: 4.0.0 Removed: \- Occurs: For [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) namespaces, when the cluster changes, because of node(s) leaving or joining the cluster (or network splits). Parameters: regime This number increments every time there is a reclustering event. This is used in strong consistency namespaces and is leveraged by the client libraries. For further details on regime, see [Strong consistency](https://aerospike.com/docs/database/learn/architecture/clustering/consistency-modes). --- expected-migrations The number of partitions expected to migrate (Transmitted, Received, Signals) as part of this reclustering event. Those correspond to the [`migrate_tx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_tx_partitions_initial), [`migrate_rx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_rx_partitions_initial), and [`migrate_signals_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_signals_remaining) statistics respectively. --- expected-appeals The number of appeals expected as part of this reclustering event. Appeals occur after a node has been cold-started. The replication state of each record is lost on cold-start and all records must assume an unreplicated state. An appeal resolves replication state from the partition’s acting master. These are important for performance; an unreplicated record must re-replicate to be read, which adds latency. During a rolling cold-restart, an operator may want to wait for the appeal phase to complete after each restart to minimize the performance impact of the procedure. Corresponds to the [`appeals_tx_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_tx_remaining) statistic but only at the initial time of the reclustering event. --- unavailable-partitions The number of partitions that are unavailable as the roster is not complete and all writes that have occurred to those partitions are not present. Partitions remaining unavailable after the cluster is formed by the full roster become dead and require the use of the [`revive`](https://aerospike.com/docs/database/reference/info#revive) command to make them available again, which could lead to inconsistencies, depending on what lead to those partition being dead. Revived nodes restore availability only when all nodes are trusted. Corresponds to the [`unavailable_partitions`](https://aerospike.com/docs/database/reference/metrics#namespace__unavailable_partitions) statistic. --- #### `rebalanced: expected-migrations dead-partitions` `WARNING` Message: ```bash {ns_name} rebalanced: regime 295 expected-migrations (826,826,826) fresh-partitions 397 expected-appeals 0 dead-partitions 425 ``` Description: Number of partitions expected to migrate (transmitted, received, and signals) as well as other migration related statistics and partition availability details. Context: partition Introduced: 4.0.0 Removed: \- Occurs: For [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) namespaces, when the cluster reforms with all roster members but resulting in dead partitions present. Parameters: regime This number increments every time there is a reclustering event. This is used in strong consistent namespace and is leveraged by the client libraries. For further details on regime, see [Strong consistency](https://aerospike.com/docs/database/learn/architecture/clustering/consistency-modes). --- expected-migrations The number of partitions expected to migrate (Transmitted, Received, Signals) as part of this reclustering event. Those correspond to the [`migrate_tx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_tx_partitions_initial), [`migrate_rx_partitions_initial`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_rx_partitions_initial), and [`migrate_signals_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__migrate_signals_remaining) statistics respectively. --- fresh-partitions Number of partitions that are created fresh or empty because a number of nodes, greater than the replication factor, has left the cluster. --- expected-appeals The number of appeals expected as part of this reclustering event. Appeals occur after a node has been cold-started. The replication state of each record is lost on cold-start and all records must assume an unreplicated state. An appeal resolves replication state from the partition’s acting master. These are important for performance; an unreplicated record must re-replicate to be read, which adds latency. During a rolling cold-restart, an operator may want to wait for the appeal phase to complete after each restart to minimize the performance impact of the procedure. Corresponds to the [`appeals_tx_remaining`](https://aerospike.com/docs/database/reference/metrics#namespace__appeals_tx_remaining) statistic but only at the initial time of the reclustering event. --- unavailable-partitions The number of partitions that are dead. Corresponds to the [`dead_partitions`](https://aerospike.com/docs/database/reference/metrics#namespace__dead_partitions) statistic. Requires the use of the [`revive`](https://aerospike.com/docs/database/reference/info#revive) command to make such partitions available again, which could lead to inconsistencies, depending on what lead to those partition being dead. --- ## Proto #### `protocol write fail` `DEBUG` Message: ```bash protocol write fail: fd 123 sz 30 errno 32 ``` Description: Client has closed the socket before the server could respond. Client may have timed out, has too many open connections, or there may be network issues. Context: proto Introduced: \- Removed: \- Parameters: fd File descriptor of the socket to the client. --- sz Size of the response that couldn’t be sent. --- errno Error number returned by OS. 32, EPIPE, is common. --- ## Query #### `could not find nbtr iterator` `WARNING` Message: ```bash Could not find nbtr iterator ``` Description: Indicates that there is a secondary index key with no corresponding digests (a bin value for which there are no records). Possibly a primary index and corresponding record got deleted, but the secondary index still exists. Subsequent round of garbage collection may correct the tree structure. May mean that partial results were returned during the query. If seen frequently, may mean that there is a structural issue with the secondary index. In this case, drop and re-create the index. Context: query Introduced: \- Removed: 5.7.0 --- #### `aggregation query job failed to start (demoted)` `DETAIL` Message: ```bash DETAIL (query): aggregation query job failed to start () ``` Description: An aggregation query job failed to start. Prior to Database 8.1.2 this was logged at WARNING level; it is now demoted to DETAIL. Context: query Introduced: 8.1.2 Removed: \- Parameters: job\_id Query job identifier. --- error Error code. --- #### `apply_ops_make_response bad number of ops` `WARNING` Message: ```bash WARNING (query): {} apply_ops_make_response: bad number of ops , can't exceed bin ops ``` Description: The number of operations in a query projection exceeds the maximum allowed bin operations for a single record. Reduce the number of projection operations. Context: query Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- count Number of operations provided. --- max Maximum allowed operations. --- digest Record digest. --- #### `basic query job failed to start (demoted)` `DETAIL` Message: ```bash DETAIL (query): basic query job failed to start () ``` Description: A basic query job failed to start. Prior to Database 8.1.2 this was logged at WARNING level; it is now demoted to DETAIL. The rate-limited ticker warning for `query-threads-limit` replaces per-job failure logging. Context: query Introduced: 8.1.2 Removed: \- Parameters: job\_id Query job identifier. --- error Error code. --- #### `basic query job no bin data and projection ops both set` `WARNING` Message: ```bash WARNING (query): basic query job no bin data and projection ops are both set ``` Description: A basic query was submitted with both “no bin data” (suppress record data) and projection operations set simultaneously, which is contradictory. Remove one of the conflicting options. Context: query Introduced: 8.1.2 Removed: \- --- #### `build_ops_response_msg can't get key` `WARNING` Message: ```bash WARNING (query): {} build_ops_response_msg: can't get key ``` Description: While building a query response with operation projection, the server could not retrieve the record key for the specified digest. The record may have been deleted or evicted between the index scan and the data read. Context: query Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- digest Record digest. --- #### `apply_ops_make_response failed` `WARNING` Message: ```bash WARNING (query): job - apply_ops_make_response failed ``` Description: The server failed to build the response for a query job that uses operation projection. This may indicate an invalid operation or an internal error. Context: query Introduced: 8.1.2 Removed: \- Parameters: job\_id Query job identifier. --- #### `ops-bg query job failed to start (demoted)` `DETAIL` Message: ```bash DETAIL (query): ops-bg query job failed to start () ``` Description: An ops-bg query job failed to start. Prior to Database 8.1.2 this was logged at WARNING level; it is now demoted to DETAIL. Context: query Introduced: 8.1.2 Removed: \- Parameters: job\_id Query job identifier. --- error Error code. --- #### `queries should only provide read ops` `WARNING` Message: ```bash WARNING (query): queries should only provide read ops - not ``` Description: A query included write operations in its projection, which is not supported. Queries accept only read operations (bin reads, CDT reads, read expressions). Remove the write operations from the query request. Context: query Introduced: 8.1.2 Removed: \- Parameters: count Number of non-read operations found. --- #### `starting basic query job` `DEBUG` Message: ```bash starting basic query job 562360700545657251 {test:sb_set5:} n-pids-requested (4096,0) rps 0 sample-max 0 socket-timeout 30000 from 127.0.0.1:51124 ``` Description: A basic query job is initiated on the specified namespace and set. Context: query Introduced: 6.0.0 Removed: \- Parameters: query job ID of the query. --- {namespace : set : index-name} The index name is either the name of the secondary index or `pi-query`. --- n-pids-requested (n-pids-requested, n-keyds-requested) - The first number is the number of partition IDs requested in this partitioned query. - The second number is the number of those partitions that start at a specified digest. This hints at whether this query is paginated, using a cursor. --- rps Records per second rate requested by the query policy. If not specified, the value of [`background-query-max-rps`](https://aerospike.com/docs/database/reference/config#namespace__background-query-max-rps) is in effect. --- sample-max The maximum number of records to return from the query, as specified by the query policy. --- socket-timeout The socket timeout requested by the client policy, in milliseconds. --- from Client IP address and port. --- #### `udf-bg query job failed to start (demoted)` `DETAIL` Message: ```bash DETAIL (query): udf-bg query job failed to start () ``` Description: A UDF background query job failed to start. Prior to Database 8.1.2 this was logged at WARNING level; it is now demoted to DETAIL. Context: query Introduced: 8.1.2 Removed: \- Parameters: job\_id Query job identifier. --- error Error code. --- ## Rbuffer #### `ring buffer file too small - digest log` `WARNING` Message: ```bash Ring buffer file /opt/aerospike/xdr/digestlog should be at least 18057 bytes Boot strap failed for digest log file (null) ``` Description: The minimum possible size of the XDR digestlog, as specified in [`xdr-digestlog-path`](https://aerospike.com/docs/database/reference/config#xdr__xdr-digestlog-path), is 18057 bytes. This is not a limitation that should ever come up in practice, because realistic sizes are in the tens or hundreds of GB. Context: rbuffer Introduced: \- Removed: \- --- #### `unable to create digest log file` `WARNING` Message: ```bash unable to create digest log file /opt/aerospike/xdr/digestlog: No such file or directory ``` Description: asd tries to create the digest log at start time if it does not already exist, but may fail due to permissions issues, a bad directory, path, etc. Context: rbuffer Introduced: \- Removed: \- Parameters: log path Path of the file that `asd` tried to create from [`xdr-digestlog-path`](https://aerospike.com/docs/database/reference/config#xdr__xdr-digestlog-path) --- system error The message from the OS giving the reason for failing to create the file --- ## Record #### `AS_PARTICLE AS_CDT_OP_LIST_APPEND failed` `WARNING` Message: ```bash {AS_PARTICLE} AS_CDT_OP_LIST_APPEND: failed ``` Description: This error may occur when a collection data type exists for the record and the write policy was set to `CREATE_ONLY`. Context: record Introduced: 5.2.0 Removed: \- --- #### `cdt_process_state_context_eval: bin empty, no create flags` `WARNING` Message: ```bash {AS_PARTICLE} cdt_process_state_context_eval() bin is empty and op has no create flags ``` Description: The map context does not exist. Context: record Introduced: \- Removed: \- --- #### `map_subcontext_by_key: non-storage elements` `WARNING` Message: ```bash {AS_PARTICLE} map_subcontext_by_key() cannot create key with non-storage elements ``` Description: A map context cannot be created with elements that are not stored, such as a wildcard. Context: record Introduced: \- Removed: \- --- #### `packed_list_get_remove_by_index_range: index out of bounds` `WARNING` Message: ```bash {AS_PARTICLE} packed_list_get_remove_by_index_range() index 155 out of bounds for ele_count 155 ``` Description: There was an attempt to remove an element at index 156 in a list of 155 elements. Since index starts from 0, index 155 means element 156. This pattern applies for any populated index. If the index is unpopulated, the error reads `index 0 out of bounds for ele_count 0`. Context: record Introduced: \- Removed: \- --- #### `record replace: drives full` `WARNING` Message: ```bash {bigdata} record replace: drives full ``` Description: No more space left in storage. Context: record Introduced: \- Removed: \- Parameters: ns Namespace being written to at the time storage filled up. --- #### `record replace: failed write` `DETAIL` Message: ```bash {namespace_name} record replace: failed write 1142f0217ababf9fda5b1a4de66e6e8d4e51765e ``` Description: Most likely appearing as a result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). The record’s digest is the last item in the log entry. To determine what set is being written to, see [How to return the set name of a record using its digest](https://support.aerospike.com/s/article/How-to-return-the-set-name-of-a-record-using-its-digest). Context: record Introduced: 5.2.0 Removed: \- Parameters: {namespace\_name} Namespace being written to. --- ## Roster #### `illegal node id` `WARNING` Message: ```bash (roster_ee.c:151) illegal node id 0 ``` Description: This error may occur during a downgrade from Database 7.2.0 if you are using SC mode and have an active-rack configured in the roster. Context: roster Introduced: 4.0.0 Removed: \- --- #### `invalid node list` `WARNING` Message: ```bash (roster_ee.c:237) {test} invalid node list M100|bb...01| ``` Description: This error may occur during a downgrade from Database 7.2.0 if you are using SC mode and have an active-rack configured in the roster. Context: roster Introduced: 4.0.0 Removed: \- --- ## Rw client #### `client write request` `DETAIL` Message: ```bash {ns_name} client 10.0.3.182:51160 write {digest}:0x8df238affec6f8e3a2c22d6c54c91c5bc4f3ff81 ``` Description: Provides details on the originating client’s IP address, the transaction type and the digest of the record being accessed. Context: rw-client Introduced: 3.16.0.1 Removed: \- Occurs: Single line per client transactions that get as far as successfully reserving a partition. Requires log level to be set to detail for the rw-context: `asinfo -v "log-set:id=0;rw-client=detail"` Parameters: client The originating client’s IP address. --- transaction {digest} The transaction type (read/write/delete/udf) as well as the digest. --- ## Rw #### `write_master: can't replace record during conflict resolution` `WARNING` Message: ```bash WARNING (rw): (write.c:939) {test} write_master: can't replace record c1ed607e1b3ccfc10fe746ee42bdbbbcdd3a1b7c if conflict resolving ``` Description: A cluster without bin-convergence enabled attempted to write to a bin-convergence enabled cluster. Context: rw Introduced: \- Removed: \- --- #### `unexpected src-lut` `WARNING` Message: ```bash WARNING (rw): (rw_utils_ee.c:530) unexpected src-lut 0 ``` Description: Source tried to insert or update a record at destination with an unexpected last update time (LUT). Context: rw Introduced: \- Removed: \- --- #### `write_master: null/empty set name not allowed` `WARNING` Message: ```bash WARNING (rw): (write.c:926) write_master: null/empty set name not allowed for namespace {namespace} ``` Description: The write fails because the set name is null or empty and [`disallow-null-setname`](https://aerospike.com/docs/database/reference/config#namespace__disallow-null-setname) is `true`. Context: rw Introduced: \- Removed: \- --- #### `write_master: drives full` `WARNING` Message: ```bash {bigdata}: write_master: drives full ``` Description: No more space left in storage. Context: rw Introduced: \- Removed: \- Parameters: ns Namespace being written to at the time storage filled up. --- #### `dup-res ack: no digest` `WARNING` Message: ```bash dup-res ack: no digest ``` Description: During a downgrade from Database 4.5.3+ to a prior version, the following protocol-related warning may be seen temporarily on a node with the prior version. This warning is harmless, the result of an older node briefly receiving 4.5.3+ protocol fabric messages from newer nodes. This message ceases after the next rebalance. Context: rw Introduced: \- Removed: \- --- #### `got rw msg with unrecognized op` `WARNING` Message: ```bash got rw msg with unrecognized op 8 ``` Description: During a downgrade from a Database 4.5.3+ server to a prior version, the following protocol-related warning may be experienced temporarily on a node with the earlier version. This warning is harmless. This warning is resultant of an older node briefly receiving 4.5.3+ protocol fabric messages from newer nodes. This message ceases after the next rebalance. Context: rw Introduced: \- Removed: \- --- #### `key mismatch - end of universe?` `WARNING` Message: ```bash key mismatch - end of universe? ``` Description: KEY\_MISMATCH error. Any update, delete or read request for a record which has key stored, the incoming key does not match with the existing stored key. Indicates a RIPEMD-160 key collision which is not likely. See [Collision Resistance of RIPEMD-160](https://online.tugraz.at/tug_online/voe_main2.getvolltext?pCurrPk=17675) for details. This message occurs in case of key / hash mismatch on the application side or some message level corruption. Context: rw Introduced: \- Removed: \- --- #### `can't get stored key` `WARNING` Message: ```bash {namespace} can't get stored key {digest}:0x5230acd92762fa6f827e902d58199dd7b928479c ``` Description: This message indicates that the index has the stored-key flag set for a record, but the record in storage does not contain the key. Normally this would never happen, but there was a bug in some older database Databases whereby this mismatch could occur when a node containing old data was cold-started and brought back into the cluster with records that had been deleted but not durably deleted. This was corrected in Database 4.4.0.8 and later. Enterprise Edition Licensees can contact Aerospike Support for further guidance when encountering this error. Context: rw Introduced: \- Removed: \- --- #### `drop while replicating` `WARNING` Message: ```bash {namespace} drop while replicating ``` Description: This message indicates that a record was dropped while replicating. For example, if a truncation is run and records are still being replicated while the truncation hits the master record. This can also happen when non durable deletes are allowed ([`strong-consistency-allow-expunge`](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency-allow-expunge)) on a [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) enabled namespace. This increments the [`client_write_error`](https://aerospike.com/docs/database/reference/metrics#namespace__client_write_error) metric. Context: rw Introduced: \- Removed: \- --- #### `write_master: bin name too long` `WARNING` Message: ```bash {} write_master: bin name too long () ``` Description: Expected when a client attempts to create a bin with a name that exceeds the 15 character limit. The error is returned to the client and the operation does not succeed. See [System limits](https://aerospike.com/docs/database/reference/limitations) Context: rw Introduced: \- Removed: \- Parameters: {namespace} Namespace being written to. --- length Length of the bin name sent by the client. --- digest The digest of the record being updated. --- #### `write_master: failed as_bin_cdt_alloc_modify_from_client` `WARNING` Message: ```bash {namespace} write_master: failed as_bin_cdt_alloc_modify_from_client() ``` Description: This error is expected if a `list_clear()` is called for a bin that isn’t a list or doesn’t exist. `list_clear()` returns an error when the bin isn’t a list or doesn’t exist. Context: rw Introduced: 5.2.0 Removed: \- Parameters: {namespace} Namespace being written to. --- #### `write_master: failed as_storage_record_write` `WARNING` Message: ```bash {namespace} write_master: failed as_storage_record_write() {digest}:0xd751c6d7eea87c82b3d6332467e8bc9a3c630e13 ``` Description: Result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). Setting the `rw` and `drv_ssd` contexts to `detail` logging provides the accompanying explanatory log messages. See [`log-set`](https://aerospike.com/docs/database/reference/info#log-set) and [`Changing Log Levels`](https://aerospike.com/docs/database/reference/logs) for how to dynamically change log levels. To determine what set is being written to, see [How to return the set name of a record using its digest](https://support.aerospike.com/s/article/How-to-return-the-set-name-of-a-record-using-its-digest). Context: rw Introduced: 3.16.0 Removed: 5.2.0 Parameters: {namespace} Namespace being written to --- {digest} Digest of the record that was rejected --- #### `write_master: record too big` `DETAIL` Message: ```bash {namespace} write_master: record too big {digest}:0xd751c6d7eea87c82b3d6332467e8bc9a3c630e13 ``` Description: Appears with the `WARNING` message about `failed as_storage_record_write()` for exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). To determine what set is being written to, see [How to return the set name of a record using its digest](https://support.aerospike.com/s/article/How-to-return-the-set-name-of-a-record-using-its-digest). Context: rw Introduced: 3.16.0 Removed: \- Parameters: {namespace} Namespace being written to --- {digest} Digest of the record that was rejected --- #### `write_master: as_storage_record_write failed` `DETAIL` Message: ```bash {namespace_name} write_master: failed as_storage_record_write() 1142f0217ababf9fda5b1a4de66e6e8d4e51765e ``` Description: Most likely appearing as a result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size). The record’s digest is the last item in the log entry. To determine what set is being written to, see [How to return the set name of a record using its digest](https://support.aerospike.com/s/article/How-to-return-the-set-name-of-a-record-using-its-digest). Context: rw Introduced: 5.2.0 Removed: \- Parameters: {namespace\_name} Namespace being written to. --- #### `read_local bad number of ops` `WARNING` Message: ```bash WARNING (rw): {} read_local: bad number of ops , can't exceed bin ops ``` Description: A read request contained more operations than the maximum allowed bin operations for a single record. This applies to both single-key reads and query projection. Reduce the number of read operations. Context: rw Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- count Number of operations provided. --- max Maximum allowed operations. --- digest Record digest. --- #### `write_master: disallowed ttl with nsup-period` `WARNING` Message: ```bash write_master: disallowed ttl with nsup-period 0 ``` Description: When [expirations in a namespace are disabled](https://aerospike.com/docs/database/manage/namespace/retention) because [`nsup-period`](https://aerospike.com/docs/database/reference/config#namespace__nsup-period) is set to the default of `0`, records with a [TTL](https://aerospike.com/docs/database/learn/architecture/data-storage/data-model#metadata) other than `0` may not be written to that namespace. This is to avoid confusion as to whether the records should be subject to expiration (or eviction). Simply set the `nsup-period` to a value other than `0` to allow records with a TTL other than `0` to be written. If there is a need to allow records with a non `0` TTL without having the nsup thread running, it is possible to set [‘allow-ttl-without-nsup\`](https://aerospike.com/docs/database/reference/config#namespace__allow-ttl-without-nsup) to true, but this is not recommended prevents those records from being properly deleted upon expiration. Context: rw Introduced: \- Removed: \- --- ## Scan #### `basic scan job failed to start` `DETAIL` Message: ```bash basic scan job 283603086331273222 failed to start (4) ``` Description: A scan job was submitted with maxRetries greater than 0 and failed or timed out on one or more nodes, causing a retry to conflict with the remaining scan that has the same transaction ID and exit with error code 4 (parameter error). Context: scan Introduced: \- Removed: \- Parameters: trid trid of the scan job being retried --- #### `error sending - connection timed out` `WARNING` Message: ```bash error sending to 10.0.0.1:45678 - fd 646 sz 1049036 Connection timed out ``` Description: A scan is trying to send data back to the client, but it cannot send the full data set. This is likely due to an aborted client-side scan. The scan must be re-run. Context: scan Introduced: \- Removed: \- Parameters: client Client address:port. --- fd File descriptor of the socket to the client. --- sz Size of the response attempted. --- error Error message returned by OS. --- #### `scan not started: rchash_put() failed` `WARNING` Message: ```bash not starting scan 261941093 because rchash_put() failed with error -4 ``` Description: Server rejects scan transaction because it is already processing. Wait for the scan to complete or abort it before issuing the scan transaction again. Context: scan Introduced: \- Removed: \- Parameters: scan ID of the scan. --- errno Error code. --- #### `scan msg has unrecognized set` `WARNING` Message: ```bash scan msg from 10.11.12.13 has unrecognized set name_of_the_set ``` Description: A scan specifying an unknown set was received. Typically a user error such as a mistyped set name, but also happens if the set does not exist on all the nodes in the cluster. For example, a set with fewer records than the number of nodes in the cluster, where some nodes don’t hold any record (master or replica) for that set. Having something configured for the set, such as access control that is set specific or a secondary index definition, causes this log warning message to disappear as the set would then be ‘known’ on the node even without any record in it. Context: scan Introduced: \- Removed: \- Parameters: address Node that the scan message came from. --- setname Set name that doesn’t exist on this node. --- #### `send error` `WARNING` Message: ```bash send error - fd 646 sz 1049036 rv 521107 ``` Description: A scan is trying to send data back to the client, but it cannot send the full data set. This is likely due to an aborted client-side scan. The scan must be re-run. Context: scan Introduced: \- Removed: \- Parameters: fd File descriptor of the socket to the client. --- sz Size of the response attempted. --- rv Size that was successfully sent. --- #### `starting basic scan job` `DEBUG` Message: ```bash starting basic scan job 8104671463142312256 {namespace:set} rps 0 sample-max 100 socket-timeout 30000 from 172.22.xx.yy:52842 ``` Description: A basic scan job is initiated on the specified namespace and set. Context: scan Introduced: 4.9.0 Removed: \- Parameters: scan job ID of the scan. --- rps Configured records per second of the scan. Configured in the scan policy. If this is not specified, the cluster maximum of the [background-scan-max-rps](https://aerospike.com/docs/database/reference/config#namespace__background-scan-max-rps) parameter is in effect. --- sample-max The maximum number of records to return from the scan, if specified. Configured in the scan policy. --- metadata-only Only if the scan is configured to initiate a metadata scan. --- fail-on-cluster-change If this is enabled in the scan policy, or using AQL, scans are halted if there is a cluster change, such as nodes leaving or joining the cluster. See [Why do I get AEROSPIKE\_ERR\_CLUSTER\_CHANGE when querying or scanning a namespace?](https://support.aerospike.com/s/article/Why-do-I-get-AEROSPIKE-ERR-CLUSTER-CHANGE-when-querying-or-scanning-a-namespace) for details. --- socket-timeout The configured socket timeout. Configured in the client policy. The default is 30 seconds. --- client Client IP address and port. --- #### `starting basic scan job namespace:set` `INFO` Message: ```bash starting basic scan job 8104671463142312256 {namespace:set} rps 0 sample-pct 100 socket-timeout 30000 from 172.22.xx.yy:52842 ``` Description: A basic scan job is initiated on the specified namespace and set. Context: scan Introduced: 4.7.0.2 Removed: 4.9.0 Parameters: scan job ID of the scan. --- rps Configured records per second of the scan. Configured in the scan policy. If this is not specified, the cluster maximum of the [background-scan-max-rps](https://aerospike.com/docs/database/reference/config#namespace__background-scan-max-rps) parameter applies. --- sample-pct The percentage of records to return from the scan, if specified. The default is 100 percent. Configured in the scan policy. --- metadata-only Only if the scan is configured to initiate a metadata scan. --- fail-on-cluster-change If this is enabled in the scan policy, or using AQL, scans are halted if there is a cluster change, such as nodes leaving or joining the cluster. See [Why do I get AEROSPIKE\_ERR\_CLUSTER\_CHANGE when querying or scanning a namespace?](https://support.aerospike.com/s/article/Why-do-I-get-AEROSPIKE-ERR-CLUSTER-CHANGE-when-querying-or-scanning-a-namespace) for details. --- socket-timeout The configured socket timeout. Configured in the client policy. The default is 30 seconds. --- client Client IP address and port --- ## Security #### `failed assertion (namespace): remove pmem block failed` `WARNING` Message: ```bash FAILED ASSERTION (namespace): (namespace_ee.c:550) can't remove persistent memory base block: Operation not permitted ``` Description: When running under systemd, this error occurs if shared memory has been allocated by a previous user, and a second user who does not have privileges to access those shared memory segments starts the Aerospike server. Aerospike reads the user from the aerospike.conf file; however, if the shared memory segments are still allocated by the previous user, it may be necessary to force a cold start. Context: security Introduced: \- Removed: \- --- #### `unknown security message command` `WARNING` Message: ```bash WARNING (security): (security.c:857) unknown security message command 20 ``` Description: Clients that enable the LDAP protocol server-side, and run on Databases of Aerospike that did not support server-side LDAP, may see this warning when they attempt to log in. This warning is transitory as the server falls back to use the non-LDAP protocol by default. Check the release notes of each client to validate the Database that supports LDAP. Context: security Introduced: \- Removed: \- --- #### `authentication failed - user login` `INFO` Message: ```bash authentication failed (user) | client: 11.22.33.44:57754 | authenticated user: | action: login | detail: user=baduser ``` Description: Provides details on the internal user not found, including failure, client with IP address and port, authenticated user, action, and user specified. Context: security Introduced: \- Removed: \- Occurs: Occurs when a failed login is attempted with an internal user that is not found in the access control list, if audit logging is configured to report those. See [`Security Configuration`](https://aerospike.com/docs/database/manage/security/rbac) for details. To enable this log message output, you must set [`report-authentication`](https://aerospike.com/docs/database/reference/config#security__report-authentication) and [`report-violation`](https://aerospike.com/docs/database/reference/config#security__report-violation) to `true`. Parameters: authenticated failed (user) The user authentication failed. --- client The client IP Address and port. --- authenticated user The authenticated user, in this example: none. --- action The action being performed, in this case login. --- detail The username involved in the failed authentication. --- #### `change password - user not changeable` `WARNING` Message: ```bash change password - user password not changeable ``` Description: Occurs when a user with _nopassword_ tries to change their own password. See [Access control - PKI authentication](https://aerospike.com/docs/database/learn/security/access-control/#pki-authentication) for details. **Example:** User with _nopassword_ tries to change their own password: ```plaintext May 06 2025 15:06:22.746 GMT-0700: WARNING (security): (security.c:3789) change password - user password not changeable ``` Context: security Introduced: 8.1.0 Removed: \- --- #### `fd send failed` `WARNING` Message: ```bash fd 361 send failed, errno 113 ``` Description: Tried to send the client a message indicating security is not supported on this CE server, but failed due to a socket issue. Context: security Introduced: \- Removed: \- Parameters: fd ID of the file descriptor the message was sent on. --- errno Linux error code for the problem, usually something like 113 (“No route to host”) or 110 (“Connection timed out”). --- #### `audit: forbidden password` `INFO` Message: ```bash (audit): forbidden password | client: 127.0.0.1:33044 | authenticated user: | action: ACTION | detail: user=George ``` Description: Occurs when a user uses a bad password or a user created with _nopassword_ tries a password login, attempts to change their own password, or another user attempts to change the password. See [Access control - PKI authentication](https://aerospike.com/docs/database/learn/security/access-control/#pki-authentication) for details. **Example:** George, a _nopassword_ user, tries to log in using a password. ```plaintext May 06 2025 14:59:54.134 GMT-0700: INFO (audit): (security.c:7451) forbidden password | client: 192.168.189.128:57880 | authenticated user: | action: login | detail: user=George ``` **Example:** An admin user “brad” tries to change the password of _nopassword_ user “George”. ```plaintext May 06 2025 14:59:54.134 GMT-0700: INFO (audit): (security.c:7451) forbidden password | client: 192.168.189.128:57880 | authenticated user: brad | action: set password | detail: user=George ``` **Example:** George, a _nopassword_ user, tries to change his own password. ```plaintext May 06 2025 15:05:10.549 GMT-0700: INFO (audit): (security.c:7451) forbidden password | client: 192.168.189.128:51124 | authenticated user: George | action: change password | detail: user=George ``` Context: security Introduced: 8.1.0 Removed: \- Parameters: client The client that attempted the operation. --- authenticated user The authenticated user performing the operation. --- action The transaction type (read/write/delete/udf) or user or data related operation. --- detail The details of the operation, either for a user or admin related operation or namespace, set and digest of the record involved if a single record transaction. --- #### `login - internal user credential mismatch` `WARNING` Message: ```bash login - internal user credential mismatch ``` Description: Provides a warning that a login failed using a valid internal user that is found in the access control list with an incorrect password. Context: security Introduced: 4.1.0.1 Removed: \- Occurs: Occurs when a failed login is attempted using an internal user that exists in the access control list but where an incorrect password has been supplied. --- #### `login - internal user not found` `WARNING` Message: ```bash login - internal user not found ``` Description: Provides a warning that a login failed with an internal user that is not found in the access control list. Context: security Introduced: 4.1.0.1 Removed: \- Occurs: Occurs when a failed login is attempted with an internal user that is not found in the access control list. To log the details of the internal user that is not found, [`report-authentication`](https://aerospike.com/docs/database/reference/config#security__report-authentication) and [`report-violation`](https://aerospike.com/docs/database/reference/config#security__report-violation) must be set to `true`. See [Security Configuration](https://aerospike.com/docs/database/manage/security/rbac) for details. --- #### `login - internal user using ldap` `WARNING` Message: ```bash login - internal user using ldap ``` Description: A login failed with an internal user that is not found in the access control list. Context: security Introduced: 4.6.0.2 Removed: \- Occurs: Occurs when a login attempt fails with an internal user that is not found in the access control list. Some conditions when the warning message is presented to the user while incorporating the use of an ACL. - When an encrypted ‘external’ (clear password encrypted) is used for LDAP but a stored hashed ‘internal’ password also exists for that user. - When an incompatible Aerospike client is attempting to connect to Database 4.6.0 or newer. See [`Overview of Access Control with LDAP and PKI`](https://aerospike.com/docs/database/manage/security/rbac) for details to ensure you are using a compatible Aerospike client version. - When clusters using Cross-Datacenter Replication (XDR) are running with conflicting Aerospike Database versions. When running XDR and an ACL, Database versions 4.1.0.1 to 4.3.0.6 are incompatible with Database 4.6.0 or later. The incompatible Database versions 4.1.0.1 to 4.3.0.6 **cannot ship to** 4.6.0.2 or later. The simplest workaround is to avoid using versions 4.1.0.1 to 4.3.0.6. See [Internal user warning returned](https://support.aerospike.com/s/article/Internal-user-warning-returned-when-using-an-Access-Control-List-ACL-with-Cross-Datacenter-Replication-XDR-non-compatible-Aerospike-Server-Databases-sending-both-internal-and-external-authentication-mode) for details. This warning message typically appears in the logs when upgrading a cluster. --- #### `login - password auth not allowed` `WARNING` Message: ```bash login - password auth not allowed ``` Description: Occurs when user with _nopassword_ tries a password login. See [Access control - PKI authentication](https://aerospike.com/docs/database/learn/security/access-control/#pki-authentication) for details. **Example:** A user with _nopassword_ tries to log in using a password. ```plaintext May 06 2025 14:59:54.134 GMT-0700: WARNING (security): (security.c:3491) login - password auth not allowed ``` Context: security Introduced: 8.1.0 Removed: \- --- #### `permitted authenticated user - create user` `INFO` Message: ```bash permitted | authenticated user: admin | action: create user | detail: user=bruce;roles=read-write ``` Description: Provides details on the operation performed, including transaction type, namespace, set and relevant record’s digest. Context: security Introduced: 3.7.0.1 Removed: \- Occurs: Occurs when permitted transaction happen under an authenticated user (in this case a user-admin related operation). See [`Security Configuration`](https://aerospike.com/docs/database/manage/security/rbac) for details. Parameters: authenticated user The authenticated user performing the operation. --- action The transaction type (read/write/delete/udf) or user or data related operation. --- detail The details of the operation, either for a user or admin related operation or namespace, set and digest of the record involved if a single record transaction. --- #### `role violation: delete action by user` `INFO` Message: ```bash role violation | authenticated user: sally | action: delete | detail: {test|setB} [D|ee50d7c1d0f427ed5c41ef8a18efd85412b973ff] ``` Description: Details on the role violation, including transaction type, namespace, set and relevant record’s digest. Context: security Introduced: 3.7.0.1 Removed: \- Occurs: Occurs when role violation happens, if audit logging configured to report those. See [`Security Configuration`](https://aerospike.com/docs/database/manage/security/rbac) for details. Parameters: authenticated user The authenticated user violating the role’s permissions. --- action The transaction type (read/write/delete/udf) or user or data related operation. --- detail The namespace, set and digest of the record involved in the role violation. --- #### `set password - user password not changeable` `WARNING` Message: ```bash set password - user password not changeable ``` Description: Occurs when a user tries to change the password of a _nopassword_ user. See [Access control - PKI authentication](https://aerospike.com/docs/database/learn/security/access-control/#pki-authentication) for details. **Example:** Admin user tries to set the password of _nopassword_ user. ```plaintext May 06 2025 15:06:22.746 GMT-0700: WARNING (security): (security.c:3731) set password - user password not changeable ``` Context: security Introduced: 8.1.0 Removed: \- --- ## Service #### `refusing client connection - proto-fd-max` `WARNING` Message: ```bash refusing client connection - proto-fd-max 50000 ``` Description: The server has reached the configured maximum for incoming file descriptors ([`proto-fd-max`](https://aerospike.com/docs/database/reference/config#service__proto-fd-max)). This corresponds directly to incoming client connections. Connections greater than the maximum are refused. Context: service Introduced: \- Removed: \- Parameters: proto-fd-max Currently configured value for [`proto-fd-max`](https://aerospike.com/docs/database/reference/config#service__proto-fd-max) --- #### `unsupported proto version` `WARNING` Message: ```bash unsupported proto version 0 from 12.0.0.1:5598 ``` Description: Indicates that messages of an unexpected type are being sent to the main server port (usually port 3000) by other nodes in the cluster. This can happen if the nodes have been misconfigured to use the main server port instead of the correct port. For example, the configuration for Mesh heartbeats should specify port 3002, not port 3000. Context: service Introduced: \- Removed: \- --- ## Sindex #### `QTR Put in hash failed` `WARNING` Message: ```bash QTR Put in hash failed with error -4. ``` Context: sindex Introduced: 4.6.0 Removed: \- Occurs: WA duplicate query being executed on the Aerospike server. On the client side, you should see error code `215 (AS_PROTO_RESULT_FAIL_QUERY_DUPLICATE)`. There are two possible causes. - 1. This error could happen when there are multiple incoming queries with same Task ID. To avoid this error, set a unique Task ID with the client when running the same query on multiple threads. The Task ID is set by using the following APIs: [setTaskID](https://aerospike.com/apidocs/java/com/aerospike/client/query/Statement.html#setTaskId-long-) for Java, and [SetTaskID](https://aerospike.com/apidocs/csharp/api/Aerospike.Client.Statement.html#Aerospike_Client_Statement_SetTaskId_System_Int64_) for C#. - 2. If there are retries enabled for secondary index query jobs. We do not recommend having retries for secondary index queries, to avoid them running for the same Job ID. --- #### `queuing namespace for sindex population by device scan` `INFO` Message: ```bash Queuing namespace {ns-name} for sindex population by device scan ``` Context: sindex Introduced: 5.3.0 Removed: \- Occurs: During startup, when the namespace’s devices are about to be scanned in order to populate secondary indexes. This happens when [sindex-startup-device-scan](https://aerospike.com/docs/database/reference/config#namespace__sindex-startup-device-scan) is `true`. --- #### `queuing namespace for sindex population by index scan` `INFO` Message: ```bash Queuing namespace {ns-name} for sindex population by index scan ``` Context: sindex Introduced: 5.3.0 Removed: \- Occurs: During startup, when the namespace’s primary index is about to be scanned in order to populate secondary indexes. This happens when [sindex-startup-device-scan](https://aerospike.com/docs/database/reference/config#namespace__sindex-startup-device-scan) is `false`. --- #### `Sindex-ticker: sindex progress` `INFO` Message: ```bash Sindex-ticker: ns=ns-name si= obj-scanned=500000 si-mem-used=47913 progress= 2% est-time=2336995 ms ``` Description: Replaced in Database 6.1.0 with [`sindex-populate`](https://aerospike.com/docs/database/reference/logs/#sindex__2067516514) Context: sindex Introduced: \- Removed: 6.1.0 Occurs: Information logged at startup when secondary indices are being rebuilt. Parameters: ns Namespace name. --- si Secondary indexes being rebuilt. --- obj-scanned Number of objects scanned. --- si-mem-used Memory used by the secondary indices. --- progress Progress in percent. --- est-time Estimated remaining time in milliseconds for the secondary indices to be fully rebuilt. --- #### `changing creation mode of set index` `INFO` Message: ```bash INFO (sindex): changing creation mode of set index for : from to ``` Description: The management method (creation mode) for a set index is transitioning, for example from `config` to `smd` when a set index previously managed via `enable-index` is adopted by an `sindex-create` command. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- set Set name. --- old\_mode Previous creation mode (`config` or `smd`). --- new\_mode New creation mode (`config` or `smd`). --- #### `failed sindex on bin - blob longer than 2048` `WARNING` Message: ```bash WARNING (sindex): (sindex.c:1658) failed sindex on bin bin-name - blob longer than 2048 ``` Context: sindex Introduced: 7.0.0 Removed: \- Occurs: A record write or update sets a bytes (blob) bin value that is 2048 bytes or larger while a secondary index exists on that bin. The write succeeds, but the value is not added to the secondary index. If the bin previously held a shorter, indexed value, that old index entry is removed. See [secondary index byte-size limits](https://aerospike.com/docs/database/reference/limitations#secondary-index). Parameters: bin-name Name of the bin whose value exceeded the secondary index size limit. --- #### `failed sindex on bin - string longer than 2048` `WARNING` Message: ```bash WARNING (sindex): (sindex.c:1643) failed sindex on bin bin-name - string longer than 2048 ``` Context: sindex Introduced: \- Removed: \- Occurs: A record write or update sets a string bin value that is 2048 bytes or larger while a secondary index exists on that bin. The write succeeds, but the value is not added to the secondary index. If the bin previously held a shorter, indexed value, that old index entry is removed. See [secondary index byte-size limits](https://aerospike.com/docs/database/reference/limitations#secondary-index). Parameters: bin-name Name of the bin whose value exceeded the secondary index size limit. --- #### `sindex-gc-done: cleaned` `INFO` Message: ```bash {ns-name} sindex-gc-done: cleaned (40000,40000) total-ms 23 ``` Context: sindex Introduced: 5.7.0 Removed: \- Occurs: Secondary index (sindex) garbage collection cycle summary. Parameters: cleaned Count of sindex entries that have been cleaned (cumulative, current). First value corresponds to the [`sindex_gc_cleaned`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_gc_cleaned) statistic. Second value corresponds to the number of sindex entries cleaned in the current round. --- total-ms Duration of the sindex garbage collection cycle in milliseconds. --- #### `sindex-gc-done: processed found deleted` `INFO` Message: ```bash {ns-name} sindex-gc-done: processed 3133360101 found 365961945 deleted 365952323 total-ms 62667962 ``` Context: sindex Introduced: 4.6.0 Removed: 5.7.0 Occurs: Secondary index (sindex) garbage collection cycle summary. Replaced by a modified version of the [`sindex-gc-done`](https://aerospike.com/docs/database/reference/logs#sindex__806010721) message in Database 5.7.0. Parameters: processed Count of sindex entries that have been checked. Corresponds to the [`sindex_gc_objects_validated`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_objects_validated) statistic. --- found Count of sindex entries found eligible for garbage collection. Corresponds to the [`sindex_gc_garbage_found`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_garbage_found) statistic. --- deleted Count of sindex entries deleted through garbage collection. Count may be lower than above number if those entries were deleted while the garbage collector was running, for example through a competing `truncate` command. Corresponds to the [`sindex_gc_garbage_cleaned`](https://aerospike.com/docs/database/reference/metrics#node_stats__sindex_gc_garbage_cleaned) statistic. --- total-ms Duration of the sindex garbage collection cycle in milliseconds. --- #### `sindex-gc-start` `INFO` Message: ```bash {ns-name} sindex-gc-start ``` Context: sindex Introduced: 4.6.0 Removed: \- Occurs: Starting secondary index (sindex) garbage collection. --- #### `set index already disabled` `WARNING` Message: ```bash WARNING (sindex): set index already disabled for : ``` Description: An attempt was made to disable a set index that is already disabled for this namespace and set. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- set Set name. --- #### `set index already enabled via method` `INFO` Message: ```bash INFO (sindex): set index already enabled via for : ``` Description: A set index creation was requested but the set index is already enabled using the same method. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: method The method that originally created the set index (`smd` or `config`). --- ns Namespace name. --- set Set name. --- #### `set index enabled via different method - cannot disable` `WARNING` Message: ```bash WARNING (sindex): set index enabled via for : - cannot disable via ``` Description: A set index exists for this namespace and set but was created using a different method. It cannot be disabled using a method other than the one that created it. Use `sindex-delete` for indexes created via `sindex-create`, or `set-config enable-index=false` for indexes created via `set-config`. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: method1 The method that originally created the set index (`smd` or `config`). --- ns Namespace name. --- set Set name. --- method2 The method that attempted to disable the index. --- #### `set index enabled via different method - cannot enable` `WARNING` Message: ```bash WARNING (sindex): set index enabled via for : - cannot enable via ``` Description: A set index already exists for this namespace and set, but was created using a different method. A set index created via `sindex-create` (SMD) cannot be re-enabled via `set-config`/`enable-index`, and vice versa. Use the same method that originally created the index. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: method1 The method that originally created the set index (`smd` or `config`). --- ns Namespace name. --- set Set name. --- method2 The method that attempted to enable the index. --- #### `SINDEX CREATE failed get-create set` `WARNING` Message: ```bash WARNING (sindex): SINDEX CREATE: failed get-create set ``` Description: The server could not find or create the specified set while processing a set index creation request via SMD. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: set Set name. --- #### `SINDEX CREATE index name already in use` `WARNING` Message: ```bash WARNING (sindex): SINDEX CREATE: iname already in use - ignoring ``` Description: A set index or secondary index with this name already exists. The create request is ignored. Use a different index name or delete the existing index first with [`sindex-delete`](https://aerospike.com/docs/database/reference/info#sindex-delete). Context: sindex Introduced: 8.1.2 Removed: \- Parameters: name The index name that is already in use. --- #### `SINDEX CREATE renaming index` `INFO` Message: ```bash INFO (sindex): SINDEX CREATE: renaming to ``` Description: A set index is being renamed during creation, typically when an existing internally-named set index is being adopted into the SMD-managed path with a user-specified name. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: old\_name Previous internal index name. --- new\_name New user-specified index name. --- #### `SINDEX CREATE request received via SMD` `INFO` Message: ```bash INFO (sindex): SINDEX CREATE: request received for : via smd ``` Description: A set index creation request was received via SMD (System Metadata) and is being processed on this node. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- set Set name. --- #### `SINDEX DROP already disabled` `INFO` Message: ```bash INFO (sindex): SINDEX DROP: already disabled for : ``` Description: A set index deletion request was received but the set index is already disabled for this namespace and set. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- set Set name. --- #### `SINDEX DROP request received via SMD` `INFO` Message: ```bash INFO (sindex): SINDEX DROP: request received for : via smd ``` Description: A set index deletion request was received via SMD (System Metadata) and is being processed on this node. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: ns Namespace name. --- set Set name. --- #### `SINDEX DROP set not found` `WARNING` Message: ```bash WARNING (sindex): SINDEX DROP: set '' not found ``` Description: A set index deletion was requested but the specified set does not exist in the namespace. Context: sindex Introduced: 8.1.2 Removed: \- Parameters: set Set name that was not found. --- #### `SINDEX DROP si not found` `WARNING` Message: ```bash WARNING (sindex): SINDEX DROP: si not found ``` Description: A set index deletion was requested but no matching set index was found. The index may have already been deleted or may not exist. Context: sindex Introduced: 8.1.2 Removed: \- --- #### `sindex-populate mem-used objects-scanned` `INFO` Message: ```bash INFO (sindex): (populate.c:240) {ns-name} sindex-populate: mem-used 219729920 objects-scanned 1637000 progress-pct 0.073 ``` Context: sindex Introduced: 6.1.0 Removed: \- Occurs: Information logged at startup when secondary indices are being rebuilt. Parameters: ns Namespace name. --- si Secondary indexes being rebuilt. --- obj-scanned Number of objects scanned. --- si-mem-used Memory used by the secondary indices. --- progress Progress in percent. --- #### `smd set name required for set index` `WARNING` Message: ```bash WARNING (sindex): smd - set name required for set index ``` Description: An SMD entry for a set index is missing the set name. Set indexes require a set name. This indicates corrupt or incomplete SMD data. Context: sindex Introduced: 8.1.2 Removed: \- --- ## Smd #### `failed to allocate system metadata cmd event` `CRITICAL` Message: ```bash failed to allocate a System Metadata cmd event ``` Description: Indicates a shortage of memory. Make sure nodes have enough memory. Context: smd Introduced: \- Removed: \- --- ## Socket #### `error adding FD to epoll instance` `CRITICAL` Message: ```bash Error while adding FD to epoll instance : () ``` Description: A double-close of file descriptors can lead to this assertion under certain conditions. If a file descriptor (FD) is closed twice in rapid succession, it usually fails silently after the first successful close. However, if the FD is reused between the first and second close due to a race condition, the second close could mistakenly target an active connection. This can result in errors like “Bad file descriptor” when the reused FD is assigned to a service thread’s epoll instance. Context: socket Introduced: \- Removed: \- Occurs: Example: `Error while adding FD 2817 to epoll instance 1105: 9 (Bad file descriptor)` See this [Bad File Descriptor knowledge base article](https://support.aerospike.com/s/article/Bad-File-Descriptor-or-similar-error-in-server-logs-potentially-leading-to-a-crash-when-using-LDAP) for details on one known cause of this assertion when using multiple LDAP servers in the Aerospike configuration and at least one of them is failing to connect against. Parameters: FD : File descriptor --- Poll-FD : epoll instance file descriptor --- error-code : OS Error Code --- error-string : OS Error code description --- #### `error while connecting - no route to host` `WARNING` Message: ```bash Error while connecting: 113 (No route to host) ``` Description: Failed to make the connection to peer node for sending heartbeat messages. Context: socket Introduced: \- Removed: \- Parameters: errno Linux error code returned by the OS. --- error message Message corresponding to errno. --- #### `socket connect error` `WARNING` Message: ```bash Error while connecting socket to 10.100.0.101:3002 ``` Description: The node is unable to connect to a configured peer node. The peer may not be up, or there may be connectivity issues. If the node had been permanently removed, see [Removing a Node](https://aerospike.com/docs/database/manage/cluster/remove-node). Context: socket Introduced: \- Removed: \- Parameters: address:port Address and port that could not be connected to. --- #### `error creating socket - too many open files` `WARNING` Message: ```bash Error while creating socket for 10.168.10.1:3002: 24 (Too many open files) ``` Description: Too many file descriptors are in use. This can lead to an assertion that would cause the node to abort. See [System running out of file descriptors](https://support.aerospike.com/s/article/System-running-out-of-file-descriptors) Context: socket Introduced: \- Removed: \- --- #### `bind: socket in use, waiting` `WARNING` Message: ```bash bind: socket in use, waiting (port:3001) ``` Description: Some other process is already using that port and asd cannot listen on it. Use `losf -i:3001` or `netstat -plant | grep :3001` to find out what process is causing the conflict. Context: socket Introduced: \- Removed: \- Parameters: port Port number that asd needs to listen on. --- #### `epoll_create() failed - too many open files` `CRITICAL` Message: ```bash epoll_create() failed: 24 (Too many open files) ``` Description: Occurs when the server has hit the system configured file descriptor limit, which leads to an assertion causing the node to abort.
- truncating ``` Description: Too many IP addresses associated with a network interface. Limit is 20 addresses per interface. Context: socket Introduced: \- Removed: \- --- ## Storage #### `compression is not available for storage-engine memory` `WARNING` Message: ```bash compression is not available for storage-engine memory ``` Description: This message means that an attempt was made to add compression on a memory storage device. This is not supported. Context: storage Introduced: 4.9.0 Removed: \- --- #### `namespace partitions shut down` `INFO` Message: ```bash {ns_name} partitions shut down ``` Description: This message is one of a sequence of messages logged during Aerospike server shutdown of storage-engine device namespaces. The message signifies that all of the namespace’s partitions and index trees have been locked, so that no records are accessible. Context: storage Introduced: 4.6.0.2 Removed: \- --- #### `storage devices flushed` `INFO` Message: ```bash {ns_name} storage devices flushed ``` Description: This message is one of a sequence of messages logged during Aerospike server shutdown of storage-engine device namespaces. The message signifies that the data in write buffers for the namespace’s devices has been successfully flushed to those devices. Context: storage Introduced: 4.6.0.2 Removed: \- --- #### `storage-engine memory - nothing to do` `INFO` Message: ```bash {ns_name} storage-engine memory - nothing to do ``` Description: This message is logged during Aerospike server shutdown of storage-engine memory namespaces. The message simply notes that there are no storage shutdown tasks needed for the namespace. Context: storage Introduced: 4.6.0.2 Removed: \- --- ## Tls #### `SSL_accept I/O unexpected EOF` `WARNING` Message: ```bash SSL_accept I/O unexpected EOF with 127.0.0.1:46630 ``` Description: The server is attempting to use a connection that has been closed by the client. This can happen if a client times out a transaction, for example if the server is too slow to respond or if there are network disruptions. Context: tls Introduced: \- Removed: \- Parameters: IP:port Source IP address and port of the client connection. --- #### `SSL_accept failed - peer did not return certificate` `WARNING` Message: ```bash SSL_accept with 127.0.0.1:34086 failed: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate ``` Description: The client sent a proper CA certificate for [mutual authentication](https://aerospike.com/docs/database/learn/security/tls) that matches the server CA certificate, but didn’t send a client certificate. Context: tls Introduced: \- Removed: \- Parameters: IP:port Source IP address and port of the client connection. --- #### `ssl_accept: certificate verify failed` `WARNING` Message: ```bash SSL_accept with 127.0.0.1:34094 failed: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed ``` Description: The client sent a proper client certificate and CA certificate for [mutual authentication](https://aerospike.com/docs/database/learn/security/tls), but sent a TLS name that doesn’t match the server’s TLS name as set in [`tls-authenticate-client`](https://aerospike.com/docs/database/reference/config#network__tls-authenticate-client) and the server’s certificate. Context: tls Introduced: \- Removed: \- Parameters: IP:port Source IP address and port of the client connection. --- #### `SSL_accept failed: unknown ca` `WARNING` Message: ```bash SSL_accept with 127.0.0.1:34100 failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca ``` Description: The client sent a proper client certificate for [mutual authentication](https://aerospike.com/docs/database/learn/security/tls), but sent a CA certificate that either doesn’t match the client certificate, or doesn’t match the server CA certificate. Context: tls Introduced: \- Removed: \- Parameters: IP:port Source IP address and port of the client connection. --- #### `SSL_read I/O unexpected EOF` `WARNING` Message: ```bash SSL_read I/O unexpected EOF with 127.0.0.1:46491 ``` Description: The server is attempting to use a connection that has been closed by the client. This can happen if a client times out a transaction, for example if the server is too slow to respond or if there are network disruptions. Context: tls Introduced: \- Removed: \- Parameters: IP:port Source IP address and port of the client connection. --- #### `tls verify: unable to get local issuer certificate` `WARNING` Message: ```bash TLS verify result: unable to get local issuer certificate ``` Description: A CA or intermediate CA is not trusted by the server. This may be caused by a self-signed certificate or CA that is not yet added to the server truststore. Context: tls Introduced: \- Removed: \- --- #### `tls SSL_shutdown I/O error - transport endpoint not connected` `WARNING` Message: ```bash WARNING (tls): (tls_ee.c:318) SSL_shutdown I/O error with (unknown): Transport endpoint is not connected ``` Description: Indicates that a socket is not connected. Adjacent messages show that the socket is being used by heartbeats, although similar messages could appear concerning any socket using TLS. Possible causes: - Interruption in network layer - Abrupt node restart if observed on heartbeat - Abrupt client restart if observed in client traffic - Server unable to send results to a client due to client timing out the socket Context: tls Introduced: \- Removed: \- --- #### `ca-file changed in context` `INFO` Message: ```bash INFO (tls): ca-file changed in context ``` Description: The TLS refresh mechanism detected that the `ca-file` has been modified for the specified TLS context. The server will rebuild the SSL context with the updated CA trust material. Context: tls Introduced: 8.1.2 Removed: \- Parameters: path Path to the ca-file. --- context TLS context name. --- #### `ca-path changed in context` `INFO` Message: ```bash INFO (tls): ca-path changed in context ``` Description: The TLS refresh mechanism detected that the `ca-path` directory has been modified for the specified TLS context. The server will rebuild the SSL context with the updated CA trust material. Context: tls Introduced: 8.1.2 Removed: \- Parameters: path Path to the ca-path directory. --- context TLS context name. --- #### `error while fetching timestamp for CA file` `WARNING` Message: ```bash WARNING (tls): error while fetching timestamp for : () ``` Description: The TLS refresh mechanism could not read the file modification timestamp for a `ca-file` or `ca-path`. This may indicate a file permission issue or the file has been removed. CA hot-reload cannot proceed until the file is accessible. Context: tls Introduced: 8.1.2 Removed: \- Parameters: path File path that could not be read. --- errno System error number. --- description Human-readable error description. --- #### `failed to load CA file` `WARNING` Message: ```bash WARNING (tls): failed to load ``` Description: The TLS refresh mechanism detected that a `ca-file` or `ca-path` changed but could not load the new CA material. The previous CA trust material remains in effect. Verify the file contents and permissions. Context: tls Introduced: 8.1.2 Removed: \- Parameters: path Path to the CA file or directory that failed to load. --- #### `retrying CA refresh in context` `INFO` Message: ```bash INFO (tls): retrying CA refresh in context ``` Description: A CA refresh failed and the server is retrying the operation for the specified TLS context. Context: tls Introduced: 8.1.2 Removed: \- Parameters: context TLS context name. --- #### `updated ca-file` `INFO` Message: ```bash INFO (tls): updated ca-file for : - ``` Description: The CA file for the specified TLS context has been successfully reloaded. The SSL context is now using the updated CA trust material. Context: tls Introduced: 8.1.2 Removed: \- Parameters: context TLS context name. --- old\_ts Previous file modification timestamp. --- new\_ts New file modification timestamp. --- #### `updated ca-path` `INFO` Message: ```bash INFO (tls): updated ca-path for : - ``` Description: The CA path directory for the specified TLS context has been successfully reloaded. The SSL context is now using the updated CA trust material. Context: tls Introduced: 8.1.2 Removed: \- Parameters: context TLS context name. --- old\_ts Previous directory modification timestamp. --- new\_ts New directory modification timestamp. --- ## Truncate #### `namespace truncate completed` `INFO` Message: ```bash {ns-name} done truncate to T deleted N ``` Description: This message indicates a request to truncate up to timestamp T has finished, after deleting N objects. Context: truncate Introduced: 6.3.0 Removed: \- Occurs: Truncate process completed. --- #### `namespace done truncate` `INFO` Message: ```bash {ns-name} done truncate ``` Context: truncate Introduced: 3.12.0 Removed: 6.3.0 Occurs: Truncate command completed. --- #### `flagging truncate to restart` `INFO` Message: ```bash {ns-name} flagging truncate to restart ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Namespace needs a second truncation pass through after the current pass completes. Truncation can act on more than one set at a time so if a truncate command is received for a set in a namespace that is currently already going through truncation (for a different set for example), a subsequent iteration is required. For example, a truncate against set s2 is ongoing when a truncate command against set s4 is triggered. In this case, from the moment the log message appears (when the command to truncate against set4 is issued), both set2 and set4 are truncated together until the end of the current pass. Then truncation restarts another pass through the namespace, so that the rest of set4 gets truncated. --- #### `starting truncate` `INFO` Message: ```bash {ns-name} starting truncate ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Truncate command being processed for the namespace. --- #### `{ns-name} truncated records` `INFO` Message: ```bash {ns-name} truncated records (10,50) ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Truncate command being processed for the namespace. Parameters: (current,total) Current truncation count (10 in this example) followed by the total number of records have been deleted by truncation since the server started (50 in this example). Those counts are only kept at the namespace level. --- #### `truncate to now command` `INFO` Message: ```bash {ns-name|set-name} got command to truncate to now (226886718769) ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Truncate command received. Appears on the node where the info command was issued. The command is distributed to other nodes using system metadata (SMD), and only the truncating/starting/restarting/truncated/done log entries appear on those nodes. Parameters: (timestamp) The LUT time in milliseconds since the Citrusleaf epoch (00:00:00 UTC on 1 Jan 2010). --- #### `tombstone covers no records on node` `INFO` Message: ```bash {ns-name|set-name} tombstone covers no records on this node ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Occurs at cold start-up and is a listing of all set truncations found in the truncation SMD file (System Meta Data) that are not covering any records. The internal mechanism is that all set truncation tombstones are marked as cenotaph before records are read from drive to build the index. The term tombstone refers here to such a truncation related entry, not to be confused with [record level tombstones](https://aerospike.com/docs/database/learn/architecture/durable-deletes#tombstones). We only cycle through those tombstones one time for a namespace during a cold-start. If the drives are all fresh, there is a cenotaph log message for every set truncation in SMD. After the drives are read, which could potentially restore records from truncated sets and strip a set-covering tombstone of its cenotaph status, all the set-truncation tombstones that are still cenotaphs (not covering any tombstones) are listed. --- #### `truncating namespace/set` `INFO` Message: ```bash {ns-name|set-name} truncating to 226886718769 ``` Context: truncate Introduced: 3.12.0 Removed: \- Occurs: Truncate command received. Appears on all the nodes after a truncate command is issued. Parameters: timestamp The LUT time in milliseconds since the Citrusleaf epoch (00:00:00 UTC on 1 Jan 2010). --- #### `undoing truncate` `INFO` Message: ```bash {ns-name|set-name} undoing truncate - was to 226886718769 ``` Context: truncate Introduced: 4.3.1.11, 4.4.0.11, 4.5.0.6, 4.5.1.5 Removed: \- Occurs: Truncate command undone. --- ## Tsvc #### `client transaction rejected - partition balance unresolved` `WARNING` Message: ```bash rejecting client transaction - initial partition balance unresolved ``` Description: When using some older Aerospike Client Libraries, there is a very small window where a node joins a cluster and other nodes begin to advertise the node but the node hasn’t finished creating its partition table and a client picked up that advertised service and made a request. This message happens only during that window and should resolve on its own very quickly. Context: tsvc Introduced: \- Removed: \- --- #### `transaction neither read nor write` `WARNING` Message: ```bash transaction is neither read nor write - unexpected ``` Description: An invalid request has been received, either a read/write request where the corresponding bit is not set, or an operate() command with no operations. The error code -4 (FAIL\_PARAMETER) is returned to the client if there is one, but this message can also be caused by non-Aerospike traffic reaching the service port. Context: tsvc Introduced: \- Removed: \- --- ## Udf #### `UDF bin limit exceeded` `WARNING` Message: ```bash UDF bin limit (512) exceeded (bin activity_map) ``` Description: As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be read or written by a UDF. Context: udf Introduced: \- Removed: 5.1.0 Parameters: bin Name of the 513th bin. --- #### `UDF timed out` `WARNING` Message: ```bash UDF timed out 734 ms ago ``` Description: An individual UDF transaction exceeded an internal timeout interval. Time gets checked at regular interval (of lua instruction sets). Context: udf Introduced: \- Removed: \- Parameters: elapsed time Milliseconds since the timeout period expired. See [`transaction-max-ms`](https://aerospike.com/docs/database/reference/config#service__transaction-max-ms) for details on when transactions could timeout on the server. --- #### `udf: update found urecord not open` `WARNING` Message: ```bash WARNING (udf): (udf_aerospike.c:226) update found urecord not open ``` Description: Indicates that aerospike:update() was called to update a record that the UDF system did not or could not open. Most likely the record expired or was truncated at the time the UDF ran. Context: udf Introduced: \- --- #### `UDF bin limit exceeded` `WARNING` Message: ```bash bin limit of 512 for UDF exceeded: 511 bins in use, 1 bins free, 3 new bins needed ``` Description: A UDF operation that adds bins to a record would result in more than 512 bins in the record. As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be read or written by a UDF. Context: udf Introduced: \- Removed: 5.1.0 Parameters: in-use bins Number of bins already in the record. --- free bins 512 - (# of in-use bins) --- new bins Number of new bins that would be need to be added for this operation to succeed. --- #### `UDF bin limit exceeded` `WARNING` Message: ```bash bin limit of 512 for UDF exceeded: 512 bins in use, 0 bins free, >4 new bins needed ``` Description: A UDF operation is trying to add bins to a record that already has 512 bins. As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be read or written by a UDF. Context: udf Introduced: \- Removed: 5.1.0 Parameters: new bins Number of new bins that would be need to be added for this operation to succeed. --- #### `drives full - record not updated` `WARNING` Message: ```bash drives full, record is not updated ``` Description: No more space left in storage (happened while executing a UDF). Context: udf Introduced: \- Removed: 5.1.0 --- #### `exceeded UDF max bins` `WARNING` Message: ```bash exceeded UDF max bins 512 ``` Description: UDF is attempting to set a bin value, which would result in writing a record with more than 512 bins. As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be written by a UDF. Context: udf Introduced: 5.1.0 Removed: \- --- #### `large number of lua function arguments` `WARNING` Message: ```bash large number of lua function arguments (22) ``` Description: As per the [Known Limitations](https://aerospike.com/docs/database/advanced/udf/limitations#lua-limitations) page, calling a Lua function with a large number of arguments can cause instability in the Lua runtime engine. Although this problem is only known to become acute at around 50 arguments, lower values may still result in issues with execution of the UDF. Context: udf Introduced: \- Removed: \- Parameters: arg count Number of arguments in the UDF call --- #### `{namespace_name} failed write` `DETAIL` Message: ```bash {namespace_name} failed write 1142f0217ababf9fda5b1a4de66e6e8d4e51765e ``` Description: Most likely appearing as a result of exceeding the [`write-block-size`](https://aerospike.com/docs/database/reference/config#namespace__write-block-size) with a UDF write. The record’s digest is the last item in the log entry. To determine what set is being written to, see [How to return the set name of a record using its digest](https://support.aerospike.com/s/article/How-to-return-the-set-name-of-a-record-using-its-digest). Context: udf Introduced: 5.2.0 Removed: \- Parameters: {namespace} Namespace being written to. --- #### `too many bins for UDF processing` `WARNING` Message: ```bash record has too many bins (513) for UDF processing ``` Description: As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be read or written by a UDF. Such records can exist in Aerospike, however. Context: udf Introduced: \- Removed: 5.1.0 Parameters: bins How many bins the record has. --- #### `too many bins for UDF` `WARNING` Message: ```bash too many bins (513) for UDF ``` Description: UDF is attempting to use a record (with Database 5.1.0 or older) or access the bins of a record (with Database 5.2.0+) with more than 512 bins. As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be accessed by a UDF, unless the UDF is read-only and accesses only the record’s metadata, with Database 5.2.0 and later. Context: udf Introduced: 5.1.0 Removed: \- Parameters: bins Number of bins in the record. --- #### `too many bins for UDF` `WARNING` Message: ```bash too many bins for UDF ``` Description: UDF is attempting to set a bin value, but has already set 512 bin values. As per the [UDF known limitations page](https://aerospike.com/docs/database/advanced/udf/limitations), records with more than 512 bins cannot be written by a UDF. Context: udf Introduced: 5.1.0 Removed: \- --- #### `udf applied with forbidden policy` `WARNING` Message: ```bash udf applied with forbidden policy ``` Description: A result of the client submitting a record UDF request with one or more forbidden policies indicated. A parameter error (code 4) is returned to the client in this situation. The forbidden policies are: - Generation - Generation gt - Create only - Update only - Create or replace - Replace only - Any of these policies can be enforced within the UDF itself using Aerospike’s LUA API. Prior to Database 5.2.0, the policy flags are ignored. Context: udf Introduced: 5.2.0 Removed: \- --- #### `udf_aerospike_rec_update: failure executing record updates` `WARNING` Message: ```bash udf_aerospike_rec_update: failure executing record updates (-3) ``` Description: UDF could not update a record, so the update was rolled back. See earlier messages in the log for further details. Context: udf Introduced: \- Removed: 5.1.0 Parameters: error code Undocumented error code. --- ## Xdr client #### `DC receive error on node` `WARNING` Message: ```bash DC receive error [2] on ``` Context: xdr-client Introduced: 5.0.0 Removed: \- Occurs: On its own (i.e if no other associated warning like bad protocol or similar), this warning is actually benign. This warning can also happen if there is a connection reset while trying to read from the socket. A potential cause could be the remote side having a node being restarted. --- #### `connected to datacenter node` `INFO` Message: ```bash connected to : ``` Context: xdr-client Introduced: 6.1.0.21, 6.2.0.16, 6.3.0.9, 6.4.0.3 Removed: \- Occurs: This is logged on a source node whenever the xdr-client tend thread establishes a successful connection to the destination node. The line is logged under the following conditions: - The first time the tend thread successfully connects to the destination node. If security is enabled on the destination, the login only happens if security is enabled in the configuration. - The tend thread has to re-establish the connection again as the login attempt failed when security was enabled on the destination node. - The tend thread has to re-establish the connection in cases such as source or destination restart, or any network interruption. The login flow can be found at [XDR login flow](https://aerospike.com/docs/database/manage/xdr/security#xdr-login-flow). Parameters: DCNAME The destination dc name, followed by the IP and port of the destination node. For example, `connected to destdc 172.17.0.5:3116` --- #### `connected to connector` `INFO` Message: ```bash connected to connector : ``` Description: This message is logged every 5 seconds and is a background health check connection used for connector destinations. Context: xdr-client Introduced: \- Removed: \- --- #### `node failed check` `WARNING` Message: ```bash failed check times ``` Description: `WARNING (xdr-client): (cluster.c:1010) aerospike-kafka-source 10.242.145.46:8080 failed check 130 times` Context: xdr-client Removed: \- Occurs: When connector DC is configured under XDR context, `xdr-client` tends every 5 seconds. After 5 consecutive fails, this warning is logged on the source node. --- #### `logged in to node - session-ttl` `INFO` Message: ```bash logged in to node : - session-ttl 86400 ``` Context: xdr-client Introduced: 6.1.0.21, 6.2.0.16, 6.3.0.9, 6.4.0.3 Removed: \- Occurs: Logged on a source node when there is a successful login from the xdr-client tend thread to a destination node. It contains the destination IP, port and the [`session-ttl`](https://aerospike.com/docs/database/reference/config#security__session-ttl) configured on the destination node during the login. The session is valid for this TTL period even if the TTL value changes on the destination node later, unless the session is interrupted by a node restart (source or destination) or auth failure which forces the tend thread to do a re-login. The login flow can be found at [XDR login flow](https://aerospike.com/docs/database/manage/xdr/security#xdr-login-flow). --- #### `login to node failed` `WARNING` Message: ```bash login to node xx.xx.xx.xx:xxxx failed: ``` Context: xdr-client Introduced: 6.1.0.21, 6.2.0.16, 6.3.0.9, 6.4.0.3 Removed: \- Occurs: Logged on a source node when there is a login failure to a destination node. It contains the destination node IP and port. The login flow can be found at [XDR login flow](https://aerospike.com/docs/database/manage/xdr/security#xdr-login-flow). Parameters: ERROR-CODE The error code returned by the destination server, such as `failed: 65` --- #### `refreshing session token` `INFO` Message: ```bash refreshing session token for : ``` Context: xdr-client Introduced: 6.1.0.21, 6.2.0.16, 6.3.0.9, 6.4.0.3 Removed: \- Occurs: Logged on a source DC node one minute before the [`session-ttl`](https://aerospike.com/docs/database/reference/config#security__session-ttl) configured on the destination node during the login. The server (destination node) sets the expiry of the access token, provided to the xdr-client, a minute shorter than the actual expiration time (renewal margin). The xdr-client tend thread refreshes the token one minute before its actual set expiry. It means the source node is issuing a LOGIN command to refresh its token for the specified destination DC node (IP:port). The login flow can be found at [XDR login flow](https://aerospike.com/docs/database/manage/xdr/security#xdr-login-flow). Parameters: DCNAME The destination DC name, followed by the IP and port of the destination node. For example, `refreshing session token for destdc 172.17.0.5:3116` --- #### `security not configured on node` `INFO` Message: ```bash security not configured on node : ``` Context: xdr-client Introduced: 6.1.0.21, 6.2.0.16, 6.3.0.9, 6.4.0.3 Removed: \- Occurs: Logged on a source node when there is auth configured on the source node under XDR context, but security is not enabled on the destination cluster node. It contains destination IP and port details. Logged only once when the connection is established by the tend thread. Does not affect the XDR working properly after the connection is established. The login flow can be found at [XDR login flow](https://aerospike.com/docs/database/manage/xdr/security#xdr-login-flow). --- ## Xdr #### `dc-state CLUSTER_UP` `INFO` Message: ```bash [DC_NAME]: dc-state CLUSTER_UP timelag-sec 2 lst 1468006386894 mlst 1468006389647 (2016-07-08 19:33:09.647 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 0 rsas-ms 0.004 rsas-pct 0.0 con 384 errcl 0 errsrv 0 sz 6 ``` Context: xdr Introduced: 3.9.0 Removed: \- Occurs: Every 1 minute, for each configured destination cluster (or DC). Parameters: \[DC\_NAME\] Name and status of the DC. Here are the different statuses: CLUSTER\_INACTIVE, CLUSTER\_UP, CLUSTER\_DOWN, CLUSTER\_WINDOW\_SHIP. Corresponds to the `dc_state` statistic. --- timelag-sec The lag in seconds. This is computed as the difference between the current time and the time-stamp of the record that was last successfully shipped. This provides a sense of how ‘far behind’ the destination cluster lags behind the source cluster. This does not correspond to the time it takes the source cluster to ‘catch up’, nor does it necessarily relate to the number of outstanding digests to be processed. Corresponds to the [`dc_timelag`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_timelag) statistic. --- lst The overall last ship time for the node (the minimum of all last ship times on this node). --- mlst The main last ship time (the last ship time of the dlogreader). --- fnlst The failed node last ship time (the minimum of the last ship times of all failed node shippers running on this node). --- wslst The window shipper last ship time (the minimum of the last ship times of all window shippers running on this node). --- shlat-ms Corresponds to the [`xdr_ship_latency_avg`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_latency_avg) statistic. --- rsas-ms Average sleep time for each write to the DC for the purpose of throttling. Corresponds to the [`dc_ship_idle_avg`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_ship_idle_avg) statistic. (Stands for remote ship average sleep ms). --- rsas-pct Percentage of throttled writes to the DC. Corresponds to the [`dc_ship_idle_avg_pct`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_ship_idle_avg_pct) statistic. (Stands for remote ship average sleep pct). --- con Number of open connection to the DC. If the DC accepts pipeline writes, there are 64 connections per destination node. Corresponds to the [`dc_as_open_conn`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_as_open_conn) statistic. --- errcl Number of client layer errors while shipping records for this DC. Errors include timeout, bad network fd, etc. Corresponds to the [`dc_ship_source_error`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_ship_source_error) statistic. --- errsrv Number of errors from the remote cluster(s) while shipping records for this DC. Errors include out-of-space, key-busy, etc. Corresponds to the [`dc_ship_destination_error`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_ship_destination_error) statistic. --- sz The cluster size of the destination DC.. Corresponds to the [`dc_as_size`](https://aerospike.com/docs/database/reference/metrics#xdr__dc_as_size) statistic. --- #### `digest log write failed - critical` `WARNING` Message: ```bash Digest Log Write Failed !!! ... Critical error ``` Context: xdr Introduced: \- Removed: \- Occurs: XDR digestlog has grown larger than its partition. See [Solution: Digestlog Partition Out Of Space](https://support.aerospike.com/s/article/Solution-Digestlog-Partition-Out-Of-Space) for more information. --- #### `failed to seek during reclaim - leaving sptr` `INFO` Message: ```bash Failed to seek during reclaim. Leaving sptr as is ``` Description: Benign message if the node on which this is logged did not receive new writes into the digest log. A background process running once a minute tries to reclaim the digest log based on a timestamp. It starts sampling the log by looking at the timestamp of the last record written in the digestlog. If it doesn’t find a single record, it stops early and prints this message. Context: xdr Introduced: \- Removed: \- --- #### `xdr-dc status` `INFO` Message: ```bash INFO (info): (dc.c:1024) xdr-dc DC1: lag 0 throughput 0 latency-ms 0 in-queue 0 outstanding 0 complete (3000,0,0,0) retries (0,0) recoveries (4096,0) hot-keys 0 ``` Context: xdr Introduced: 5.0.0 Removed: 5.1.0 Parameters: lag See [`lag`](https://aerospike.com/docs/database/reference/metrics#xdr__lag). --- throughput See [`throughput`](https://aerospike.com/docs/database/reference/metrics#xdr__throughput). --- latency-ms See [`latency_ms`](https://aerospike.com/docs/database/reference/metrics#xdr__latency_ms) metric. --- in-queue See [`in_queue`](https://aerospike.com/docs/database/reference/metrics#xdr__in_queue). --- in-progress See [`in_progress`](https://aerospike.com/docs/database/reference/metrics#xdr__in_progress). --- complete Composed of the following metrics: - [`success`](https://aerospike.com/docs/database/reference/metrics#xdr__success) - [`abandoned`](https://aerospike.com/docs/database/reference/metrics#xdr__abandoned) - [`not_found`](https://aerospike.com/docs/database/reference/metrics#xdr__not_found) - [`filtered_out`](https://aerospike.com/docs/database/reference/metrics#xdr__filtered_out) --- retries Composed of the following metrics: - [`retry_conn_reset`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_conn_reset) - [`retry_dest`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_dest) --- recoveries Composed of the following metrics: - [`recoveries`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries) - [`recoveries_pending`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries_pending) --- hot-keys See [`hot_keys`](https://aerospike.com/docs/database/reference/metrics#xdr__hot_keys). --- #### `xdr-dc status - lag throughput latency` `INFO` Message: ```bash INFO (info): (dc.c:1353) xdr-dc dc2: lag 12 throughput 710 latency-ms 19 in-queue 250563 in-progress 81150 complete (1002215,0,0,0) retries (0,0,23) recoveries (2048,0) hot-keys 4655 ``` Context: xdr Introduced: 5.1.0 Removed: 5.3.0 Parameters: lag See [`lag`](https://aerospike.com/docs/database/reference/metrics#xdr__lag). --- throughput See [`throughput`](https://aerospike.com/docs/database/reference/metrics#xdr__throughput). --- latency-ms See [`latency_ms`](https://aerospike.com/docs/database/reference/metrics#xdr__latency_ms). --- in-queue See [`in_queue`](https://aerospike.com/docs/database/reference/metrics#xdr__in_queue). --- in-progress See [`in_progress`](https://aerospike.com/docs/database/reference/metrics#xdr__in_progress). --- complete Composed of the following metrics: - [`success`](https://aerospike.com/docs/database/reference/metrics#xdr__success) - [`abandoned`](https://aerospike.com/docs/database/reference/metrics#xdr__abandoned) - [`not_found`](https://aerospike.com/docs/database/reference/metrics#xdr__not_found) - [`filtered_out`](https://aerospike.com/docs/database/reference/metrics#xdr__filtered_out) --- retries Composed of the following metrics: - [`retry_no_node`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_no_node) - [`retry_conn_reset`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_conn_reset) - [`retry_dest`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_dest) --- recoveries Composed of the following metrics: - [`recoveries`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries) - [`recoveries_pending`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries_pending) --- hot-keys See [`hot_keys`](https://aerospike.com/docs/database/reference/metrics#xdr__hot_keys). --- #### `xdr-dc status summary` `INFO` Message: ```bash INFO (info): (dc.c:1353) xdr-dc dc2: nodes 8 lag 12 throughput 710 latency-ms 19 in-queue 250563 in-progress 81150 complete (1002215,0,0,0) retries (0,0,23) recoveries (2048,0) hot-keys 4655 ``` Description: In the XDR log line we use the following values when multiple namespaces are shipped to a DC: - nodes refers to DC-level alone - lag is the max of all namespaces - latency is the average across namespaces - everything else is the sum across namespaces Context: xdr Introduced: 5.3.0 Removed: 6.0.0 Parameters: nodes See [`nodes`](https://aerospike.com/docs/database/reference/metrics#xdr__nodes). --- lag See [`lag`](https://aerospike.com/docs/database/reference/metrics#xdr__lag). --- throughput See [`throughput`](https://aerospike.com/docs/database/reference/metrics#xdr__throughput). --- latency-ms See [`latency_ms`](https://aerospike.com/docs/database/reference/metrics#xdr__latency_ms). --- in-queue See [`in_queue`](https://aerospike.com/docs/database/reference/metrics#xdr__in_queue). --- in-progress See [`in_progress`](https://aerospike.com/docs/database/reference/metrics#xdr__in_progress). --- complete Composed of the following metrics: - [`success`](https://aerospike.com/docs/database/reference/metrics#xdr__success) - [`abandoned`](https://aerospike.com/docs/database/reference/metrics#xdr__abandoned) - [`not_found`](https://aerospike.com/docs/database/reference/metrics#xdr__not_found) - [`filtered_out`](https://aerospike.com/docs/database/reference/metrics#xdr__filtered_out) --- retries Composed of the following metrics: - [`retry_no_node`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_no_node) - [`retry_conn_reset`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_conn_reset) - [`retry_dest`](https://aerospike.com/docs/database/reference/metrics#xdr__retry_dest) --- recoveries Composed of the following metrics: - [`recoveries`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries) - [`recoveries_pending`](https://aerospike.com/docs/database/reference/metrics#xdr__recoveries_pending) --- hot-keys See [`hot_keys`](https://aerospike.com/docs/database/reference/metrics#xdr__hot_keys). --- #### `failed to create set` `CRITICAL` Message: ```bash {NAMESPACE} failed to create set ``` Description: A record belonging to a previously-unknown set was received using XDR, but the limit on set names has already been reached locally, so the new set could not be created. If it happens while Aerospike is starting up, this message may be followed by an abort with SIGUSR1. Context: xdr Introduced: \- Removed: \- Parameters: namespace The namespace where the set could not be created. --- #### `XDR digestlog cannot keep up - dropping record` `WARNING` Message: ```bash XDR digestlog cannot keep up with writes. Dropping record. ``` Description: Aerospike is writing data faster to the XDR digestlog than the underlying disk can handle. If using raw-disk backed xdr storage, consider switching over to file-backed xdr storage for the [xdr-digestlog-path](https://aerospike.com/docs/database/reference/config#xdr__xdr-digestlog-path) parameter. This takes advantage of filesystem caching for reads and writes. Otherwise, you may need to use a faster disk or join multiple disks using RAID-0 to allow for faster read/write to the XDR digestlog. Also ensure that dmesg is properly checked for disk failures and a SMART disk test is performed. Context: xdr Introduced: \- Removed: 5.0.0 --- #### `cluster does not support pipelining` `WARNING` Message: ```bash 'XXXX' cluster does not support pipelining ``` Description: This message usually indicates that one of the destinations is running an older version of XDR (pre-3.8) or there exists a misconfiguration of the kernel configs /proc/sys/net/core/wmem\_max and /proc/sys/net/core/rmem\_max between source and destination clusters. Context: xdr Introduced: \- Removed: \- --- #### `detail: sh lg lproc rproc errcl errsrv` `INFO` Message: ```bash detail: sh 5588691 ul 12 lg 11162298 rlg 54 rlgi 0 rlgo 54 lproc 11162198 rproc 45 lkdproc 0 errcl 54 errsrv 0 hkskip 6303 hkf 6299 flat 0 ``` Context: xdr Introduced: 3.9.0 Removed: \- Parameters: sh The cumulative number of records that have been attempted to be shipped since this node started, across all datacenters. If a record is shipped to 3 different datacenters, then this number increments by 3. Corresponds to the sum of the [`xdr_ship_success`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_success), [`xdr_ship_source_error`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_source_error) and [`xdr_ship_destination_error`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_destination_error) statistics. --- ul The number of record’s digests that have been written to the node but not logged yet to the digestlog (unlogged). --- lg The number of record’s digests that have been logged. Includes both master and replica records but a node only ships records when it owns the master partition and processes records belonging to its replica partitions only when a neighboring source node goes down. --- rlog Relogged digests. The number of record’s digests that have been relogged on this node due to temporary failures when attempting to ship. Corresponds to the [`dlog_relogged`](https://aerospike.com/docs/database/reference/metrics#xdr__dlog_relogged) statistic. --- rlgi Relogged incoming digests. The number of record’s digest that another node sent to this node (typically prole side relog or partition ownership change). Corresponds to the [`relogged_incoming`](https://aerospike.com/docs/database/reference/metrics#xdr__relogged_incoming) statistic. --- rlgo Relogged outgoing digests. The number of record’s digest log entries that were sent to another node (typically prole side relog or partition ownership change). Corresponds to the [`relogged_outgoing`](https://aerospike.com/docs/database/reference/metrics#xdr__relogged_outgoing) statistic. --- lproc The number of record’s digests that have been processed locally. A processed digest does not necessarily imply a shipped record (for example, replica digests don’t get shipped unless a source node is down, and hotkeys also don’t have all their updates necessarily shipped). Corresponds to the [`dlog_processed_main`](https://aerospike.com/docs/database/reference/metrics#xdr__dlog_processed_main) statistics. --- rproc The number of replica record’s digests that have been processed by this node. A node processes records belonging to its replica partitions only when a neighboring source node goes down. Corresponds to the [`dlog_processed_replica`](https://aerospike.com/docs/database/reference/metrics#xdr__dlog_processed_replica) statistic. --- lkdproc The number of record’s digests that have been processed as part of a linked down session. A link down session is spawned when a full destination cluster is down or not reachable. Corresponds to the [`dlog_processed_link_down`](https://aerospike.com/docs/database/reference/metrics#xdr__dlog_processed_link_down) statistic. --- errcl The number of errors encountered when attempting to ship due to the embedded client. For example, if the local XDR embedded client is having issues or delays in establishing connections. Corresponds to the [`xdr_ship_source_error`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_source_error) statistic. --- errsrv The number of errors encountered when attempting to ship due to the destination cluster. For example if the destination cluster is temporarily overloaded. Corresponds to the [`xdr_ship_destination_error`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_destination_error) statistic. --- hkskip Hotkey skipped. Represents the number of record’s digests that are skipped due to an already existing entry in the reader’s thread cache (meaning a version of this record was just shipped). Corresponds to the [xdr\_hotkey\_skip](https://aerospike.com/docs/database/reference/metrics#xdr__hotkey_skip) statistic. --- hkf Hotkey fetched. Represents the number of record’s digest that are actually fetched and shipped because their cache entries expired and were dirty. Corresponds to the [xdr\_hotkey\_fetch](https://aerospike.com/docs/database/reference/metrics#xdr__hotkey_fetch) statistics. --- flat The average time in milliseconds to fetch records locally (this is an exponential moving average - 95/5). Corresponds to the [`xdr_read_latency_avg`](https://aerospike.com/docs/database/reference/metrics#xdr__read_latency_avg) statistic. --- #### `dlog: free-pct reclaimed glst` `INFO` Message: ```bash dlog: free-pct 93 reclaimed 2456 glst 1490623097271 (2017-03-27 13:58:17 GMT) ``` Description: Provides digest log (dlog) related information. Context: xdr Introduced: 3.1.02.1 Removed: \- Occurs: Every 1 minute. Parameters: free-pct Percentage of the digest log free and available for use. Corresponds to the [`dlog_free_pct`](https://aerospike.com/docs/database/reference/metrics#xdr__dlog_free_pct) statistic. --- reclaimed Indicates how many digests were safely ‘removed’ from the digestlog. As shipping successfully proceeds, and records are shipped, digests which are no longer necessary can have their space reclaimed in the digest log. A linked down (destination cluster down or unreachable) is an example where digest log space cannot be reclaimed. --- glst The minimum last ship time across all nodes in the cluster. Corresponds to the [`xdr_global_lastshiptime`](https://aerospike.com/docs/database/reference/metrics#xdr__global_lastshiptime) statistics. This specifies up to what point slots in the digest log can be reclaimed, by keeping track of the oldest last ship time across all nodes in the cluster. --- #### `dlog-q: capacity used-elements read/write-offsets` `DEBUG` Message: ```bash dlog-q: capacity 64 used-elements 1 read-offset 0 write-offset 1 ``` Description: Status information on the dlog-q. The dlog-q queue is the in-memory digest log queue. Digests of records that have been written get put on this in-memory queue. The dlogwriter picks them from there and puts them in the on-disk digest log. See below for the details on the 4 numbers. Context: xdr Introduced: 3.9.0 Removed: \- Occurs: Every 1 minute. Parameters: capacity The size of the queue. --- used-elements The number of elements in the queue. --- read-offset The read pointer of the queue. In general, the number of elements in the queue is the difference between the read pointer and the write pointer (that is, the number of elements that have been written to the queue but haven’t yet been read). --- write-offset The write pointer of the queue. In general, the number of elements in the queue is the difference between the read pointer and the write pointer (that is, the number of elements that have been written to the queue but haven’t yet been read). --- #### `xdr-tomb-raid-done: dropped tombs` `INFO` Message: ```bash {namespace} xdr-tomb-raid-done: dropped total-ms ``` Description: `INFO (xdr): (xdr_ee.c:206) {users_eu} xdr-tomb-raid-done: dropped 0 total-ms 308923` Context: xdr Removed: \- Occurs: XDR tomb raider has finished its cycle that is expected to run every [xdr-tomb-raider-period](https://aerospike.com/docs/database/reference/config#namespace__xdr-tomb-raider-period) seconds having dropped _n_ records and took _t_ ms to finish. Parameters: n Number of records dropped by the current cycle of XDR tomb raider. --- t Number of ms XDR tomb raider cycle took to finish. --- #### `DC abandon result` `WARNING` Message: ```bash {namespaceName} DC DC1 abandon result -4 ``` Context: xdr Introduced: 5.0.0 Removed: \- Occurs: In [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) enabled namespaces, if XDR finds a record which has not been replicated, re-replication from master to replica is triggered. XDR prints this warning message when this happens. XDR attempts to ship that record again. See [XDR transaction delays](https://aerospike.com/docs/database/manage/xdr/lifecycle/#transaction-delays) for details on how XDR handles record in [strong-consistency](https://aerospike.com/docs/database/reference/config#namespace__strong-consistency) enabled namespaces. See [XDR 5.0 Error codes](https://support.aerospike.com/s/article/FAQ-XDR-5-error-codes). --- #### `summary: throughput inflight dlog metrics` `INFO` Message: ```bash summary: throughput 3722 inflight 164 dlog-outstanding 100 dlog-delta-per-sec -10.0 ``` Context: xdr Introduced: 3.9.0 Removed: 5.0.0 Parameters: throughput The current throughput, shipping to destination cluster(s). When shipping to multiple clusters, the throughput represents the combined throughput to all destination clusters. Corresponds to the [`xdr_throughput`](https://aerospike.com/docs/database/reference/metrics#xdr__throughput) statistic. --- inflight The number of records that are inflight, meaning that have been sent to the destination cluster(s) but for which a response has not been received yet. Corresponds to the [`xdr_ship_inflight_objects`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_inflight_objects) statistic. --- dlog-outstanding The number of record’s digests yet to be processed in the digest log. In parenthesis, the average change normalized to digests per second over the 10 seconds interval separating those log lines. Corresponds to the [`xdr_ship_outstanding_objects`](https://aerospike.com/docs/database/reference/metrics#xdr__ship_outstanding_objects) statistic. --- dlog-delta-per-sec The variation of the dlog-outstanding normalized on a per second basis. Gives an idea whether the number of entries in the digestlog is increasing or decreasing over time and at what pace. ---