Skip to content

Client errors

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

Client Error Codes

Individual client error codes are provided in each client’s API references:

Example:

Client receiving Server memory errors

When the node reaches a stop write or the node’s device is not able to catch up with writing to disk, the client receives server memory errors. Errors vary depending on your client. For example, in the Java client, this appears as Error Code 8 SERVER_MEM_ERROR.

  • Verify that your cluster is running the latest OS firmware and there’s no general disk failure.
  • Run iostat to get your server’s CPU and I/O stats.
  • Check the cluster’s disk and memory usage using the monitoring tool (asadm) to see if it’s higher than the stop-writes percentage. If it is, your namespace may be running out of space on that node.
  • Check the queue delay values on every node and watch if it’s increasing:
$ asadm
$ Admin> watch 2 show statistics like queue

or

Terminal window
$ asinfo -v "statistics" -l | grep queue
  • Check the log file /var/log/aerospike/aerospike.log for “write-q” and check if any objects are stuck in the queue waiting.

In addition, you can address the server errors by setting the stop write threshold or by deleting or evicting the objects faster.

Once you have adjusted the settings, continue monitoring the cluster and the write latency statistics. Also, verify that the used memory bytes is decreasing as more objects are getting evicted:

$ Admin> watch 2 show statistics like memory

Client is logging KEY_BUSY result code messages

KEY_BUSY (error code 14) means that the server rejected a command because too many commands for the same record were already pending. This error usually indicates a hot key, a record that receives a disproportionate share of requests.

  • Writes, deletes, and User-Defined Function (UDF) calls can fail with KEY_BUSY in any namespace.

  • Reads in a stable Available and Partition-tolerant (AP) namespace never fail with KEY_BUSY, so a hot read key in that namespace doesn’t increment fail_key_busy.

  • Reads during migrations, and some reads in strong consistency (SC) namespaces, can fail with KEY_BUSY. For the full rules, see Diagnose hot keys.

  • When the server rejects a command with KEY_BUSY, it increments the namespace fail_key_busy statistic. The following command runs five times, 2 seconds apart, so you can check whether the statistic is growing on each node:

    Terminal window
    asadm -e "watch 2 5 show statistics namespace like fail_key_busy"
  • The namespace parameter transaction-pending-limit sets how many commands can be pending for the same record, 20 by default. Use a higher value only as a temporary measure while you fix the access pattern. The following command sets the limit to 25 for one namespace on every node. Replace NAMESPACE with the namespace name.

Terminal window
asadm -e "enable; manage config namespace NAMESPACE param transaction-pending-limit to 25"

To find the hot key and change the access pattern, see Diagnose hot keys.

Client request

If a network glitch occurs during a client request to the Aerospike server, the info request from the client fails. This network issue can be caused by congestion or packet loss. The client retries every second. If the message “DUNNED NODE” appears in the client log (happiness value becomes 0), the client cannot reach a node in the cluster and will need to reconnect. This is a normal part of the client behavior when there is some interruption talking to the cluster nodes. You will see error messages similar to the following in the Aerospike log (/var/log/aerospike/aerospike.log). These are generally temporary issues.

2012-09-25 09:01:41 EDT AEROSPIKE [143] info: could not connect to abc1.def.domain.com : connect timed out
2012-09-25 09:01:41 EDT AEROSPIKE [143] node BB9738A8E9B2100 info get failed, decrease happiness abc1.def.domain.com/10.0.0.53:3000
2012-09-25 09:34:56 EDT AEROSPIKE [243] node BB9738A8E9B2100 has valid network activity but not fully happy : 250