Client errors
For the complete documentation index see: llms.txt
All documentation pages available in markdown.
Client Error Codes
Individual client error codes are provided in each client’s API references:
Example:
- Java Client Error Codes: Result Codes
- C# Client Error Codes: Result Codes
Client receiving Server memory errors
When the node reaches a stop write or the node’s device is not able to catch up with writing to disk, the client receives server memory errors. Errors vary depending on your client. For example, in the Java client, this appears as Error Code 8 SERVER_MEM_ERROR.
- Verify that your cluster is running the latest OS firmware and there’s no general disk failure.
- Run iostat to get your server’s CPU and I/O stats.
- Check the cluster’s disk and memory usage using the monitoring tool (asadm) to see if it’s higher than the stop-writes percentage. If it is, your namespace may be running out of space on that node.
- Check the
queuedelay values on every node and watch if it’s increasing:
$ asadm$ Admin> watch 2 show statistics like queueor
$ asinfo -v "statistics" -l | grep queue- Check the log file
/var/log/aerospike/aerospike.logfor “write-q” and check if any objects are stuck in the queue waiting.
In addition, you can address the server errors by setting the stop write threshold or by deleting or evicting the objects faster.
Once you have adjusted the settings, continue monitoring the cluster and the write latency statistics. Also, verify that the used memory bytes is decreasing as more objects are getting evicted:
$ Admin> watch 2 show statistics like memoryClient is logging KEY_BUSY result code messages
KEY_BUSY (error code 14) means that the server rejected a command because too many commands for the same record were already pending.
This error usually indicates a hot key, a record that receives a disproportionate share of requests.
-
Writes, deletes, and User-Defined Function (UDF) calls can fail with
KEY_BUSYin any namespace. -
Reads in a stable Available and Partition-tolerant (AP) namespace never fail with
KEY_BUSY, so a hot read key in that namespace doesn’t incrementfail_key_busy. -
Reads during migrations, and some reads in strong consistency (SC) namespaces, can fail with
KEY_BUSY. For the full rules, see Diagnose hot keys. -
When the server rejects a command with
KEY_BUSY, it increments the namespacefail_key_busystatistic. The following command runs five times, 2 seconds apart, so you can check whether the statistic is growing on each node:Terminal window asadm -e "watch 2 5 show statistics namespace like fail_key_busy" -
The namespace parameter
transaction-pending-limitsets how many commands can be pending for the same record, 20 by default. Use a higher value only as a temporary measure while you fix the access pattern. The following command sets the limit to 25 for one namespace on every node. ReplaceNAMESPACEwith the namespace name.
asadm -e "enable; manage config namespace NAMESPACE param transaction-pending-limit to 25"To find the hot key and change the access pattern, see Diagnose hot keys.
Client request
If a network glitch occurs during a client request to the Aerospike server, the info request from the client fails. This network issue can be caused by congestion or packet loss. The client retries every second. If the message “DUNNED NODE” appears in the client log (happiness value becomes 0), the client cannot reach a node in the cluster and will need to reconnect. This is a normal part of the client behavior when there is some interruption talking to the cluster nodes. You will see error messages similar to the following in the Aerospike log (/var/log/aerospike/aerospike.log). These are generally temporary issues.
2012-09-25 09:01:41 EDT AEROSPIKE [143] info: could not connect to abc1.def.domain.com : connect timed out2012-09-25 09:01:41 EDT AEROSPIKE [143] node BB9738A8E9B2100 info get failed, decrease happiness abc1.def.domain.com/10.0.0.53:30002012-09-25 09:34:56 EDT AEROSPIKE [243] node BB9738A8E9B2100 has valid network activity but not fully happy : 250