Misc Issues
This page describes a few miscellaneous issues/problems that may arise.
How do I alter the write commit level?
For every write before responding to the client, Aerospike will wait for the prole reply (replica write) . You can enable the fire-forget feature, which will return to the client as soon as the master write is successful.
Tools package 6.0.x or later is required to use asadm's manage config commands. Otherwise, use the equivalent asinfo - set-config command.
$ asadm -e "enable; manage config namespace <namespaceName> param write-commit-level-override to master" -h [seed-host ip]
See documentation on Per-Transaction Consistency Guarantees for more details.
Failures for the same key when accessed concurrently (transaction-pending-limit)
When many threads try to concurrently access the same key (generally these will be write operations, though reads could possibly also be relevant in certain strong consistency scenarios), there could be failures. This is due to the configuration transaction-pending-limit.
For example, if transaction-pending-limit is 9 and there are 10 writes for the same key in the queue, then transaction-pending-limit is considered exceeded and any other requested operations on that key will fail.
The default is 20. Increase this configuration to accommodate the behavior of your application. The statistic that tracks these failures is err_rw_pending_limit. If you set transaction-pending-limit to zero, the transaction queue check is not performed.
See Knowledge-Base article on Hot Key Error Code 14 for more details.
What to do when you get a Stack Trace
At times, you see a stack trace in the log that is not meaningful or helpful. The log might show a stack trace that looks like this:
Stack trace (non-dedicated):
/usr/bin/asd() [0x403928]
/lib64/libc.so.6() [0x3f8a432750]
/usr/bin/asd() [0x4102fb]
/usr/bin/asd() [0x412d35]
/usr/bin/asd() [0x413a3c]
/usr/bin/asd() [0x41755b]
/lib64/libpthread.so.0() [0x3f8ac06a3a]
/lib64/libc.so.6(clone+0x6d) [0x3f8a4de65d]
End of stack trace.
Before Aerospike can help you diagnose the problem, you must run gdb.
If you don't have gdb installed, you must install it: CentOS:
$ sudo yum install gdb
Then you must use gdb to analyze the stack trace:
$ gdb asd
From the gdb prompt: info line * [ID numbers from the stack trace in the log]
The output should look like this:
(gdb) info line *0x403928
Line 105 of "base/as.c" starts at address 0x403928 and ends at 0x40392b .
(gdb) info line * 0x4102fb
Line 1387 of "base/thr_rw.c" starts at address 0x4102fb and ends at 0x41030a .
(gdb) info line * 0x412d35
Line 425 of "base/thr_rw.c" starts at address 0x412d35 and ends at 0x412d3b .
(gdb) info line * 0x413a3c
Line 693 of "base/thr_rw.c" starts at address 0x413a3c and ends at 0x413a41 .
Send the output to Aerospike Tech Support for help resolving this issue.
"key field too big 1248. Is it for real?" in log
This occurs when the keys you are using are long. These messages are for informational purposes only and can be ignored. Writes will still continue. No loss of data happens. The same key will still be processed as normal.
Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?
Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?