Miscellaneous issues

This page describes a few miscellaneous issues/problems that may arise.

How do I alter the write commit level?

For every write, before responding to the client, Aerospike waits for the prole reply (replica write). You can enable the fire-forget feature, which returns to the client as soon as the master write is successful.

$ asadm -e "enable; manage config namespace <namespaceName> param write-commit-level-override to master" -h [seed-host ip]

See Per-Transaction Consistency Guarantees for more details.

Failures for the same key when accessed concurrently (transaction-pending-limit)

There could be failures when many threads try to concurrently access the same key. Typically these are write commands, though reads can be relevant in certain strong consistency scenarios. This is due to the transaction-pending-limit configuration.

For example, if transaction-pending-limit is set to its default value of 20, and there are 21 writes for the same key in the queue, then transaction-pending-limit is considered exceeded and any other requested operations on that key will fail. Increase this configuration value to accommodate the behavior of your application. The statistic that tracks these failures is err_rw_pending_limit. If you set transaction-pending-limit to zero, the transaction queue check is not performed.

See Knowledge-Base article on Hot Key Error Code 14 for more details.

What to do when you get a stack trace

At times, you may see a stack trace in the log that is not meaningful or helpful. The log might show a stack trace that looks like this:

Stack trace (non-dedicated):
 /usr/bin/asd() [0x403928]
 /lib64/libc.so.6() [0x3f8a432750]
 /usr/bin/asd() [0x4102fb]
 /usr/bin/asd() [0x412d35]
 /usr/bin/asd() [0x413a3c]
 /usr/bin/asd() [0x41755b]
 /lib64/libpthread.so.0() [0x3f8ac06a3a]
 /lib64/libc.so.6(clone+0x6d) [0x3f8a4de65d]
 End of stack trace.

Before Aerospike Support can help you diagnose the problem, you must use the GNU Product Debugger (gdb) to analyze the stack trace:

$ gdb asd
From the gdb prompt:  info line * [ID numbers from the stack trace in the log]

gdb output

 (gdb) info line *0x403928
 Line 105 of "base/as.c" starts at address 0x403928  and ends at 0x40392b .
 (gdb) info line * 0x4102fb
 Line 1387 of "base/thr_rw.c" starts at address 0x4102fb  and ends at 0x41030a .
 (gdb) info line * 0x412d35
 Line 425 of "base/thr_rw.c" starts at address 0x412d35  and ends at 0x412d3b .
 (gdb) info line * 0x413a3c
 Line 693 of "base/thr_rw.c" starts at address 0x413a3c  and ends at 0x413a41 .

Send the output to Aerospike Support for help resolving this issue.

”key field too big 1248. Is it for real?” in log

This occurs when the keys you are using are long. These messages are for informational purposes only and can be ignored. Writes will still continue. No loss of data happens. The same key will still be processed as normal.

Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?
Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?