Miscellaneous issues

This page describes a few miscellaneous issues/problems that may arise.

How do I alter the write commit level?

For every write, before responding to the client, Aerospike waits for the prole reply (replica write). You can enable the fire-forget feature, which returns to the client as soon as the master write is successful.

Note

Tools package 6.0.x or later is required to use asadm’s manage config commands. Otherwise, use the equivalent asinfo - set-config command.

$ asadm -e "enable; manage config namespace NAMESPACE_NAME param write-commit-level-override to master" -h [seed-host ip]

See Per-Transaction Consistency Guarantees for more details.

Failures for the same key when accessed concurrently (transaction-pending-limit)

There could be failures when many threads try to concurrently access the same key. These are typically write commands. Reads fail this way only while the server resolves duplicates during migrations, and in some strong consistency cases. This is due to the transaction-pending-limit configuration.

For example, suppose transaction-pending-limit is set to its default value of 20. If the server holds 21 pending writes for the same key, the limit is exceeded. Any other write, delete, or User-Defined Function (UDF) call on that key then fails. The server counts these failures in the fail_key_busy statistic and returns KEY_BUSY (error code 14) to the client. If you set transaction-pending-limit to zero, the transaction queue check is not performed. Use a higher limit only as a temporary measure.

To find the record and reduce its load, see Diagnose hot keys.

What to do when you get a stack trace

At times, you may see a stack trace in the log that is not meaningful or helpful. The log might show a stack trace that looks like this:

Stack trace (non-dedicated):
 /usr/bin/asd() [0x403928]
 /lib64/libc.so.6() [0x3f8a432750]
 /usr/bin/asd() [0x4102fb]
 /usr/bin/asd() [0x412d35]
 /usr/bin/asd() [0x413a3c]
 /usr/bin/asd() [0x41755b]
 /lib64/libpthread.so.0() [0x3f8ac06a3a]
 /lib64/libc.so.6(clone+0x6d) [0x3f8a4de65d]
 End of stack trace.

Before Aerospike Support can help you diagnose the problem, you must use the GNU Product Debugger (gdb) to analyze the stack trace:

$ gdb asd
From the gdb prompt:  info line * [ID numbers from the stack trace in the log]

gdb output

 (gdb) info line *0x403928
 Line 105 of "base/as.c" starts at address 0x403928  and ends at 0x40392b .
 (gdb) info line * 0x4102fb
 Line 1387 of "base/thr_rw.c" starts at address 0x4102fb  and ends at 0x41030a .
 (gdb) info line * 0x412d35
 Line 425 of "base/thr_rw.c" starts at address 0x412d35  and ends at 0x412d3b .
 (gdb) info line * 0x413a3c
 Line 693 of "base/thr_rw.c" starts at address 0x413a3c  and ends at 0x413a41 .

Send the output to Aerospike Support for help resolving this issue.

”key field too big 1248. Is it for real?” in log

This occurs when the keys you are using are long. These messages are for informational purposes only and can be ignored. Writes will still continue. No loss of data happens. The same key will still be processed as normal.

Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?
Oct 11 2012 13:46:02 GMT: INFO (proto): (base/transaction.c:133) key field too big 1248. Is it for real?