Managing Batch Operations
Batch-Index protocol
A batch is a series of requests that are sent together to the Aerospike database. A batch groups multiple operations into one unit and passes it in a single network trip to each database node. See Batch Operations for more information.
Tuning batches
Background
An incoming batch request from a client is assigned to a specific batch response thread if the status is not "full". A thread is declared "full" when batch-max-buffers-per-queue (255 default) are in use in a queue.
Each batch response thread has a queue of 128KiB buffers. Existing batch requests assigned to a response thread can allocate buffers beyond batch-max-buffers-per-queue, as needed. The “full” designation prevents a queue or response thread from assigning new incoming batch requests. If all response threads are at "full" status, new incoming batch requests are not accepted and an error is returned to the client.
When response threads need a new buffer, they take a buffer from a single pool of unused buffers. This pool is empty at startup and serves all the response threads. Response threads allocate the buffers, and then return them to the pool of unused buffers when done.
A destination client is associated with each 128KB buffer allowing the same response thread to serve multiple batch requests. Excess buffers are destroyed if the number of buffers in the unused pool exceeds batch-max-unused-buffers (default 256). Records that require more than 128KB are allocated a "huge" buffer which is destroyed after use and is not saved in the unused buffer pool.
Operational considerations
The three main configuration options for tuning batch transactions on the server side are the following:
| Name | Default | Max | Dynamic | Description |
|---|---|---|---|---|
batch-max-buffers-per-queue | 255 | true | Maximum number of 128KiB response buffers allowed in each batch queue before it is designated "full". Additional buffers beyond batch-max-buffers-per-queue can be allocated for accepted batch requests, if needed. If all batch queues are full, new batch requests are rejected with error code 152 (AS_ERR_BATCH_QUEUES_FULL). | |
batch-max-unused-buffers | 256 | true | Maximum number of 128KiB response buffers allowed in the unused buffer pool for reuse by any response thread. If the limit is reached, new buffers created by response threads at runtime are destroyed upon completion of the batch request. This limits the size of the unused buffer pool that serves all response threads. | |
batch-index-threads | Number of CPUs | 256 | true | Number of batch index response worker threads. Each thread has its own queue. These threads only handle returning batch response buffers to the client using sockets. The maximum memory consumption can be computed as: batch-index-threads x batch-max-buffer-per-queue x 128KB. Tuning batch threads to 0 will disable batch functionality, rejecting batch commands with error code 150 (AS_ERR_BATCH_DISABLED) |
The default values should be adequate for most workloads, but you may need to adjust those parameters for more demanding workloads to avoid situations where all batch index queues are full, or intermittent CPU spikes. It is also important to understand the memory impact of those parameters.
For example, with 4 index threads and 512 buffers per queue, the total memory used by batch operations is 4 x 512 x 128 KiB = 256 MiB. With 8 index threads and 1024 buffers per queue, the total memory used is 8 x 1024 x 128 KiB = 1 GiB.
The tools package 6.0.x or later is required to use Aerospike Admin (asadm) manage config commands.
Otherwise, use the equivalent asinfo - set-config command.
To display the current batch settings, use the show config like batch command in asadm:
Admin> show config like batch
~~~~~~~~~~~~~~~~Service Configuration~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch-index-threads : 4
batch-max-buffers-per-queue: 255
batch-max-requests : 5000
batch-max-unused-buffers : 256
batch-priority : 200
batch-threads : 4
query-batch-size : 100
To monitor your batch statistics, use the asadm command show statistics
like batch_index.
In the following example, 62 buffers are created (batch_index_created_buffers) and none are destroyed (batch_index_destroyed_buffers).
There are 44 unused buffers (batch_index_unused_buffers) and 18 buffers (batch_index_queue) being used by the 4 index threads.
Admin> show statistics like batch_index
~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch_index_complete : 3534847
batch_index_created_buffers : 62
batch_index_destroyed_buffers: 0
batch_index_error : 46
batch_index_huge_buffers : 0
batch_index_initiate : 3534911
batch_index_queue : 5:5,4:4,4:4,5:5
batch_index_timeout : 0
batch_index_unused_buffers : 44
In the following example, there are 33 unused buffers.
In the batch_index_queue, there are 16 + 5 + 5 = 26 buffers in use.
~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~
NODE : host.aerospike.com:3000
batch_index_complete : 3520267
batch_index_created_buffers : 62
batch_index_destroyed_buffers: 0
batch_index_error : 44
batch_index_huge_buffers : 0
batch_index_initiate : 3520335
batch_index_queue : 12:16,6:5,0:0,6:5
batch_index_timeout : 0
batch_index_unused_buffers : 33
In the following example, there are 191 unused buffers and the currently active buffers are (165620 - 165324) = 296.
The default batch-max-unused-buffers is 256 so there are buffers being created that will be destroyed.
There are also buffers larger than 128 KiB being created to accommodate records larger than 128 KiB.
Those are tracked by the batch_index_huge_buffers metric.
Huge buffers are destroyed after completed and are not be returned to the buffer pool.
The buffer pool can contain a maximum of 256 128 KiB buffers by default.
They get used when available and returned to the pool when completed, if the pool size is still below the maximum configured value.
If the batch_index_created_buffers and batch_index_destroyed_buffers metrics are incrementing quickly, it is an indication of high garbage collection activity.
This may potentially lead to CPU spikes.
Tuning the batch-max-unused-buffers configuration option to a higher value may help.
batch_index_complete : 964670
batch_index_created_buffers : 165620
batch_index_destroyed_buffers: 165324
batch_index_error : 17168
batch_index_huge_buffers : 154797
batch_index_initiate : 1020295
batch_index_queue : 9:23,4:33,11:33,7:6
batch_index_timeout : 38426
batch_index_unused_buffers : 191
If the batch_index_huge_buffers metric is high, you may want to avoid using batch.
Large records already saturate socket data buffers in single record mode, so batching them together does not provide any benefit and results in higher memory usage on the server.
To increase the number of idle batch response buffers, use the following asadm command:
asadm -e "enable; manage config service param batch-max-unused-buffers to 512"
You can experiment with the effect of huge buffers on the batch index protocol with the benchmarking application. Try the following benchmarking command:
./run_benchmarks -h 127.0.0.1 -p 3000 -n test -k 10000000 -b 1 -o B:180000 -w RU,99 -g 5000 -T 50 -z 80 -B 10 -t 500000;
Run the command a few times and check your batch metrics:
Admin> show config like batch
batch_index_complete : 258277
batch_index_created_buffers : 11906
batch_index_destroyed_buffers: 11823
batch_index_error : 0
batch_index_huge_buffers : 11823
batch_index_initiate : 258277
batch_index_queue : 0:0,0:0,0:0,0:0
batch_index_timeout : 0
batch_index_unused_buffers : 83
The metrics show that batch_index_huge_buffers and batch_index_destroyed_buffers
are the same. This indicates that huge buffers are being created and then destroyed,
reducing the efficiency of the batch operations.
Metrics
The server provides the following batch metrics.
| Name | Description |
|---|---|
batch_index_initiate | Number of batch requests received. |
batch_index_queue | Number of batch requests and response buffers remaining on each batch queue. Format: <q1 requests> :<q1 buffers> ,<q2 requests> :<q2 buffers>,... |
batch_index_complete | Number of completed batch requests. |
batch_index_timeout | Number of timed-out batch requests. |
batch_index_error | Number of batch requests rejected because of errors. |
batch_index_unused_buffers | Number of available 128KB response buffers in the buffer pool. |
batch_index_huge_buffers | Number temporary response buffers created that exceeded 128KB. Huge buffers are created when one of the records is retrieved that is greater than 128KB. Huge records do not benefit from batching and can result in excessive memory thrashing on the server. |
batch_index_created_buffers | Number of 128KB response buffers created. Response buffers are created when there are no buffers left in the pool. If this number consistently increases and there is available memory, then batch-max-unused-buffers should be increased. |
batch_index_destroyed_buffers | Number of 128KB response buffers destroyed. Response buffers are destroyed when there is no slot left to put the buffer back into the pool. The maximum response buffer pool size is batch-max-unused-buffers. |
batch-index | Batch performance histogram. |
Batch sub transactions
The server also provides the following batch sub transactions for read, writes, deletes, udfs, and (Lua) language. Refer to Metrics Reference.
| Name | Description |
|---|---|
batch_sub_delete_success | Number of batch delete sub transactions that were completed. |
batch_sub_delete_error | Number of batch delete sub transactions that failed with an error. |
batch_sub_delete_timeout | Number of batch delete sub transactions that timed out. |
batch_sub_delete_not_found | Number of batch delete sub transactions that were not found. |
batch_sub_delete_filtered_out | Number of batch delete sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression. |
batch_sub_lang_read_success | Number of batch sub language read transactions that were completed. |
batch_sub_lang_write_success | Number of batch sub language write transactions that were completed. |
batch_sub_lang_delete_success | Number of batch sub language delete transactions that were completed. |
batch_sub_lang_error | Number of batch language sub transactions that failed with an error. |
batch_sub_udf_complete | Number of batch UDF sub transactions that were completed. |
batch_sub_udf_error | Number of batch UDF sub transactions that failed with an error. |
batch_sub_udf_timeout | Number of batch UDF sub transactions that timed out. |
batch_sub_udf_filtered_out | Number of batch UDF sub transactions that were filtered out. Transactions filtered out at the bin level by a filter expression. |
batch_sub_write_success | Number of batch write sub transactions that were completed. |
batch_sub_write_error | Number of batch write sub transactions that failed with an error. |
batch_sub_write_timeout | Number of batch write sub transactions that timed out. |
batch_sub_write_filtered_out | Number of batch write sub transactions that were filtered out. Transactions filtered out at the bin level by a predicate expression. |
batch_sub_proxy_complete | Number of proxied batch sub transactions that completed. |
batch_sub_proxy_error | Number of proxied batch sub transactions that failed with an error. |
batch_sub_proxy_timeout | Number of proxied batch sub transactions that timed out. |
batch_sub_read_error | Number of batch read sub transaction that failed with an error. |
batch_sub_read_not_found | Number of batch read sub transactions that resulted in not found. |
batch_sub_read_success | Number of successful batch read sub transactions. |
batch_sub_read_timeout | Number of batch read sub transactions that timed out. |
batch_sub_tsvc_error | Number of batch read sub transactions that failed with an error in the transaction service, before attempting to handle the transaction. For example protocol errors or security permission mismatch. |
batch_sub_tsvc_timeout | Number of batch read sub transactions that timed out in the transaction service, before attempting to handle the transaction. For example, protocol errors or security permission mismatch. |
retransmit_all_batch_sub_dup_res | Obsolete as of Database 6.0 Number of retransmits that occurred during batch sub transactions that were being duplicate resolved. Note this includes retransmits originating on the client as well as proxying nodes. |
retransmit_batch_sub_dup_res | Number of retransmits that occurred during batch sub transactions that were being duplicate resolved. Replaced with retransmit_all_batch_sub_dup_res. |
retransmit_all_batch_sub_read_dup_res | Number of retransmits that occurred during batch read subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_write_dup_res | Number of retransmits that occurred during batch write subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_delete_dup_res | Number of retransmits that occurred during batch delete subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_udf_dup_res | Number of retransmits that occurred during batch UDF subtransactions that were being duplicate-resolved. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_write_repl_write` | Number of retransmits that occurred during batch write (insert/update/upsert/replace) subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_delete_repl_write` | Number of retransmits that occurred during batch delete subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_udf_repl_write` | Number of retransmits that occurred during batch UDF subtransactions that were being replica-written. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_read_repl_ping` | Number of retransmits that occurred during SC linearized reads. Includes retransmits originating on the client as well as proxying nodes. |
retransmit_all_batch_sub_read_repl_ping` | Number of retransmits that occurred during SC linearized read subtransactions within a batch operation. Includes retransmits originating on the client as well as proxying nodes. |
early_tsvc_batch_sub_error` | Number of errors early in the transaction for batch sub transactions. For example, bad/unknown namespace name or security authentication errors. |
from_proxy_batch_sub_delete_success | Number of records successfully deleted by batch sub delete transaction proxied from another node. |
from_proxy_batch_sub_delete_error | Number of records that were not deleted and failed with an error by batch sub delete transaction. |
from_proxy_batch_sub_delete_timeout | Number of records that were not deleted due to time out by batch sub delete transaction. |
from_proxy_batch_sub_delete_not_found | Number of records that were not deleted because that were not found by batch sub transaction. |
from_proxy_batch_sub_delete_filtered_out | Number of records that were not deleted because they were filtered out by batch sub transaction. Transactions filtered out at the bin level by a filter expression. |
from_proxy_sub_lang_read_success | Number of records that were completed by batch sub language read transactions. |
from_proxy_sub_lang_write_success | Number of records that were completed by batch sub language write transactions. |
from_proxy_sub_lang_delete_success | Number of records that succeeded by batch sub language delete transactions proxied from another node. |
from_proxy_sub_lang_error | Number of records that failed with an error by batch sub language transaction. |
from_proxy_sub_udf_complete | Number of records that were completed by batch sub udf transaction. |
from_proxy_sub_udf_error | Number of records that failed with an error by batch sub udf transaction. |
from_proxy_sub_udf_timeout | Number of records of batch UDF sub transactions proxied from another node that timed out, before attempting to handle this transaction. |
from_proxy_sub_udf_filtered_out | Number of records of batch UDF sub transactions proxied from another node that did not happen because the record was filtered out using a filter expression. |
from_proxy_sub_write_success | Number of records successfully written by batch sub transactions proxied from another node. |
from_proxy_sub_write_error | Number of batch write sub transactions proxied from another node that failed with an error. |
from_proxy_sub_write_timeout | Number of batch write sub transactions proxied from another node that timed out. |
from_proxy_sub_write_filtered_out | Number of batch write sub transactions proxied from another node that did not happen because the record was filtered out using a predicate expression. |
Batch log file histograms
Periodically, the Aerospike server writes histograms to the log file. Refer to Latency Monitoring for latency histograms of batch and its sub transactions. In addition, review the Batch Transaction Analysis.
| Name | Description |
|---|---|
batch_sub_write_master | Time taken for writing all the copies of a record to the master . This only applies for strong consistency enabled namespaces.\n |
batch_sub_udf_master | Time taken from partition reserved or after duplicate resolution to an actual master record applied.\n |
batch_sub_repl_write | Time taken from the master record written to replica(s) written.\n |