Aerospike client best practices
Enable Client Log
Each AerospikeClient instance runs a background cluster tend thread that periodically polls all nodes for cluster status. This background thread generates log messages that reflect node additions/removal and any errors when retrieving node status, peers, partition maps and racks. It's critical that user applications enable this log to receive these important messages.
See Enable Log.
Share Client Instance
Each AerospikeClient instance spawns a maintenance thread that periodically makes info requests to all server nodes for cluster status. Multiple client instances create additional load on the server. Use only one client instance per cluster in an application and share that instance among multiple threads. AerospikeClient is thread-safe.
Set Maximum Connections
Set ClientPolicy.maxConnsPerNode
and/or ClientPolicy.asyncMaxConnsPerNode
to the maximum number of connections allowed (default 300) per server node. Sync and async connections are tracked separately. The number of connections used per node depends on concurrent commands in progress plus sub-commands used for parallel multi-node commands (batch, scan, and query). One connection will be used for each command. A request will fail if the maximum number of connections would be exceeded.
User-Defined Key
By default, the user-defined key is not stored on the server. It is converted to a hash digest which is used to identify a record. If the user-defined key must persist on the server, use one of the following methods:
- Set
WritePolicy.sendKey
to true. The key is sent to the server for storage on writes, and retrieved on multi-record scans and queries. - Explicitly store and retrieve the user-defined key in a bin.
Avoid Value/Bin Generic Object Constructors
Do not use Value
or Bin
constructors that take in an object. These constructors are slower than hard-coded constructors because the object must be queried (using instanceof
) for its real type. They also use the default Java serializer, which is the slowest of all serialization implementations. Instead, serialize the object with a better serializer and use the byte[] constructor.
Replace Mode
In cases where all record bins are created or updated by a command, enable Replace mode on the command to increase performance. The server then does not have to read the old record before updating. Do not use Replace mode when updating a subset of bins.
WritePolicy policy = new WritePolicy();
policy.recordExistsAction = RecordExistsAction.REPLACE;
client.put(policy, key, bins);
Policy
Each database command takes in a policy as the first argument. If the policy is identical for a group of commands, reuse them instead of instantiating policies for each command.
Also, set policy defaults in ClientPolicy
and pass in a null policy on each command.
ClientPolicy policy = new ClientPolicy();
policy.readPolicyDefault.socketTimeout = 50;
policy.readPolicyDefault.totalTimeout = 110;
policy.readPolicyDefault.sleepBetweenRetries = 10;
policy.writePolicyDefault.socketTimeout = 200;
policy.writePolicyDefault.totalTimeout = 450;
policy.writePolicyDefault.sleepBetweenRetries = 50;
AerospikeClient client = new AerospikeClient(policy, "hostname", 3000);
client.put(null, new Key("test", "set", 1), new Bin("bin", 5));
If a policy needs to change from the default (such as setting an expected generation), create a new policy that copies the default policy (copy on write). This avoids the case where shared policy instance modifications affect other commands running in parallel.
public void putIfGeneration(Key key, int generation, Bin... bins) {
WritePolicy policy = new WritePolicy(client.writePolicyDefault);
policy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL;
policy.generation = generation;
client.put(policy, key, bins);
}
Circuit Breaker
Employ a circuit breaker that activates when a maximum error count is reached for a node and rejects requests to that node until the specified error window expires. The following ClientPolicy fields can create a circuit breaker.
maxErrorRate
Maximum number of errors allowed per node per errorRateWindow
. Errors include connection errors, timeouts and device overload. If maximum errors are reached, further requests to that node are retried to another node depending on replica policy. If maxRetries are exhausted, a backoff exception AerospikeException.Backoff
is thrown with error code ResultCode.MAX_ERROR_RATE
.
errorRateWindow
The number of cluster tend iterations that defines the window for maxErrorRate
. One tend iteration is defined as the tend interval (default 1 second) plus the time to tend all nodes. At the end of the window, the error count is reset to zero and backoff state is removed on all nodes.
The user application could optionally use a fallback cluster to handle traffic when the circuit breaker is employed.
Close RecordSet/ResultSet
RecordSet/ResultSet query iterators should always be closed after the iterator is no longer used. Failure to close the iterator when an exception occurs while processing query results may cause the query buffer to fill up and prevent server nodes from completing the query.
try (RecordSet rs = client.query(null, stmt)) {
while (rs.next()) {
...
}
}
or
RecordSet rs = client.query(null, stmt);
try {
while (rs.next()) {
...
}
}
finally {
rs.close();
}
Operate
Use AerospikeClient.operate()
to batch multiple operations (add/get) on the same record in a single call.
Using virtual threads
Aerospike Java client 8 introduces support for virtual threads in sync commands.
To use virtual threads with the Java client, specify aerospike-client-jdk21
as the artifactId
and Aerospike client version 8.0.1 in your application's .pom
file. The Java client version 8 does not require any changes to your application
code, but getting the maximum performance benefit of virtual threads
requires some code updates.
Remove thread local variables
Virtual threads are lightweight and are usually created and destroyed with a higher frequency than traditional OS threads. Thread local variables are useful for threads that exist for a long period of time.
The Java client itself has replaced thread local buffers with heap allocated buffers. Short-lived virtual threads are likely to require a heap allocation to create the buffer and still retain the overhead of maintaining the buffer on the virtual thread, which are likely only used for one database command.
Remove sync thread pools
Sync thread pools are no longer necessary, because virtual threads are created and destroyed with minimal performance impact. The Java client no longer uses a sync thread pool. Instead, it splits sync batch/scan/query commands into multiple node commands and runs each node command in a virtual thread in parallel.
Use the virtual thread fanout pattern
A common task is to run multiple sub-tasks in parallel and then wait until all sub-tasks have completed. JDK 21 provides an optimized virtual thread implementation to accomplish this task. The Java client uses this pattern to execute sync batch/scan/query commands in parallel.
ThreadFactory threadFactory = Thread.ofVirtual().name("Aerospike-", 0L).factory();
try (ExecutorService es = Executors.newThreadPerTaskExecutor(threadFactory)) {
for (IBatchCommand command : commands) {
es.execute(command);
}
}
When the try
block completes, the implicit ExecutorService close()
efficiently waits for all node commands to complete.
Replace synchronized locks with ReentrantLock
Replace long running synchronized locks with ReentrantLock
. When synchronized locks use virtual threads with the synchronized
keyword, they lock the underlying OS thread, potentially reducing performance and causing deadlocks.
Run database commands in virtual threads
The application determines the thread type used to run database commands. Create virtual threads instead of OS threads to run these commands.
Old OS thread:
Thread thread = Thread.ofPlatform();
New virtual thread:
Thread thread = Thread.ofVirtual();
Benchmarking
The Java benchmarking tool now includes
a command line option, -vt <count>
, to run benchmarks in virtual threads.
These benchmarks show an approximately 18% performance boost when using
virtual threads versus OS threads for sync commands. Asynchronous command
performance has not changed.