Read-write transactions
For the complete documentation index see: llms.txt
All documentation pages available in markdown.
In this tutorial, you run read-modify-write (R-M-W) updates from several concurrent workers. You see how concurrent updates lose writes, then use the record’s generation to prevent that loss. Choose your client in any tab group. The page remembers your choice.
The Java SDK and Python SDK tabs use the Aerospike Developer SDKs. The Java and Python tabs use the legacy clients.
Prerequisites
- An Aerospike Database running on your computer and listening on port 3000, with a namespace named
test. The Java SDK and Python SDK tabs require Aerospike Database 8.2.0 or later. For the other clients, see the Client Compatibility Matrix. To set up a database, see the Quickstart. - Your client installed. To install it, follow the Install the client step of the Hello, World! tutorial.
- Familiarity with the Hello, World! and Basic operations tutorials.
Introduction
By default, Aerospike runs a single command on a single record atomically and in isolation. A command can have several operations on that record. Some updates fit in one command, such as incrementing a number, or using an expression to compute a bin from other bins in the record.
An update that needs general logic, such as concatenating alternate letters, or data from outside the record can’t run as one command of operations or expressions. A User-Defined Function (UDF) can run complex logic in a transactional update of a single record, but UDFs aren’t suitable for every situation, for reasons such as performance and ease of use. Most such updates therefore use the R-M-W pattern: read the record, modify the bins on the client, and write the updated record back to the server.
Aerospike Database 8.0 and later also supports multi-record transactions. This tutorial covers only updates to a single record.
This tutorial first shows how read-write commands can lose writes when several clients run at the same time. It then shows how to make a write conditional on the record’s version. An interleaved read and write is rejected, so no write is lost.
The problem of lost writes
In a concurrent setting, several clients can run a read-modify-write on the same record and get in each other’s way. The transactions can interleave, so a read-modify-write attempt is lost if another client updates the record without having read the first client’s update.
To show this, the tutorial uses the record’s generation, which is the record’s version. The generation is record metadata, and the database increments it on each successful update of the record.
The record has an integer bin named gen_times_2. It holds 2 times the generation that the record has after an update. A client first reads the record’s generation, and then writes a new value that is 2 times the generation the record has after the write.
With a single client, the bin satisfies this rule after every read-modify-write. With several clients, a client can write a value that it computed from an older generation, so the bin no longer holds 2 times the current generation. That’s a lost write.
The tutorial detects lost writes by checking the relationship between the new generation and the value in the bin. Then it shows how a conditional write with a generation check solves the problem.
Import the client modules
import java.util.ArrayList;import java.util.List;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import java.util.concurrent.locks.LockSupport;import java.util.function.Supplier;
import com.aerospike.client.sdk.AerospikeException;import com.aerospike.client.sdk.Cluster;import com.aerospike.client.sdk.ClusterDefinition;import com.aerospike.client.sdk.DataSet;import com.aerospike.client.sdk.Key;import com.aerospike.client.sdk.Record;import com.aerospike.client.sdk.Session;import com.aerospike.client.sdk.policy.Behavior;import randomimport threadingimport time
from aerospike_sdk import Behavior, DataSet, GenerationErrorfrom aerospike_sdk.sync import ClusterDefinitionuse std::future::Future;use std::time::{Duration, SystemTime, UNIX_EPOCH};
use aerospike::operations::scalar;use aerospike::{as_bin, as_key, Bins, Client, ClientPolicy, GenerationPolicy, Key, ReadPolicy, ResultCode, WritePolicy};use tokio::time::sleep;using System;using System.Threading;using Aerospike.Client;import ( "fmt" "log" "math/rand" "sync" "time"
"github.com/aerospike/aerospike-client-go/v8" "github.com/aerospike/aerospike-client-go/v8/types")import Aerospike from 'aerospike';
const { operations: op } = Aerospike;#include <pthread.h>#include <stdbool.h>#include <stdint.h>#include <stdio.h>#include <stdlib.h>#include <time.h>#include <aerospike/aerospike.h>#include <aerospike/aerospike_key.h>#include <aerospike/as_error.h>#include <aerospike/as_key.h>#include <aerospike/as_operations.h>#include <aerospike/as_policy.h>#include <aerospike/as_record.h>import java.util.ArrayList;import java.util.List;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import java.util.concurrent.locks.LockSupport;import java.util.function.Supplier;
import com.aerospike.client.AerospikeClient;import com.aerospike.client.AerospikeException;import com.aerospike.client.Bin;import com.aerospike.client.Key;import com.aerospike.client.Operation;import com.aerospike.client.Record;import com.aerospike.client.ResultCode;import com.aerospike.client.policy.GenerationPolicy;import com.aerospike.client.policy.WritePolicy;import randomimport threadingimport time
import aerospikefrom aerospike import exception as exfrom aerospike_helpers.operations import operationsConnect to the database
Connect to the database on port 3000 of your computer. This is the default port. If your database runs on another host or port, change the address.
Cluster cluster = new ClusterDefinition("127.0.0.1", 3000).connect();Session session = cluster.createSession(Behavior.DEFAULT);cluster = ClusterDefinition("127.0.0.1", 3000).connect()session = cluster.create_session(Behavior.DEFAULT)let client = Client::new(&ClientPolicy::default(), &"127.0.0.1:3000").await?;AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);client, err := aerospike.NewClient("127.0.0.1", 3000)if err != nil { log.Fatal(err)}const client = await Aerospike.connect({ hosts: '127.0.0.1:3000' });as_config config;as_config_init(&config);as_config_add_host(&config, "127.0.0.1", 3000);
aerospike as;aerospike_init(&as, &config);
as_error err;if (aerospike_connect(&as, &err) != AEROSPIKE_OK) { fprintf(stderr, "Error connecting: %s\n", err.message); return 1;}AerospikeClient client = new AerospikeClient("127.0.0.1", 3000);config = {"hosts": [("127.0.0.1", 3000)]}client = aerospike.client(config).connect()Write the test record
Write one record with an integer bin named gen_times_2, set to 1. The record is new, so its generation is 1.
Key key = DataSet.of("test", "rmw").id("id-1");
session.upsert(key).bin("gen_times_2").setTo(1).execute();key = DataSet.of("test", "rmw").id("id-1")
session.upsert(key).put({"gen_times_2": 1}).execute()let key = as_key!("test", "rmw", "id-1");
client.put(&WritePolicy::default(), &key, &[as_bin!("gen_times_2", 1)]).await?;Key key = new Key("test", "rmw", "id-1");
client.Put(null, key, new Bin("gen_times_2", 1));key, err := aerospike.NewKey("test", "rmw", "id-1")if err != nil { log.Fatal(err)}
err = client.PutBins(nil, key, aerospike.NewBin("gen_times_2", 1))if err != nil { log.Fatal(err)}const key = new Aerospike.Key('test', 'rmw', 'id-1');
await client.put(key, { gen_times_2: 1 });as_key key;as_key_init_str(&key, "test", "rmw", "id-1");
as_record rec;as_record_inita(&rec, 1);as_record_set_int64(&rec, "gen_times_2", 1);if (aerospike_key_put(&as, &err, NULL, &key, &rec) != AEROSPIKE_OK) { fprintf(stderr, "Error writing record: %s\n", err.message); as_record_destroy(&rec); return 1;}as_record_destroy(&rec);Key key = new Key("test", "rmw", "id-1");
client.put(null, key, new Bin("gen_times_2", 1));key = ("test", "rmw", "id-1")
client.put(key, {"gen_times_2": 1})Run workers concurrently
To simulate concurrent clients, the test starts several workers. Each worker runs the read-modify-write function 10 times in a row. A function returns true if its write was lost, and false if it wasn’t. The test reports whether any worker lost a write.
The functions in this tutorial return a value, instead of printing, so they run on any thread.
static final int TXNS_PER_WORKER = 10;
static void sleepMs(long ms) { LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(ms));}
static void runTest(int workers, Supplier<Boolean> rmw) { ExecutorService pool = Executors.newFixedThreadPool(workers); List<CompletableFuture<Boolean>> futures = new ArrayList<>(); for (int w = 0; w < workers; w++) { futures.add(CompletableFuture.supplyAsync(() -> { boolean lost = false; for (int i = 0; i < TXNS_PER_WORKER; i++) { lost |= rmw.get(); } return lost; }, pool)); } boolean lostWrites = false; for (CompletableFuture<Boolean> future : futures) { lostWrites |= future.join(); } pool.shutdown(); System.out.println("Workers: " + workers + ", lost writes: " + (lostWrites ? "yes" : "no"));}TXNS_PER_WORKER = 10
def run_test(workers, rmw_fn): results = []
def work(): lost = False for _ in range(TXNS_PER_WORKER): lost |= rmw_fn() results.append(lost)
threads = [threading.Thread(target=work) for _ in range(workers)] for thread in threads: thread.start() for thread in threads: thread.join() print(f"Workers: {workers}, lost writes: {'yes' if any(results) else 'no'}")const TXNS_PER_WORKER: usize = 10;
fn random_ms(max: u64) -> u64 { let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().subsec_nanos(); u64::from(nanos / 1000) % max}
async fn run_test<F, Fut>(workers: usize, client: &Client, key: &Key, rmw: F)where F: Fn(Client, Key) -> Fut + Copy + Send + 'static, Fut: Future<Output = bool> + Send,{ let mut handles = Vec::new(); for _ in 0..workers { let (client, key) = (client.clone(), key.clone()); handles.push(tokio::spawn(async move { let mut lost = false; for _ in 0..TXNS_PER_WORKER { lost |= rmw(client.clone(), key.clone()).await; } lost })); } let mut lost_writes = false; for handle in handles { lost_writes |= handle.await.unwrap(); } println!("Workers: {}, lost writes: {}", workers, if lost_writes { "yes" } else { "no" });}The jitter helper takes the microseconds from the clock, so the example needs no random-number crate.
const int TxnsPerWorker = 10;
static void RunTest(int workers, Func<bool> rmw){ bool[] lost = new bool[workers]; Thread[] threads = new Thread[workers]; for (int w = 0; w < workers; w++) { int worker = w; threads[w] = new Thread(() => { for (int i = 0; i < TxnsPerWorker; i++) { if (rmw()) { lost[worker] = true; } } }); threads[w].Start(); } foreach (Thread thread in threads) { thread.Join(); } Console.WriteLine("Workers: " + workers + ", lost writes: " + (Array.IndexOf(lost, true) >= 0 ? "yes" : "no"));}const txnsPerWorker = 10
func runTest(workers int, rmw func() bool) { lost := make([]bool, workers) var wg sync.WaitGroup for w := 0; w < workers; w++ { wg.Add(1) go func(w int) { defer wg.Done() for i := 0; i < txnsPerWorker; i++ { if rmw() { lost[w] = true } } }(w) } wg.Wait()
lostWrites := "no" for _, l := range lost { if l { lostWrites = "yes" } } fmt.Printf("Workers: %d, lost writes: %s\n", workers, lostWrites)}const TXNS_PER_WORKER = 10;const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
async function runTest(workers, rmw) { const results = await Promise.all(Array.from({ length: workers }, async () => { let lost = false; for (let i = 0; i < TXNS_PER_WORKER; i++) { lost = (await rmw()) || lost; } return lost; })); console.log(`Workers: ${workers}, lost writes: ${results.some(Boolean) ? 'yes' : 'no'}`);}Node.js runs JavaScript on one thread. The workers are asynchronous tasks, and their reads and writes interleave while they wait for the database.
static const int TXNS_PER_WORKER = 10;#define MAX_WORKERS 8
typedef bool (*rmw_fn)(aerospike*, const as_key*);
typedef struct { aerospike* as; const as_key* key; rmw_fn rmw; bool lost;} worker_t;
static void sleep_ms(long ms){ struct timespec ts = {ms / 1000, (ms % 1000) * 1000000L}; nanosleep(&ts, NULL);}
static long random_ms(long max){ struct timespec now; clock_gettime(CLOCK_REALTIME, &now); return (now.tv_nsec / 1000) % max;}
static void* work(void* arg){ worker_t* worker = arg; for (int i = 0; i < TXNS_PER_WORKER; i++) { if (worker->rmw(worker->as, worker->key)) { worker->lost = true; } } return NULL;}
static void run_test(int workers, aerospike* as, const as_key* key, rmw_fn rmw){ pthread_t threads[MAX_WORKERS]; worker_t state[MAX_WORKERS]; if (workers > MAX_WORKERS) { fprintf(stderr, "At most %d workers are supported\n", MAX_WORKERS); exit(1); } for (int w = 0; w < workers; w++) { state[w] = (worker_t){as, key, rmw, false}; if (pthread_create(&threads[w], NULL, work, &state[w]) != 0) { fprintf(stderr, "Error starting a worker\n"); exit(1); } } bool lost_writes = false; for (int w = 0; w < workers; w++) { pthread_join(threads[w], NULL); lost_writes |= state[w].lost; } printf("Workers: %d, lost writes: %s\n", workers, lost_writes ? "yes" : "no");}The jitter helper takes the microseconds from the clock. The sleep and jitter helpers are also used by the functions in the next steps. The test allows up to eight workers.
static final int TXNS_PER_WORKER = 10;
static void sleepMs(long ms) { LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(ms));}
static void runTest(int workers, Supplier<Boolean> rmw) { ExecutorService pool = Executors.newFixedThreadPool(workers); List<CompletableFuture<Boolean>> futures = new ArrayList<>(); for (int w = 0; w < workers; w++) { futures.add(CompletableFuture.supplyAsync(() -> { boolean lost = false; for (int i = 0; i < TXNS_PER_WORKER; i++) { lost |= rmw.get(); } return lost; }, pool)); } boolean lostWrites = false; for (CompletableFuture<Boolean> future : futures) { lostWrites |= future.join(); } pool.shutdown(); System.out.println("Workers: " + workers + ", lost writes: " + (lostWrites ? "yes" : "no"));}TXNS_PER_WORKER = 10
def run_test(workers, rmw_fn): results = []
def work(): lost = False for _ in range(TXNS_PER_WORKER): lost |= rmw_fn() results.append(lost)
threads = [threading.Thread(target=work) for _ in range(workers)] for thread in threads: thread.start() for thread in threads: thread.join() print(f"Workers: {workers}, lost writes: {'yes' if any(results) else 'no'}")Read and write without a check
The first function is a simple read-modify-write. It does the following:
- Reads the record, which returns the record’s generation.
- Waits up to 100 milliseconds at random to simulate time an application spends computing a new value.
- Computes the new value of
gen_times_2, which is 2 times the generation the write creates: 2 times the sum of the generation the function read and 1. - Writes the new value, and gets the record’s new generation in the same command.
- Returns
trueif the new value divided by 2 isn’t the new generation. That means another client wrote the record between the read and the write, and the first write was lost. Otherwise, it returnsfalse.
static boolean rmwSimple(Session session, Key key) { Record record = session.query(key).execute().getFirstRecord(); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); Record written = session.upsert(key).bin("gen_times_2").setTo(newValue).execute().getFirstRecord(); return newValue / 2 != written.generation;}def rmw_simple(session, key): record = session.query(key).execute().first_or_raise().record_or_raise() read_gen = record.generation
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) written = session.upsert(key).bin("gen_times_2").set_to(new_value).execute().first_or_raise().record_or_raise() return new_value // 2 != written.generationA write that has no read operation still returns the record’s generation.
async fn rmw_simple(client: Client, key: Key) -> bool { let record = client.get(&ReadPolicy::default(), &key, Bins::All).await.expect("read failed");
sleep(Duration::from_millis(random_ms(100))).await;
let new_value = 2 * (record.generation + 1); let ops = [scalar::put(&as_bin!("gen_times_2", i64::from(new_value)))]; let written = client.operate(&WritePolicy::default(), &key, &ops).await.expect("write failed"); new_value / 2 != written.generation}A write that has no read operation still returns the record’s generation. The function starts with the generation as an unsigned integer, so new_value is also unsigned.
static bool RmwSimple(AerospikeClient client, Key key){ Record record = client.Get(null, key); int readGen = record.generation;
Thread.Sleep(Random.Shared.Next(0, 100));
int newValue = 2 * (readGen + 1); Record written = client.Operate(null, key, Operation.Put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation;}func rmwSimple(client *aerospike.Client, key *aerospike.Key) bool { record, err := client.Get(nil, key) if err != nil { log.Fatal(err) }
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
newValue := 2 * (int(record.Generation) + 1) written, err := client.Operate(nil, key, aerospike.PutOp(aerospike.NewBin("gen_times_2", newValue))) if err != nil { log.Fatal(err) } return newValue/2 != int(written.Generation)}async function rmwSimple(client, key) { const record = await client.get(key);
await sleep(Math.random() * 100);
const newValue = 2 * (record.gen + 1); const written = await client.operate(key, [op.write('gen_times_2', newValue)]); return newValue / 2 !== written.gen;}static bool rmw_simple(aerospike* as, const as_key* key){ as_error err; as_record* record = NULL; if (aerospike_key_get(as, &err, NULL, key, &record) != AEROSPIKE_OK) { fprintf(stderr, "Error reading record: %s\n", err.message); exit(1); } int read_gen = record->gen; as_record_destroy(record);
sleep_ms(random_ms(100));
int64_t new_value = 2 * (read_gen + 1); as_operations ops; as_operations_inita(&ops, 1); as_operations_add_write_int64(&ops, "gen_times_2", new_value); as_record* written = NULL; as_status status = aerospike_key_operate(as, &err, NULL, key, &ops, &written); as_operations_destroy(&ops); if (status != AEROSPIKE_OK) { fprintf(stderr, "Error writing record: %s\n", err.message); exit(1); } bool lost = new_value / 2 != written->gen; as_record_destroy(written); return lost;}The write command returns the record’s metadata, including the generation, even though it has no read operation.
static boolean rmwSimple(AerospikeClient client, Key key) { Record record = client.get(null, key); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); Record written = client.operate(null, key, Operation.put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation;}def rmw_simple(client, key): (_, meta, _) = client.get(key)
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (meta["gen"] + 1) (_, meta, _) = client.operate(key, [operations.write("gen_times_2", new_value)]) return new_value // 2 != meta["gen"]The operate function returns the key, the metadata, and the bins. The metadata has the new generation.
Run the test without a check
With greater concurrency, a larger share of the read-write transactions are lost. With one worker, none are lost. With two or more workers, the transactions interleave, and the value in the gen_times_2 bin doesn’t stay at 2 times the generation. The test prints yes when any worker lost a write.
for (int workers = 1; workers <= 4; workers++) { runTest(workers, () -> rmwSimple(session, key));}for workers in range(1, 5): run_test(workers, lambda: rmw_simple(session, key))for workers in 1..=4 { run_test(workers, &client, &key, rmw_simple).await;}for (int workers = 1; workers <= 4; workers++){ RunTest(workers, () => RmwSimple(client, key));}for workers := 1; workers <= 4; workers++ { runTest(workers, func() bool { return rmwSimple(client, key) })}for (let workers = 1; workers <= 4; workers++) { await runTest(workers, () => rmwSimple(client, key));}for (int workers = 1; workers <= 4; workers++) { run_test(workers, &as, &key, rmw_simple);}for (int workers = 1; workers <= 4; workers++) { runTest(workers, () -> rmwSimple(client, key));}for workers in range(1, 5): run_test(workers, lambda: rmw_simple(client, key))Workers: 1, lost writes: noWorkers: 2, lost writes: yesWorkers: 3, lost writes: yesWorkers: 4, lost writes: yesLost writes depend on timing. With two or more workers, you almost always see yes. If a row shows no, run the program again.
Read and write with a generation check
To prevent lost writes, make the write conditional on the record not having changed since the read. This is a check-and-set operation. The write succeeds only if the record’s generation is still the generation the function read. Otherwise, the write fails with a generation error, and the client must retry the whole read-modify-write.
The function retries a failed write up to three times. Before each retry, it uses exponential backoff: 40, 80, and then 160 milliseconds.
If the write still fails after the retries, the function gives up. If the function gives up on a write, that write is not applied, so it is not a lost write. The function returns false in that case. No write is overwritten or lost, and gen_times_2 always stays at 2 times the generation.
static boolean rmwWithGenCheck(Session session, Key key) { for (int retry = 0; retry <= 3; retry++) { Record record = session.query(key).execute().getFirstRecord(); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); try { Record written = session.upsert(key).bin("gen_times_2").setTo(newValue) .ensureGenerationIs(readGen).execute().getFirstRecord(); return newValue / 2 != written.generation; } catch (AerospikeException.GenerationException e) { if (retry < 3) sleepMs((1L << (retry + 1)) * 20); } } return false;}The ensureGenerationIs method makes the write conditional on the generation. A mismatch throws AerospikeException.GenerationException.
def rmw_with_gen_check(session, key): for retry in range(4): record = session.query(key).execute().first_or_raise().record_or_raise() read_gen = record.generation
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) try: written = ( session.upsert(key).bin("gen_times_2").set_to(new_value) .ensure_generation_is(read_gen).execute().first_or_raise().record_or_raise() ) return new_value // 2 != written.generation except GenerationError: if retry < 3: time.sleep((2 ** (retry + 1)) * 0.02) return FalseThe ensure_generation_is method makes the write conditional on the generation. A mismatch raises GenerationError.
async fn rmw_with_gen_check(client: Client, key: Key) -> bool { for retry in 0..=3u64 { let record = client.get(&ReadPolicy::default(), &key, Bins::All).await.expect("read failed");
sleep(Duration::from_millis(random_ms(100))).await;
let new_value = 2 * (record.generation + 1); let mut policy = WritePolicy::default(); policy.generation_policy = GenerationPolicy::ExpectGenEqual; policy.generation = record.generation; let ops = [scalar::put(&as_bin!("gen_times_2", i64::from(new_value)))]; match client.operate(&policy, &key, &ops).await { Ok(written) => return new_value / 2 != written.generation, Err(e) if e.server_result_code() == Some(ResultCode::GenerationError) => { if retry < 3 { sleep(Duration::from_millis((1 << (retry + 1)) * 20)).await; } } Err(e) => panic!("write failed: {e}"), } } false}The write policy has a generation policy and an expected generation. A mismatch returns a server error with the result code GenerationError.
static bool RmwWithGenCheck(AerospikeClient client, Key key){ for (int retry = 0; retry <= 3; retry++) { Record record = client.Get(null, key); int readGen = record.generation;
Thread.Sleep(Random.Shared.Next(0, 100));
int newValue = 2 * (readGen + 1); WritePolicy policy = new WritePolicy { generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL, generation = readGen }; try { Record written = client.Operate(policy, key, Operation.Put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; } catch (AerospikeException e) when (e.Result == ResultCode.GENERATION_ERROR) { if (retry < 3) Thread.Sleep((1 << (retry + 1)) * 20); } } return false;}The write policy has a generation policy and an expected generation. A mismatch throws an AerospikeException with the result code GENERATION_ERROR.
func rmwWithGenCheck(client *aerospike.Client, key *aerospike.Key) bool { for retry := 0; retry <= 3; retry++ { record, err := client.Get(nil, key) if err != nil { log.Fatal(err) }
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
newValue := 2 * (int(record.Generation) + 1) policy := aerospike.NewWritePolicy(0, 0) policy.GenerationPolicy = aerospike.EXPECT_GEN_EQUAL policy.Generation = record.Generation written, err := client.Operate(policy, key, aerospike.PutOp(aerospike.NewBin("gen_times_2", newValue))) if err == nil { return newValue/2 != int(written.Generation) } if !err.Matches(types.GENERATION_ERROR) { log.Fatal(err) } if retry < 3 { time.Sleep(time.Duration((1<<(retry+1))*20) * time.Millisecond) } } return false}The write policy has a generation policy and an expected generation. A mismatch returns an error that matches types.GENERATION_ERROR.
async function rmwWithGenCheck(client, key) { for (let retry = 0; retry <= 3; retry++) { const record = await client.get(key);
await sleep(Math.random() * 100);
const newValue = 2 * (record.gen + 1); try { const written = await client.operate(key, [op.write('gen_times_2', newValue)], { gen: record.gen }, { gen: Aerospike.policy.gen.EQ }); return newValue / 2 !== written.gen; } catch (error) { if (error.code !== Aerospike.status.AEROSPIKE_ERR_RECORD_GENERATION) { throw error; } if (retry < 3) await sleep(2 ** (retry + 1) * 20); } } return false;}The operate function takes the expected generation as metadata and a generation policy in the write policy. A mismatch rejects with the status AEROSPIKE_ERR_RECORD_GENERATION.
static bool rmw_with_gen_check(aerospike* as, const as_key* key){ for (int retry = 0; retry <= 3; retry++) { as_error err; as_record* record = NULL; if (aerospike_key_get(as, &err, NULL, key, &record) != AEROSPIKE_OK) { fprintf(stderr, "Error reading record: %s\n", err.message); exit(1); } int read_gen = record->gen; as_record_destroy(record);
sleep_ms(random_ms(100));
int64_t new_value = 2 * (read_gen + 1); as_operations ops; as_operations_inita(&ops, 1); as_operations_add_write_int64(&ops, "gen_times_2", new_value); ops.gen = read_gen;
as_policy_operate policy; as_policy_operate_init(&policy); policy.gen = AS_POLICY_GEN_EQ;
as_record* written = NULL; as_status status = aerospike_key_operate(as, &err, &policy, key, &ops, &written); as_operations_destroy(&ops); if (status == AEROSPIKE_OK) { bool lost = new_value / 2 != written->gen; as_record_destroy(written); return lost; } if (status != AEROSPIKE_ERR_RECORD_GENERATION) { fprintf(stderr, "Error writing record: %s\n", err.message); exit(1); } if (retry < 3) sleep_ms((1L << (retry + 1)) * 20); } return false;}The operations hold the expected generation, and the policy compares it. A mismatch returns AEROSPIKE_ERR_RECORD_GENERATION.
static boolean rmwWithGenCheck(AerospikeClient client, Key key) { for (int retry = 0; retry <= 3; retry++) { Record record = client.get(null, key); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); WritePolicy policy = new WritePolicy(); policy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL; policy.generation = readGen; try { Record written = client.operate(policy, key, Operation.put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; } catch (AerospikeException e) { if (e.getResultCode() != ResultCode.GENERATION_ERROR) { throw e; } if (retry < 3) sleepMs((1L << (retry + 1)) * 20); } } return false;}The write policy has a generation policy and an expected generation. A mismatch throws an AerospikeException with the result code ResultCode.GENERATION_ERROR.
def rmw_with_gen_check(client, key): for retry in range(4): (_, meta, _) = client.get(key) read_gen = meta["gen"]
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) try: (_, meta, _) = client.operate( key, [operations.write("gen_times_2", new_value)], meta={"gen": read_gen}, policy={"gen": aerospike.POLICY_GEN_EQ}, ) return new_value // 2 != meta["gen"] except ex.RecordGenerationError: if retry < 3: time.sleep((2 ** (retry + 1)) * 0.02) return FalseThe operate function takes the expected generation in its meta argument and a generation policy in its policy argument. A mismatch raises RecordGenerationError.
Run the test with the generation check
Run the same test with the generation check. No worker loses a write. Under heavy concurrency, a worker can still run out of retries and give up on a write, but giving up never corrupts the record.
for (int workers = 2; workers <= 4; workers++) { runTest(workers, () -> rmwWithGenCheck(session, key));}for workers in range(2, 5): run_test(workers, lambda: rmw_with_gen_check(session, key))for workers in 2..=4 { run_test(workers, &client, &key, rmw_with_gen_check).await;}for (int workers = 2; workers <= 4; workers++){ RunTest(workers, () => RmwWithGenCheck(client, key));}for workers := 2; workers <= 4; workers++ { runTest(workers, func() bool { return rmwWithGenCheck(client, key) })}for (let workers = 2; workers <= 4; workers++) { await runTest(workers, () => rmwWithGenCheck(client, key));}for (int workers = 2; workers <= 4; workers++) { run_test(workers, &as, &key, rmw_with_gen_check);}for (int workers = 2; workers <= 4; workers++) { runTest(workers, () -> rmwWithGenCheck(client, key));}for workers in range(2, 5): run_test(workers, lambda: rmw_with_gen_check(client, key))Workers: 2, lost writes: noWorkers: 3, lost writes: noWorkers: 4, lost writes: noDelete the record and close the connection
Delete the test record, then close the client.
session.delete(key).execute();cluster.close();session.delete(key).execute()cluster.close()client.delete(&WritePolicy::default(), &key).await?;client.close().await?;client.Delete(null, key);client.Close();_, err = client.Delete(nil, key)if err != nil { log.Fatal(err)}client.Close()await client.remove(key);client.close();if (aerospike_key_remove(&as, &err, NULL, &key) != AEROSPIKE_OK) { fprintf(stderr, "Error deleting record: %s\n", err.message); return 1;}
as_key_destroy(&key);aerospike_close(&as, &err);aerospike_destroy(&as);client.delete(null, key);client.close();client.remove(key)client.close()Complete program
This program combines the steps. It writes the test record, runs four workers without a check and then with the generation check, and deletes the record.
import java.util.ArrayList;import java.util.List;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import java.util.concurrent.locks.LockSupport;import java.util.function.Supplier;
import com.aerospike.client.sdk.AerospikeException;import com.aerospike.client.sdk.Cluster;import com.aerospike.client.sdk.ClusterDefinition;import com.aerospike.client.sdk.DataSet;import com.aerospike.client.sdk.Key;import com.aerospike.client.sdk.Record;import com.aerospike.client.sdk.Session;import com.aerospike.client.sdk.policy.Behavior;
public class ReadModifyWrite { static final int TXNS_PER_WORKER = 10;
static void sleepMs(long ms) { LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(ms)); }
static void runTest(int workers, Supplier<Boolean> rmw) { ExecutorService pool = Executors.newFixedThreadPool(workers); List<CompletableFuture<Boolean>> futures = new ArrayList<>(); for (int w = 0; w < workers; w++) { futures.add(CompletableFuture.supplyAsync(() -> { boolean lost = false; for (int i = 0; i < TXNS_PER_WORKER; i++) { lost |= rmw.get(); } return lost; }, pool)); } boolean lostWrites = false; for (CompletableFuture<Boolean> future : futures) { lostWrites |= future.join(); } pool.shutdown(); System.out.println("Workers: " + workers + ", lost writes: " + (lostWrites ? "yes" : "no")); }
static boolean rmwSimple(Session session, Key key) { Record record = session.query(key).execute().getFirstRecord(); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); Record written = session.upsert(key).bin("gen_times_2").setTo(newValue).execute().getFirstRecord(); return newValue / 2 != written.generation; }
static boolean rmwWithGenCheck(Session session, Key key) { for (int retry = 0; retry <= 3; retry++) { Record record = session.query(key).execute().getFirstRecord(); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); try { Record written = session.upsert(key).bin("gen_times_2").setTo(newValue) .ensureGenerationIs(readGen).execute().getFirstRecord(); return newValue / 2 != written.generation; } catch (AerospikeException.GenerationException e) { if (retry < 3) sleepMs((1L << (retry + 1)) * 20); } } return false; }
public static void main(String[] args) { try (Cluster cluster = new ClusterDefinition("127.0.0.1", 3000).connect()) { Session session = cluster.createSession(Behavior.DEFAULT); Key key = DataSet.of("test", "rmw").id("id-1");
session.upsert(key).bin("gen_times_2").setTo(1).execute();
runTest(4, () -> rmwSimple(session, key)); runTest(4, () -> rmwWithGenCheck(session, key));
session.delete(key).execute(); } }}import randomimport sysimport threadingimport time
from aerospike_sdk import Behavior, DataSet, GenerationErrorfrom aerospike_sdk.exceptions import AerospikeErrorfrom aerospike_sdk.sync import ClusterDefinition
TXNS_PER_WORKER = 10
def run_test(workers, rmw_fn): results = []
def work(): lost = False for _ in range(TXNS_PER_WORKER): lost |= rmw_fn() results.append(lost)
threads = [threading.Thread(target=work) for _ in range(workers)] for thread in threads: thread.start() for thread in threads: thread.join() print(f"Workers: {workers}, lost writes: {'yes' if any(results) else 'no'}")
def rmw_simple(session, key): record = session.query(key).execute().first_or_raise().record_or_raise() read_gen = record.generation
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) written = session.upsert(key).bin("gen_times_2").set_to(new_value).execute().first_or_raise().record_or_raise() return new_value // 2 != written.generation
def rmw_with_gen_check(session, key): for retry in range(4): record = session.query(key).execute().first_or_raise().record_or_raise() read_gen = record.generation
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) try: written = ( session.upsert(key).bin("gen_times_2").set_to(new_value) .ensure_generation_is(read_gen).execute().first_or_raise().record_or_raise() ) return new_value // 2 != written.generation except GenerationError: if retry < 3: time.sleep((2 ** (retry + 1)) * 0.02) return False
cluster = ClusterDefinition("127.0.0.1", 3000).connect()try: session = cluster.create_session(Behavior.DEFAULT) key = DataSet.of("test", "rmw").id("id-1")
session.upsert(key).put({"gen_times_2": 1}).execute()
run_test(4, lambda: rmw_simple(session, key)) run_test(4, lambda: rmw_with_gen_check(session, key))
session.delete(key).execute()except AerospikeError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1)finally: cluster.close()use std::future::Future;use std::time::{Duration, SystemTime, UNIX_EPOCH};
use aerospike::operations::scalar;use aerospike::{as_bin, as_key, Bins, Client, ClientPolicy, GenerationPolicy, Key, ReadPolicy, ResultCode, WritePolicy};use tokio::time::sleep;
const TXNS_PER_WORKER: usize = 10;
fn random_ms(max: u64) -> u64 { let nanos = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().subsec_nanos(); u64::from(nanos / 1000) % max}
async fn run_test<F, Fut>(workers: usize, client: &Client, key: &Key, rmw: F)where F: Fn(Client, Key) -> Fut + Copy + Send + 'static, Fut: Future<Output = bool> + Send,{ let mut handles = Vec::new(); for _ in 0..workers { let (client, key) = (client.clone(), key.clone()); handles.push(tokio::spawn(async move { let mut lost = false; for _ in 0..TXNS_PER_WORKER { lost |= rmw(client.clone(), key.clone()).await; } lost })); } let mut lost_writes = false; for handle in handles { lost_writes |= handle.await.unwrap(); } println!("Workers: {}, lost writes: {}", workers, if lost_writes { "yes" } else { "no" });}
async fn rmw_simple(client: Client, key: Key) -> bool { let record = client.get(&ReadPolicy::default(), &key, Bins::All).await.expect("read failed");
sleep(Duration::from_millis(random_ms(100))).await;
let new_value = 2 * (record.generation + 1); let ops = [scalar::put(&as_bin!("gen_times_2", i64::from(new_value)))]; let written = client.operate(&WritePolicy::default(), &key, &ops).await.expect("write failed"); new_value / 2 != written.generation}
async fn rmw_with_gen_check(client: Client, key: Key) -> bool { for retry in 0..=3u64 { let record = client.get(&ReadPolicy::default(), &key, Bins::All).await.expect("read failed");
sleep(Duration::from_millis(random_ms(100))).await;
let new_value = 2 * (record.generation + 1); let mut policy = WritePolicy::default(); policy.generation_policy = GenerationPolicy::ExpectGenEqual; policy.generation = record.generation; let ops = [scalar::put(&as_bin!("gen_times_2", i64::from(new_value)))]; match client.operate(&policy, &key, &ops).await { Ok(written) => return new_value / 2 != written.generation, Err(e) if e.server_result_code() == Some(ResultCode::GenerationError) => { if retry < 3 { sleep(Duration::from_millis((1 << (retry + 1)) * 20)).await; } } Err(e) => panic!("write failed: {e}"), } } false}
#[tokio::main]async fn main() -> Result<(), Box<dyn std::error::Error>> { let client = Client::new(&ClientPolicy::default(), &"127.0.0.1:3000").await?; let key = as_key!("test", "rmw", "id-1");
client.put(&WritePolicy::default(), &key, &[as_bin!("gen_times_2", 1)]).await?;
run_test(4, &client, &key, rmw_simple).await; run_test(4, &client, &key, rmw_with_gen_check).await;
client.delete(&WritePolicy::default(), &key).await?; client.close().await?; Ok(())}using System;using System.Threading;using Aerospike.Client;
public class ReadModifyWrite{ const int TxnsPerWorker = 10;
static void RunTest(int workers, Func<bool> rmw) { bool[] lost = new bool[workers]; Thread[] threads = new Thread[workers]; for (int w = 0; w < workers; w++) { int worker = w; threads[w] = new Thread(() => { for (int i = 0; i < TxnsPerWorker; i++) { if (rmw()) { lost[worker] = true; } } }); threads[w].Start(); } foreach (Thread thread in threads) { thread.Join(); } Console.WriteLine("Workers: " + workers + ", lost writes: " + (Array.IndexOf(lost, true) >= 0 ? "yes" : "no")); }
static bool RmwSimple(AerospikeClient client, Key key) { Record record = client.Get(null, key); int readGen = record.generation;
Thread.Sleep(Random.Shared.Next(0, 100));
int newValue = 2 * (readGen + 1); Record written = client.Operate(null, key, Operation.Put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; }
static bool RmwWithGenCheck(AerospikeClient client, Key key) { for (int retry = 0; retry <= 3; retry++) { Record record = client.Get(null, key); int readGen = record.generation;
Thread.Sleep(Random.Shared.Next(0, 100));
int newValue = 2 * (readGen + 1); WritePolicy policy = new WritePolicy { generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL, generation = readGen }; try { Record written = client.Operate(policy, key, Operation.Put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; } catch (AerospikeException e) when (e.Result == ResultCode.GENERATION_ERROR) { if (retry < 3) Thread.Sleep((1 << (retry + 1)) * 20); } } return false; }
public static void Main() { AerospikeClient client = new AerospikeClient("127.0.0.1", 3000); try { Key key = new Key("test", "rmw", "id-1");
client.Put(null, key, new Bin("gen_times_2", 1));
RunTest(4, () => RmwSimple(client, key)); RunTest(4, () => RmwWithGenCheck(client, key));
client.Delete(null, key); } finally { client.Close(); } }}package main
import ( "fmt" "log" "math/rand" "sync" "time"
"github.com/aerospike/aerospike-client-go/v8" "github.com/aerospike/aerospike-client-go/v8/types")
const txnsPerWorker = 10
func runTest(workers int, rmw func() bool) { lost := make([]bool, workers) var wg sync.WaitGroup for w := 0; w < workers; w++ { wg.Add(1) go func(w int) { defer wg.Done() for i := 0; i < txnsPerWorker; i++ { if rmw() { lost[w] = true } } }(w) } wg.Wait()
lostWrites := "no" for _, l := range lost { if l { lostWrites = "yes" } } fmt.Printf("Workers: %d, lost writes: %s\n", workers, lostWrites)}
func rmwSimple(client *aerospike.Client, key *aerospike.Key) bool { record, err := client.Get(nil, key) if err != nil { log.Fatal(err) }
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
newValue := 2 * (int(record.Generation) + 1) written, err := client.Operate(nil, key, aerospike.PutOp(aerospike.NewBin("gen_times_2", newValue))) if err != nil { log.Fatal(err) } return newValue/2 != int(written.Generation)}
func rmwWithGenCheck(client *aerospike.Client, key *aerospike.Key) bool { for retry := 0; retry <= 3; retry++ { record, err := client.Get(nil, key) if err != nil { log.Fatal(err) }
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
newValue := 2 * (int(record.Generation) + 1) policy := aerospike.NewWritePolicy(0, 0) policy.GenerationPolicy = aerospike.EXPECT_GEN_EQUAL policy.Generation = record.Generation written, err := client.Operate(policy, key, aerospike.PutOp(aerospike.NewBin("gen_times_2", newValue))) if err == nil { return newValue/2 != int(written.Generation) } if !err.Matches(types.GENERATION_ERROR) { log.Fatal(err) } if retry < 3 { time.Sleep(time.Duration((1<<(retry+1))*20) * time.Millisecond) } } return false}
func main() { client, err := aerospike.NewClient("127.0.0.1", 3000) if err != nil { log.Fatal(err) } defer client.Close()
key, err := aerospike.NewKey("test", "rmw", "id-1") if err != nil { log.Fatal(err) }
err = client.PutBins(nil, key, aerospike.NewBin("gen_times_2", 1)) if err != nil { log.Fatal(err) }
runTest(4, func() bool { return rmwSimple(client, key) }) runTest(4, func() bool { return rmwWithGenCheck(client, key) })
if _, err := client.Delete(nil, key); err != nil { log.Fatal(err) }}import Aerospike from 'aerospike';
const { operations: op } = Aerospike;
const TXNS_PER_WORKER = 10;const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
async function runTest(workers, rmw) { const results = await Promise.all(Array.from({ length: workers }, async () => { let lost = false; for (let i = 0; i < TXNS_PER_WORKER; i++) { lost = (await rmw()) || lost; } return lost; })); console.log(`Workers: ${workers}, lost writes: ${results.some(Boolean) ? 'yes' : 'no'}`);}
async function rmwSimple(client, key) { const record = await client.get(key);
await sleep(Math.random() * 100);
const newValue = 2 * (record.gen + 1); const written = await client.operate(key, [op.write('gen_times_2', newValue)]); return newValue / 2 !== written.gen;}
async function rmwWithGenCheck(client, key) { for (let retry = 0; retry <= 3; retry++) { const record = await client.get(key);
await sleep(Math.random() * 100);
const newValue = 2 * (record.gen + 1); try { const written = await client.operate(key, [op.write('gen_times_2', newValue)], { gen: record.gen }, { gen: Aerospike.policy.gen.EQ }); return newValue / 2 !== written.gen; } catch (error) { if (error.code !== Aerospike.status.AEROSPIKE_ERR_RECORD_GENERATION) { throw error; } if (retry < 3) await sleep(2 ** (retry + 1) * 20); } } return false;}
const client = await Aerospike.connect({ hosts: '127.0.0.1:3000' });try { const key = new Aerospike.Key('test', 'rmw', 'id-1');
await client.put(key, { gen_times_2: 1 });
await runTest(4, () => rmwSimple(client, key)); await runTest(4, () => rmwWithGenCheck(client, key));
await client.remove(key);} finally { client.close();}#include <pthread.h>#include <stdbool.h>#include <stdint.h>#include <stdio.h>#include <stdlib.h>#include <time.h>#include <aerospike/aerospike.h>#include <aerospike/aerospike_key.h>#include <aerospike/as_error.h>#include <aerospike/as_key.h>#include <aerospike/as_operations.h>#include <aerospike/as_policy.h>#include <aerospike/as_record.h>
static const int TXNS_PER_WORKER = 10;#define MAX_WORKERS 8
typedef bool (*rmw_fn)(aerospike*, const as_key*);
typedef struct { aerospike* as; const as_key* key; rmw_fn rmw; bool lost;} worker_t;
static void sleep_ms(long ms){ struct timespec ts = {ms / 1000, (ms % 1000) * 1000000L}; nanosleep(&ts, NULL);}
static long random_ms(long max){ struct timespec now; clock_gettime(CLOCK_REALTIME, &now); return (now.tv_nsec / 1000) % max;}
static void* work(void* arg){ worker_t* worker = arg; for (int i = 0; i < TXNS_PER_WORKER; i++) { if (worker->rmw(worker->as, worker->key)) { worker->lost = true; } } return NULL;}
static void run_test(int workers, aerospike* as, const as_key* key, rmw_fn rmw){ pthread_t threads[MAX_WORKERS]; worker_t state[MAX_WORKERS]; if (workers > MAX_WORKERS) { fprintf(stderr, "At most %d workers are supported\n", MAX_WORKERS); exit(1); } for (int w = 0; w < workers; w++) { state[w] = (worker_t){as, key, rmw, false}; if (pthread_create(&threads[w], NULL, work, &state[w]) != 0) { fprintf(stderr, "Error starting a worker\n"); exit(1); } } bool lost_writes = false; for (int w = 0; w < workers; w++) { pthread_join(threads[w], NULL); lost_writes |= state[w].lost; } printf("Workers: %d, lost writes: %s\n", workers, lost_writes ? "yes" : "no");}
static bool rmw_simple(aerospike* as, const as_key* key){ as_error err; as_record* record = NULL; if (aerospike_key_get(as, &err, NULL, key, &record) != AEROSPIKE_OK) { fprintf(stderr, "Error reading record: %s\n", err.message); exit(1); } int read_gen = record->gen; as_record_destroy(record);
sleep_ms(random_ms(100));
int64_t new_value = 2 * (read_gen + 1); as_operations ops; as_operations_inita(&ops, 1); as_operations_add_write_int64(&ops, "gen_times_2", new_value); as_record* written = NULL; as_status status = aerospike_key_operate(as, &err, NULL, key, &ops, &written); as_operations_destroy(&ops); if (status != AEROSPIKE_OK) { fprintf(stderr, "Error writing record: %s\n", err.message); exit(1); } bool lost = new_value / 2 != written->gen; as_record_destroy(written); return lost;}
static bool rmw_with_gen_check(aerospike* as, const as_key* key){ for (int retry = 0; retry <= 3; retry++) { as_error err; as_record* record = NULL; if (aerospike_key_get(as, &err, NULL, key, &record) != AEROSPIKE_OK) { fprintf(stderr, "Error reading record: %s\n", err.message); exit(1); } int read_gen = record->gen; as_record_destroy(record);
sleep_ms(random_ms(100));
int64_t new_value = 2 * (read_gen + 1); as_operations ops; as_operations_inita(&ops, 1); as_operations_add_write_int64(&ops, "gen_times_2", new_value); ops.gen = read_gen;
as_policy_operate policy; as_policy_operate_init(&policy); policy.gen = AS_POLICY_GEN_EQ;
as_record* written = NULL; as_status status = aerospike_key_operate(as, &err, &policy, key, &ops, &written); as_operations_destroy(&ops); if (status == AEROSPIKE_OK) { bool lost = new_value / 2 != written->gen; as_record_destroy(written); return lost; } if (status != AEROSPIKE_ERR_RECORD_GENERATION) { fprintf(stderr, "Error writing record: %s\n", err.message); exit(1); } if (retry < 3) sleep_ms((1L << (retry + 1)) * 20); } return false;}
int main(void){ as_config config; as_config_init(&config); as_config_add_host(&config, "127.0.0.1", 3000);
aerospike as; aerospike_init(&as, &config);
as_error err; if (aerospike_connect(&as, &err) != AEROSPIKE_OK) { fprintf(stderr, "Error connecting: %s\n", err.message); return 1; }
int exit_code = 1; as_key key; as_key_init_str(&key, "test", "rmw", "id-1");
as_record rec; as_record_inita(&rec, 1); as_record_set_int64(&rec, "gen_times_2", 1); as_status status = aerospike_key_put(&as, &err, NULL, &key, &rec); as_record_destroy(&rec); if (status != AEROSPIKE_OK) { fprintf(stderr, "Error writing record: %s\n", err.message); goto cleanup; }
run_test(4, &as, &key, rmw_simple); run_test(4, &as, &key, rmw_with_gen_check);
if (aerospike_key_remove(&as, &err, NULL, &key) != AEROSPIKE_OK) { fprintf(stderr, "Error deleting record: %s\n", err.message); goto cleanup; } exit_code = 0;
cleanup: as_key_destroy(&key); aerospike_close(&as, &err); aerospike_destroy(&as); return exit_code;}import java.util.ArrayList;import java.util.List;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import java.util.concurrent.locks.LockSupport;import java.util.function.Supplier;
import com.aerospike.client.AerospikeClient;import com.aerospike.client.AerospikeException;import com.aerospike.client.Bin;import com.aerospike.client.Key;import com.aerospike.client.Operation;import com.aerospike.client.Record;import com.aerospike.client.ResultCode;import com.aerospike.client.policy.GenerationPolicy;import com.aerospike.client.policy.WritePolicy;
public class ReadModifyWrite { static final int TXNS_PER_WORKER = 10;
static void sleepMs(long ms) { LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(ms)); }
static void runTest(int workers, Supplier<Boolean> rmw) { ExecutorService pool = Executors.newFixedThreadPool(workers); List<CompletableFuture<Boolean>> futures = new ArrayList<>(); for (int w = 0; w < workers; w++) { futures.add(CompletableFuture.supplyAsync(() -> { boolean lost = false; for (int i = 0; i < TXNS_PER_WORKER; i++) { lost |= rmw.get(); } return lost; }, pool)); } boolean lostWrites = false; for (CompletableFuture<Boolean> future : futures) { lostWrites |= future.join(); } pool.shutdown(); System.out.println("Workers: " + workers + ", lost writes: " + (lostWrites ? "yes" : "no")); }
static boolean rmwSimple(AerospikeClient client, Key key) { Record record = client.get(null, key); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); Record written = client.operate(null, key, Operation.put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; }
static boolean rmwWithGenCheck(AerospikeClient client, Key key) { for (int retry = 0; retry <= 3; retry++) { Record record = client.get(null, key); int readGen = record.generation;
sleepMs(ThreadLocalRandom.current().nextInt(0, 100));
int newValue = 2 * (readGen + 1); WritePolicy policy = new WritePolicy(); policy.generationPolicy = GenerationPolicy.EXPECT_GEN_EQUAL; policy.generation = readGen; try { Record written = client.operate(policy, key, Operation.put(new Bin("gen_times_2", newValue))); return newValue / 2 != written.generation; } catch (AerospikeException e) { if (e.getResultCode() != ResultCode.GENERATION_ERROR) { throw e; } if (retry < 3) sleepMs((1L << (retry + 1)) * 20); } } return false; }
public static void main(String[] args) { try (AerospikeClient client = new AerospikeClient("127.0.0.1", 3000)) { Key key = new Key("test", "rmw", "id-1");
client.put(null, key, new Bin("gen_times_2", 1));
runTest(4, () -> rmwSimple(client, key)); runTest(4, () -> rmwWithGenCheck(client, key));
client.delete(null, key); } }}import randomimport sysimport threadingimport time
import aerospikefrom aerospike import exception as exfrom aerospike_helpers.operations import operations
TXNS_PER_WORKER = 10
def run_test(workers, rmw_fn): results = []
def work(): lost = False for _ in range(TXNS_PER_WORKER): lost |= rmw_fn() results.append(lost)
threads = [threading.Thread(target=work) for _ in range(workers)] for thread in threads: thread.start() for thread in threads: thread.join() print(f"Workers: {workers}, lost writes: {'yes' if any(results) else 'no'}")
def rmw_simple(client, key): (_, meta, _) = client.get(key)
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (meta["gen"] + 1) (_, meta, _) = client.operate(key, [operations.write("gen_times_2", new_value)]) return new_value // 2 != meta["gen"]
def rmw_with_gen_check(client, key): for retry in range(4): (_, meta, _) = client.get(key) read_gen = meta["gen"]
time.sleep(random.uniform(0, 0.1))
new_value = 2 * (read_gen + 1) try: (_, meta, _) = client.operate( key, [operations.write("gen_times_2", new_value)], meta={"gen": read_gen}, policy={"gen": aerospike.POLICY_GEN_EQ}, ) return new_value // 2 != meta["gen"] except ex.RecordGenerationError: if retry < 3: time.sleep((2 ** (retry + 1)) * 0.02) return False
config = {"hosts": [("127.0.0.1", 3000)]}
try: client = aerospike.client(config).connect()except ex.ClientError as e: print(f"Failed to connect to the cluster with {config['hosts']}: {e}", file=sys.stderr) sys.exit(1)
try: key = ("test", "rmw", "id-1") client.put(key, {"gen_times_2": 1})
run_test(4, lambda: rmw_simple(client, key)) run_test(4, lambda: rmw_with_gen_check(client, key))
client.remove(key)except ex.AerospikeError as e: print(f"Error: {e}", file=sys.stderr) sys.exit(1)finally: client.close()Workers: 4, lost writes: yesWorkers: 4, lost writes: noWhat you learned
You ran read-modify-write transactions from several concurrent workers and saw writes get lost. You made the write conditional on the record’s generation and added retries with exponential backoff when the check fails, so no write was lost.
Next steps
- Read the blog posts Developers: Understanding Aerospike Transactions and Twelve Do’s of Consistency in Aerospike.
- Learn more about transactions in Transactions.
- Learn how the Developer SDKs update records, including optimistic locking, in Update records.
- Run multi-record transactions with the Developer SDKs in Use transactions.