Skip to content

Upgrade to Database 8.2.0 and later

For the complete documentation index see: llms.txt

All documentation pages available in markdown.

This page describes the Aerospike Database 8.2.0 changes that need attention during or after a rolling upgrade. Most clusters can use a standard rolling upgrade after they complete the checks that apply to them.

Applies to: Rolling upgrades to Database 8.2.0 or later from Database 7.1.0.x through 8.1.2.x. If you upgrade from an earlier release, also review the special upgrade page for each interim release in Special upgrades and downgrades.

Audience: Cluster operators performing the upgrade, and developers who own String, List, or Map data or Lua UDFs.

Outcome: Every node runs Database 8.2.0 or later, String bins with invalid UTF-8 and List or Map values over the depth limit are identified and have a repair plan. The final node completes its initial system metadata (SMD) sync before it serves traffic.

Check the sections that apply to your cluster:

String operations and UTF-8 validation

Database 8.2.0 adds server-side String operations and String expressions. Both validate UTF-8. Validation applies when a String operation or String expression reads the bin. The upgrade does not change how the server stores or returns existing String bins, including bins that hold bytes that are not valid UTF-8.

Before any String read or modify operation runs, the server validates the existing bin value:

ConditionResulting error code
String bin contains invalid UTF-8AS_ERR_INVALID_ENCODING (29)
Operation argument such as a needle, pattern, or pad string is invalid UTF-8AS_ERR_PARAMETER (4)
Operation is applied to the wrong bin typeAS_ERR_INCOMPATIBLE_TYPE (12)

Plain record and bin writes allow invalid UTF-8 in String bins. The write succeeds, and the server logs a rate-limited warning. Later String operations on that bin fail with AS_ERR_INVALID_ENCODING until you repair the data. Store human-readable text in String bins, and store arbitrary binary data in Blob/Bytes bins.

The server checks RFC 3629 well-formedness:

  • Complete byte sequences only.
  • Unicode scalar values from U+0000 through U+10FFFF, excluding the surrogates U+D800 through U+DFFF.
  • No overlong encodings. An overlong encoding uses more bytes than a character needs, such as two bytes for an ASCII character.

The server does not enforce normalization form, grapheme cluster validity, or Unicode assignment.

String expressions also validate UTF-8 when they read a String bin:

  • In an operate projection, a String expression that reads invalid UTF-8 returns AS_ERR_OP_NOT_APPLICABLE (26).
  • In a query filter, the expression evaluates to unknown, and the query excludes the record from its results.

Identify String bins with invalid UTF-8

Aerospike Database does not scan stored data for invalid UTF-8. Use the following signals to find affected bins after the upgrade and before you enable String operations in application code.

  • Server log: When a write stores invalid UTF-8 in a String bin, a node that runs Database 8.2.0 or later logs the rate-limited warning string_from_wire - invalid UTF-8 detected in string data; string APIs will fail on this bin. The warning shows that an application or XDR source still writes invalid bytes, but it does not identify the record.
  • Query: Run a primary index (PI) query over the namespace or set with a filter expression that selects records where the bin_type of the bin is String. In the query projection, read the string_strlen expression on that bin with the EVAL_NO_FAIL read flag. Records that return no value for the expression hold invalid UTF-8. The query reads every record in the namespace or set, so run it outside peak traffic.
  • Client errors: A String operation that fails with AS_ERR_INVALID_ENCODING, or a String expression in an operate projection that fails with AS_ERR_OP_NOT_APPLICABLE, identifies the record by the key your application sent.

Repair legacy String bins

Repair requires a connected client, the affected record keys, knowledge of the original encoding, and support for get and put bin operations.

On a staging cluster that matches your production server and client versions:

  1. Repair a representative sample of affected records.

  2. Confirm that the transcoded values match application expectations, beyond strlen succeeding.

  3. Run the repair against production data after the sample values pass validation.

For namespace-wide or set-wide migration, repair affected records in batches before enabling String operations in application code.

Repair requires the bin’s original bytes. Clients differ in whether a read can return them, so check this behavior before planning a repair:

ClientReading a String bin that holds invalid UTF-8
C, GoReturns the bytes unchanged.
Java, C#, Node.jsReturns a string with every malformed sequence replaced by U+FFFD. The original bytes are not recoverable from it.
PythonRaises AEROSPIKE_ERR_CLIENT with the message Unknown type for value. The bin cannot be read.
RustReturns a UTF-8 decoding error. The bin cannot be read.

  1. Read the bin using a plain record read. Do not use String read operations such as strlen or to_blob, because they validate UTF-8 first and fail with AS_ERR_INVALID_ENCODING.

    record, err := client.Get(nil, key)
    if err != nil {
        return err
    }
    

    // A Go string holds arbitrary bytes, so this is the stored value as written.
    raw := []byte(record.Bins["legacy_text"].(string))

    The C client works the same way. as_record_get_string returns the stored bytes, and as_string_len returns their length.

  2. Write the original bytes to a separate Blob bin. Keep the Blob bin until you verify the repair.

    err := client.PutBins(nil, key, as.NewBin("legacy_text_backup", raw))

    A Blob bin stores the bytes as given. Writing them to a String bin does not preserve them after a client has decoded the value, so the Blob bin is the only reliable holding place.

  3. Transcode the bytes to UTF-8 in your application.

    Identify the source encoding before transcoding. The wrong encoding produces valid UTF-8 that contains the wrong characters. The server accepts that value, and you cannot reverse it without the original bytes.

  4. Write the transcoded value back as a String bin, or as a Blob bin if the payload is not text.

  5. Verify the repair by running a read-only String operation such as strlen on the bin. Success confirms valid UTF-8.

  6. After you verify the repair, remove the backup bin.

If the transcoded text is wrong, restore the value from the backup bin before you remove it. A Go string carries arbitrary bytes, so writing it back to the String bin restores the stored value.

record, err := client.Get(nil, key, "legacy_text_backup")
if err != nil {
    return err
}


backup := record.Bins["legacy_text_backup"].([]byte)
err = client.PutBins(nil, key, as.NewBin("legacy_text", string(backup)))

A value read through a client that substitutes replacement characters is not a rollback source, because those bytes are already lost.

SymptomLikely causeAction
AS_ERR_INVALID_ENCODING on operateLegacy or corrupted String binRepair the bin as described in this section. Check first whether your client can read the original bytes.
AS_ERR_PARAMETER on operateInvalid UTF-8 in an operation argumentValidate input before sending it.
String operation rejected by the serverNode runs a release prior to Database 8.2.0Finish the rolling upgrade before you enable String operations.

CDT nesting depth limit

Database 8.2.0 and later limit List and Map nesting to 64 levels. The top-level List or Map in a bin counts as level 1, and each nested List or Map adds one level. See Nesting depth limit for how depth is counted.

The server checks depth when a List or Map value arrives in a request. Values already stored deeper than 64 levels stay readable, and migrations during the rolling upgrade move them unchanged. On Database 8.2.0 and later, the following fail for a value deeper than 64 levels:

  • Whole-bin writes, including a read-modify-write cycle that writes back a value deeper than 64 levels.
  • CDT operations whose value or context path is deeper than 64 levels.
  • XDR shipments to a destination that runs Database 8.2.0 or later.
  • Restores of a backup that contains the value.

See CDT nesting depth limited to 64 levels for the error code and message that each request type returns.

Identify List and Map values exceeding the depth limit

Aerospike Database does not scan stored data for nesting depth, and it does not log values past a depth of 64 at the default log level. Use the following signals:

  • Client errors: Writes fail with the message list/map nested too deeply. A whole-bin write returns AS_ERR_UNKNOWN (1), and a CDT operation returns AS_ERR_PARAMETER (4).
  • XDR: When a destination that runs Database 8.2.0 or later rejects a shipped record, the source abandons the record instead of retrying it. The source’s abandoned metric increases, and the source logs a rate-limited abandon result warning with the DC name and the error code.
  • Application check: To find values deeper than 64 levels before they cause errors, read the bins in your application and count nesting levels on the client.

Flatten values over the depth limit

Restructure each List or Map value to 64 levels or fewer before it is rewritten, shipped, or restored. For XDR, flatten these values on the source cluster before you upgrade the destination to Database 8.2.0.

System metadata readiness on node join

In Database 8.2.0 and later, a new or returning node waits for initial SMD sync before it answers service or admin connections and before it receives incoming record migrations. The wait applies only when every node in the cluster runs Database 8.2.0 or later.

During a mixed-version rolling upgrade, joining nodes do not wait. After you upgrade the final node, that node performs the initial SMD sync before it serves traffic. Clients that connect too early receive a timeout rather than a connection refusal. There is no configuration to disable the wait.

Run asinfo -v "smd-info" on a node that runs Database 8.2.0 or later. While the cluster is mixed, the command reports mixed_cluster=true. After every node runs Database 8.2.0 or later, it reports mixed_cluster=false, and it reports initial_sync_done=true once sync finishes. Nodes that run earlier releases do not report these fields. See the smd-info reference for field definitions, and System metadata readiness on node join for log messages and diagnostics.

XDR stack overflow risk (AER-6898)

No action is needed if you upgrade from Database 8.1.2.x, which includes the fix for AER-6898. This section also does not apply to Community Edition or to clusters that do not use XDR.

Enterprise Edition and Federal Edition clusters that use XDR and run one of the following releases must follow the safe upgrade procedure instead of a standard rolling upgrade:

  • Database 7.1.0.14 through 7.1.0.22
  • Database 7.2.0.8 through 7.2.0.16
  • Database 8.0.0.3 through 8.0.0.14
  • Database 8.1.0.0 through 8.1.1.x

See XDR stack overflow risk (AER-6898) for detection steps and the full list of affected versions.

Lua UDF sandbox hardening

Database 8.2.0 changes the default of allow-unsafe-lua to false, so every UDF runs in hardened mode after the upgrade. UDFs that use os.*, io.*, debug.*, load*, or package.loadlib, and UDFs packaged as native .so modules or precompiled Lua bytecode, stop working. Before you upgrade, audit your registered UDFs with the migration checklist.

To keep the permissive behavior, add allow-unsafe-lua true to the mod-lua context of each node’s configuration file when you upgrade that node. The parameter is not dynamic, and only Database 7.1.0.25, 7.2.0.19, 8.0.0.17, 8.1.2.2, and later patch releases on those lines recognize it, so add it as part of the upgrade restart.

Replace --experimental with --preview

Database 8.2.0 rejects the --experimental startup flag. Replace it with --preview and a comma-separated list of preview features with no spaces:

Terminal window
asd --preview yaml-config,index-checkpoint --config-file /etc/aerospike/aerospike.yaml

Update deployment scripts, container entrypoints, and systemd units as you upgrade each node. Database 8.1.1 and 8.1.2 require --experimental for YAML configuration, so change the flag only on nodes that run Database 8.2.0.

Upgrade

  1. Back up the configuration file and startup scripts on each node.

  2. Upgrade one node at a time using the standard rolling upgrade procedure. As you upgrade each node, replace --experimental with --preview in its startup command, and add allow-unsafe-lua true to its mod-lua context if your UDFs need the permissive sandbox.

  3. Confirm that each upgraded node rejoins the cluster before you upgrade the next node.

  4. On the final node, wait until asinfo -v "smd-info" reports initial_sync_done=true and clients connect successfully.

  5. After every node runs Database 8.2.0 or later, enable String operations in application code. Nodes that run earlier releases reject String operations.