Validation tool (asvalidation)

This page describes Aerospike’s validation tool (asvalidation), which scans all records in a namespace and validates bins with Collection Data Type (CDT) values (List and Map bins). Optionally, it attempts to repair any damage detected. Records with unrecoverable CDT errors are written to output if an output file is specified. Records without CDTs or detected errors are ignored.

Download asvalidation

Download asvalidation, or visit its repo aerospike/aerospike-tools-validation .

Modes of operation

You can run asvalidation in validation mode or fix mode. Aerospike recommends first running in validation mode, limited by --max-records, to see the kinds of errors it discovers. Afterwards, run it in fix mode.

• Validation mode discovers problems and produces a report. This is the default mode.
• Fix mode attempts to correct discovered problems where possible. It is triggered by the --cdt-fix-ordered-list-unique option. Fix counts should report zero in the summary printout.

Run in validation mode

In the following example, myNamespace is checked and output is written to asvalidationOutput.txt.

• asvalidation was run in validation mode, because the --cdt-fix-ordered-list-unique option was not specified.
• It found 100 lists: 100 Lists.
• It found no Maps: 0 Maps.
• 10 of the lists were corrupted: 10 Need Fix under Lists.
• The reason for corruption is shown as out of order: 10 Order.
• Because it was run in validation mode, no fixes were applied: 0 Fixed.
• No List or Map was nested deeper than 64 levels, and the deepest had 3 levels: 0 Nested deeper than 64 levels and Deepest list/map nesting: 3 levels.

> asvalidation -n myNamespace -o  asvalidationOutput.txt
...
2024-05-01 22:12:28 GMT [INF] [24662] Found 10 invalid record(s) from 1 node(s), 2620 byte(s) in total (~262 B/rec)
2024-05-01 22:12:28 GMT [INF] [24662] CDT Mode: validate
2024-05-01 22:12:28 GMT [INF] [24662]        100 Lists
2024-05-01 22:12:28 GMT [INF] [24662]          0   Unfixable
2024-05-01 22:12:28 GMT [INF] [24662]          0     Has non-storage
2024-05-01 22:12:28 GMT [INF] [24662]          0     Corrupted
2024-05-01 22:12:28 GMT [INF] [24662]          0     Invalid Keys
2024-05-01 22:12:28 GMT [INF] [24662]         10   Need Fix
2024-05-01 22:12:28 GMT [INF] [24662]          0     Fixed
2024-05-01 22:12:28 GMT [INF] [24662]          0     Fix failed
2024-05-01 22:12:28 GMT [INF] [24662]         10     Order
2024-05-01 22:12:28 GMT [INF] [24662]          0     Padding
2024-05-01 22:12:28 GMT [INF] [24662]          0 Maps
2024-05-01 22:12:28 GMT [INF] [24662]          0   Unfixable
2024-05-01 22:12:28 GMT [INF] [24662]          0     Has duplicate keys
2024-05-01 22:12:28 GMT [INF] [24662]          0     Has non-storage
2024-05-01 22:12:28 GMT [INF] [24662]          0     Corrupted
2024-05-01 22:12:28 GMT [INF] [24662]          0     Invalid Keys
2024-05-01 22:12:28 GMT [INF] [24662]          0   Need Fix
2024-05-01 22:12:28 GMT [INF] [24662]          0     Fixed
2024-05-01 22:12:28 GMT [INF] [24662]          0     Fix failed
2024-05-01 22:12:28 GMT [INF] [24662]          0     Order
2024-05-01 22:12:28 GMT [INF] [24662]          0     Padding
2024-05-01 22:12:28 GMT [INF] [24662]          0 Nested deeper than 64 levels
2024-05-01 22:12:28 GMT [INF] [24662]          0   Rejected by server 8.2+
2024-05-01 22:12:28 GMT [INF] [24662]          0   Only via empty lists/maps (rejected by server 8.2.1+, client 7.6+ and UDFs)
2024-05-01 22:12:28 GMT [INF] [24662] Deepest list/map nesting: 3 levels

Note

Numbers under a heading do not necessarily add up to the count of the line. For example, there could be a 1 Need Fix record with both an Order and Padding error.

Run in fix mode

In the following example, myNamespace is checked and output is written to asvalidationOutput.txt.

• Fixing is triggered by the --cdt-fix-ordered-list-unique option.

• Fixes are applied to the server. Failed fixes can be due to (but not limited to) an unsupported server version.

asvalidation -n myNamespace -o asvalidationOutput.txt -r --cdt-fix-ordered-list-unique
validation of 127.0.0.1 (namespace: myNamespace, set: [all], bins: [all], after: [none], before: [none]) to asvalidationOutput.txt
2024-05-01 22:08:25 GMT [INF] [10999] [src/main/aerospike/as_cluster.c:132][as_cluster_add_nodes_copy] Add node BB909000027000A 127.0.0.1:3000
2024-05-01 22:08:25 GMT [INF] [10999] Processing 1 node(s)
2024-05-01 22:08:25 GMT [INF] [10999] Node ID             Objects        Replication
2024-05-01 22:08:25 GMT [INF] [10999] BB909000027000A     130            1
2024-05-01 22:08:25 GMT [INF] [10999] Namespace contains 130 record(s)
2024-05-01 22:08:25 GMT [INF] [10999] Created new output file temp.txt
2024-05-01 22:08:25 GMT [INF] [11018] Starting validation for node BB909000027000A
2024-05-01 22:08:25 GMT [INF] [11018] Completed validation for node BB909000027000A, records: 30, size: 9200 (~306 B/rec)
2024-05-01 22:08:26 GMT [INF] [11017] 23% complete (~8 KiB/s, ~30 rec/s, ~306 B/rec)
2024-05-01 22:08:26 GMT [INF] [11017] ~3s remaining
2024-05-01 22:08:26 GMT [INF] [11017] Found 30 invalid record(s) from 1 node(s), 9200 byte(s) in total (~306 B/rec)
2024-05-01 22:08:26 GMT [INF] [11017] CDT Mode: fix
2024-05-01 22:08:26 GMT [INF] [11017]        110 Lists
2024-05-01 22:08:26 GMT [INF] [11017]          0   Unfixable
2024-05-01 22:08:26 GMT [INF] [11017]          0     Has non-storage
2024-05-01 22:08:26 GMT [INF] [11017]          0     Corrupted
2024-05-01 22:08:26 GMT [INF] [11017]          0     Invalid Keys
2024-05-01 22:08:26 GMT [INF] [11017]         10   Need Fix
2024-05-01 22:08:26 GMT [INF] [11017]         10     Fixed
2024-05-01 22:08:26 GMT [INF] [11017]          0     Fix failed
2024-05-01 22:08:26 GMT [INF] [11017]         10     Order
2024-05-01 22:08:26 GMT [INF] [11017]          0     Padding
2024-05-01 22:08:26 GMT [INF] [11017]         20 Maps
2024-05-01 22:08:26 GMT [INF] [11017]         10   Unfixable
2024-05-01 22:08:26 GMT [INF] [11017]         10     Has duplicate keys
2024-05-01 22:08:26 GMT [INF] [11017]          0     Has non-storage
2024-05-01 22:08:26 GMT [INF] [11017]          0     Corrupted
2024-05-01 22:08:26 GMT [INF] [11017]          0     Invalid Keys
2024-05-01 22:08:26 GMT [INF] [11017]         10   Need Fix
2024-05-01 22:08:26 GMT [INF] [11017]          0     Fixed
2024-05-01 22:08:26 GMT [INF] [11017]          0     Fix failed
2024-05-01 22:08:26 GMT [INF] [11017]         10     Order
2024-05-01 22:08:26 GMT [INF] [11017]          0     Padding
2024-05-01 22:08:26 GMT [INF] [11017]          0 Nested deeper than 64 levels
2024-05-01 22:08:26 GMT [INF] [11017]          0   Rejected by server 8.2+
2024-05-01 22:08:26 GMT [INF] [11017]          0   Only via empty lists/maps (rejected by server 8.2.1+, client 7.6+ and UDFs)
2024-05-01 22:08:26 GMT [INF] [11017]          0   Fix skipped
2024-05-01 22:08:26 GMT [INF] [11017] Deepest list/map nesting: 3 levels

Check nesting depth

Starting with version 2.3.0, asvalidation finds Lists and Maps nested deeper than 64 levels. Aerospike Database 8.2 and later reject most values nested deeper than this limit, and earlier servers accepted any depth. 8.2.0 accepts nesting that exceeds the limit only through empty Lists or Maps without flags. The upgrade to Database 8.2 succeeds with such records in place, but afterward the following problems occur:

• Rewriting the record, including a client read-modify-write, fails with error 1 (“list/map nested too deeply”).
• asrestore into a Database 8.2 cluster stops when it reaches the record, so later records in the backup are never restored.
• Cross Datacenter Replication (XDR) doesn’t ship the record, or any later update to it, to a Database 8.2 cluster, and doesn’t retry.
• C client 7.6 and later, some other client libraries, and user-defined functions (UDFs) can’t read the record. A UDF sees the bin as nil, and a UDF that writes the bin back deletes it.

Run asvalidation on every namespace at each of the following points:

• Before you upgrade to Database 8.2.
• Before an XDR source ships to a Database 8.2 cluster.
• Before you move applications to C client 7.6 or later.
• Before you rely on UDFs on Database 8.2, 8.1.2.5, 8.0.0.20, 7.2.0.22, and later releases.
• Again after the upgrade. Some writes on Database 8.2 can still store bins that are too deep: Lua UDF writes, and on Database 8.2.0, a List or Map operation that nests a 64-level value inside another.

Depth checking runs in every mode. To run only the depth check, which is faster, use --cdt-check-depth-only. To make the exit code report deep bins, add --fail-on-cdt-depth.

asvalidation -n NAMESPACE -o deep.asb --cdt-check-depth-only --fail-on-cdt-depth

Records with deep bins go to the output file like any other problem record. If many records are affected, the output can approach the size of the namespace. To split it into multiple files, use --directory with --file-limit. These records can’t be restored into a Database 8.2 cluster as they are, so the output isn’t a usable backup of them.

Note

Older versions of asvalidation can crash on deeply nested data while checking Map keys. With those versions, add --no-cdt-check-map-keys.

The summary reports the depth findings in the following lines:

Summary line	Description
Nested deeper than N levels	Number of List and Map bins nested deeper than the limit.
Rejected by server 8.2+	Deep bins where a List or Map past the limit holds elements or flags.
Only via empty lists/maps	Deep bins that exceed the limit only through empty Lists or Maps without flags.
Fix skipped	Fix mode only. Fixes not applied because the record has a bin in the Rejected by server 8.2+ category.
Deepest list/map nesting	The deepest nesting found, or more than N levels if it exceeds the limit.

The tool doesn’t fix deep bins. Reduce the nesting in your application instead. In fix mode, asvalidation never changes records that have a bin in the Rejected by server 8.2+ category.

asvalidation exits with code 0 when the run completes, 1 on errors, and 3 with --fail-on-cdt-depth when it finds bins nested deeper than the limit.

With --machine, the end of the run also writes the following line:

CDT_DEPTH:LIMIT:DEEPEST:REJECTED_BY_SERVER:ONLY_VIA_EMPTY

LIMIT is the depth limit. DEEPEST is the deepest nesting found, and a value of LIMIT plus 1 means deeper than the limit. REJECTED_BY_SERVER and ONLY_VIA_EMPTY are the Rejected by server 8.2+ and Only via empty lists/maps counts.

Minimal options for asvalidation

Following is the minimal set of asvalidation options.

Option	Description
--cdt-fix-ordered-list-unique	Fix lists where elements were not stored in order and remove duplicate elements. Without this option, the tool only validates, not fixes.
--no-cdt-check-map-keys	Do not check CDT Map keys.
--cdt-check-depth-only	Only check List and Map nesting depth, and Map keys unless --no-cdt-check-map-keys is set. Skips the order, duplicate key, padding, and non-storage checks. Mutually exclusive with --cdt-fix-ordered-list-unique.
--fail-on-cdt-depth	Exit with code 3 if any List or Map is nested deeper than 64 levels.
-o	Output file name for validation report.
-d	Output directory.
--help	Get a comprehensive list of options for tool.

Namespace data selection options

Option	Default	Description
-n NAMESPACE or --namespace NAMESPACE	-	Mandatory. Namespace to validate.
-s SETS or --set SETS	All sets	Set(s) to validate. May pass in a comma-separated list of sets to validate.
-B BIN1,BIN2,... or --bin-list BIN1,BIN2,...	All bins	Bins to validate.
-M or --max-records N	0 = all records.	Approximate limit for the number of records to process.

Partition scanning validation options

Scan a list of partition filters. Partition filters can be ranges, individual partitions, or records after a specific digest within a single partition.

This option is mutually exclusive with --node-list.

Option	Default
-X, --partition-list LIST	not used

Default partitions to scan: 0 to 4095 (all partitions)

• LIST format: FILTER1,FILTER2,…
• FILTER format: BEGIN-PARTITION -PARTITION-COUNT|DIGEST
  • BEGIN-PARTITION: 0 to 4095.
  • Either the optional PARTITION-COUNT: 1 to 4096. Default: 1
  • Or the optional DIGEST: Base64-encoded string of desired digest to start at in specified partition.

Note

Each partition filter operates as a single scan call and cannot be parallelized using the parallel option. To increase parallelism, divide the range into multiple partition filters.

Example	Description
-X 361	Validate only partition 361.
-X 361-10	Validate 10 partitions, starting with 361 and including 370.
-X VSmeSvxNRqr46NbOqiy9gy5LTIc=	Validate all records after the digest VSmeSvxNRqr46NbOqiy9gy5LTIc= in its partition.
-X 0-1000,2222,EjRWeJq83vEjRRI0VniavN7xI0U=	Validate partitions 0 to 999 (1000 partitions starting from 0), partition 2222, and all records after the digest EjRWeJq83vEjRRI0VniavN7xI0U= in its partition.

Connection options

Option	Default	Description
-h HOST1:TLSNAME1:PORT1,... or --host HOST1:TLSNAME1:PORT1,...	127.0.0.1	The host that acts as the entry point to the cluster. Any nodes in the cluster can be specified. The remaining nodes are discovered automatically.
-p PORT or --port PORT	3000	Port to connect to.
-U USER or --user USER	-	User name with read permission. Mandatory if the server has security enabled.
-P PASSWORD or--password	-	Password to authenticate the given user. The first form passes the password on the command line. The second form prompts for the password.
-A or --auth	INTERNAL	Set authentication mode when user and password are defined. Modes are (INTERNAL, EXTERNAL, EXTERNAL_INSECURE, PKI) and the default is INTERNAL. This mode must be set EXTERNAL when using LDAP.
--parallel N	1	Maximum number of scans to run in parallel. If only one partition range is given, or the entire namespace is being validated, the range of partitions is evenly divided by this number to be processed in parallel. Otherwise, each filter cannot be parallelized individually, so you may only achieve as much parallelism as there are partition filters.
-l HOST1:[TLSNAME1:]PORT1,... or --node-list HOST1:[TLSNAME1:]PORT1,...	-	Validate the given cluster nodes only. Mutually exclusive with --partition-list.
--tls-enable	disabled	Indicates a TLS connection should be used.
-S or --services-alternate	false	Set to true to connect to Aerospike node’s alternate-access-address.
--prefer-racks RACKID1,...	disabled	A comma separated list of rack IDs to prefer when reading records. This is useful for limiting cross datacenter network traffic.

Output options

Option	Default	Description
-d PATH or --directory PATH	-	Directory to store the .asb validation files. If the directory does not exist, it is created before use. Mandatory, unless --output-file or --estimate is given.
-o PATH or --output-file PATH	-	The single file to write to. - means stdout. Mandatory, unless --directory or --estimate is given.
-q DESIRED-PREFIX or --output-file-prefix DESIRED-PREFIX	-	Must be used with the --directory option. A desired prefix for all output files.
-F LIMIT or --file-limit LIMIT	250 MB	File size limit (in MiB) for --directory. If a .asb validation file crosses this size threshold, asvalidation switches to a new file.
-r or --remove-files	-	Clear directory or remove output file. By default, asvalidation refuses to write to a non-empty directory or to overwrite an existing validation file. This option clears the given --directory or removes an existing --output-file. Mutually exclusive to --continue.
--remove-artifacts	-	Clear directory or remove output file, like --remove-files, without running a validation. This option is mutually exclusive to --continue and --estimate.
-N BANDWIDTH or --nice BANDWIDTH	-	Throttles asvalidation’s write operations to the validation file(s) to not exceed the given bandwidth in MiB/s. Effectively also throttles the scan on the server side as asvalidation refuses to accept more data than it can write.

Other options

Option	Default	Description
-v or --verbose	disabled	Output considerably more information about the running validation.
-m or --machine PATH	-	Output machine-readable status updates to the given path, typically a FIFO.
-L or --records-per-second RPS	0	Limit total returned records per second (RPS). If RPS is zero, a records-per-second limit is not applied.
-V or --version	-	Print ASVALIDATION version information.
-C or --compact	disabled	Do not apply base-64 encoding to BLOBs; results in smaller output files.

Configuration file options

Configurations are read from the following files in the given order:

• /etc/aerospike/astools.conf
• ~/.aerospike/astools.conf

Option	Default	Description
-no-config-file	disabled	Do not read any config file.
--instance NAME	-	Section with this instance is read. In case instance a is specified, sections cluster_a, asvalidation_a are read.
--config-file PATH	-	Read this file after default configuration file.
--only-config-file PATH	-	Read only this configuration file.

Error descriptions

Reason	Description	Disposition
Has non-storage	The bin contains an infinite or wildcard element which is not allowed as storage.	Requires manual intervention to fix.
Has duplicate keys	A map bin has duplicate key entries.	Unfixable without manual intervention.
Corrupted	A problem not attributable to any other error categories.	Requires manual intervention to fix.
Invalid Keys	The bin has a Map with at least one invalid key.	Requires manual intervention to fix.
Order	The bin has elements out of order.	Can be fixed by reordering the list with the --cdt-fix-ordered-list-unique option.
Padding	The bin has garbage bytes after the valid List or Map.	Can be fixed by truncating the extra bytes.
Rejected by server 8.2+	A List or Map that holds elements or flags is nested deeper than 64 levels. Every Database 8.2 release refuses the record when it is rewritten, restored, or shipped by XDR. An empty ordered List or Map holds its order flags, so it counts here.	Not fixable by asvalidation. Reduce the nesting in your application before upgrading. Fix mode never changes these records.
Only via empty lists/maps	Only empty Lists or Maps without flags are nested deeper than 64 levels. Database 8.2.0 accepts the record, but Database 8.2.1 and later reject it. C client 7.6 and later, some other client libraries, and UDFs can’t read it, and a UDF that writes the bin back deletes it.	Not fixable by asvalidation. Reduce the nesting in your application.