Aerospike Document API for JSON Documents

For an interactive Jupyter notebook experience:

This notebook illustrates the Aerospike Document API, which is used to store JSON documents in the Aerospike Database and perform operations on them, with code examples.

This notebook requires the Aerospike Database running locally with Java kernel and Aerospike Java Client. To create a Docker container that satisfies the requirements and holds a copy of Aerospike notebooks, visit the Aerospike Notebooks Repo.

Introduction

The Aerispike Document API library is available in this github repository. It is described in blog posts here and here.

This goal of this notebook is to demonstrate the Document API and JSONPath capabilities.

The main topics in this notebook include:

Document API methods and code examples
JSONPath overview and code examples
Additional JSONPath capabilities and code examples

Prerequisites

This tutorial assumes familiarity with the following topics:

Setup

Ensure database is running

This notebook requires that the Aerospike Database is running.

import io.github.spencerpark.ijava.IJava;
import io.github.spencerpark.jupyter.kernel.magic.common.Shell;
IJava.getKernelInstance().getMagics().registerMagics(Shell.class);
%sh asd

Download and Install Additional Components

Aerospike Java client 6.1.6 and the document api library 2.0.0.

%%loadFromPOM
<dependencies>
  <dependency>
    <groupId>com.aerospike</groupId>
    <artifactId>aerospike-client</artifactId>
    <version>6.1.6</version>
  </dependency>
  <dependency>
  <groupId>com.aerospike</groupId>
  <artifactId>aerospike-document-api</artifactId>
  <version>2.0.0</version>
</dependency>
</dependencies>

Imports

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.stream.Stream;
import java.util.stream.Collectors;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.JsonNode;
import com.jayway.jsonpath.JsonPath;

import com.aerospike.client.AerospikeException;
import com.aerospike.client.AerospikeClient;
import com.aerospike.client.Key;
import com.aerospike.client.BatchRecord;
import com.aerospike.documentapi.batch.BatchOperation;
import com.aerospike.documentapi.batch.GetBatchOperation;
import com.aerospike.documentapi.batch.PutBatchOperation;
import com.aerospike.documentapi.batch.AppendBatchOperation;
import com.aerospike.documentapi.batch.DeleteBatchOperation;

import com.aerospike.documentapi.*;
import com.aerospike.documentapi.util.JsonConverters;

Define Convenience Functions

Define convenience functions readJSONFromAFile, printDocuments, and truncateTestData.

// define a convenience function to get the document bin from all records and print
void printDocuments() {
    List<BatchOperation> batchOpsList = new ArrayList<>();
    batchOpsList.add(new GetBatchOperation(key1, Collections.singletonList(documentBinName), "$"));
    batchOpsList.add(new GetBatchOperation(key2, Collections.singletonList(documentBinName), "$"));
    batchOpsList.add(new GetBatchOperation(key3, Collections.singletonList(documentBinName), "$"));
    List<BatchRecord> results = documentClient.batchPerform(batchOpsList, true);
    results.stream().forEach(s -> System.out.println(s.record.bins.get("documentBin")));
}

// convenience function to read JSON doc from a file
String readJSONFromAFile(String filePath) throws IOException {
    StringBuilder contentBuilder = new StringBuilder();
    Stream<String> stream = Files.lines(Paths.get(filePath), StandardCharsets.UTF_8);
    stream.forEach(s -> contentBuilder.append(s).append("\n"));
    return contentBuilder.toString();
}

// convenience function to truncate test data
void truncateTestData() {
    try {
        aerospikeClient.truncate(null, NAMESPACE, SET, null);
    }
    catch (AerospikeException e) {
        // ignore
    }
}

Document API

The Aerospike Document API provides CRUD operations at arbitrary points within a JSON document as described in this blog post.

Interface

The document API is as represented in the interface IAerospikeDocumentClient, which is shown below for convenience. The API has the CRUD methods get, put, append, and delete, and allows a JSONPath argument to specify parts of the document simply and expressively to apply these methods. Starting with release 2.0, a powerful batch functionality is also available through the batchPerform method that allows multiple operations to be performed in a single request, where each operation is get/put/append/delete on a record.

    /**
     * Retrieve an object matched by JSON path.
     *
     * @param key      Aerospike Key.
     * @param binName  name of a bin storing json.
     * @param jsonPath JSON path matching the required elements.
     * @return object matched by jsonPath.
     * @throws DocumentApiException if there was an error.
     */
    Object get(Key key, String binName, String jsonPath);

    /**
     * Retrieve a map of objects matched by JSON path.
     *
     * @param key      Aerospike Key.
     * @param binNames names of bins storing json (all bins with the same document structure).
     * @param jsonPath JSON path matching the required elements.
     * @return A map of objects matched by jsonPath with bin names as keys.
     * @throws DocumentApiException if there was an error.
     */
    Map<String, Object> get(Key key, Collection<String> binNames, String jsonPath);

    /**
     * Put a JSON document.
     *
     * @param key        Aerospike Key.
     * @param binName    name of a bin to store json.
     * @param jsonObject JSON object (document) to put.
     */
    void put(Key key, String binName, JsonNode jsonObject);

    /**
     * Put an object at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binName  name of a bin storing json.
     * @param jsonPath A JSON path to put the given JSON object in.
     * @param object   An object to put in the given JSON path.
     * @throws DocumentApiException if there was an error.
     */
    void put(Key key, String binName, String jsonPath, Object object);

    /**
     * Put an object at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binNames names of bins storing json (all bins with the same document structure).
     * @param jsonPath JSON path to put the given JSON object in.
     * @param object   the object to be put at the given JSON path.
     * @throws DocumentApiException if there was an error.
     */
    void put(Key key, Collection<String> binNames, String jsonPath, Object object);

    /**
     * Append an object to a collection at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binName  name of a bin storing json.
     * @param jsonPath JSON path that includes a collection to append the given JSON object to.
     * @param object   the object to be appended at the given JSON path.
     * @throws DocumentApiException if there was an error.
     */
    void append(Key key, String binName, String jsonPath, Object object);

    /**
     * Append an object to a collection at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binNames names of bins storing json (all bins with the same document structure).
     * @param jsonPath JSON path that includes a collection to append the given JSON object to.
     * @param object   the object to be appended at the given JSON path.
     * @throws DocumentApiException if there was an error.
     */
    void append(Key key, Collection<String> binNames, String jsonPath, Object object);

    /**
     * Delete an object at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binName  name of a bin storing json.
     * @param jsonPath JSON path for the object deletion.
     * @throws DocumentApiException if there was an error.
     */
    void delete(Key key, String binName, String jsonPath);

    /**
     * Delete an object at a particular path in JSON document.
     *
     * @param key      Aerospike Key.
     * @param binNames names of bins storing json (all bins with the same document structure).
     * @param jsonPath JSON path for the object deletion.
     * @throws DocumentApiException if there was an error.
     */
    void delete(Key key, Collection<String> binNames, String jsonPath);

    /**
     * Perform batch operations.
     *
     * <p>Operations order is preserved only for those 1-step operations
     * (with JSONPath that contains only array and/or map elements)
     * that have unique Aerospike keys within a batch.</p>
     * <p>Every 2-step operation (with JSONPath containing wildcards, recursive descent, filters, functions, scripts)
     * should have unique Aerospike key within a batch.
     *
     * @param batchOperations a list of batch operations to apply.
     * @param parallel        whether batch processing stream operations should run in parallel.
     * @return a list of corresponding {@link BatchRecord} results.
     * @throws DocumentApiException     if there was an error.
     * @throws IllegalArgumentException if the batch has multiple two-step operations with the same key.
     */
    List<BatchRecord> batchPerform(List<BatchOperation> batchOperations, boolean parallel);
}

Example JSON Document

The example JSON document used is stored in file doc_api_example_tommyleejones.json, and has this json:

{
  "forenames": [
    "Tommy",
    "Lee"
  ],
  "surname": "Jones",
  "date_of_birth": {
    "day": 15,
    "month": 9,
    "year": 1946
  },
  "selected_filmography":{
    "2012":["Lincoln","Men In Black 3"],
    "2007":["No Country For Old Men"],
    "2002":["Men in Black 2"],
    "1997":["Men in Black","Volcano"],
    "1994":["Natural Born Killers","Cobb"],
    "1991":["JFK"],
    "1980":["Coal Miner's Daughter","Barn Burning"]
  },
  "imdb_rank":{
    "source":"https://www.imdb.com/list/ls050274118/",
    "rank":51
  },
  "best_films_ranked": [
    {
      "source": "http://www.rottentomatoes.com",
      "films": ["The Fugitive","No Country For Old Men","Men In Black","Coal Miner's Daughter","Lincoln"]
    },
    {
      "source":"https://medium.com/the-greatest-films-according-to-me/10-greatest-films-of-tommy-lee-jones-97426103e3d6",
      "films":["The Three Burials of Melquiades Estrada","The Homesman","No Country for Old Men","In the Valley of Elah","Coal Miner's Daughter"]
    }
  ]
}

Store JSON Document

Iniialize Document Client

Create a document client using an Aerospike client.

final String NAMESPACE = "test";
final String SET = "document-api";

AerospikeClient aerospikeClient = new AerospikeClient("localhost", 3000);
System.out.println("Initialized Aerospike client and connected to the cluster.");
AerospikeDocumentClient documentClient = new AerospikeDocumentClient(aerospikeClient);
System.out.println("Initialized document client from the Aerospike client.");;

Output:
Initialized Aerospike client and connected to the cluster. Initialized document client from the Aerospike client.

Read JSON Document

Using the convenience function readJSONFromAFile defined above, read a json file.

// Read the json document into a string.
String jsonString = readJSONFromAFile("doc_api_example_tommyleejones.json");
System.out.println("Read JSON doc from file.");

// Convert JSON string to a JsonNode
JsonNode jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
System.out.format("JSON doc: %s\n", jsonNode);;

Output:
Read JSON doc from file. JSON doc: {"forenames":["Tommy","Lee"],"surname":"Jones","date_of_birth":{"day":15,"month":9,"year":1946},"selected_filmography":{"2012":["Lincoln","Men In Black 3"],"2007":["No Country For Old Men"],"2002":["Men in Black 2"],"1997":["Men in Black","Volcano"],"1994":["Natural Born Killers","Cobb"],"1991":["JFK"],"1980":["Coal Miner's Daughter","Barn Burning"]},"imdb_rank":{"source":"https://www.imdb.com/list/ls050274118/","rank":51},"best_films_ranked":[{"source":"http://www.rottentomatoes.com","films":["The Fugitive","No Country For Old Men","Men In Black","Coal Miner's Daughter","Lincoln"]},{"source":"https://medium.com/the-greatest-films-according-to-me/10-greatest-films-of-tommy-lee-jones-97426103e3d6","films":["The Three Burials of Melquiades Estrada","The Homesman","No Country for Old Men","In the Valley of Elah","Coal Miner's Daughter"]}]}

Add JSON Document to Database

// Construct a key
Key tommyLeeJonesDBKey = new Key(NAMESPACE, SET, "tommy-lee-jones.json");

// Add the document to database
String documentBinName = "documentBin";
documentClient.put(tommyLeeJonesDBKey, documentBinName, jsonNode);
System.out.println("Stored JSON doc in database.");

Output:
Stored JSON doc in database.

Read

// indexed array elements
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.best_films_ranked[0].films[0]" );

Output:
The Fugitive

// an array element denoted by a simple path
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.1980");

Output:
[Coal Miner's Daughter, Barn Burning]

// complex json path: wild-card
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.*");

Output:
{1997=[Men in Black, Volcano], 2007=[No Country For Old Men], 1994=[Natural Born Killers, Cobb], 2002=[Men in Black 2], 1991=[JFK], 1980=[Coal Miner's Daughter, Barn Burning], 2012=[Lincoln, Men In Black 3]}

// complex json path: matching elements in a subtree
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$..source");

Output:
["http:\/\/www.rottentomatoes.com","https:\/\/medium.com\/the-greatest-films-according-to-me\/10-greatest-films-of-tommy-lee-jones-97426103e3d6","https:\/\/www.imdb.com\/list\/ls050274118\/"]

Update

// add year 2019 and films
List<String> _2019Films = new Vector<String>();
_2019Films.add("Ad Astra");
documentClient.put(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.2019", _2019Films);
// read the update
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.2019" );

Output:
[Ad Astra]

// update the imdb rank
documentClient.put(tommyLeeJonesDBKey, documentBinName, "$.imdb_rank.rank", 45);
// read the update
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.imdb_rank.rank" );

Output:
45

// append films to best_films_ranked
documentClient.append(tommyLeeJonesDBKey, documentBinName, "$.best_films_ranked[0].films", "Rolling Thunder");
documentClient.append(tommyLeeJonesDBKey, documentBinName, "$.best_films_ranked[0].films", "The Three Burials");
// read the updates
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.best_films_ranked[0].films" );

Output:
[The Fugitive, No Country For Old Men, Men In Black, Coal Miner's Daughter, Lincoln, Rolling Thunder, The Three Burials]

// put a json-node object
String jsonString = "[\"The Comeback Trail\",\"Wander\"]";
//String jsonString = "{\"a\":[\"The Comeback Trail\",\"Wander\"]}";
//String jsonString = "{\"last-updated\": 2020,\"updated-by\": \"A.Fan\"}";
// convert JSON string to a JsonNode
JsonNode jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
//put the json-node
documentClient.put(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.2020", jsonNode);
// read the update
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$..selected_filmography" );

Output:
[{"1997":["Men in Black","Volcano"],"2019":["Ad Astra"],"2007":["No Country For Old Men"],"1994":["Natural Born Killers","Cobb"],"2002":["Men in Black 2"],"1991":["JFK"],"1980":["Coal Miner's Daughter","Barn Burning"],"2012":["Lincoln","Men In Black 3"],"2020":["The Comeback Trail","Wander"]}]

// put a json-node object
String jsonString = "{\"last-updated\": 2020,\"updated-by\": \"A.Fan\"}";
// convert JSON string to a JsonNode
JsonNode jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
//put the json-node
documentClient.put(tommyLeeJonesDBKey, documentBinName, "$.notes", jsonNode);
// read the update
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$.notes" );

Output:
{"last-updated":2020,"updated-by":"A.Fan"}

// complex json path: wild-card
documentClient.append(tommyLeeJonesDBKey, documentBinName, "$..selected_filmography[*]", "TEST");
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$..selected_filmography[*]" );

Output:
[["Men in Black","Volcano","TEST"],["Ad Astra","TEST"],["No Country For Old Men","TEST"],["Natural Born Killers","Cobb","TEST"],["Men in Black 2","TEST"],["JFK","TEST"],["Coal Miner's Daughter","Barn Burning","TEST"],["Lincoln","Men In Black 3","TEST"],["The Comeback Trail","Wander","TEST"]]

// complex json path: matching elements in a subtree
//append to all arrays
documentClient.append(tommyLeeJonesDBKey, documentBinName, "$..films", "TEST");
// read the update
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$..films" );

Output:
[["The Fugitive","No Country For Old Men","Men In Black","Coal Miner's Daughter","Lincoln","Rolling Thunder","The Three Burials","TEST"],["The Three Burials of Melquiades Estrada","The Homesman","No Country for Old Men","In the Valley of Elah","Coal Miner's Daughter","TEST"]]

Delete

// complex json path: wild-card
documentClient.delete(tommyLeeJonesDBKey, documentBinName, "$.selected_filmography.[*][2]");
documentClient.get(tommyLeeJonesDBKey, documentBinName, "$..selected_filmography" );

Output:
[{"1997":["Men in Black","Volcano"],"2019":["Ad Astra","TEST"],"2007":["No Country For Old Men","TEST"],"1994":["Natural Born Killers","Cobb"],"2002":["Men in Black 2","TEST"],"1991":["JFK","TEST"],"1980":["Coal Miner's Daughter","Barn Burning"],"2012":["Lincoln","Men In Black 3"],"2020":["The Comeback Trail","Wander"]}]

// print state before deletion
Object docObject = documentClient.get(tommyLeeJonesDBKey, documentBinName, "$");
System.out.format("Before delete: \n%s\n", docObject);
// delete all descendants of the document root
documentClient.delete(tommyLeeJonesDBKey, documentBinName, "$..*");
// print state after deletion
docObject = documentClient.get(tommyLeeJonesDBKey, documentBinName, "$");
System.out.format("After delete: \n%s\n", docObject);;

Output:
Before delete:

{best_films_ranked=[{films=[The Fugitive, No Country For Old Men, Men In Black, Coal Miner's Daughter, Lincoln, Rolling Thunder, The Three Burials, TEST], source=http://www.rottentomatoes.com}, {films=[The Three Burials of Melquiades Estrada, The Homesman, No Country for Old Men, In the Valley of Elah, Coal Miner's Daughter, TEST], source=https://medium.com/the-greatest-films-according-to-me/10-greatest-films-of-tommy-lee-jones-97426103e3d6}], date_of_birth={month=9, day=15, year=1946}, forenames=[Tommy, Lee], imdb_rank={rank=45, source=https://www.imdb.com/list/ls050274118/}, notes={last-updated=2020, updated-by=A.Fan}, selected_filmography={1997=[Men in Black, Volcano], 2019=[Ad Astra, TEST], 2007=[No Country For Old Men, TEST], 1994=[Natural Born Killers, Cobb], 2002=[Men in Black 2, TEST], 1991=[JFK, TEST], 1980=[Coal Miner's Daughter, Barn Burning], 2012=[Lincoln, Men In Black 3], 2020=[The Comeback Trail, Wander]}, surname=Jones}

After delete:

{}

Batch Operations

In Document API versions 2.0+, batch operations are supported: Document operations involving multiple records can be performed in a single batch request. An operation in a batch can be get, put, append, or delete, and specifies the operand using a record key, document bin(s), and JSON path.

Simple and Complex JSON Paths in Operations

A JSON path that resolves in a single element in the target document is considered a simple JSON path with a 1-step operation because the operation can be performed in a single server request. Examples of simple JSON paths are any path resolving to a single element such as a scalar type, an object, or an array.

A JSON path that resolves in multiple elements in the target document is considered a complex JSON path requiring a 2-step operation because the operation takes two steps: the first step to resolve all operand elements within the document, and the second step to perform individual operations on the operand elements. Examples of complex JSON paths are paths resolving to multiple elements such as wild-cards (*), all instances in a subtree (..attr), filters (?(@.attr condition), and functions (such as @.length).

Constraints on 2-Step Operations

Multiple operations on the same operands are not allowed with complex JSON paths that must be processed in two steps. Specifically, a batch of records that has the same record appearing more than once is not allowed. The reason being that such operations can yield different results depending on the execution sequence of the multiple client-server interactions.

Note also that the document api allows the same operation to be performed on multiple document bins within a record. However, a batch request may not have the same bin appearing more than once. Such redundant appearance of a bin will amount to performing the same operation multiple times, which does not make sense.

Parallel execution

A batch request can be submitted with a parallel flag. If this flag is set to false, operations on records on a single node are performed sequentially in a single thread. Otherwise they may be executed in parallel. Sequential execution must be specified if the batch operations depend on the specific order of execution. For example, first append an element to an array, and then retrieve the length of the new array.

Batch Operations to Add Multiple Document Records

Below we add multiple documents in a batch operation.

final String SET = "document_api_batch";
// Insert three documents as separate records with keys 1, 2, and 3
Key key1 = new Key(NAMESPACE, SET, 1);
Key key2 = new Key(NAMESPACE, SET, 2);
Key key3 = new Key(NAMESPACE, SET, 3);
// Create the documents.
// 1 
String jsonString = "{\"id\": 1, \"obs\": {\"t1\": 101, \"t2\": 102}, \"arr\": [11, 12] }";
JsonNode jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
documentClient.put(key1, documentBinName, jsonNode);
// 2
jsonString = "{\"id\": 2, \"obs\": {\"t1\": 201, \"t2\": 202}, \"arr\": [21, 22] }";
jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
documentClient.put(key2, documentBinName, jsonNode);
// 3
jsonString = "{\"id\": 3, \"obs\": {\"t1\": 301, \"t2\": 302}, \"arr\": [31, 32] }";
jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
documentClient.put(key3, documentBinName, jsonNode);

printDocuments();

Output:
{arr=[11, 12], id=1, obs={t1=101, t2=102}} {arr=[21, 22], id=2, obs={t1=201, t2=202}} {arr=[31, 32], id=3, obs={t1=301, t2=302}}

Batch Operations with Simple JSON Paths, Non-Unique Records

Below is an example of a batch request where multiple operations are performed on different records, and two operations are performed on one record:

key1:
- put a new element t3=103 in obs object
- key1: delete the second element in arr array
key2: put 0 in the first position of arr array
key3: append 33 to arr array

// batch operations
// Insert
BatchOperation op1 = new PutBatchOperation(
    key1, 
    Collections.singletonList(documentBinName),
    "$.obs.t3",
    "103"
);

// Update
BatchOperation op2 = new PutBatchOperation(
    key2,
    Collections.singletonList(documentBinName),
    "$.arr[0]",
    0
);

// Append
BatchOperation op3 = new AppendBatchOperation(
    key3,
    Collections.singletonList(documentBinName),
    "$.arr",
    33
);

// Delete
BatchOperation op4 = new DeleteBatchOperation(
    key1,
    Collections.singletonList(documentBinName),
    "$.arr[1]"
);

// Collecting operations and running
List<BatchOperation> batchOpsList = new ArrayList<>();
batchOpsList.add(op1);
batchOpsList.add(op2);
batchOpsList.add(op3);
batchOpsList.add(op4);

List<BatchRecord> results = documentClient.batchPerform(batchOpsList, true);

// examine document bins - ensure that:
//    id=1: obs object has a new element t3=103
//          the second element in arr array is removed
//    id=2: arr array has 0 in the first position
//    id=3: arr array has 33 as the last element
printDocuments();

Output:
{arr=[11], id=1, obs={t1=101, t2=102, t3=103}} {arr=[0, 22], id=2, obs={t1=201, t2=202}} {arr=[31, 32, 33], id=3, obs={t1=301, t2=302}}

Batch operations with Complex JSON Paths, Unique Records

In the following example we illustrate JSON paths with wildcard and function.

key1: make all elements of arr array 0, or arr[*] = 0
key2: remove t3 from obs, or obs.t3

// batch operations
// id=1: make all arr elments 0
BatchOperation op1 = new PutBatchOperation(
    key1, 
    Collections.singletonList(documentBinName),
    "$.arr[*]",
    0
);

// id=2: remove t3 from obs
BatchOperation op2 = new DeleteBatchOperation(
    key2,
    Collections.singletonList(documentBinName),
    "$.obs.t3"
);


// collect operations
List<BatchOperation> batchOpsList = new ArrayList<>();
batchOpsList.add(op1);
batchOpsList.add(op2);

// execute
List<BatchRecord> results = documentClient.batchPerform(batchOpsList, true);

// examine document bins - ensure that:
//    id=1: arr elements should be all 0
//    id=2: t3 is removed from obs 
printDocuments();

Output:
{arr=[0], id=1, obs={t1=101, t2=102, t3=103}} {arr=[0, 22], id=2, obs={t1=201, t2=202}} {arr=[31, 32, 33], id=3, obs={t1=301, t2=302}}

Error case: Batch Operations with Complex JSON Paths, Non-Unique Records

Below we submit two operations on the same record with one or more 2-step operations involving a complex JSON path. Note it disallows the request by generating an exception.

// batch operations
// id=1: a 2-step operation with a complex json path
BatchOperation op1 = new PutBatchOperation(
    key1, 
    Collections.singletonList(documentBinName),
    "$.arr[*]",
    0
);

// id=1: another operation 
BatchOperation op2 = new AppendBatchOperation(
    key1, 
    Collections.singletonList(documentBinName),
    "$.arr",
    100
);

// collect operations
List<BatchOperation> batchOpsList = new ArrayList<>();
batchOpsList.add(op1);
batchOpsList.add(op2);

// excute - should produce an exception
try {
    List<BatchRecord> results = documentClient.batchPerform(batchOpsList, true);
}
catch (java.lang.Exception e) {
    System.out.format("Error: %s\n", e);
}

Output:
Error: java.lang.IllegalArgumentException: Multiple two-step operations with the same key are not allowed

JSONPath Queries

Please refer to this introduction to JSONPath: JSONPath, XPATH for JSON.

Syntax

The following table summarizes its syntax.

JSONPath	Description
\$	the root object/element
@	the current object/element
. or []	child operator
..	recursive descent
*	wildcard. All objects/elements regardless their names
[]	subscript operator in array
[,]	alternate names or array indices as a set
[start🔚step]	array slice operator
[?(`<expression>`{=html})]	boolean filter expression
(`<expression>`{=html})	script expression

Use Examples

The following table summarizes its key uses.

JSONPath	Result
$.store.book[*].author	the authors of all books in the store
$..author	all authors
$.store.*	all things in store, which are some books and a red bicycle
$.store..price	the price of everything in the store
$..book[2]	the third book
$..book[(@.length-1)], $..book[-1:]	the last book in order
$..book[0,1], $..book[:2]	the first two books
$..book[?(@.isbn)]	filter all books with isbn number
$..book[?(@.price<10)]	filter all books cheaper than 10
$..*	all members of JSON structure
$..book.length()	the number of books

Code Examples

Below are the code examples from the blog post Aerospike Document API: JSONPath Queries.

Example JSON Document

The example JSON document it uses is stored in file doc_api_example_store.json, and has this json:

{
 "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95,
        "ref": [1,2]
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99,
        "ref": [2,4,16]
      },
      {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99,
        "ref": [1,3,5]
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99,
        "ref": [1,2,7]
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  },
  "expensive": 10
}

Store JSON Document

Read JSON Document from File

String jsonString = readJSONFromAFile("doc_api_example_store.json");
System.out.println("Read JSON doc from file.");

Output:
Read JSON doc from file.

Add JSON Document to Database

// Store the JSON doc in the database and read it back.

// Create a document client via an existing aerospikeClient
AerospikeDocumentClient documentClient = new AerospikeDocumentClient(aerospikeClient);

// Convert JSON string to a JsonNode
JsonNode jsonNode = JsonConverters.convertStringToJsonNode(jsonString);
System.out.println(jsonNode);

// Construct an appropriate key
Key documentKey = new Key(NAMESPACE, SET, "jsonExampleKey");

String documentBinName = "documentBin";
// Add to database
documentClient.put(documentKey, documentBinName, jsonNode);
System.out.println("Stored JSON doc to database.");

Output:
{"store":{"book":[{"category":"reference","author":"Nigel Rees","title":"Sayings of the Century","price":8.95,"ref":[1,2]},{"category":"fiction","author":"Evelyn Waugh","title":"Sword of Honour","price":12.99,"ref":[2,4,16]},{"category":"fiction","author":"Herman Melville","title":"Moby Dick","isbn":"0-553-21311-3","price":8.99,"ref":[1,3,5]},{"category":"fiction","author":"J. R. R. Tolkien","title":"The Lord of the Rings","isbn":"0-395-19395-8","price":22.99,"ref":[1,2,7]}],"bicycle":{"color":"red","price":19.95}},"expensive":10} Stored JSON doc to database.

Query All Subnodes

// Get all products, both books and bicycles
String jsonPath = "$.store.*";
Object objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
{bicycle={color=red, price=19.95}, book=[{ref=[1, 2], category=reference, title=Sayings of the Century, author=Nigel Rees, price=8.95}, {ref=[2, 4, 16], category=fiction, title=Sword of Honour, author=Evelyn Waugh, price=12.99}, {ref=[1, 3, 5], category=fiction, title=Moby Dick, author=Herman Melville, price=8.99, isbn=0-553-21311-3}, {ref=[1, 2, 7], category=fiction, title=The Lord of the Rings, author=J. R. R. Tolkien, price=22.99, isbn=0-395-19395-8}]}

Query Specific Field

// Get the authors of all books
String jsonPath = "$.store.book[*].author";
Object objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
["Nigel Rees","Evelyn Waugh","Herman Melville","J. R. R. Tolkien"]

Modify Field

// 3. Modify the authors of all books to “J.K. Rowling”
// Get the authors of all books
String jsonPath = "$.store.book[*].author";
String jsonObject = "J.K. Rowling";
// Modify the authors of all books to "J.K. Rowling"
documentClient.put(documentKey, documentBinName, jsonPath, jsonObject);
Object objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
["J.K. Rowling","J.K. Rowling","J.K. Rowling","J.K. Rowling"]

Query with Exists Predicate

// 4. Get all the books that have an ISBN number
jsonPath = "$..book[?(@.isbn)]";
objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
[{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"J.K. Rowling","price":8.99,"isbn":"0-553-21311-3"},{"ref":[1,2,7],"category":"fiction","title":"The Lord of the Rings","author":"J.K. Rowling","price":22.99,"isbn":"0-395-19395-8"}]

Query with Boolean Predicate

// 5. Get all the books in store cheaper than 10
jsonPath = "$.store.book[?(@.price < 10)]";
objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"J.K. Rowling","price":8.95},{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"J.K. Rowling","price":8.99,"isbn":"0-553-21311-3"}]

Query with RegEx

// 6. Get all the books matching regex (ignore case)
jsonPath = "$..book[?(@.author =~ /.*ROWLING/i)]";
objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.println(objectFromDB);

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"J.K. Rowling","price":8.95},{"ref":[2,4,16],"category":"fiction","title":"Sword of Honour","author":"J.K. Rowling","price":12.99},{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"J.K. Rowling","price":8.99,"isbn":"0-553-21311-3"},{"ref":[1,2,7],"category":"fiction","title":"The Lord of the Rings","author":"J.K. Rowling","price":22.99,"isbn":"0-395-19395-8"}]

Delete Field

// 7. Delete the price field of every object exists in store
// Get the price of everything
String jsonPath = "$.store..price";
Object objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.format("Before delete: %s\n", objectFromDB);
// Delete the price field of every object exists in the store
documentClient.delete(documentKey, documentBinName, jsonPath);
Object objectFromDB = documentClient.get(documentKey, documentBinName, jsonPath);
System.out.format("After delete: %s\n", objectFromDB);;

Output:
Before delete: [19.95,8.95,12.99,8.99,22.99] After delete: []

More Query and Transformation Examples

Consider the document:

    {arr=[{a=1, b=10}, 
          {a=2, b=20}], 
     id=4, 
     obs={
         c=[100, 200], 
         id=x}
    }

JSON queries:

"$.arr[*].a" -> 
    [1, 2]
    
"$.obs.c[\*]" -> 
    [100, 200]
    
"$..id" -> 
    [4, x]
    
"$..c.length()" -> 
    2

Transformations using JSON queries:

put(“$.arr[*].a”, 0) -> 
    {arr=[{a=0, b=10}, {a=0, b=20}], id=4, obs={c=[100, 200], id=x}}
    
put(“$.obj.c[\*]”, 0) -> 
    {arr=[{a=0, b=10}, {a=0, b=20}], id=4, obs={c=[0, 0], id=x}}
    
put(“$..id”, 0) -> 
    {arr=[{a=0, b=10}, {a=0, b=20}], id=0, obs={c=[0, 0], id=0}}
    
put)”$..a”, 1) -> 
    {arr=[{a=1, b=10}, {a=1, b=20}], id=0, obs={c=[0, 0], id=0}}
    
put("$..c", List.of(11, 111)) -> 
    {arr=[{a=1, b=10}, {a=1, b=20}], id=0, obs={c=[11, 111], id=0}}
    
append("$.obs.c", 0) -> 
    arr=[{a=1, b=10}, {a=1, b=20}], id=0, obs={c=[11, 111, 0], id=0}}

More JSONPath Capabilities and Examples

Restore the original store document for the following examples.

documentClient.put(documentKey, documentBinName, jsonNode);
documentClient.get(documentKey, documentBinName, "$");

Output:
{expensive=10, store={bicycle={color=red, price=19.95}, book=[{ref=[1, 2], category=reference, title=Sayings of the Century, author=Nigel Rees, price=8.95}, {ref=[2, 4, 16], category=fiction, title=Sword of Honour, author=Evelyn Waugh, price=12.99}, {ref=[1, 3, 5], category=fiction, title=Moby Dick, author=Herman Melville, price=8.99, isbn=0-553-21311-3}, {ref=[1, 2, 7], category=fiction, title=The Lord of the Rings, author=J. R. R. Tolkien, price=22.99, isbn=0-395-19395-8}]}}

Functions

JSONPath has functions min, sum, size, length, etc. See the JSONPath repo for a full list.

// length(): find number of references to Moby Dick
jsonPath = "$..store.book[?(@.title == 'Moby Dick')].ref.length()";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
0

// avg(): find average price of books
jsonPath = "$..book..price.avg()";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
13.48

Filter Operators

Operators like in and anyof are supported. See the JSONPath repo for a full list.

// in: books with category in ['reference', 'biography']
jsonPath = "$..book[?(@.category in ['reference', 'biography'])]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"Nigel Rees","price":8.95}]

// in: books that have 2 in their reference list
jsonPath = "$..book[?(2 in @.ref)]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"Nigel Rees","price":8.95},{"ref":[2,4,16],"category":"fiction","title":"Sword of Honour","author":"Evelyn Waugh","price":12.99},{"ref":[1,2,7],"category":"fiction","title":"The Lord of the Rings","author":"J. R. R. Tolkien","price":22.99,"isbn":"0-395-19395-8"}]

// anyof: books whose ref list has any of 1, 3, 7, or 100.
jsonPath = "$..book[?(@.ref anyof [1, 3, 7, 100])]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"Nigel Rees","price":8.95},{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"Herman Melville","price":8.99,"isbn":"0-553-21311-3"},{"ref":[1,2,7],"category":"fiction","title":"The Lord of the Rings","author":"J. R. R. Tolkien","price":22.99,"isbn":"0-395-19395-8"}]

Predicates

// books with price between 10 and 20
jsonPath = "$..book[?(@.price > 10 && @.price < 20)]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[2,4,16],"category":"fiction","title":"Sword of Honour","author":"Evelyn Waugh","price":12.99}]

// comparing with another element
jsonPath = "$..book[?(@.price < $.expensive)]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[1,2],"category":"reference","title":"Sayings of the Century","author":"Nigel Rees","price":8.95},{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"Herman Melville","price":8.99,"isbn":"0-553-21311-3"}]

// composable conditions: books in fiction category with isbn and priced less than 10
jsonPath = "$..book[?(@.category == 'fiction' && @.isbn && @.price < 10 )]";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
[{"ref":[1,3,5],"category":"fiction","title":"Moby Dick","author":"Herman Melville","price":8.99,"isbn":"0-553-21311-3"}]

Try Your Own Path

// substitute your json path
jsonPath = "$";
documentClient.get(documentKey, documentBinName, jsonPath );

Output:
{expensive=10, store={bicycle={color=red, price=19.95}, book=[{ref=[1, 2], category=reference, title=Sayings of the Century, author=Nigel Rees, price=8.95}, {ref=[2, 4, 16], category=fiction, title=Sword of Honour, author=Evelyn Waugh, price=12.99}, {ref=[1, 3, 5], category=fiction, title=Moby Dick, author=Herman Melville, price=8.99, isbn=0-553-21311-3}, {ref=[1, 2, 7], category=fiction, title=The Lord of the Rings, author=J. R. R. Tolkien, price=22.99, isbn=0-395-19395-8}]}}

Cleaning Up

Remove tutorial data and close connection.

//truncateTestData();
//aerospikeClient.close();
//System.out.println("Removed tutorial data and closed server connection.");

Further Exploration and Resources

Here are some links for further exploration.

Resources

Blog posts
- Aerospike Document API
- Aerospike Document API: JSONPath Queries
Related notebooks
Aerospike Developer Hub
- Java Developers Resources
Github repos

Exploring Other Notebooks

Visit Aerospike notebooks repo to run additional Aerospike notebooks. To run a different notebook, download the notebook from the repo to your local machine, and then click on File->Open in the notebook menu, and select Upload.

Aerospike Document API for JSON Documents

For an interactive Jupyter notebook experience:

Introduction​

Prerequisites​

Setup​

Ensure database is running​

Download and Install Additional Components​

Imports​

Define Convenience Functions​

Document API​

Interface​

Example JSON Document​

Store JSON Document​

Iniialize Document Client​

Read JSON Document​

Add JSON Document to Database​

Read​

Update​

Delete​

Batch Operations​

Simple and Complex JSON Paths in Operations​

Constraints on 2-Step Operations​

Parallel execution​

Batch Operations to Add Multiple Document Records​

Batch Operations with Simple JSON Paths, Non-Unique Records​

Batch operations with Complex JSON Paths, Unique Records​

Error case: Batch Operations with Complex JSON Paths, Non-Unique Records​

JSONPath Queries​

Syntax​

Use Examples​

Code Examples​

Example JSON Document​

Store JSON Document​

Read JSON Document from File​

Add JSON Document to Database​

Query All Subnodes​

Query Specific Field​

Modify Field​

Query with Exists Predicate​

Query with Boolean Predicate​

Query with RegEx​

Delete Field​

More Query and Transformation Examples​

More JSONPath Capabilities and Examples​

Functions​

Filter Operators​

Predicates​

Try Your Own Path​

Cleaning Up​

Further Exploration and Resources​

Resources​

Exploring Other Notebooks​

Introduction

Prerequisites

Setup

Ensure database is running

Download and Install Additional Components

Imports

Define Convenience Functions

Document API

Interface

Example JSON Document

Store JSON Document

Iniialize Document Client

Read JSON Document

Add JSON Document to Database

Read

Update

Delete

Batch Operations

Simple and Complex JSON Paths in Operations

Constraints on 2-Step Operations

Parallel execution

Batch Operations to Add Multiple Document Records

Batch Operations with Simple JSON Paths, Non-Unique Records

Batch operations with Complex JSON Paths, Unique Records

Error case: Batch Operations with Complex JSON Paths, Non-Unique Records

JSONPath Queries

Syntax

Use Examples

Code Examples

Example JSON Document

Store JSON Document

Read JSON Document from File

Add JSON Document to Database

Query All Subnodes

Query Specific Field

Modify Field

Query with Exists Predicate

Query with Boolean Predicate

Query with RegEx

Delete Field

More Query and Transformation Examples

More JSONPath Capabilities and Examples

Functions

Filter Operators

Predicates

Try Your Own Path

Cleaning Up

Further Exploration and Resources

Resources

Exploring Other Notebooks