Vector data in Aerospike

Overview

This page describes how Aerospike Vector Search (AVS) stores data in the Aerospike Database (ASDB).

tip

Your data requirements may differ from your index storage needs. For more information, see Configure Aerospike Database for AVS.

Vector records

A record in AVS is equivalent to a record in ASDB, but the AVS record has one or more additional bins (analogous to one or more additional columns). When developing, you will interact with AVS using simple insert, upsert, and delete calls.

The following AQL call illustrates a vector record in ASDB:

aql> select * from test.image-data limit 2

[
    [
        {
          "image_id": "GOPR0671.jpg",
          "image_name": "GOPR0671.jpg",
          "image_embedding": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 10 0A 80 10 C0 11 3F 62 32 26 BF 88 C6 17 BE 9B C7 B4 BD 3C BB 2C BE 7E AD 89 BE 7F FE C2 BE EF E1 22 BE A3 BE F7 BD 20 E6 5B BE 30 15 2C 3D 20 41 25 BC 4A C5 B3 3D EF 68 86 3E 9C BF 32 3E 08 A9 89 BE 07 4F 04 BF E6 33 73 3E 71 ",
          "relative_path": "GOPR0671.jpg",
          "_avs_uk_": "GOPR0671.jpg"
        },
        {
          "image_id": "GOPR0886.jpg",
          "image_name": "GOPR0886.jpg",
          "image_embedding": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 10 0A 80 10 28 5E 1B 3D 90 84 2F BC 64 1E D3 BE E5 51 05 BE 9B 45 59 3E CC BB 95 BE B6 C4 27 3F D3 E9 F8 3E E4 BC 57 BE DB F0 83 3E 50 AB 85 3C 08 DF 8A BD 9A D8 A2 3D C7 0A 0A 3E 10 25 AC 3C 58 03 46 3D 72 D3 7A 3E 1C D7 73 3E",
          "relative_path": "GOPR0886.jpg",
          "_avs_uk_": "GOPR0886.jpg"
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

caution

You can edit a vector record directly, but this may affect search results. Aerospike recommends that you upsert changes using an AVS client.

Index records

In addition to vector records, your HNSW index contains additional records that include lists of neighbors along with the vector used for calculating distances with the search vector.

The following AQL call illustrates an index record in ASDB:

aql> select * from test.prism-image-search limit 2

[
    [
        {
          "layer": 0,
          "id": "0A 04 74 65 73 74 12 0A 69 6D 61 67 65 2D 64 61 74 61 1A 14 45 C1 99 8F 6E 2B CE 32 4D 74 38 C3 BA D8 C0 C9 BD 77 A7 64",
          "vectorDigest": "D9 08 A9 8C F8 59 CD CB 9F 45 9A 7E 8B C5 C4 50 E9 C5 46 C7",
          "vector": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 10 0A 80 10 28 5E 1B 3D 90 84 2F BC 64 1E D3 BE E5 51 05 BE 9B 45 59 3E CC BB 95 BE B6 C4 27 3F D3 E9 F8 3E E4 BC 57 BE DB F0 83 3E 50 AB 85 3C 08 DF C9 BE A6 84 E9 3E 3C 66 5C BE EC 3D 39 3E 6E 8F C8 3D 0A 0A 3E 10 25 AC 3C 58 03 46 3D 72 D3 7A 3E 1C D7 73 3E",
          "neighbors": [
            "43 82 A6 49 17 37 95 01 1A C9 98 43 A4 CF C8 5F 1B B0 FC 3A",
            "43 B6 F7 45 BB A2 8E FD DD 92 35 DD 81 31 95 31 BF 3A 52 46",
            "71 8A E2 42 C2 9F 45 22 D6 A8 85 DD 1F BB FB A1 74 AF 5D E1"
          ],
          "lut": 112564816309846016,
          "indexIdWithTs": "test:prism-image-search:112564802777513984",
          "isMainIndex": true
        },
        {
          "layer": 0,
          "id": "0A 04 74 65 73 74 12 0A 69 6D 61 67 65 2D 64 61 74 61 1A 14 3C C0 D3 8F 13 17 FB 49 E8 66 56 85 CD 4C 41 03 98 31 DF C7",
          "vectorDigest": "13 24 95 7B 2D A9 1A 8B 07 3C 72 12 E0 8C 7B 38 28 C5 14 DA",
          "vector": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 10 0A 80 10 C0 E6 AA 3D 10 82 92 3E 20 0D 0A 3C B8 A9 0B 3E 8C 0F CB BC 6D 91 19 BE 78 45 93 3E 17 34 61 3E E7 57 93 3D DC 0E 84 3E 0D 64 40 3F 60 10 8E BE 80 29 BD BE C1 66 AA BD AA 05 D4 3E DC C8 F0 BC A1 F5 3C 00 D3 1F BD 7C C1 1B BE 30 A3 FA BB 30 31 41 BC",
          "neighbors": [
            "F2 01 B4 6F 02 A6 9B 9C 2C 2D CA 57 41 03 A9 1F BB E1 D0 D3",
            "ED 8C 32 27 9E 18 F1 97 E8 80 10 F2 1A 48 C5 21 B4 8E 45 E5",
            "73 2F 36 57 11 8B 4B AF FF EC 01 0B 67 CC 27 F0 44 96 9B D1",
            "24 26 29 4D 8D CF 07 19 30 0B 7A 60 F4 A3 5D AC 2D 91 85 94",
            "18 B8 1F AC 18 9C 88 C7 9F 50 51 29 1E 2D 0A BB 81 30 07 A3"
          ],
          "lut": 112564814372601856,
          "indexIdWithTs": "test:prism-image-search:112564802777513984",
          "isMainIndex": true
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

caution

Do not edit or change any values in an index record. Changing an index record directly will affect search results. You need to delete, or update a record if you wish to re-index the record.

Index metadata records

AVS maintains metadata on each of your indexes, which should be configured in a unique namespace in your Aerospike cluster. Even with a large number of indexes, the data in this namespace is not likely to grow very large as it only contains details about the name and properties of your indexes.

aql> select * from avs-meta

[
    [
        {
          "PK": "cluster-name",
          "index-log": "0A 82 01 08 80 80 A4 BA E4 CA 8D C8 01 12 76 0A 74 0A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68 18 80 03 20 01 2A 0F 71 75 6F 74 65 5F 65 6D 62 65 64 64 69 6E 67 32 0A 71 75 6F 74 65 2D 64 61 74 61 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 4A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68 0A 7C 08 80 80 FC EB F9 C8 92 C8 01 12 70 0A 6E 0A 1A 0A 04 74 65 73 74 12 12 70 72 69 73 6D 2D 69 6D 61 67 65 2D 73 65 61 72 63 68 18 80 04 20 01 2A 0F 69 6D 61 67 65 5F 65 6D 62 65 64 64 69 6E 67 32 0A 69 6D 61 67 65 2D 64 61 74 61 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 4A 1A 0A 04 74 65 73 74 12 12 70 72 69 73 6D 2D 69 6D 61 67 65 2D 73 65 61 72 63 68 0A B4 01 08 80 80 B8 FC 8F CC 92 C8 01 12 A7 01 0A A4 01 0A 1A 0A 04 74 65 73 74 12 12 71 75 6F 74 65 2D 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32 18 80 03 20 01 2A 06 76 65 63 74 6F 72 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 42 0A 0A 03 64 69 6D 12 03 33 38 34 42 12 0A 04 64 61 74 65 12 0A 30 35 2F 30 34 2F 32 30 32 34 42 15 0A 05 6D 6F 64 65 6C 12 0C 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32 42 12 0A 08 64 69 73 74 61 6E 63 65 12 06 63 6F 73 69 6E 65 4A 1A 0A 04 74 65 73 74 12 12 71 75 6F 74 65 2D 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32",
          "index-log-ts": 112678945501609984
        },
        {
          "PK": "cluster-name-22aa699d1388",
          "index-log": "0A 82 01 08 80 80 A4 BA E4 CA 8D C8 01 12 76 0A 74 0A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68 18 80 03 20 01 2A 0F 71 75 6F 74 65 5F 65 6D 62 65 64 64 69 6E 67 32 0A 71 75 6F 74 65 2D 64 61 74 61 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 4A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68 0A 7C 08 80 80 FC EB F9 C8 92 C8 01 12 70 0A 6E 0A 1A 0A 04 74 65 73 74 12 12 70 72 69 73 6D 2D 69 6D 61 67 65 2D 73 65 61 72 63 68 18 80 04 20 01 2A 0F 69 6D 61 67 65 5F 65 6D 62 65 64 64 69 6E 67 32 0A 69 6D 61 67 65 2D 64 61 74 61 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 4A 1A 0A 04 74 65 73 74 12 12 70 72 69 73 6D 2D 69 6D 61 67 65 2D 73 65 61 72 63 68 0A B4 01 08 80 80 B8 FC 8F CC 92 C8 01 12 A7 01 0A A4 01 0A 1A 0A 04 74 65 73 74 12 12 71 75 6F 74 65 2D 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32 18 80 03 20 01 2A 06 76 65 63 74 6F 72 3A 12 08 10 10 64 18 64 22 0A 08 A0 8D 06 10 B0 EA 01 18 00 42 0A 0A 03 64 69 6D 12 03 33 38 34 42 12 0A 04 64 61 74 65 12 0A 30 35 2F 30 34 2F 32 30 32 34 42 15 0A 05 6D 6F 64 65 6C 12 0C 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32 42 12 0A 08 64 69 73 74 61 6E 63 65 12 06 63 6F 73 69 6E 65 4A 1A 0A 04 74 65 73 74 12 12 71 75 6F 74 65 2D 6D 69 6E 69 4C 4D 2D 4C 36 2D 76 32",
          "index-log-ts": 112678945501544448
        },
        {
          "PK": "cluster-name",
          "hlc": 112678945488371712
        },
        {
          "PK": "cluster-name-22aa699d1388",
          "hlc": 112678945488371712
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

caution

Do not edit any of the data in this namespace directly. The data is encoded and making changes will result in errors when you attempt to use your indexes.

Aerospike data layout

tip

Review Aerospike Terminology to understand how AVS takes advantage of ASDB concepts when laying out relevant data storage.

Data is composed by adopting the following ASDB concepts:

AVS Meta Namespace: A specific meta namespace is reserved for storing meta information about indexes used by AVS. This takes up a small amount of storage and a recommended configuration is provided in the Storage Configuration guide.
Namespaces: AVS supports up to 31 namespaces of separate data. Configuring new namespaces requires updating each ASDB node. In a namespace, you define characteristics about your storage, consistency, and availability of your data. In some cases, it may make sense to use a unique namespace for your index.
Sets: Sets provide an organizational tool for monitoring and setting quota on data used within a namespace. By default, index data is stored in a set based on your index name.
Vector Bin: An index is built on top of a specified field in a vector record which is stored in a specific bin. A bin is similar to a column in a relational database.

Overview​

Vector records​

Index records​

Index metadata records​

Aerospike data layout​

Overview

Vector records

Index records

Index metadata records

Aerospike data layout