Skip to main content
We are excited to be a part of AWS re:Invent 2024. Visit us at booth #1844 in Las Vegas.More info
Loading

Vector data in Aerospike​

Overview​

This page describes how Aerospike Vector Search (AVS) stores data in the Aerospike Database (ASDB).

tip

Your data requirements may differ from your index storage needs. For more information, see Configure Aerospike Database for AVS.

Vector records​

A record in AVS is equivalent to a record in ASDB, but the AVS record has one or more additional bins (analogous to one or more additional columns). When developing, you will interact with AVS using simple insert, upsert, and delete calls.

The following AQL call illustrates a vector record in ASDB:

aql> select * from test.quote-data limit 2

[
    [
        {
          "quote_id": 185,
          "quote": "Look after my heart - I've left it with you.",
          "author": "Stephenie Meyer, Eclipse",
          "tags": [
            "bella",
            " edward",
            " love",
            " twilight-saga"
          ],
          "quote_embedding": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 0C 0A 80 0C 37 C3 8F BD 7F 46 96 BA 10 02 3A 3C 30 B5 88 3C CB A2 FD 3A 1C BF 28 3D F6 A4 30 BC 31 DA EF BD DC D6 05 3E 91 C4 0E BE DD 1C 2C BD 1A 1F A8 3B E7 59 B4 BD E8 C9 A7 BC 18 84 95 BC 94 5D BA 3C EE 25 79 3D 18 A2 5B BD 90 6C A2 BD D4 0E 95 3D 01 2F B3 BC 12 12 8B 3D 34 BE BB BC AD E6 02 3D B1 6E 86 3D AE 8B 8D 3B DD 09 08 3D CD 65 52 BD 5F 70 1A BD 62 E8 73 BC A6 10 01 3D ",
          "_avs_uk_": 185
        },
        {
          "quote_id": 359,
          "quote": "I am not an angel,' I asserted; 'and I will not be one till I die: I will be myself. Mr. Rochester, you must neither expect nor exact anything celestial of me - for you will not get it, any more than I shall get it of you: which I do not at all anticipate.",
          "author": "Charlotte Brontë, Jane Eyre",
          "tags": [
            "character",
            " empowerment",
            " flaws",
            " freedom",
          ],
          "quote_embedding": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 0C 0A 80 0C D7 E6 37 BC 8D C5 26 BB B2 07 52 BB 31 A8 23 3D B8 3A 28 BC A0 8A A8 BB FC AF 83 3D 0F 93 98 BD C4 A4 DF 3D 83 BC A8 BD 1E 7C 37 3D D7 5F AB 3B ED D3 AA BC B1 D6 A5 BD 4D D2 0A BC 2B CE 53 3D 8E 4F BB 3B FC 66 21 BC 93 B2 44 BD 35 9C BF 3D 0F 7D A4 BC A9 A9 F0 3B 29 92 D5 BD 03 C0 55 3C DE F5 3D BC 68 B6 BA BC 33 1C 55 BC 08 C3 77 BB C5 D2 05 3C 94 40 A1 3C 49 3C 3A 3C ",
          "_avs_uk_": 359
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]
caution

While other record properties can be edited directly, vector data is encoded in Aerospike and should not be edited directly with the native Aerospike clients or using AQL. We recommend that you upsert changes using an AVS client.

Index records​

In addition to vector records, your HNSW index contains additional records that include lists of neighbors along with the vector used for calculating distances with the search vector.

The following AQL call illustrates an index record in ASDB:

aql> select * from test.quote-semantic-search limit 2

[
    [
        {
          "layer": 0,
          "highestLayer": 0,
          "id": "0A 04 74 65 73 74 12 0A 71 75 6F 74 65 2D 64 61 74 61 1A 14 12 F0 F3 50 F1 EE 56 C8 97 80 1E 99 9D DA AE 9D 93 A8 2C 97",
          "vectorDigest": "57 DD 58 32 48 24 91 F4 C9 CD D4 F7 A0 BD 0F 9F DC C6 1F 55",
          "vector": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 0C 0A 80 0C 37 C3 8F BD 7F 46 96 BA 10 02 3A 3C 30 B5 88 3C CB A2 FD 3A 1C BF 28 3D F6 A4 30 BC 31 DA EF BD DC D6 05 3E 91 C4 0E BE DD 1C 2C BD 1A 1F A8 3B E7 59 B4 BD E8 C9 A7 BC 18 84 95 BC 94 5D BA 3C EE 25 79 3D 18 A2 5B BD 90 6C A2 BD D4 0E 95 3D 01 2F B3 BC 12 12 8B 3D 34 BE BB BC AD E6 02 3D B1 6E 86 3D AE 8B 8D 3B DD 09 08 3D CD 65 52 BD 5F 70 1A BD 62 E8 73 BC A6 10 01 3D 35 67 CF ",
          "neighbors": [
            "2B B9 04 63 C7 16 32 C2 60 7C B6 63 84 39 A4 3C 95 91 7A 38",
            "0B C2 25 48 4A 4D D6 09 FE 47 5D 88 11 58 77 26 7F AA 8E B3",
            "19 5E 2D 20 54 81 94 61 8B A5 51 12 0E 28 11 79 00 33 F5 EB",
            "1A AF F0 C6 10 7B 75 ED 66 51 1F 2A 94 E9 8A 2E 4B 50 6D 88",
            "55 3F 8A 43 FD EB D7 D5 9D 5B 9B 4C CD 41 4C 42 84 E1 6A 1F",
            "D4 B1 14 5D 29 89 7E 05 97 BD 91 A2 0F 43 53 9F F9 B0 E8 99",
            "8E FC 64 C0 DF 8C C2 CC E0 54 18 89 C6 24 EF 7A 62 07 AA A5",
            "F6 30 2B 5A 1D 15 AB BE 13 AF ED 8A 0C B5 1C 99 39 F9 A4 D0",
            "2D C2 E1 80 79 51 49 5B F9 AE 4B 1C CB DA 26 CB CB 41 D7 AA",
            "A6 69 27 83 63 C9 34 49 C1 A6 79 47 0F 28 A7 20 56 F3 21 6F",
            "58 46 6B 84 D6 99 B1 33 9D B7 EF 12 EB 56 F0 03 39 0D 48 52",
            "2F BB A1 1A 82 84 EC 17 F3 4F 55 71 03 E3 54 46 6C 65 DC 99",
            "CF 11 4B CE 6B FE 34 8F 93 4F E0 37 0D B0 1D D6 80 76 AF 27",
            "3E 86 A3 0E 4C 02 70 A0 A8 EB D4 91 E7 15 6E D6 D4 68 1D 60",
            "FB ED 32 07 76 44 A9 52 0B D0 46 1C 13 4D BC 9D 48 59 1F FB",
            "4E D2 AA 34 39 E2 A3 D8 93 7F 7B 37 E5 17 A2 56 08 D5 76 16"
          ],
          "lut": 113113235523371008,
          "indexIdWithTs": "test:quote-semantic-search:113113217615396864",
          "isMainIndex": true
        },
        {
          "layer": 0,
          "highestLayer": 0,
          "id": "0A 04 74 65 73 74 12 0A 71 75 6F 74 65 2D 64 61 74 61 1A 14 0B 50 DC 8F BC F8 94 6A 85 92 39 DB 37 73 E6 86 D3 29 73 6F",
          "vectorDigest": "F6 BA A8 77 A3 13 2C 53 59 E3 BA 5E BE 3C D0 46 B4 ED 68 AF",
          "vector": "63 2E 61 65 72 6F 2E 76 65 63 2E 6D 61 67 69 63 12 83 0C 0A 80 0C D7 E6 37 BC 8D C5 26 BB B2 07 52 BB 31 A8 23 3D B8 3A 28 BC A0 8A A8 BB FC AF 83 3D 0F 93 98 BD C4 A4 DF 3D 83 BC A8 BD 1E 7C 37 3D D7 5F AB 3B ED D3 AA BC B1 D6 A5 BD 4D D2 0A BC 2B CE 53 3D 8E 4F BB 3B FC 66 21 BC 93 B2 44 BD 35 9C BF 3D 0F 7D A4 BC A9 A9 F0 3B 29 92 D5 BD 03 C0 55 3C DE F5 3D BC 68 B6 BA BC 33 1C 55 BC 08 C3 77 BB C5 D2 05 3C 94 40 A1 3C 49 3C 3A 3C 76 56 20 ",
          "neighbors": [
            "2B 57 A5 BA 82 2B 3E 3D 2E EA 43 F6 92 A9 85 27 75 B9 29 FC",
            "AF 80 63 C6 68 A9 89 B5 E7 B9 FE 37 D8 2C 5F BE 14 18 AB 5A",
            "6D AE 2B F4 30 62 0F C4 BD 3A E8 BB 5B 2C 98 55 4C 99 E4 C4",
            "31 3A 76 3D D0 AF 86 A1 31 A2 16 DD 29 D1 D4 B6 02 43 B3 7D",
            "D0 0E E9 99 B1 1C AF A8 D0 C1 69 B1 37 39 F7 C9 2C 8F 91 02",
            "56 F4 9F 97 9F D0 78 BC 4E A2 56 FC C7 15 04 A7 FA 33 2C 72",
            "95 44 42 91 E3 57 FF 6F D3 0D 52 2F 74 BD 59 D7 F2 2C D2 15",
            "72 26 93 9B BF FE 4E 17 8F CC B8 B9 14 AB 16 F1 59 42 21 73",
            "AD 6B 74 64 6C AD 41 B9 DD 28 90 4C D4 E0 41 58 54 F9 37 F4",
            "88 77 9B F0 27 08 19 44 77 9F 38 B4 20 AD 86 6A 58 9B D6 B0",
            "A0 64 6F 33 E5 96 64 BA DE 8D 17 3A 33 53 44 A5 7B ED 79 9D",
            "E6 FC DE 23 6E 0D 39 54 9C 4E EA 7A 29 C6 54 ED 61 6D AE A6",
            "94 36 DE CC 9F D0 A8 67 0E AC C1 8F B6 71 15 17 24 81 E7 C5",
            "59 9A EF 46 15 84 C5 E5 23 BD AF EE 9A E3 2D 62 23 79 B9 F7"
          ],
          "lut": 113113233563320320,
          "indexIdWithTs": "test:quote-semantic-search:113113217615396864",
          "isMainIndex": true
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]



danger

Do not edit or change any values in an index record. Changing an index record directly will affect search results. You need to delete, or update a record if you wish to re-index the record.

Index metadata records​

AVS maintains metadata on each of your indexes, which should be configured in a unique namespace in your Aerospike cluster. Even with a large number of indexes, the data in this namespace is not likely to grow very large as it only contains details about the name and properties of your indexes.

aql> select * from avs-meta limit 2

[
    [
        {
          "PK": "ptrial-aql-222af6af1388",
          "index-log": "0A 54 08 80 80 FC F3 DC F7 F6 C8 01 12 48 0A 46 0A 13 0A 04 74 65 73 74 12 0B 62 61 73 69 63 5F 69 6E 64 65 78 18 02 2A 06 76 65 63 74 6F 72 3A 10 08 10 10 64 18 64 22 08 08 A0 8D 06 10 B0 EA 01 4A 13 0A 04 74 65 73 74 12 0B 62 61 73 69 63 5F 69 6E 64 65 78 0A 80 01 08 80 80 F8 9A F4 FB F6 C8 01 12 74 0A 72 0A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68 18 80 03 20 01 2A 0F 71 75 6F 74 65 5F 65 6D 62 65 64 64 69 6E 67 32 0A 71 75 6F 74 65 2D 64 61 74 61 3A 10 08 10 10 64 18 64 22 08 08 A0 8D 06 10 B0 EA 01 4A 1D 0A 04 74 65 73 74 12 15 71 75 6F 74 65 2D 73 65 6D 61 6E 74 69 63 2D 73 65 61 72 63 68",
          "index-log-ts": 113113266746163200
        },
        {
          "PK": "ptrial-aql",
          "hlc": 113113266656706560
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]
danger

Do not edit any of the data in this namespace directly. The data is encoded, and making changes will result in errors when you attempt to use your indexes.

Aerospike data layout​

tip

Review Aerospike Terminology to understand how AVS takes advantage of ASDB concepts when laying out relevant data storage.

Data is composed by adopting the following ASDB concepts:

  • AVS Meta Namespace: A specific meta namespace is reserved for storing meta information about indexes used by AVS. This takes up a small amount of storage and a recommended configuration is provided in the Storage Configuration guide.

  • Namespaces: AVS supports up to 31 namespaces of separate data. Configuring new namespaces requires updating each ASDB node. In a namespace, you define characteristics about your storage, consistency, and availability of your data. In some cases, it may make sense to use a unique namespace for your index.

  • Sets: Sets provide an organizational tool for monitoring and setting quota on data used within a namespace. By default, index data is stored in a set based on your index name.

  • Vector Bin: An index is built on top of a specified field in a vector record which is stored in a specific bin. A bin is similar to a column in a relational database.