# Configure secondary indexes The [secondary indexes](https://aerospike.com/docs/database/learn/architecture/data-storage/secondary-index/) of an Aerospike namespace can be stored in three different mediums: shared memory (SHMem) by default, Intel® Optane™ Persistent Memory (PMem), or Flash device (on NVMe SSDs). Separate namespaces within the same cluster can use different types of secondary index storage. ::: note Secondary index storage (`sindex-type`) is not configurable in Aerospike Database Community Edition (CE). Its secondary index storage is always volatile process memory, and the cluster node is not able to perform a [fast restart](https://aerospike.com/docs/database/manage/database/fast-start). ::: To specify a secondary index (sindex) storage method, use the namespace configuration parameter [`sindex-type`](https://aerospike.com/docs/database/reference/config#namespace__sindex-type). - The default `sindex-type shmem` stores secondary index metadata in shared memory segments. - `sindex-type pmem` specifies PMem storage for namespace secondary indexes. - `sindex-type flash` specifies NVMe SSD storage for namespace secondary indexes. For sizing information, see [secondary index capacity Planning](https://aerospike.com/docs/database/manage/planning/capacity/secondary-indexes). ::: caution On a [systemd-based OS](https://support.aerospike.com/s/article/The-asd-process-cold-starts-after-a-dirty-exit-which-appears-to-be-clean) you might need to increase **TimeoutSec** from the default of 10 minutes if you have a very large database. This setting is in `/usr/lib/systemd/system/aerospike.service`. This prevents systemd from killing the `asd` process prematurely while the service is shutting down. ::: ## Secondary index in memory By default the namespace secondary index storage type in Aerospike Database Enterprise Edition variants (EE, FE, SE) is shared memory, equivalent to explicitly setting `sindex-type shmem`. As of Database 6.1.0 the use of secondary indexes in memory is compatible with [fast restarts](https://aerospike.com/docs/database/manage/database/fast-start/). ## Secondary index on Flash When the namespace secondary index storage is configured to `sindex-type flash`, Aerospike writes index metadata in multiple files spread across the configured NVMe SSD devices. This requires device partitions to be set up with an appropriate filesystem and mounted. ### Set kernel parameters The following Linux kernel parameters are required in an index on Flash deployment. `enforce-best-practices` verifies that these kernel parameters have the expected values. Terminal window ```bash /proc/sys/vm/dirty_bytes = 16777216 /proc/sys/vm/dirty_background_bytes = 1 /proc/sys/vm/dirty_expire_centisecs = 1 /proc/sys/vm/dirty_writeback_centisecs = 10 ``` - When running as non-root, you must prepare these values before running the Aerospike server. - When running as root, the server configures them automatically. Either way, if these parameters can’t be correctly set manually, or automatically by the server, the node will not be able to start up with a `sindex-type flash` configuration. ::: caution While it is advisable to adjust the kernel’s `min_free_kbytes` parameter in any configuration, it is especially important to do so when storing the primary index on Flash memory (All Flash). The Linux kernel will attempt to use all free space by caching disk writes. With All Flash configuration, this may result in an out of memory (OOM) kill if there isn’t enough free memory left for normal system operations. For this reason, Aerospike recommends setting `min_free_kbytes=1153434` (1.1GB). For more information, see [Best practices for Aerospike and Linux](https://aerospike.com/docs/database/learn/best-practices#min_free_kbytes). ::: ### Prepare and mount Flash devices Aerospike instantiates at least 4 different arena allocations (as files) for each device partition configured for use by the namespace. This helps reduce contention against the same arena, which improves performance during heavy insertion loads. An XFS filesystem is recommended because it has been shown to provide better concurrent access to files than EXT4. Terminal window ```bash sudo mkfs.xfs -f /dev/nvme0 sudo mount /dev/nvme0 /mnt/nvme0 ``` - Using more physical devices will improve performance through increased parallelism of disk IO. - Assigning more partitions per physical device doesn’t necessarily improve performance. ### Subcontext configuration In the [`sindex-type`](https://aerospike.com/docs/database/reference/config#namespace__sindex-type) subcontext define - A [`mount`](https://aerospike.com/docs/database/reference/config#namespace__mount) point for each mounted Flash device. Mount points can be shared across multiple namespaces. - A [`mounts-budget`](https://aerospike.com/docs/database/reference/config#namespace__mounts-budget) (or [`mounts-size-limit`](https://aerospike.com/docs/database/reference/config#namespace__mounts-size-limit) before Database 7.0.0) directive to indicate this namespace’s share of device storage space available across the given mount points. - Ensure the budget is less than or equal to the size of the filesystem. If mount points are not shared between namespaces, then simply specify the total available space. - An optional eviction threshold as a percent of the budget can be defined through [`evict-mounts-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-mounts-pct) (or [`mounts-high-water-pct`](https://aerospike.com/docs/database/reference/config#namespace__mounts-high-water-pct) before Database 7.0.0). ### Example #### Database 7.0.0 and later ```ruby namespace test { sindex-type flash { mount /mnt/nvme0 mount /mnt/nvme1 mount /mnt/nvme2 mount /mnt/nvme3 mounts-budget 1T } } ``` #### Prior to Database 7.0.0 ```ruby namespace test { sindex-type flash { mount /mnt/nvme0 mount /mnt/nvme1 mount /mnt/nvme2 mount /mnt/nvme3 mounts-size-limit 1T } } ``` ## Secondary index in PMem When the namespace secondary index storage is configured to `sindex-type pmem`, Aerospike writes index metadata in multiple files spread across the configured PMem devices. This requires device partitions to be set up with an appropriate filesystem and mounted. Aerospike requires PMem to be accessible using DAX (Direct Access), that is, using block devices such as `/dev/pmem0`: - The NVDIMM regions must be configured as `AppDirect` regions, as in the following example from a machine with a 750-GiB `AppDirect` region: Terminal window ```bash sudo ipmctl show -region SocketID ISetID PersistentMemoryType Capacity FreeCapacity HealthState 0 0x59727f4821b32ccc AppDirect 750.0 GiB 0.0 GiB Healthy ``` - The NVDIMM regions must be turned into `fsdax` namespaces, as in the following example from the same machine: Terminal window ```bash sudo ndctl list [ { "dev":"namespace0.0", "mode":"fsdax", "blockdev":"pmem0", ... } ] ``` ### Filesystem configuration ::: note We recommend an XFS file system because it has been shown to provide better concurrent access to files compared to ext4. ::: The PMem block device must contain a filesystem that is capable of DAX (Direct Access), such as XFS or ext4. On the machine in the above example, this could be accomplished in the usual way: #### Prepare an XFS filesystem Terminal window ```bash sudo mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0 ``` #### Prepare an ext4 filesystem Terminal window ```bash sudo mkfs.ext4 /dev/pmem0 ``` #### Mount the filesystem Finally, the file system must be mounted with the `dax` mount option. The `dax` mount option is important. Without this option, the Linux page cache is involved in all I/O to and from persistent memory, which would drastically reduce performance. In the following example, we use `/mnt/pmem0` as the mount point. Terminal window ```bash sudo mount -o dax /dev/pmem0 /mnt/pmem0 ``` Remember to make the mount persistent to survive system reboots by adding it to `/etc/fstab`. The mount point configuration line can be copied from `/etc/mtab` to `/etc/fstab`. ### Mount point configuration In the [`sindex-type`](https://aerospike.com/docs/database/reference/config#namespace__sindex-type) subcontext define - A [`mount`](https://aerospike.com/docs/database/reference/config#namespace__mount) point for each PMem device. Secondary index metadata will be evenly distributed across all of them. Mount points can be shared across multiple namespaces. - A [`mounts-budget`](https://aerospike.com/docs/database/reference/config#namespace__mounts-budget) (or [`mounts-size-limit`](https://aerospike.com/docs/database/reference/config#namespace__mounts-size-limit) before Database 7.0.0) directive to indicate this namespace’s share of PMem storage space available across the given mount points. - Ensure the budget is less than or equal to the size of the filesystem. If mount points are not shared between namespaces, then simply specify the total available space. - An optional eviction threshold as a percent of the budget can be defined through [`evict-mounts-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-mounts-pct) (or [`mounts-high-water-pct`](https://aerospike.com/docs/database/reference/config#namespace__mounts-high-water-pct) before Database 7.0.0). ### Example The following configuration snippet extends the earlier example and makes all of `/mnt/pmem0` memory (for example, 750 GiB) available to the namespace: #### Database 7.0.0 and later ```ruby namespace test { sindex-type pmem { mount /mnt/pmem0 mounts-budget 750G } } ``` #### Prior to Database 7.0.0 ```ruby namespace test { sindex-type pmem { mount /mnt/pmem0 mounts-size-limit 750G } } ``` ## Manage secondary indexes This section describes using [`asadm`](https://aerospike.com/docs/database/tools/asadm/live-mode/#create-a-secondary-index) for managing secondary indexes. For information on storage consumption of secondary indexes (memory, PMem or SSD, depending on the `sindex-type`) see the [capacity planning guide](https://aerospike.com/docs/database/manage/planning/capacity). Use the [`indexes-memory-budget`](https://aerospike.com/docs/database/reference/config#namespace__indexes-memory-budget) configuration parameter to limit memory usage by all indexes (primary, set, secondary). If `indexes-memory-budget` is exceeded, the namespace goes into `stop-writes` mode and refuses new write commands until memory usage falls below the configured threshold. ::: note Index management commands are run synchronously. When the command is complete, verify the state of the index before using it. Avoid creating several indexes at one time, or creating indexes while nodes are migrating. ::: ### Create a secondary index Use [`manage sindex create`](https://aerospike.com/docs/database/tools/asadm/live-mode/#create-a-secondary-index) to create an index. For example: Terminal window ```bash Admin> enable Admin+> manage sindex create numeric ageidx ns test set employees bin age ``` ### List secondary indexes Use [`show sindex`](https://aerospike.com/docs/database/tools/asadm/live-mode/#show-sindex) to list the secondary indexes: Terminal window ```bash Admin> show sindex ~~~~~~~~Secondary Indexes (2025-05-17 19:18:08 UTC)~~~~~~~~~ Index|Namespace| Set| Bin| Bin| Index|State Name| | | | Type| Type| ageidx |test |employees| age|numeric|default|RW officeidx|test |employees| office|string |default|RW phoneidx |test |employees|phonenum|numeric|default|RW Number of rows: 3 Admin> show sindex like age ~~~~~~~~Secondary Indexes (2025-05-17 19:18:12 UTC)~~~~~~~~~ Index|Namespace| Set| Bin| Bin| Index|State Name| | | | Type| Type| ageidx |test |employees| age|numeric|default|RW Number of rows: 1 ``` ### Get secondary index statistics Use [`show statistics`](https://aerospike.com/docs/database/tools/asadm/live-mode/#statistics) to display secondary index statistics: Terminal window ```bash Admin> show statistics sindex like bval ~test employees ageidx SIndex Statistics (2025-05-17 20:13:16 UTC)~ Node |1.0.0.127.in-addr.arpa:3100 entries_per_bval|167 Number of rows: 2 ~test employees officeidx SIndex Statistics (2025-05-17 20:13:16 UTC)~ Node |1.0.0.127.in-addr.arpa:3100 entries_per_bval|1429 Number of rows: 2 ~test employees phoneidx SIndex Statistics (2025-05-17 20:13:16 UTC)~ Node |1.0.0.127.in-addr.arpa:3100 entries_per_bval|1 Number of rows: 2 ``` Alternatively, use the [`sindex-stat`](https://aerospike.com/docs/database/reference/info#sindex-stat) info command directly: Terminal window ```bash Admin+> asinfo -l -v sindex-stat:namespace=test;indexname=officeidx 1.0.0.127.in-addr.arpa:3100 (127.0.0.1) returned: entries=10000 used_bytes=16777216 entries_per_bval=1429 entries_per_rec=1 load_pct=100 load_time=30 stat_gc_recs=0 ``` ### Drop a secondary index Use [`manage sindex delete`](https://aerospike.com/docs/database/tools/asadm/live-mode/#drop-a-secondary-index) to drop a secondary index. For example: Terminal window ```bash Admin> enable Admin+> manage sindex delete ageidx ns test set employees ``` ## Enable performance histograms ::: note Deprecated in Database 6.0.0 and later. ::: To enable writing secondary index performance histograms to the log: Terminal window ```bash asadm -e 'enable; "asinfo -v "sindex-histogram:ns=NAMESPACE;[set=demo];indexname=INDEX_NAME;enable=true"' ``` To disable writing secondary index performance histograms to the log: Terminal window ```bash asadm -e 'enable; asinfo -v "sindex-histogram:ns=NAMESPACE;[set=demo];indexname=INDEX_NAME;enable=false"' ``` ## Related documentation - [Queries](https://aerospike.com/docs/develop/learn/queries/) - [Secondary index system limits](https://aerospike.com/docs/database/reference/limitations#secondary-index)