---
title: "Monitor index memory allocation"
description: "Read the Aerospike index allocation, used, and tail metrics together to explain process RSS, predict arena growth, and interpret deletes."
---

# Monitor index memory allocation

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

Each namespace stores its primary index, secondary index, and set index in a separate memory pool, called an _arena_. Each arena grows in fixed-size chunks, called _stages_, and reserves a whole stage at a time. Aerospike fills each stage from start to end, and the _allocation point_ is the position where the next new entry goes. Aerospike Database 8.2.0 reports three figures for each arena: allocated bytes for stages reserved, used bytes for live index elements, and tail bytes for unused space in the newest stage. Use them together to explain the `asd` process footprint and to predict when an arena reserves another stage. The gap between allocated and used memory can be several GiB.

**Applies to:** Aerospike Database 8.2.0 and later. Allocation metrics for the primary and secondary indexes require Aerospike Database Enterprise Edition, because [`index-type`](https://aerospike.com/docs/database/reference/config#namespace__index-type) and [`sindex-type`](https://aerospike.com/docs/database/reference/config#namespace__sindex-type) are configurable only in Enterprise Edition. `set_index_alloc_bytes`, [`process_rss_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__process_rss_bytes), and the control group (cgroup) metrics are available in both editions.

**Audience:** Operators and SREs sizing nodes, setting memory alerts, or investigating `asd` memory use against a configured budget.

## The three numbers

For each index arena, the server reports up to three values.

| Value | Metric | Meaning |
| --- | --- | --- |
| Allocated | `*_alloc_bytes` | Total memory the arena has reserved: number of stages multiplied by stage size. |
| Used | `*_used_bytes` | Memory holding live index elements. |
| Tail | `*_tail_bytes` | Unused space in the newest stage: how much can still be allocated before another stage is reserved. |

Record deletion frees space behind the allocation point. Calculate that reusable space as:

```text
reusable space = alloc - used - tail
```

The following diagram shows where each value lives in an arena with three stages.

```text
stage 0            stage 1            stage 2 (newest)

+-----------------+-----------------+-----------------+

| live | reusable | live | reusable | live |   | tail |

+-----------------+-----------------+-----------------+

                                          ^

                                          allocation point

 <----------- alloc = stage count * stage size ------->
```

Each new stage is reserved after the previous stage is full. Earlier stages sit entirely behind the allocation point, so only the newest stage has a tail.

## Metrics by arena

| Arena | Allocated | Used | Tail |
| --- | --- | --- | --- |
| Primary index, `shmem` | [`index_shmem_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_shmem_alloc_bytes) | [`index_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_used_bytes) | [`index_shmem_tail_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_shmem_tail_bytes) |
| Primary index, `pmem` | [`index_pmem_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_pmem_alloc_bytes) | `index_used_bytes` | [`index_pmem_tail_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_pmem_tail_bytes) |
| Primary index, `flash` | [`index_flash_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__index_flash_alloc_bytes) | `index_used_bytes` | Not reported |
| Secondary index, `shmem` | [`sindex_shmem_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_shmem_alloc_bytes) | [`sindex_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_used_bytes) | [`sindex_shmem_tail_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_shmem_tail_bytes) |
| Secondary index, `pmem` | [`sindex_pmem_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_pmem_alloc_bytes) | `sindex_used_bytes` | [`sindex_pmem_tail_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_pmem_tail_bytes) |
| Secondary index, `flash` | [`sindex_flash_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_flash_alloc_bytes) | `sindex_used_bytes` | [`sindex_flash_tail_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__sindex_flash_tail_bytes) |
| Set index | [`set_index_alloc_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__set_index_alloc_bytes) | [`set_index_used_bytes`](https://aerospike.com/docs/database/reference/metrics#namespace__set_index_used_bytes) | Not reported |

A namespace reports the allocation metrics that match its configured backing type. For example, `index-type shmem` reports `index_shmem_alloc_bytes`.

Flash primary index and set index allocate as follows:

-   **Flash primary index.** A flash-backed primary index allocates by 4 KiB chunk within a stage. `index_flash_alloc_bytes` reports the sum of those allocated chunks, so it is a different quantity from the `shmem` and `pmem` stage-footprint metrics. Remaining unused space is spread across chunks. The secondary index arena allocates from one allocation point, so a flash-backed secondary index reports a tail.
-   **Set index.** A namespace holds one set-index arena per data-bearing partition tree per indexed set, and each arena has its own allocation point. `set_index_alloc_bytes` already includes a 4 KiB stage floor per arena, and remaining per-arena unused space is small next to that floor.

## How the budget relates to allocation

Eviction and stop-writes act on used bytes. The eviction supervisor (`nsup`) compares the combined RAM index usage against [`indexes-memory-budget`](https://aerospike.com/docs/database/reference/config#namespace__indexes-memory-budget) using the same numerator as [`indexes_memory_used_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__indexes_memory_used_pct), and [`evict-indexes-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-indexes-memory-pct) acts on that value.

`*_alloc_bytes` reports the larger stage footprint. That allocated figure can sit above `indexes-memory-budget` while `indexes_memory_used_pct` remains below 100, because the budget is evaluated against used bytes.

## Predict arena growth from the tail

Index stages default to 1 GiB ([`index-stage-size`](https://aerospike.com/docs/database/reference/config#namespace__index-stage-size)), so index growth happens in 1 GiB steps. The tail shows how much of the current stage remains.

-   **A tail above `0` means the current stage still has unused space.** Allocation at the allocation point consumes the tail, and the arena reserves another stage only after the current stage is full. A namespace with 2 GiB of primary index tail absorbs 2 GiB of new index elements, roughly 33 million records at 64 bytes each, before growing.
-   **A tail of `0` means the next allocation may reuse deleted space or reserve a new stage.** The primary index arena keeps 64 independent free lists. Allocation selects one in round-robin order, and frees return elements to a list determined by element ID, so an allocation can find its list empty and reserve a new stage while the other 63 lists still hold reusable elements.

Read the two values together:

| Tail | Calculated reusable space (`alloc - used - tail`) | Reading | Action |
| --- | --- | --- | --- |
| Trending to `0`, used still climbing | Any | Another stage is imminent | Provision for `alloc` plus one `index-stage-size` |
| Flat | Growing | Deletes are landing and being recycled | None. The footprint stays at the highest level reached |
| Near a full stage | Any | A stage was just reserved | Roughly 16.7 million records of capacity added per 1 GiB stage |
| `0` | Large | Capacity exists, but an individual insert can miss its free list | Alert. Other measures can still look fine |

### How reusable the freed space is differs by arena

Freed space is reused differently in each arena.

| Arena | Free-list structure | When reusable space prevents a new stage |
| --- | --- | --- |
| Secondary index | One free list, one allocation point | Always. Reusable space is interchangeable with tail, so the derived number is usable free space. |
| Primary index | 64 free lists; allocation selects round-robin, frees return by element ID | While the tail is above `0`, reusable space absorbs bulk growth. After the tail reaches `0`, an individual allocation can miss its free list. |
| Set index | One free list per arena, up to one arena per partition tree per indexed set | Within one partition tree and set. A delete in partition 7 creates free space for inserts into that same tree. |

### Accuracy of the calculated reusable space value

For the primary index, `reusable space = alloc - used - tail` overstates true space freed by deletes slightly:

-   The allocation point advances in batches of 512 elements, and the remainder of each batch is pushed onto a free list before it is handed out. Across 64 free lists this accounts for at most 64 × 512 × 64 bytes, or about 2 MiB per namespace, which is negligible against a 1 GiB stage.
-   Element 0 is reserved and never allocated, so it counts as one element of reusable space.

For the secondary index arena, use `reusable space = alloc - used - tail` as an estimate. The arena reserves element zero (about 4 KiB), which the formula counts as reusable space, and `alloc`, `used`, and `tail` are independent unlocked snapshots.

## Memory after deletes

After a mass delete against the primary index, `index_{shmem,pmem}_alloc_bytes` stays at its highest level reached. Freed elements go onto free lists and remain available for reuse up to the previous used value.

The three arenas free space at different times:

| Arena | How space is freed | Consequence for the metric |
| --- | --- | --- |
| Primary index | Freed elements go onto one of 64 free lists | `index_{shmem,pmem}_alloc_bytes` only increases and stays at the highest level reached. `index_flash_alloc_bytes` can decrease when partition trees are destroyed and chunks are freed. `index_*_tail_bytes` only shrinks between stage additions for shmem and pmem, because the allocation point only moves forward |
| Secondary index | Every stage is released when the last element is freed | `sindex_*_alloc_bytes` can drop to `0` in one step, and `sindex_*_tail_bytes` goes to `0` with it |
| Set index | Stages return when the tree is destroyed, which happens when the set index is removed or the partition stops holding data | `set_index_alloc_bytes` follows partition ownership in steps. It drops on rebalance or removal |

### Behavior across a warm restart

The three arenas behave differently across a [warm restart](https://aerospike.com/docs/database/manage/database/fast-start):

-   **Primary index and secondary index** stages are reattached, so `index_*_alloc_bytes` and `sindex_*_alloc_bytes` continue from their pre-restart values. The primary index highest level reached, including its accumulated reusable space, survives.
-   **Set index** stages are heap allocations rebuilt during warm restart before the info service starts, so metrics scrapes do not show a drop to `0` or a climb during resume. The post-restart `set_index_alloc_bytes` and `set_index_used_bytes` values can differ from pre-restart values.

Expect the set index graphs to be discontinuous across a warm restart where the other two are continuous.

## Reconcile against process RSS

[`process_rss_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__process_rss_bytes) reports the resident memory of the `asd` process. For container capacity, compare [`cgroup_memory_used_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__cgroup_memory_used_bytes) with [`cgroup_memory_limit_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__cgroup_memory_limit_bytes); the control group’s total includes page cache and any other processes in the group, so it can exceed `asd`’s RSS. The [`heap_*_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__heap_allocated_kbytes) metrics report the allocator’s view and omit shared memory segments and memory-mapped data.

::: caution
`*_alloc_bytes` statistics report allocator capacity (whole stages reserved for shmem and pmem, or heap stages for the set index), not resident memory. Mapped shmem and pmem stages contribute to [`process_rss_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__process_rss_bytes) only for pages the process has touched. Do not sum per-namespace `*_alloc_bytes` with `process_rss_bytes` to estimate total memory use.
:::

A value of `0` for `process_rss_bytes` means `/proc/self/statm` could not be read or parsed. Treat `0` as a failed read.

### Control group limits

[`cgroup_memory_used_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__cgroup_memory_used_bytes) and [`cgroup_memory_limit_bytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__cgroup_memory_limit_bytes) report the kernel’s raw usage and limit when the server finds a usable control group with a finite limit. [`cgroup-mem-tracking`](https://aerospike.com/docs/database/reference/config#service__cgroup-mem-tracking) controls whether [`system_free_mem_pct`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_free_mem_pct) and [`system_free_mem_kbytes`](https://aerospike.com/docs/database/reference/metrics#node_stats__system_free_mem_kbytes) use control-group-aware arithmetic for stop-writes and free-memory reporting.

::: caution
With the default (`cgroup-mem-tracking` `false`), `system_free_mem_pct` and `system_free_mem_kbytes` report host-level free memory even inside a container, while `cgroup_memory_*_bytes` may still be published when control group files are readable.

When `cgroup-mem-tracking` is enabled and a finite limit is found, `system_free_mem_pct` subtracts reclaimable page cache from usage and clamps the result by host availability. That value can differ from `100 - (cgroup_memory_used_bytes * 100 / cgroup_memory_limit_bytes)` because subtracting `inactive_file` can raise the free percentage, while clamping to host `MemAvailable` can lower it on a busy host.
:::

## Expressing allocation as a percentage

The allocation and tail metrics are byte counts. [`index_flash_alloc_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__index_flash_alloc_pct) is the allocation percentage the server reports. To express another allocation figure as a fraction of a budget, divide by [`indexes-memory-budget`](https://aerospike.com/docs/database/reference/config#namespace__indexes-memory-budget) or the relevant mount budget. Both are available through `get-config`.

[`indexes_memory_used_pct`](https://aerospike.com/docs/database/reference/metrics#namespace__indexes_memory_used_pct) uses the same denominator and used bytes as its numerator.

## View allocation with asadm

Tools 13.1.0 (`asadm` 5.1.0) and later display these metrics without manual arithmetic:

-   [`info namespace usage`](https://aerospike.com/docs/database/tools/asadm/live-mode#namespace) adds an `Alloc` column to the Primary Index and Secondary Index groups, next to `Used`, from the allocation metric that matches each namespace’s backing type.
-   [`info memory -v`](https://aerospike.com/docs/database/tools/asadm/live-mode#memory) shows allocated and used bytes for the memory-backed primary, secondary, and set index arenas of each node, next to host memory, cgroup memory, and process heap.

The `Alloc%` column in `info memory` is a percentage of the node’s memory capacity, not of `indexes-memory-budget`.

## Next steps

-   [Key metrics to monitor](https://aerospike.com/docs/database/observe/key-metrics)
-   [`info memory` in asadm](https://aerospike.com/docs/database/tools/asadm/live-mode#memory)
-   [Metrics reference](https://aerospike.com/docs/database/reference/metrics)
-   [Configuring the primary index](https://aerospike.com/docs/database/manage/namespace/primary-index)
-   [Configuring the secondary index](https://aerospike.com/docs/database/manage/namespace/secondary-index)