---
title: "Troubleshooting startup problems"
description: "Troubleshoot Aerospike database startup issues, including shared memory, network interfaces, and permission errors."
---

# Troubleshooting startup problems

> For the complete documentation index see: [llms.txt](https://aerospike.com/docs/llms.txt)
> 
> All documentation pages available in markdown.

This page provides solutions to common Aerospike daemon (`asd`) startup failures.

The Aerospike daemon (`asd`) is the core database process. On systemd-based Linux distributions, start the service using `systemctl start aerospike`. On SysV init systems, use `/etc/init.d/aerospike start`. For more information, see [Aerospike Daemon Management](https://aerospike.com/docs/database/8.1.2/manage/database/daemon).

## The asd process will not start

### Server aborts because it could not allocate enough shared memory

**ISSUE**

In Aerospike Database 7.0.0 or later, if you’re using an in-memory namespace, the following error means that your operating system’s `kernel.shmmax` or `kernel.shmall` is set too low to pre-allocate the in-memory data storage for this namespace.

```plaintext
May 09 2024 22:31:59 GMT: CRITICAL (drv-mem): (drv_mem_ee.c:1078) {test} could not allocate 1342177280-byte shmem stripe

May 09 2024 22:31:59 GMT: WARNING (as): (signal.c:259) SIGUSR1 received, aborting Aerospike Enterprise Edition build 7.0.0.8 os ubuntu20.0>
```

**SOLUTION** Check the current value of `kernel.shmmax`:

Terminal window

```bash
sysctl -n kernel.shmmax
```

The value must be at least one eighth of `data-size` for pure in-memory namespaces, or at least the `filesize` or `device` size for storage-backed persistence. If the command returns a value that is too low (for example, `2147483648`):

Terminal window

```bash
sysctl -w kernel.shmmax=17592186044416
```

Terminal window

```bash
sysctl -n kernel.shmall
```

If `shmall` is too small for your namespace size, increase it. With a page size of 4096 bytes (`getconf PAGE_SIZE`), 2097152 pages is only 8 GiB:

Terminal window

```bash
sysctl -w kernel.shmall=4294967296
```

See [configuring namespace data storage](https://aerospike.com/docs/database/8.1.2/manage/namespace/storage/config) for more details.

### The header has not been zeroized

**ISSUE** If you try to start Aerospike Database 6.0.0 or later, the following error means that one of the devices configured as namespace storage is not recognized as an Aerospike device. This may be a good thing, as you would not want a misconfiguration to result in the server writing over the wrong device, such as the root partition.

```plaintext
Apr 13 2022 05:42:46 GMT: CRITICAL (drv_ssd): (drv_ssd.c:2216) /dev/nvme1n1p1: not an Aerospike device but not erased - check config or erase device
```

**SOLUTION**

1.  Verify that the devices in the namespace configuration are actually supposed to be used by the server.
    
2.  Verify that the devices have been properly [initialized](https://aerospike.com/docs/database/8.1.2/manage/planning/ssd/manage).
    

### Permission denied for /var/lock/subsys/aerospike

**ISSUE** You try to start `asd` and get the following error:

```plaintext
touch: cannot touch `/var/lock/subsys/aerospike': Permission denied
```

**SOLUTION** Confirm you are starting the process as the correct user with the appropriate permissions, using `sudo` when needed.

::: note
You must be logged in as the root user to start the daemon.
:::

## In-memory `data-size` sizing issues

Set `data-size` according to each node’s available memory so that namespace `stop-writes` thresholds trigger before the host runs out of memory. The following symptoms indicate a mismatch between `data-size` and host capacity.

| Symptom | Likely cause | Action |
| --- | --- | --- |
| Low `data_used_pct` alongside high host memory usage | `data-size` exceeds the node’s available memory | Right-size `data-size` per node; see [Capacity planning](https://aerospike.com/docs/database/8.1.2/manage/planning/capacity#required-memory) |
| `memory` reported by the [`best-practices`](https://aerospike.com/docs/database/reference/info#best-practices) info command, or [`failed_best_practices`](https://aerospike.com/docs/database/reference/metrics#node_stats__failed_best_practices) is `true` | Combined namespace data and index memory specs exceed system memory | Reduce `data-size` or add memory; see [Namespace memory budget](https://aerospike.com/docs/database/8.1.2/learn/best-practices#namespace-memory-budget) |
| Namespace stop-writes thresholds have not engaged, even though the host is low on memory | `data-size` exceeds host capacity, so namespace-level thresholds based on `data-size` do not engage before host memory is exhausted | Right-size `data-size`; see [`stop-writes-sys-memory-pct`](https://aerospike.com/docs/database/reference/config#namespace__stop-writes-sys-memory-pct) for the host-memory backstop |
| `existing memory stripe size differs from config` at restart | `data-size` was changed without clearing shared memory first | Follow [How to change `data-size` configuration in a running cluster](https://support.aerospike.com/hc/en-us/articles/49939190589339-How-to-change-data-size-config-in-a-running-cluster). On Kubernetes, see [Change `data-size` on a running cluster](https://aerospike.com/docs/kubernetes/manage/storage/data-in-memory#change-data-size-on-a-running-cluster). |
| System stop-writes with headroom remaining in the namespace | Host memory is exhausted even though the namespace has room within `data-size` | Recovery requires manual intervention; see [Stop-writes thresholds and `data-size`](https://aerospike.com/docs/database/8.1.2/manage/namespace/retention#stop-writes-thresholds-and-data-size) |

To learn how `data-size` relates to namespace thresholds and host memory, see [In-memory data storage and stripes](https://aerospike.com/docs/database/8.1.2/manage/namespace/storage/config#in-memory-data-storage-and-stripes).

## Failed to get the feature-key

**ISSUE**

If you try to start `asd` and get the following error, you must provide a feature-key file:

```plaintext
Apr 09 2021 06:35:12 GMT: CRITICAL (config): (features_ee.c:142) failed to get feature key /etc/aerospike/features.conf
```

::: note
Starting with Database 6.1.0, a simple feature-key file is included. This feature-key file only allows deployment of a single-node cluster.
:::

**SOLUTION** For more information, see [Configuring the Feature-Key File](https://aerospike.com/docs/database/8.1.2/manage/planning/feature-key/).

## Problem with network interface

**ISSUE** The server won’t start due to inability to get the physical address, and you see the following message in the log file:

```plaintext
Jun 22 2014 02:34:10 GMT: WARNING (cf:misc): (id.c::249) Tried eth,bond,wlan and list of all available interfaces on device.Failed to retrieve physical address with errno 19 No such device

Jun 22 2014 02:34:10 GMT: CRITICAL (config): (cfg.c:3363) could not get unique id and/or ip address

Jun 22 2014 02:34:10 GMT: WARNING (as): (signal.c::120) SIGINT received, shutting down

Jun 22 2014 02:34:10 GMT: WARNING (as): (signal.c::123) startup was not complete, exiting immediately
```

**SOLUTION**

1.  Check the name of your network interface:

Terminal window

```bash
ifconfig -a
```

2.  Specify the [`node-id-interface`](https://aerospike.com/docs/database/reference/config#service__node-id-interface) in the configuration.

The interface name in the following example is p2p1:

```plaintext
service {

        ...

          node-id-interface p2p1

        ...

    }

    ...
```

For more information on the configuration, see [Configuration Reference](https://aerospike.com/docs/database/reference/config).

## Not enough file descriptors error in log

**ISSUE** Look for the following message in the Aerospike log:

```plaintext
Aug 24 2012 16:43:10 GMT: INFO (as): (base/as.c:172) File descriptor limit is : 1024 and proto-fd-max is : 2048

Aug 24 2012 16:43:10 GMT: CRITICAL GLOBAL (as): (base/as.c:174) Not enough file descriptors, Starting with 1024 and needs 2048

critical error: backtrace: frame 0 /usr/bin/asd() [0x460cef]

critical error: backtrace: frame 1 /usr/bin/asd() [0x404b59]

critical error: backtrace: frame 2 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f620b53030d] critical error: backtrace: frame 3 /usr/bin/asd() [0x403e79]
```

At Aerospike server start, this value must not exceed the system’s file descriptor limit.

**SOLUTION** To avoid a startup problem, there are two alternatives:

-   Decrease the value of `proto-fd-max` in your Aerospike configuration file.
-   Increase the process file descriptor limit (Linux system setting).

Prior to Aerospike Database 4.9.0, for a dynamic change, this limit was enforced only if the new value was lower than the system setting.

## Stuck in defrag loop at startup

When Aerospike Database starts up, it requires a minimum percentage of available write blocks (`avail-pct`) to join the cluster. We recommend keeping storage utilization around 50% to allow efficient defragmentation. The minimum percentage required at startup depends on the value of [`defrag-startup-minimum`](https://aerospike.com/docs/database/reference/config#namespace__defrag-startup-minimum).

In Database 5.7.0 and later, this value is 0 by default, meaning the server will never get stuck in a defrag loop at startup. If nonzero, a typical value might be 10, which is the default in server versions prior to 5.7.0:

```plaintext
defrag-startup-minimum 10
```

If a node does not start up for a long time and the log file at /var/log/aerospike/aerospike.log shows something like:

```plaintext
Aug 22 2012 21:16:38 GMT: INFO (as): (base/as.c:265) waiting for defrag: namespace devices percent 0 waiting for 10

Apr 24 2013 17:19:59 GMT: INFO (drv_ssd): (storage/drv_ssd.c:1544) read_bin: could not read first

Apr 24 2013 17:20:00 GMT: WARNING (drv_ssd): (storage/drv_ssd.c:1390) **** ssd_read: record f8de4ae1d039c87 has no block associated, fail

Apr 24 2013 17:20:00 GMT: WARNING (drv_ssd): (storage/drv_ssd.c:1390) **** ssd_read: record f8de4ae1d039c87 has no block associated, fail
```

When the server starts, it tries to defrag and won’t join the cluster until `avail-pct` reaches the `defrag-startup-minimum` threshold. If storage utilization is too high, the node may not be able to defrag enough write blocks to raise `avail-pct` to that threshold.

To get out of the defrag loop:

-   For an in-memory database with persistence, in general, the filesize should be 8x the amount of memory. For example, if the memory for data is 20 GB, then the filesize should be 160 GB. Increase the filesize to the 8x limit or higher. If your database has very high traffic, you may require higher than 8x.
-   For storage-backed namespaces, lower the storage eviction threshold in the [Aerospike configuration file](https://aerospike.com/docs/database/8.1.2/manage/namespace/retention) so the node evicts records and frees write blocks. In Database 7.0.0 and later, set [`evict-used-pct`](https://aerospike.com/docs/database/reference/config#namespace__evict-used-pct); in earlier releases, set [`high-water-disk-pct`](https://aerospike.com/docs/database/reference/config#namespace__high-water-disk-pct). If startup still stalls, lower the threshold further.

_Eviction_ is typically a back-stopping strategy - data should _expire_ before utilization reaches eviction thresholds. This problem is a symptom of a larger problem:

-   If your storage is insufficient, re-evaluate your storage/capacity strategy, as described in [Managing storage capacity](https://aerospike.com/docs/database/8.1.2/manage/namespace/storage/config).
-   Contact Aerospike for the capacity planning spreadsheet to help you re-configure your cluster capacity.

## With a non-standard network device, the server won’t start

**ISSUE**

On some distributions and with some unusual network devices, the server won’t start and the log shows the following error. This means the network interface name on the node is something other than “eth”, “bond”, or “wlan”.

```plaintext
Aug 22 2012 06:34:18 GMT: WARNING (cf:misc): (id.c:163) can't get physical address, tried eth, bond, wlan. fatal: 19 No such device
```

**SOLUTION**

Add the network-interface-name parameter to your configuration file. In the example below, the device is named vlan708:

```plaintext
network {

           service {

                   address 10.0.2.131

                   port 3000

                   network-interface-name vlan708

                   }

           }
```

If you have multiple network devices, such as `eth0` and `eth1`, Aerospike will choose one of them. A common situation is a node that has two Ethernet ports, one for the Internet and one for internal traffic. In this case, Aerospike needs to access the Internet port, but it may choose the wrong one, resulting in a node that does not see traffic correctly. To specify a specific network device, add the [`access-address`](https://aerospike.com/docs/database/reference/config#network__access-address) parameter to your configuration file.

## Warning messages on starting Aerospike

If you start Aerospike and get the following warnings, they are related to memory management in the Linux kernel.

1.  **Warning about SHMMAX:**

SHMMAX is the maximum size of a single shared memory segment. If you see the following output when you start a server node, it means that the system was configured with a shared memory maximum block size that’s less than the 1GB required by Aerospike. The start script dynamically raises the limit to 1GB.

```plaintext
sudo service aerospike start

kernel.shmmax too low, setting to 1GB

kernel.shmmax = 1073741824

Starting aerospike:     [OK]
```

Unless the machine is rebooted, you should only see this happen on the first start.

2.  **Warning about SHMALL:**

SHMALL is the sum of all shared memory segments on the whole system. If you see the following output when you start a server node, it means the start script dynamically raised the maximum number of shared memory pages. Unless the machine is rebooted, you should only see this happen on the first start. It’s possible to see both limits raised during a single (first) start.

```plaintext
sudo service aerospike start

kernel.shmall too low, setting to 4G pages

kernel.shmall = 4294967296

Starting aerospike:     [OK]
```

## Other problems

For other problems, check for consistent settings between all nodes in the cluster, including service and network settings. Make sure that the namespaces are configured the same on each node.

Verify that no firewall is interfering with communication between nodes (ports 3000 through 3004).