Server logs with systemd

This page describes how to access server logs on systemd-based Linux distributions with journalctl. Managing asd using systemctl is described in Aerospike daemon management.

Under systemd, journald is the standard facility for uniformly managing logs for all Linux daemon processes. A structured and indexed centralized journal is more performant than working solely with text-based log files.

Access Aerospike log with journalctl

journalctl -u aerospike -a -o cat -f

The -a option ensures nothing is suppressed from the Aerospike log messages, even if some lines are very long.
The -o cat option presents the raw Aerospike log without prepending the journal’s timestamp and other metadata to each line.
The -f option will “follow” the log as it is generated.

When running under systemd, it is still possible to direct the Aerospike server to create (and rotate) log files. For more information on managing Aerospike server log files, see Configuring Log Files.

Using asloglatency under systemd

To use the asloglatency tool under systemd, extract the desired portion of the log into a temporary file using journalctl:

journalctl -u aerospike -a -o cat --since "2023-03-17" --until "2023-03-18" | grep GMT > /tmp/aerospike.20230317.log

When finished, run asloglatency on the extracted log file:

asloglatency -h writes_master -f head -l /tmp/aerospike.20160317.log

More information about the Journal daemon

Storage location

The default location for Journal files is /run/log/journal. When you configure /etc/systemd/journald.conf to use persistent storage, files are stored in /var/log/journal, and fallback to /run/log/journal. /run/log/journal is used for logging during early boot when the disk is not yet writable.

Log rotation

Journal rotates logs according to the configurations in /etc/systemd/journald.conf. You can configure logs to rotate based upon file size, directory size, and age of logs. You can also configure logs to live in memory and to not persist after reboot.

If you configure logs to rotate based on file size, Journal governs file size with the option SystemMaxFileSize in the journald.conf configuration file. This option sets a maximum file size for Journal files. SystemMaxUse sets the maximum storage allocated for Journal files. Supported size notation include no-suffix for bytes, K for Kibibyte (KiB), M for Mebibyte (MiB), G for Gibibyte (GiB), T for Tebibyte (TiB).

You can also use the Maxfilesec option to set a maximum age for entries in a single Journal file before rotating to the next one. This setting can be in: year, month, day, hour, or minute. The MaxRetentionSec option sets the maximum age for stored Journal files.

Aerospike recommends that you configure journald.conf to rotate logs every 24 hours, and to keep the logs for 90 days. You can refer to the sample journald.conf file at the end of this document.

Reviewing prior logs

When you review Journal files with journalctl, prior logs are automatically included. If the time period that you are reviewing spans two or more Journal files, journalctl displays the entries seamlessly.

Persistence in Journal

Persistence determines if and where you store the Journal files. You can modify /etc/systemd/journald.conf to configure persistence.

Config item	Value	Result
`Storage`	`none`	Journal does not keep data
`Storage`	`volatile`	Journal data is only held in memory
`Storage`	`persistent`	logs are held in `/var/log/journal`
		use `/run/log/journal` for fallback during early boot if needed

Sample systemd Journal configuration file

This file is part of systemd, typically located at /etc/systemd/journald.conf.

[Journal]
Storage=auto
# Compress=yes
# Seal=yes
# SplitMode=login
# SyncIntervalSec=5m
# RateLimitInterval=30s
# RateLimitBurst=100
SystemMaxUse=2G
# SystemKeepFree=
# SystemMaxFileSize=
# RuntimeMaxUse=
# RuntimeKeepFree=
# RuntimeMaxFileSize=
MaxRetentionSec=90day
MaxFileSec=24h
# ForwardToSyslog=yes
# ForwardToKMsg=no
# ForwardToConsole=no
# TTYPath=/dev/console
# MaxLevelStore=debug
# MaxLevelSyslog=debug
# MaxLevelKMsg=notice
# MaxLevelConsole=info