Skip to main content
We are excited to be a part of AWS re:Invent 2024. Visit us at booth #1844 in Las Vegas.More info
Loading

Log Analyzer Commands

note

The log analysis feature of asadm is deprecated. Use show latencies in asadm to see live latencies. You can also use the Aerospike Grafana dashboards if you have Prometheus installed, or the AGI stack in AeroLab to further explore log files.

The asadm command has a built-in interactive shell for analysis of Aerospike server logs.

Starting the log analyzer​

To start the log analyzer shell, run the following command:

asadm -l -f <asLogpath-1> <asLogpath-2> <asLogpath-3> ...

where <asLogpath*> are the space-separated paths of single log files or directories of log files.

Per-session lists of log files​

In each session, the log analyzer keeps lists of log files for analysis:

  • All list: the complete list of all log files added to the session.
  • Selected list: those log files currently active for analysis.

In the log analyzer shell itself, you can add, select, or remove logs.

Summary of log analyzer commands​

The log analyzer has the following commands. Some of them are similar to common UNIX shell commands.

Log analyzer commandDescription
addAdd more log files to the All and Selected lists.
countDisplay number of lines that match a specified pattern and time range.
diffDisplay counts of a specified search pattern and the deltas of that pattern in consecutive log lines.
grepDisplay lines that match a specified pattern and time range.
helpDisplay usage information about a specified command.
histogramCalculate and display histogram measurements for a specific time range.
listDisplay names of logs in the All and Selected lists.
pagerpager on: Sets paging size of output before pausing. Paging size is specified with the -p option.

pager off: Turn off pausing at paging size.
removeRemove logs from the All and Selected lists.
selectSelect log files from the All list to put in the Selected lists.

Command help​

The help command displays usage information about the specified command.

Log-analyzer> help grep

Displays all lines from server logs matched with input strings.

Usage:  grep 

        -s     - Space-separated strings to search in log files. Format: -s
                 'string1' 'string2' ... 'stringn'
        -a     - Set 'AND'ing of search strings (provided with -s): Finding
                 lines with all search strings in it. Default is 'OR'ing:
                 Finding lines with at least one search string in it.
        -v     - Non-matching strings (space separated).
        -i     - Perform case-insensitive matching of search strings (-s) and
                 non-matching strings (-v)
        -u     - Set to find unique lines.
        -f     - Log time from which to analyze. May use the following formats:
                 'Sep 22 2011 22:40:14', -3600, or '-1:00:00'
                 Default: head
        -d     - Maximum time period to analyze. May use the following formats:
                 3600 or 1:00:00.
        -n     - Comma-separated node numbers. You can get these numbers by the
                 list command. Example: -n '1,2,5'. If not set, then runs on all
                 server logs in the selected list.
        -p     - Showing output in pages with p entries per page
                 Default: 10


About these examples​

For details about command options in these examples, use help <command_name>.

Omission of lengthy output lines in some examples is indicated by three periods:

Line 1
.
.
.
Line n

add​

The add command makes new logs from different nodes of the cluster available for analysis. It adds logs to both the All to the Selected lists.

Note: You can add only a single log file per node.

Arguments to add are the space-separated paths of a log or a directory of logs.

Note for Aerospike version 3.7.1 and earlier: the displayed name of a log file is the MD5 hash of the specified path.

In the following example, a directory of logs and a single log are added.

Log-analyzer> add /home/vagrant/shared/data/testlogs/ /var/log/aerospike/aerospike2.log
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/1.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/2.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/4.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/5.log.
INFO: Added Log File /home/vagrant/shared/data/testlogs/files/6.log.
6 server logs added for server analysis.
INFO: Added Log File /var/log/aerospike/aerospike2.log.
1 server log added for server analysis.

count​

The count command displays the number of lines that match a specified pattern and time range.

You can specify any of the following:

  • Multiple match patterns.
  • Log selection.
  • Ignore case.
  • Invert match.
  • Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
  • Time range.
  • Use the-p option to set the paging number of lines to display, with default 10 lines.
Log-analyzer> count -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 -t 20
~~~~~~~~~~~~~~~~~~~~~~~~~cluster (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   c1d635c2a270008   |   c81635c2a270008   |   
Jan 31 2017 07:00:00:   2                 |   2                 |   
Jan 31 2017 07:00:20:   2                 |   2                 |   
Jan 31 2017 07:00:40:   2                 |   2                 |   
Jan 31 2017 07:01:00:   2                 |   2                 |   
Jan 31 2017 07:01:20:   4                 |   4                 |   
total               :   12                |   12                |

diff​

The diff command displays the deltas of a specified pattern in consecutive values for search.

It supports following five patterns of search string S and values Vi :

  • S V0
  • S (V0)
  • S (V0, V1, …, Vn)
  • S (V0
  • V0(V1) S

You can specify any of the following:

  • Log selection.
  • Ignore case.
  • Output filter.
  • Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
  • Time range.
  • The-p option to set the paging number of lines to display, with default 10 lines.
Log-analyzer> diff -s "fds - proto" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~fds - proto Diff (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   c1d635c2a270008   .               |   c81635c2a270008   .               |   
.                   :   Total             Diff            |   Total             Diff            |   
Jan 31 2017 07:00:00:   [0, 510, 510]     [0, 510, 510]   |   [0, 508, 508]     [0, 508, 508]   |   
Jan 31 2017 07:00:10:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       |   
Jan 31 2017 07:00:20:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       |   
Jan 31 2017 07:00:30:   [0, 510, 510]     [0, 0, 0]       |   [0, 508, 508]     [0, 0, 0]       
.
.
.

grep​

The grep command displays lines that match a specified pattern. You can merge lines from a different file based on timestamp and display lines in ascending order of the timestamp.

You can specify any of the following:

  • Multiple match patterns.
  • Log selection.
  • Ignore case.
  • Invert match.
  • Filter lines that are identical so only a single line is displayed, similar to the Unix uniq command.
  • Time range.
  • The-p option to set the paging number of lines to display, with default 10 lines.
Log-analyzer> grep -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100
  c81635c2a270008::Jan 31 2017 07:00:05 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
  c1d635c2a270008::Jan 31 2017 07:00:09 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
  c81635c2a270008::Jan 31 2017 07:00:15 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec
.
.
.

Histogram​

The histogram command analyzes server logs and displays histogram measurements. For background information and types of histograms, see Histograms.

The histogram command parses histogram lines for a specified time range and calculates the percentages of average and maximum operations in each time range.

Note: By default, histogram displays only three buckets of latency percentages. For an example of displaying more buckets with finer granularity, see Increasing buckets and granularity of measurement.

You can use the -N <namespace> option for a latency histogram for a specific namespace.

Use the -p option to set the paging number of lines to display (default 10 lines).

Log-analyzer> histogram -h reads -f head -d 150 -p 20 -f 'Jan 27 2016 22:26:00'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~reads Latency (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                :   node0    .        .         .         |   node1    .        .         .         |   node2    .        .         .         |   
.                   :   % >1ms   % >8ms   % >64ms   ops/sec   |   % >1ms   % >8ms   % >64ms   ops/sec   |   % >1ms   % >8ms   % >64ms   ops/sec   |   
Jan 27 2016 22:26:10:   9.09     6.06     0.00      3.3       |   []       []       []        []        |   []       []       []        []        |   
Jan 27 2016 22:26:20:   0.00     0.00     0.00      0.8       |   3.70     0.00     0.00      2.7       |   11.11    0.00     0.00      0.9       |   
Jan 27 2016 22:26:30:   0.00     0.00     0.00      0.0       |   0.00     0.00     0.00      0.1       |   0.00     0.00     0.00      0.0       |   
.
.
.
avg                 :   0.61     0.40     0.00      1.0       |   0.42     0.00     0.00      1.0       |   0.89     0.00     0.00      1.0       |   
max                 :   9.09     6.06     0.00      19.8      |   3.70     0.00     0.00      19.0      |   11.11    0.00     0.00      14.5      |

Syntax variations for histograms for a specific namespace​

This same output can be displayed with either of the following command lines:

  • Namespace prefix on histogram type. The braces are required: histogram -h {<namespace>}-write
  • Namespace with -N: histogram -h write -N <namespace>

Increasing buckets and granularity of measurement​

In a "stretch cluster" configuration, where nodes are geographically distant from each other, you might need percentage of latencies larger than histogram's default display.

Example

You can use a combination of pager on, -b, and -e to more sharply isolate where the latency occurs.

  1. So you can avoid wrapped lines and scroll the display horizontally and vertically, turn on paging: pager on

  2. Use a combination of -b and -e to specify the granularity of percentages and number of buckets. For example, a combination of-b 4 -e 2 displays four buckets of latency in milliseconds in increasing 1, 4, 16, and 64 ratios. The output shows these buckets:

    % >1ms % >4ms % >16ms % >64ms.

  3. Scroll vertically with the arrow keys to see all the buckets and horizontally to see all the values.

  4. Turn off pager mode: pager off

list​

The list command displays the names of server logs currently added and selected for processing.

Log-analyzer> list
Added Logs:
1  : bb9635c2a270008     /var/log/aerospike/aerospike.log

Selected Logs:
     bb9635c2a270008     /var/log/aerospike/aerospike.log

pager​

The pager command sets paging size of output before pausing or turns the paging off.

  • pager on: Sets paging size of output before pausing. Paging size is specified with the -p option. When pager mode is on, you can also scroll the output display vertically or horizontally.

    The command pager on is especially useful when looking at more than the log analyzer's default number of histogram buckets. For an example, see histogram.

  • pager off: Turn off pausing at paging size.

remove​

The remove command removes logs from the All and Selected lists.

To remove all server logs, specify the all argument.

Log-analyzer>
Log-analyzer> remove 1 3 7
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/6.log.
INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/3.log.
INFO: Removed Log File /var/log/aerospike/aerospike.log.
Log-analyzer>

select​

The select command provides a way to select server log from all list to selected list. It removes existing logs from the All list and adds newly selected logs.

Log-analyzer> list
Added Logs:
1  : MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
2  : MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
3  : c1d635c2a270008     /var/log/aerospike/aerospike1.log
.
.
.
Log-analyzer> select 1 2 3
Log-analyzer>     
Log-analyzer> list
Added Logs:
1  : MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
2  : MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
3  : c1d635c2a270008     /var/log/aerospike/aerospike1.log

Selected Logs:
     MD5_0b32a78b084     /home/vagrant/shared/data/testlogs/files/6.log
     MD5_13e2b3a769b     /home/vagrant/shared/data/testlogs/files/5.log
     c81635c2a270008     /var/log/aerospike/aerospike2.log
.
.
.
Last updated on