# Log analyzer mode commands ::: note Log analyzer mode for `asadm` is deprecated. Use `show latencies` in `asadm` to see live latencies. You can also use the [Aerospike Grafana dashboards](https://grafana.com/orgs/aerospike/dashboards) if you have Prometheus installed, or [the AGI stack in AeroLab](https://github.com/aerospike/aerolab/tree/master/docs/agi) to further explore log files. ::: The `asadm` command has a built-in interactive shell for analysis of Aerospike server logs. ## Starting the log analyzer To start the log analyzer shell, run the following command: `asadm -l -f ...` where `` are the space-separated paths of single log files or directories of log files. ### Per-session lists of log files In each session, the log analyzer keeps lists of log files for analysis: - **All** list: the complete list of all log files added to the session. - **Selected** list: those log files currently active for analysis. In the log analyzer shell itself, you can add, select, or remove logs. ### Summary of log analyzer commands The log analyzer has the following commands. Some of them are similar to common UNIX shell commands. | Log analyzer command | Description | | --- | --- | | `add` | Add more log files to the **All** and **Selected** lists. | | `count` | Display number of lines that match a specified pattern and time range. | | `diff` | Display counts of a specified search pattern and the deltas of that pattern in consecutive log lines. | | `grep` | Display lines that match a specified pattern and time range. | | `help` | Display usage information about a specified command. | | `histogram` | Calculate and display histogram measurements for a specific time range. | | `list` | Display names of logs in the **All** and **Selected** lists. | | `pager` | `pager on`: Sets paging size of output before pausing. Paging size is specified with the `-p` option. `pager off`: Turn off pausing at paging size. | | `remove` | Remove logs from the **All** and **Selected** lists. | | `select` | Select log files from the **All** list to put in the **Selected** lists. | ### Command help The `help` command displays usage information about the specified command. ```text Log-analyzer> help grep Displays all lines from server logs matched with input strings. Usage: grep -s - Space-separated strings to search in log files. Format: -s 'string1' 'string2' ... 'stringn' -a - Set 'AND'ing of search strings (provided with -s): Finding lines with all search strings in it. Default is 'OR'ing: Finding lines with at least one search string in it. -v - Non-matching strings (space separated). -i - Perform case-insensitive matching of search strings (-s) and non-matching strings (-v) -u - Set to find unique lines. -f - Log time from which to analyze. May use the following formats: 'Sep 22 2011 22:40:14', -3600, or '-1:00:00' Default: head -d - Maximum time period to analyze. May use the following formats: 3600 or 1:00:00. -n - Comma-separated node numbers. You can get these numbers by the list command. Example: -n '1,2,5'. If not set, then runs on all server logs in the selected list. -p - Showing output in pages with p entries per page Default: 10 ``` ### About these examples For details about command options in these examples, use `help `. Omission of lengthy output lines in some examples is indicated by three periods: ```plaintext Line 1 . . . Line n ``` ## add The `add` command makes new logs from different nodes of the cluster available for analysis. It adds logs to both the **All** and the **Selected** lists. You can add only a single log file per node. Arguments to `add` are the space-separated paths of a log or a directory of logs. ::: note In Database 3.7.1 and earlier, the displayed name of a log file is the MD5 hash of the specified path. ::: Example: A directory of logs and a single log are added ```text Log-analyzer> add /home/vagrant/shared/data/testlogs/ /var/log/aerospike/aerospike2.log INFO: Added Log File /home/vagrant/shared/data/testlogs/files/1.log. INFO: Added Log File /home/vagrant/shared/data/testlogs/files/2.log. INFO: Added Log File /home/vagrant/shared/data/testlogs/files/3.log. INFO: Added Log File /home/vagrant/shared/data/testlogs/files/4.log. INFO: Added Log File /home/vagrant/shared/data/testlogs/files/5.log. INFO: Added Log File /home/vagrant/shared/data/testlogs/files/6.log. 6 server logs added for server analysis. INFO: Added Log File /var/log/aerospike/aerospike2.log. 1 server log added for server analysis. ``` ## count The `count` command displays the number of lines that match a specified pattern and time range. You can specify any of the following: - Multiple match patterns. - Log selection. - Ignore case. - Invert match. - Filter lines that are identical so only a single line is displayed, similar to the Unix `uniq` command. - Time range. - Use the`-p` option to set the paging number of lines to display, with default 10 lines. ```text Log-analyzer> count -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 -t 20 ~~~~~~~~~~~~~~~~~~~~~~~~~cluster (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~ NODE : c1d635c2a270008 | c81635c2a270008 | Jan 31 2017 07:00:00: 2 | 2 | Jan 31 2017 07:00:20: 2 | 2 | Jan 31 2017 07:00:40: 2 | 2 | Jan 31 2017 07:01:00: 2 | 2 | Jan 31 2017 07:01:20: 4 | 4 | total : 12 | 12 | ``` ## diff The `diff` command displays the deltas of a specified pattern in consecutive values for search. It supports following five patterns of search string _S_ and values _Vi_ : - _S_ _V0_ - _S_ (_V0_) - _S_ (_V0_, _V1_, …, _Vn_) - _S_ (_V0_ - _V0_(_V1_) _S_ You can specify any of the following: - Log selection. - Ignore case. - Output filter. - Filter lines that are identical so only a single line is displayed, similar to the Unix `uniq` command. - Time range. - The`-p` option to set the paging number of lines to display, with default 10 lines. ```text Log-analyzer> diff -s "fds - proto" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~fds - proto Diff (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NODE : c1d635c2a270008 . | c81635c2a270008 . | . : Total Diff | Total Diff | Jan 31 2017 07:00:00: [0, 510, 510] [0, 510, 510] | [0, 508, 508] [0, 508, 508] | Jan 31 2017 07:00:10: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0] | Jan 31 2017 07:00:20: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0] | Jan 31 2017 07:00:30: [0, 510, 510] [0, 0, 0] | [0, 508, 508] [0, 0, 0] . . . ``` ## grep The `grep` command displays lines that match a specified pattern. You can merge lines from a different file based on timestamp and display lines in ascending order of the timestamp. You can specify any of the following: - Multiple match patterns. - Log selection. - Ignore case. - Invert match. - Filter lines that are identical so only a single line is displayed, similar to the Unix `uniq` command. - Time range. - The`-p` option to set the paging number of lines to display, with default 10 lines. ```text Log-analyzer> grep -s "reads" -n "5,6" -f "Jan 31 2017 07:00:00" -d 100 c81635c2a270008::Jan 31 2017 07:00:05 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec c1d635c2a270008::Jan 31 2017 07:00:09 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec c81635c2a270008::Jan 31 2017 07:00:15 GMT: INFO (info): (hist.c:137) histogram dump: reads (0 total) msec . . . ``` ## histogram The `histogram` command analyzes server logs and displays histogram measurements. For background information and types of histograms, see [Histograms](https://aerospike.com/docs/database/observe/latency). The `histogram` command parses histogram lines for a specified time range and calculates the percentages of average and maximum operations in each time range. ::: note By default, `histogram` displays only three buckets of latency percentages. For an example of displaying more buckets with finer granularity, see [Increase buckets and granularity of measurement](#increase-buckets-and-granularity-of-measurement). ::: Use the `-N ` option for a latency histogram for a specific namespace. Use the `-p` option to set the paging number of lines to display (default 10 lines). ```text Log-analyzer> histogram -h reads -f head -d 150 -p 20 -f 'Jan 27 2016 22:26:00' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~reads Latency (Page-1)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NODE : node0 . . . | node1 . . . | node2 . . . | . : % >1ms % >8ms % >64ms ops/sec | % >1ms % >8ms % >64ms ops/sec | % >1ms % >8ms % >64ms ops/sec | Jan 27 2016 22:26:10: 9.09 6.06 0.00 3.3 | [] [] [] [] | [] [] [] [] | Jan 27 2016 22:26:20: 0.00 0.00 0.00 0.8 | 3.70 0.00 0.00 2.7 | 11.11 0.00 0.00 0.9 | Jan 27 2016 22:26:30: 0.00 0.00 0.00 0.0 | 0.00 0.00 0.00 0.1 | 0.00 0.00 0.00 0.0 | . . . avg : 0.61 0.40 0.00 1.0 | 0.42 0.00 0.00 1.0 | 0.89 0.00 0.00 1.0 | max : 9.09 6.06 0.00 19.8 | 3.70 0.00 0.00 19.0 | 11.11 0.00 0.00 14.5 | ``` ### Syntax variations for histograms for a specific namespace This same output can be displayed with either of the following command lines: - Namespace prefix on histogram type. The braces are required: `histogram -h {}-write` - Namespace with `-N`: `histogram -h write -N ` ### Increase buckets and granularity of measurement In a “stretch cluster” configuration, where nodes are geographically distant from each other, you might need percentage of latencies larger than `histogram`’s default display. #### Example Use a combination of `pager on`, `-b`, and `-e` to more sharply isolate where the latency occurs. 1. To avoid wrapped lines and scroll the display horizontally and vertically, turn on paging: `pager on` 2. Use a combination of `-b` and `-e` to specify the granularity of percentages and number of buckets. For example, a combination of`-b 4 -e 2` displays four buckets of latency in milliseconds in increasing 1, 4, 16, and 64 ratios. The output shows these buckets: `% >1ms % >4ms % >16ms % >64ms`. 3. Scroll vertically with the arrow keys to see all the buckets and horizontally to see all the values. 4. Turn off pager mode: `pager off` ## list The `list` command displays the names of server logs currently added and selected for processing. ```text Log-analyzer> list Added Logs: 1 : bb9635c2a270008 /var/log/aerospike/aerospike.log Selected Logs: bb9635c2a270008 /var/log/aerospike/aerospike.log ``` ## pager The `pager` command sets paging size of output before pausing or turns the paging off. - `pager on`: Sets paging size of output before pausing. Paging size is specified with the `-p` option. When pager mode is on, you can also scroll the output display vertically or horizontally. The command `pager on` is especially useful when looking at more than the log analyzer’s default number of histogram buckets. For an example, see [histogram](#histogram). - `pager off`: Turn off pausing at paging size. ## remove The `remove` command removes logs from the **All** and **Selected** lists. To remove all server logs, specify the `all` argument. ```text Log-analyzer> Log-analyzer> remove 1 3 7 INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/6.log. INFO: Removed Log File /home/vagrant/shared/data/testlogs/files/3.log. INFO: Removed Log File /var/log/aerospike/aerospike.log. Log-analyzer> ``` ## select The `select` command provides a way to select server log from all list to selected list. It removes existing logs from the **All** list and adds newly selected logs. ```text Log-analyzer> list Added Logs: 1 : MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log 2 : MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log 3 : c1d635c2a270008 /var/log/aerospike/aerospike1.log . . . Log-analyzer> select 1 2 3 Log-analyzer> Log-analyzer> list Added Logs: 1 : MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log 2 : MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log 3 : c1d635c2a270008 /var/log/aerospike/aerospike1.log Selected Logs: MD5_0b32a78b084 /home/vagrant/shared/data/testlogs/files/6.log MD5_13e2b3a769b /home/vagrant/shared/data/testlogs/files/5.log c81635c2a270008 /var/log/aerospike/aerospike2.log . . . ```