Aerospike Admin (asadm)
Over the past year, asadm has been expanded with a handful of useful features, to make it the only command-line tool you need to monitor and administer your Aerospike Database. I wrote about some previously added features in a post early this year. There, I discussed user/role administration, UDF management, and secondary-index management. The latest expansion of asadm is split into a few releases.
2.3.0:
Dynamic Configuration (with tab-completion)
2.4.2
Validating for Best-Practices
2.5.0
2.6.0
Added --enable flag to accompany --executeoption.
Now, this is quite a hefty list. Rather than talk about them all, I would like to discuss Dynamic Configuration, Roster Management, Truncation, and Job Management. The rest are simple enough, and you can take a look at the technical documentation linked above. It is also worth noting that there are some great operations manuals showing asadm in action. Here are a few:
Let’s dive in . . .
Dynamic Configuration
Before now, the only way to dynamically change Aerospike configuration was to use info commands. Asinfo is a lightweight tool for making raw info requests to an Aerospike node. It’s not ideal, but it works. Here at Aerospike, we are trying to move our customers away from using asinfo, as much as possible. That is why in asadm 2.3.0 we introduced the manage config command, a counterpart to the pre-existing show config command. The manage config command allows you to change configuration parameters and logging levels, create and delete XDR datacenters, add/remove nodes from datacenters, and add/remove namespaces from datacenters. The documentation to do all this is pretty good, so rather than repeat it here, I would just like to talk about the UI design and the tab-completion feature.
The UI was designed to be intuitive enough, so that by knowing the context of any given parameter, you should be able to infer the correct command. To find the context of the parameter take a look at the configuration reference or your aerospike.conf file.
Say for instance you would like to change the value of a configuration parameter foo in the network > fabric context to bar. Well, think of it as navigating to the given context, then changing the value of the parameter.
Admin+> manage config network fabric param foo to barNow, there are other cases where the context is also assigned a name, like a namespace, set, and dc. In these cases just enter the name following the context:
Admin+> manage config namespace test set testset param foo to barBut Wait . . . There is Tab-Completion
The tab-completion for manage config is an extension of what is already provided by asadm. manage config tab-completion works on contexts, subcontexts, parameters, and enumerated values, but only if they are dynamically configurable. The goal of this is to help our customers quickly change the configuration of the cluster, without all the headaches of the set-config info command. So, if you are wondering which parameters are dynamically configurable or what are the acceptable values for any given parameter, asadm can help!
Rather than walk you through all the tab-completion features, here is an example workflow (sorry if the text is small):
Namespace and Set Truncation
The new manage truncate command, added in asadm 2.3.0, allows users to truncate namespaces and sets, truncate records with a last-update-time (LUT) before a given DateTime, and undo in-progress truncations. Before manage truncate, a user would be required to use asinfo. The asinfo commands are not as easy to use and have no protections. manage truncation, on the other hand, provides in-place protections to prevent the user from truncating records unintentionally.
In the example below, we want to truncate records in the namespace test with an LUT before July 4th, 2021 UTC-07:00.
Note: This added protection is disabled by default in asadm 2.3.x and 2.4.x. To enable it, type enable --warn
Admin+> manage truncate ns test before 2021-07-04T00:00:00-0700 iso-8601
You're about to truncate up to 97345 records from namespace test with LUT before 00:00:00.000000 UTC-07:00 on July 04, 2021
Confirm that you want to proceed by typing 31d05e, or cancel by typing anything else.
31d05e
Successfully started truncation for namespace testCool, eh? Now we can use an ISO-8601 DateTime. However, you can still use a unix-epoch value, as before.
Admin+> manage truncate ns test before 1625382000 unix-epoch
You're about to truncate up to 97345 records from namespace test with LUT before 07:00:00.000000 UTC on July 04, 2021
Confirm that you want to proceed by typing 46fc72, or cancel by typing anything else.
46fc72
Successfully started truncation for namespace testOh No!
If you just truncated a namespace or set unintentionally you can use the manage truncate ns <ns> [set <set>] undo command to revert the command.
Roster Management
Roster management, added in asadm 2.5.0, is done using the new manage roster commands. manage roster allows you to add/remove nodes from the pending roster, and to set the pending roster to equal the observed nodes list. Coupled with the new show roster and manage recluster commands, roster management now only requires a few simple commands. Let's look at one example.
Configuring the Roster to the Cluster’s Observed Nodes
Previously, setting the roster was a bit of a pain. You would first have to run the asinfo roster: command
Admin+> asinfo -v "roster:namespace=test"
node7.aerospike.com:3000 (192.168.10.7) returned: roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202 node6.aerospike.com:3000 (192.168.10.6) returned: roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202 node5.aerospike.com:3000 (192.168.10.5) returned: roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202 node4.aerospike.com:3000 (192.168.10.4) returned: roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202 node2.aerospike.com:3000 (192.168.10.2) returned: roster=null:pending_roster=null:observed_nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202,BB9040016AE4202,BB9020016AE4202Then, you would need to copy the observed_nodes list and call the roster-set: command
Admin+> asinfo -v "roster-set:namespace=test;nodes=BB9070016AE4202,BB9060016AE4202,BB9050016AE4202, BB9040016AE4202,BB9020016AE4202" with BB9020016AE4202
node7.aerospike.com:3000 (192.168.10.7) returned:
oknode6.aerospike.com:3000 (192.168.10.6) returned:
oknode5.aerospike.com:3000 (192.168.10.5) returned:
oknode4.aerospike.com:3000 (192.168.10.4) returned:
oknode2.aerospike.com:3000 (192.168.10.2) returned:
okThen, you would need to verify the pending roster was set correctly. Finally, you would need to issue a recluster:
Admin+> asinfo -v 'recluster:'
node7.aerospike.com:3000 (192.168.10.7) returned:
ignored-by-non-principalnode6.aerospike.com:3000 (192.168.10.6) returned:
ignored-by-non-principal node5.aerospike.com:3000 (192.168.10.5) returned:
ignored-by-non-principalnode4.aerospike.com:3000 (192.168.10.4) returned:
ignored-by-non-principalnode2.aerospike.com:3000 (192.168.10.2) returned:
okNow, with the new manage roster commands the same steps can be reduced to just two operations:
Admin+> manage roster stage observed ns test
Pending roster now contains observed nodes.
Run "manage recluster" for your changes to take affect.
Admin+> manage recluster
Successfully started reclusterThere you have it! Nice and easy roster management.
Viewing and Killing Jobs
We plan to devolve aql, removing its admin functions, and refocusing it on being a command-line data browser. Accordingly, we have moved the responsibility of job management to asadm in version 2.5.0. Two new sets of commands have been added, show jobs and manage jobs. Viewing jobs in asadm is more convenient than aql, as you can see below.
Hint: Run pager on, before running show jobs, in order to easily scroll the table left and right.
The show jobs scans command will show all scan jobs, organized in various ways. Vertically, jobs are first grouped by “Namespace” and then by “Type”. Horizontally, jobs are ordered by Progress % and then by Time Since Done to show the most relevant scans. We also added some nice colors :)
There You Have It
At Aerospike we are working hard to make our customers’ lives easier. Whether that is helping you achieve high availability in your globally distributed application, or just making cluster management less of a chore. If you have questions (or recommendations), feel free to submit an issue in the aerospike/aerospike-admin repo on GitHub. There are more features in the works, so please stay tuned.
