Aerospike is pleased to announce Aerospike 5.1, with general availability in late July, 2020. The central theme of this release is enhanced security in the Enterprise Edition, which has an Identity and Access Management (IAM) system based on users and roles.
Aerospike has been continually upgrading security, in keeping with the mantra that “security is a process, not an end point.” Security technology evolved alongside computing from the beginning, with a sharp inflection point coinciding with the advent of the internet. It is by and large a cat and mouse game, with ever more sophisticated attacks requiring ever more elaborate countermeasures.
Against this backdrop, key trends are motivating Aerospike’s ongoing security efforts:
Mission-critical workloads requiring Enterprise-grade security (Aerospike Database must be compatible with existing corporate-wide security infrastructure).
The increasing sophistication of exploits, e.g. recent covert speculative attacks based on processor vulnerabilities.
Workloads migrating to cloud environments, invalidating IAM systems based on dedicated hardware and fixed IP addresses.
A lower barrier to entry for hackers, with a wide availability of tools, and clandestine marketplaces for 0-day exploits.
HashiCorp Vault Integration
To secure, store and tightly control access to passwords, certificates and encryption keys, Aerospike 5.1 now supports HashiCorp Vault.
“Aerospike has partnered with HashiCorp to allow selected configuration parameters to be migrated from using secrets stored on the local filesystem to HashiCorp Vault. We are pleased for the Aerospike Enterprise integration with Vault. With this integration Aerospike customers will be able to leverage Vault to reduce the attack surface, allow IAM policies to be defined and centrally administered and help solve the distribution problem of Aerospike deployments.”
—Asvin Ramesh, Director, Technology Partners at HashiCorp
Aerospike has long supported features such as encryption-at-rest, TLS for network links, LDAP authentication, and more. Prior to Aerospike 5.1, the configuration information for these features resided in files on the local filesystem. Keeping them in sync with enterprise-wide security policies is a significant, often labor-intensive support burden. Additionally, the files are potential targets for exploits, either to alter and gain unauthorized access, or to extract sensitive data such as passwords and private keys.
Aerospike 5.1 allows selected configuration files to be migrated from the local filesystem to HashiCorp Vault secrets. This reduces the attack surface, solves the distribution problem, and allows IAM policies to be defined and centrally administered independently of Aerospike deployments. HashiCorp Vault was chosen because it has both Open Source and Enterprise versions that are already deployed in large customers as a cross-organization policy arbiter.
In this block diagram of Aerospike 5.1 components, it can be seen where Vault fits in. Vault has a client-server architecture and Aerospike operates as a Vault client. Aerospike supports the Vault KV Secrets Engine (Version 1 and Version 2) with a RESTful API. Local file accesses are replaced by communication over encrypted network links (in blue) to Vault servers to retrieve secrets.
In its initial integration, Aerospike allows the following classes of parameters to be managed by and stored within Vault:
Aerospike’s LDAP server password
XDR remote destination passwords
Encryption-at-rest key
Network TLS certificates, keys, and passwords
Customers who have already deployed Vault in their enterprise can take advantage of Aerospike’s integration by creating new Vault secrets using their pre-existing infrastructure. Customers new to Vault would need to define and deploy Vault servers before taking advantage of it in Aerospike 5.1. Having a working knowledge of Vault is essential before attempting to deploy it with Aerospike.
Aerospike needs to know the Vault service endpoint (expressed as a URL), and be mutually authenticated with that service. This information is supplied via new parameters in the service stanza of the Aerospike configuration file and is read at start up time.
Every node in an Aerospike cluster must supply these parameters, but there is no requirement that they point to the same Vault server. However, Vault Secrets must be globally unique, so if two nodes in a cluster have different Vault secrets, those names must be unique.
To move a parameter from a local file to Vault, the first step is to define a new Vault secret containing the same data. Depending on the parameter, this might be a single value like a password, or a more complex entity (e.g. an X.509 Certificate). Because Vault APIs are based on JSON, all the values of a secret must be Base64-encoded.
The second step is to replace the corresponding reference in the Aerospike configuration file with a Vault reference. For example, the LDAP query-user-password-file configuration parameter changes from:
security {... ldap {...
query-user-password-file /etc/aerospike/ldap-pwd.txt } }to
security {... ldap {...
query-user-password-file vault:ldap-pwd } }
where the ldap-pwd secret is the password corresponding to the contents of the ldap-pwd.txt file. The pseudo-protocol identifier “vault:” is the new Aerospike 5.1 syntax for denoting Vault secrets. Although moving to Vault is highly recommended, the existing file-based access continues to be supported. This allows secrets to be migrated to Vault in any order, piecemeal or all at once.
Permission Change for Truncate-* Info Commands
The truncate family of Info commands can now be issued by users with either data-admin or write privileges. Prior to Aerospike 5.1, only data-admin privilege was accepted for performing this operation. The motivation for this change is administrative flexibility: many deployments would like to limit the number of users with data-admin privileges, and allowing truncate with the less powerful write privilege achieves this goal.
There is a slight nuance with all the truncate-* Info commands with respect to privileges. If the user has write privilege the namespace to be truncated must exist on the target node(s). If the user has data-admin privilege the command can be issued on any node.
UDF Security & Consistency Enhancements
Aerospike’s security policies are based on a multi-faceted, defense-in-depth paradigm. Open Source components are regularly scanned for vulnerabilities, and public databases such as CVE are monitored. Aerospike also is engaged with the wider Security community, including security researchers. In response to dialog with one such researcher, several security enhancements were made to Lua:
Lua
io.* functions are no longer availableOnly the following os library Lua functions are available:
os.clock()os.date()os.difftime()os.time()
Further information on this issue can be found in CVE-2020-13151.
Another set of changes allow all UDF failure scenarios to be correctly handled. This means that UDFs are now fully unwindable, making them compatible with Strong Consistency namespaces.
Minor Features
Aerospike Database 5.1 includes some minor (but useful) features: see the release notes for the complete list and additional details.
Histograms have been augmented with the new latencies Info command. It returns latency and throughput information any (or all) enabled histograms, based on the most recent time slice. The older latency Info command is still available but deprecated, and generates only default output. In a related change, the new microsecond-histograms configuration parameter allows histograms to operate at microsecond granularity (the default is millisecond granularity).
Several new XDR-related configuration parameters have been added, two of which are noteworthy. The period-ms configuration parameter controls the granularity with which data is shipped to remote DCs. A source DC has a thread for each remote DC: each of these threads executes a loop that ships all data that has accumulated for that destination since the last iteration. period-ms specifies the minimum time (including execution time) between successive iterations of these threads.
This interval is colloquially referred to as lap time. If a thread finishes shipping data before period-ms milliseconds have elapsed, it will sleep for the duration. period-ms previously defaulted to 100ms. It should be changed with care: too low a value can cause excessive CPU consumption, and too high can increase network jitter.
Another new configuration parameter, sc-replication-wait-ms, specifies how many milliseconds to allow for replication before trying to ship a record with strong consistency. As before, care should be taken before overriding.
Evaluating performance is a standard part of Aerospike server development methodology. Sometimes optimizing code is an explicit goal (e.g. when overhauling a subsystem), but even when this is not the case new releases are benchmarked to guard against regressions. Because of the relative nature of bottlenecks, it often transpires that functional changes can serendipitously boost performance too. In Aerospike 5.1, some users may find performance is improved for UDFs on large lists and maps. Likewise, access to internal bin-related structures has been sped up. Finally, a re-arrangement of locks should result in better write performance when records are stored in DRAM or PMEM.
Useful Links
Reminder: Upgrade/Downgrade Considerations
Aerospike Database 5 made changes to low-level data structures and protocols. Aerospike 4.9 functions as a bridge release. The following rules must be observed to preserve the ability to upgrade/downgrade between Aerospike 5.1 and earlier server versions:
In a cluster, all nodes running releases earlier than 4.9 must first upgrade to 4.9 before any node is upgraded to 5.1.
In a cluster, all nodes running Aerospike 5.1 or greater must downgrade to 4.9 before any node is downgraded to earlier releases. Downgrading to releases earlier than 4.9 may also require erasing storage devices.
Many of the security features are based on IAM users & roles, which are available only in the Aerospike Enterprise Edition. See the release notes for a complete list of 5.1 features, and which editions they are supported on.
