New Aerospike benchmark demonstrates real-time performance at petabyte scale on just 20 AWS nodes with Intel processors

Aerospike has unveiled the results of a petabyte-scale benchmark that illustrates its Real-time Data Platform delivering 4 million to 5 million transactions per second (TPS) with sub-millisecond latencies for read-only and mixed workloads.

Achieved in collaboration with AWS and Intel^®, this benchmark shows exceptional results on a remarkably small 20-node AWS cluster powered by Intel^® Xeon Scalable Processors. Furthermore, it proves that even with significant data growth and extreme workloads, the Aerospike Database performs with hundreds of nodes less than other databases, saving up to $10M per application in infrastructure costs.

The benchmark (workload and configuration):

Benchmarks are most meaningful when they are based on real customer use cases.

Aerospike factored varied requirements into its benchmark, which modeled user profile and campaign data in separate databases (namespaces) on a single Aerospike 20-node cluster.

• User profile stores are maintained by Ad Tech demand-side platforms to help advertisers determine how much (if anything) they wish to bid on an available online advertising space.

• Campaign data contains critical transactional information, including budgets and bidding data for digital ads, and this data is managed in a system of record.

This type of workload is utilized not only in Ad Tech; it is common in many industries, including financial services, telecommunications, retail, and others.

From a database platform perspective, the requirements of such applications are extreme, calling for ultra-low data access latencies, 24×7 availability, high levels of scalability, and operational ease with up to 80% fewer instances or nodes.

The benchmark involved read-only and an 80/20 mix of read/write operations against a user profile database populated with 500 billion unique records of varying sizes to yield 1 PB of uncompressed user data. With replication, the total uncompressed size was 2 PB. In addition, Aerospike also benchmarked a 50/50 mix of read/write operations against a campaign database with 6 billion unique records totaling 1.5 TB uncompressed; with replication, the total uncompressed size was 3 TB. Workloads against both databases were run concurrently, as is typical in production scenarios.

For Aerospike’s real-time data platform, AWS provided the best instance types due to their large NVMe SSDs and powerful Intel CPUs.

Figure 1 – The Petabyte scale benchmark configuration on AWS

As shown in Figure 1, all benchmark tests were run on a 20-node AWS cluster with Aerospike

Enterprise Server 5.4.0.2-1. Each EC2 i3en.24xlarge node with Intel^® Xeon^® Scalable processors featured 768 GB of DRAM and 8 x 7500 GB NVMe SSDs. For clients, the benchmark employed 40 Amazon EC2 c5n.9xlarge nodes using Intel^® Xeon^® Scalable processors with 96 GB DRAM each and EBS-only storage; operations were executed using Aerospike’s C client.

The results:

As shown in Table 1, Aerospike’s Real-time Data Platform delivers 4 million to 5 million transactions per second (TPS) with sub-millisecond latencies for read-only and mixed workloads. These fast speeds and strong transaction volumes are particularly remarkable given the 20-node server size. Other NoSQL (or SQL) alternatives would require substantially larger footprints to manage similar transaction volumes and might still be unable to consistently deliver submillisecond latencies.

Table 1 – Benchmark results. User profile and campaign workloads ran concurrently

Dramatically Fewer Instances and Nodes – The TCO:

As cost is an important factor in modern IT infrastructures, it makes sense to compare Aerospike with alternative solutions from a TCO perspective. Aerospike petabyte-scale benchmark showed the platform delivering 4 – 5 million transactions per second (TPS) for read-only and mixed workloads with sub-millisecond latencies on a 20-node AWS cluster. Compared with alternative approaches (such as Apache Cassandra), Aerospike’s footprint is remarkably small, driving substantial savings in TCO.

Figure 2 – TCO (3 Year) comparison between Aerospike and Cassandra

As Figure 2 depicts, the 3-year TCO for running such a cluster in production is a modest $4.17 million, even with 15% annual growth in data volumes. By contrast, an Apache Cassandra-based architecture would require an estimated $15.29 million over the same 3-year period. That’s $11 million in added expenses over the Aerospike implementation. (For more details on the cost calculations, please refer to the petabyte-scale benchmark.)

Aerospike’s architecture – what made these results possible

The results above are unique; there are no similar benchmarks from NoSQL vendors at the same level of performance at scale with such a low server footprint. These results are possible due to Aerospike’s architecture.

Deployed to support systems of engagement or systems of record (with Strong Consistency), Aerospike delivers exceptional scalability and runtime performance with low TCO because it is optimized to exploit modern hardware, including multi-core processors with non-uniform memory access (NUMA), non-volatile memory extended (NVMe) Flash drives, persistent memory (PMem), network application device queues (ADQ), and more. Such optimizations, coupled with Aerospike’s multi-threaded architecture and other features, provide firms with distinct advantages for both on-premises and cloud deployments.

For example, Aerospike automatically distributes data evenly across its shared-nothing architecture clusters, dynamically rebalances workloads, and accommodates software upgrades and most cluster changes without downtime. In the cloud (in this case, on AWS), Aerospike exploits caching on ephemeral devices, backing up data on Elastic Block Store (EBS) volumes.

Furthermore, Aerospike’s flexible configuration options enable administrators to choose where to keep indexes and user data — all in traditional memory (DRAM), all in PMem, all on Flash (SSDs), or in hybrid configurations with indexes in DRAM or PMem and user data on Flash.

Such flexibility enables firms to use Aerospike in different ways to support various business needs without incurring excess hardware costs or compromising on application requirements. It also helps firms standardize on a common database platform at the edge and at the core to simplify operations and reduce overall expenses. For further details, see Introducing Aerospike’s architecture.

Customers

The benchmark is another illustration of the real-world experiences of Aerospike’s global customers. The report illustrates how companies like PayPal, ThreatMetrix(Lexis Nexis), Verizon Media (Yahoo), and Signal (TransUnion) significantly reduced their server counts and cut costs by millions of dollars. In most cases, these customers turned to Aerospike after other solutions failed to effectively scale up to meet their rapidly growing data and application needs.

Next steps

Download the full benchmark report and watch the presentation from the recent 2021 Aerospike Digital Summit for detailed descriptions of the workloads, configurations, and performance.

For more information on Aerospike on AWS click here.