Qualifying cloud instances

The Aerospike Cloud Qualifying Process evaluates cloud providers’ instance and device performance by simulating a steady read and write workload, and measuring response time. These are tests that should be performed to get an idea of the level of performance you can expect from cloud instances.

If your instance is not one that has been tested by Aerospike, you can test it yourself by downloading the open sourced Aerospike Cloud Qualification package to test your instances, or by utilizing YCSB directly.

Qualified Instance Types

Cloud instance types will give you a certain amount of resources (CPU, Memory, Disk and Networking), but it can be hard to translate that into expected performance. The Cloud Qualification Process will take these instances and qualify it capable of a certain operational level, as an overall metric. It does not focus on individual resource components or optimizations.

This page will hopefully provide a starting point into what you can expect from each instance and how the instance class/family is more suitable toward different object sizes and use cases.

The Cloud Qualification Process is done using Aerospike’s fork of the Yahoo! Cloud Serving Benchmark.

The testing methodology is as follows:

Workload	Access Pattern	Record Count*	Record Size*	Cluster Size	YCSB Duration
50/50 R/U	Uniform	60% of memory	50% of disk	3	3 * Record Count

* Based on a 2 node cluster

The testing client is a single instance of a top-end system focused on compute power and network.

The entire test is repeated at least 3 times for verification

Aerospike is configured to utilize local SSD storage (when available), but is otherwise left as default, as are the instances that Aerospike is deployed on. You may see improved performance from additional tuning. You may also choose to alter these numbers and percentages in your own run of YCSB.

An instance is considered passing at an operational level (eg: 30,000 ops/s) if the average read latency does not exceed 1 millisecond from the client. 95 and 99 percentiles, as well as object size and count, are provided as reference.

Larger operational levels are better.

Your results may vary from these published numbers due to differences in server, topology, and even variation between systems of the same type.

The table below shows the results from the Aerospike Cloud Qualification Process on some popular instance types.

Qualified Instances

AWS

Instance Type	Client Transactions +	95th Percentile (ms)	99th Percentile (ms)	Object Size (bytes)	Object Count	Disk Size	Cost ++
i3.xlarge	68,000	3.3	5.4	1424	240,000,000	950GB	$0.312/hr
c3.4xlarge +++	66,000	4.1	11	554	240,000,000	2x160GB	$0.84/hr
i2.xlarge	36,000	3.5	5.8	1424	240,000,000	800GB	$0.424/hr

As measured from the client, 50/50 read/update averaging under 1ms.

++ Price in us-west-2, at time of writing.

+++ All available ephemeral drives were utilized concurrently.

Your numbers may not necessarily match these exactly, due to variance within AWS’s environment.

Azure

Instance Type	Client Transactions +	95th Percentile (ms)	99th Percentile (ms)	Object Size (bytes)	Object Count	Disk Size	Cost ++
Standard_Gs1	31,000	3.5	9.1	13 +++	240,000,000	56GB	$0.61/hr
Standard_L4s	41,000	1.2	5.8	1500	240,000,000	678GB	$0.344/hr

As measured from the client, 50/50 read/update averaging under 1ms.

++ Price in West US, at time of writing.

+++ Low object size is due to fitting 240m objects into a mere 56GB disk. Larger object sizes can be used with the following optimizations:

shorten set name from 9 characters
Enable single-bin for the namespace. This will make 28 additional bytes available.
3 additional bytes are available for Integer or Float types.
This results in up to an additional 39 bytes for data per record without impacting storage space (using a single character set name).

Your numbers may not necessarily match these exactly, due to variance within Azure’s environment.

GCP

Instance Type	Client Transactions +	95th Percentile (ms)	99th Percentile (ms)	Object Size (bytes)	Object Count	Disk Size	Cost ++
n1_standard_8	82,000	3.2	5.9	768	240,000,000	375GB	$0.492/hr +++

As measured from the client, 50/50 read/update averaging under 1ms.

++ Price in us-central1-a, at time of writing.

+++ Cost includes $0.112/hr for a single local ssd.

Your numbers may not necessarily match these exactly, due to variance within Google’s environment.

IBM Cloud

Instance Type	Client Transactions +	95th Percentile (ms)	99th Percentile (ms)	Object Size (bytes)	Object Count	Disk Size	Cost ++
Xeon E3 1270v3 1Gig Networking VM to Baremetal	75,000	2.7	4.3	500	240,000,000	1.2TB	$415/mo ($0.576/hr)
Xeon E3 1270v3 1Gig Networking Baremetal to Baremetal	130,000	3.4	5.5	500	240,000,000	1.2TB	$415/mo ($0.576/hr)
Xeon E3 1270v3 1Gig Networking Baremetal to Baremetal	30,000	3.0	4.6	3000	240,000,000	1.2TB	$415/mo ($0.576/hr)
2x Xeon E5 2620v4 10Gig Networking Baremetal	222,000	1.7	3.2	500	240,000,000	1.2TB	$1109/mo ($1.54/hr)
2x Xeon E5 2620v4 10Gig Networking Baremetal +++	100,000	2.6	3.4	3000	240,000,000	1.2TB	$1109/mo ($1.54/hr)

As measured from the client, 50/50 read/update averaging under 1ms.

++ Price in SJC01/SJC04, at time of writing. System, RAM, Networking and Disk are all separate billable items, shown as a combined sum. Servers were on monthly provisioning, with hourly costs extrapolated using 30-day months.

+++ SSD Device overloaded and testing was terminated early. Testing proceeded at a faster pace than the device was capable of while still able to maintain latency profile. This caused defragmentation to fall behind and exhausted free blocks, so the Aerospike server entered into stop_writes. Read more about stop-writes and how it relates to

defragmentation.

Your numbers may not necessarily match these exactly, due to variance within IBM’s environment.