Real-time fast data queries for optimization at scale

Apoorva Anupindi
Sr. Campaign Marketing Manager
September 1, 2022|6 min read

Today’s marketing and advertising professionals are challenged to keep pace with the ever-increasing demands of the digital consumer. For the modern consumer, general personalization in advertising is no longer enough, they demand individualized experiences. Meeting this demand requires advertisers to develop and scale an ad tech stack with the ability to store terabytes of data about hundreds of millions of individuals. More importantly, it requires the ability to query and action these consumer profiles with the lowest latency possible.

In a recent webinar sponsored by DBTA, industry experts from AWS, Zeta, and Aerospike discussed how Zeta omnichannel marketing platform utilizes Aerospike’s Real-time Data Platform and Amazon Web Services to curate real-time, individualized digital experiences, and how they envision the technology scaling to meet the exponential growth of consumer data.

Daniel Landsman, Head of AdTech solutions at Aerospike, says the ad tech ecosystem is in flux. With many recent changes in the industry, such as GDPR, CCPA, and cookie deprecation, companies are having to reconsider how they target audiences, and how to do it at scale.

Wilson Mai, Principal Software Engineer at Zeta, and Gerry Luow, Head of WW Solutions Architecture for AdTech and Martech at AWS, both agree that “change is the only constant” in the digital advertising world. In 2018, Forbes estimated more than 2.5 quintillion bytes of data were being created every day and Mai states that “handling even a fraction of that data would be a challenge,” and that’s where technologies like AWS and Aerospike become essential to help host and scale a company’s data architecture.

Moderator Stephen Faig, Director of Database Trends and Applications and Unisphere Research, asks the panelists a series of questions about challenges in the ad tech industry, architecting for change, and what to look for in technology solutions.

As companies are processing more and more data, their main challenges include not only keeping up with the incoming data but also handling it in near real time with low latency. As industry requirements continue to change, companies must also plan their infrastructure for said change. Mai explains that Zeta uses Aerospike for the flexibility of making changes without having to rebuild their database architecture, and with AWS they can scale for growth. Luow stresses that building infrastructure in a fast-changing environment comes down to planning for change, structuring for speed, and delivering your minimum viable product as quickly as possible so you can continue to iterate.


Figure 1. Zeta data flow

Mai states that critical requirements include uptime and scale – their products and services need to be highly scalable and highly available so that clients are not impacted by downtime. They also need to account for spiky traffic patterns, meaning they not only need to scale up but also scale fast.

When discussing how to architect for change, Luow says applications should be API-driven and be consistent with autonomous services so you can change and scale them independently. It’s also important to make applications stateless so you can scale faster and scale horizontally, and keep data as regional as possible.

Mai talks about finding the right technology for your requirements. The technology should be able to handle failure scenarios without any downtime or data loss. He mentions that, with Aerospike, even as the company has grown, they’ve been able to scale up and keep the same infrastructure throughout each growth phase. They found that Aerospike’s Real-time Data Platform offered high performance, low latency, and was just as fast as distributed cache lookups, and “there was no need to add that caching layer between our APIs and the database,” which simplified their stack even further. Luow concurs that the ability to handle high volume workloads with low latency performance is one of the main reasons he would select Aerospike.

Landsman describes Aerospike’s unique hybrid memory architecture and how it optimizes for both memory and flash storage together in a single storage layer, which provides the speed of an in-memory architecture and the persistence of a non-volatile architecture. This also enables a higher density per node and significantly reduces costs, even with increasing amounts of data, essentially future-proofing application infrastructure. The speed and persistence Aerospike is able to offer is critical for large-scale, low latency intensive workloads that have to be operational 24/7 globally, especially for ad tech requirements such as real-time bidding, profile stores, recommendation engines, fraud, attribution, analytics, etc.

aerospike-real-time-data-platform 100kb-1024x502

Figure 2. Aerospike Real-time Data Platform

One of the main takeaways from the webinar is the importance of scale. Luow says “the things that make ad tech fun is massive volumes.” In the ad tech industry, you see anywhere from 1 million to 30 million queries per second – that’s billions of queries per day. “It’s not unusual to actually see a year over year growth of 20 to 40% and as much as 70%,” says Luow. He states that the ability to scale infrastructure is a base requirement and the partnership between Aerospike and AWS enables customers to facilitate this type of scale.

Mai adds that with this level of data growth, Zeta is always testing new technologies to offer their clients solutions with insights at this scale, which enables them to reach a much wider audience, “but targeting them and personalization is the key, and providing an omnichannel solution becomes a necessity.” They’ve built their recommendation engine with scale in mind – services are separated and hosted on different instance types and tailored specs on AWS, the engine is designed to have very low latency, and with Aerospike they are able to handle a high workload at that level of scale.

In final thoughts, Mai states it’s important to select the right database and run it on the right hardware. It’s also important to keep experimenting with new features. Luow agrees that experimentation is key and adds that it can have a significant impact on platform performance and efficiency.

Aerospike offers multiple deployment options to get started with Aerospike on AWS for AdTech, including Aerospike Cloud Managed Services, which offers a risk-free approach, and the Aerospike Kubernetes Operator, to deploy on hybrid cloud or multi-cloud environments. Learn more about the deployment options with Aerospike on AWS at