Optimizing buy-side AdTech with Aerospike's real-time performance

It’s a pretty obvious statement to say that buy-side AdTech empowers advertisers to move faster, bid smarter, and outperform the competition with data-driven precision. It’s a market where speed and intelligence win. It just so happens that choosing the right database can give you a leg-up over your competition.

Whether you work for an advertiser, agency, or any demand-side platform (DSP) interacting with a supply-side platform (SSP), there are a few top prerogatives to keep in mind.

First and foremost is efficient pricing and performance. If you’re like every other buy-side AdTech, you’re trying to maximize return on ad spend (ROAS) by leveraging real-time bidding, optimizing bid strategies, and ensuring cost-effective access to premium audiences. Low latency? Check. High throughput? Yup. But have you considered the required infrastructure costs or how the performance is affected as the system scales? The more data you can process, the faster, and better you’ll do. On the flip side of that equation - picking the wrong database could negatively impact all corners of your business. 

This blog is a hands-on introduction to RTB for developers on the buy-side of AdTech and how a real-time database like Aerospike can help immeasurably.

The buy-side and sell-side of AdTech

In the AdTech ecosystem, the buy-side and sell-side play crucial roles in the RTB process. The buy-side consists of advertisers and agencies looking to purchase ad space inventory to reach their target audience. They use DSPs to bid on impressions in real time.

On the other hand, the sell-side includes publishers and ad networks offering ad inventory. They use SSPs to manage and sell their inventory to the highest bidder.

Why Aerospike is ideal for AdTech real-time use cases

Aerospike's unique architecture and features make it the perfect database for AdTech. Here are some key reasons:

1. Low latency: Aerospike's Hybrid Memory Architecture ensures sub-millisecond read and write latencies, enabling real-time decision-making.

2. High throughput: With its ability to handle millions of transactions per second, Aerospike can effortlessly manage the high volume of bid requests and frequency capping in RTB on the buy-side. On the sell-side, Aerospike can quickly supply data to enrich impressions sent to ad exchanges and publisher networks.

3. Scalability: Aerospike's linear scalability allows it to grow with the increasing demands of buy-side and sell-side, ensuring consistent performance.

4. Reliability: Aerospike's strong consistency and fault-tolerance features ensure data integrity and availability, even in the face of hardware failures.

This blog continues to explain how RTB is handled using Aerospike.

Buy-side canonical model

Below is a simplified general model of the buy-side entities. As we will focus only on RTB, the important entities are shown in red and yellow here.

Campaign: Represents a marketing campaign with an ID, name, list of lineitem IDs, budget, startDate, endDate, and associated Advertiser.

Lineitem: This represents individual ad placements within a campaign. Think of a line item as an execution plan for a portion of a campaign.

Audience: This represents the target user audience for line items with a unique ID, name, and list of segments. Each audience can be associated with one or more UserProfiles.

Segment: Represents a specific group within the audience, such as age range, gender, interests, etc.

Advertiser: Represents the entities that create campaigns to advertise products/services. An advertiser can have multiple campaigns and associated creatives.

Creative: Represents the advertisement itself, which includes an ID, name, URL, format, size, type, and associated line items.

UserProfile

User profiles contain demographic information, interests, activities (websites visited and timestamps), purchases, a list of Lineitem IDs, and audience IDs that associate the audience segments with the user. Key parts of a user profile include:

1. Demographics: Contains age range, gender, location, etc., which are attributes of a UserProfile.

2. Location: Represents the city, state, country, etc., for a UserProfile.

3. LineitemIds: a List of Lineitem IDs eligible for this user.

The UserProfile is a document that stores the fields in Aerospike b veins. Fields such as interests, location, demographics, and lineitemIds (and others) are stored in the collection data type b Bins. A single record stores all bins.

Campaigns and Lineitems

Brands or advertisers create advertising campaigns to buy advertising space on a retailer's or publisher's website, device app, or online video service (YouTube et al.). They aim to promote products, services, or awareness to sell products or gain followers (a.k.a. influencers, religion, politics).

Every campaign has a set of distinctive goals, including:

1. Awareness: The initial contact between potential consumers and a brand, product, service, or affiliation

2. Discovery: Potential customers visit the website, app, or social media channels (YouTube, Instagram, TikTok, etc.)

3. Consideration: Prospects evaluate the offerings

4. Conversion: A prospect becomes a customer and buys a product, service, or subscription

5. Retention: Retaining existing customers

A campaign has an ID, name, budget, start and end dates, and a set of Lineitems or AdSets.

A Lineitem (or AdSet) is an execution plan for a portion of the campaign that defines the parameters for delivering ads to a target audience. It includes the budget, bid strategy, targeting criteria (e.g., demographics, interests, geographic location), ad creatives, and scheduling. By setting these parameters, advertisers can optimize their campaigns to reach the right audience at the right time, maximizing the effectiveness and efficiency of their ad spend. A Lineitem has an ID, name, start date, end date, status, and budget. It is associated with one or more creatives (banner ads, videos, etc) and an audience.

The Lineitem is a central component in the configuration, real-time and measurement of a campaign.

Real-time envelope

What is real-time? The bid cycle occurs when a publisher renders a web page, a mobile app component, or a video from a streaming service. From a user's perspective, it happens in the blink of an eye.

The DSP maintains a real-time UserProfile and Lineitem set in Aerospike. The UserProfile set usually contains 10 billion to 50 billion user profiles. Old profiles expire, and new and updated profiles are added and maintained by offline processing. The Lineitem set contains line items for active campaigns. It is populated and maintained with input from a campaign configuration application and/or a public API. Typically, there are between 50,000 and 300,000 lines.

Lineitems have a state indicating their eligibility for bidding. The campaign configuration controls the states and can only be activated during the period bounded by the start and end dates. The state of a line item may be changed at any time, either by a human, an API, or a set of rules. A limited number (20-50) of eligible Lineitems are associated with a user profile. This association is made by an offline process proprietary to the DSP using data science techniques.

A DSP provides several bidder service instances that listen for bid requests from an external source such as an ad exchange or publisher network or, in the case of direct or header bidding, directly from the SSP representing the retailer or publisher.

The bidder instance receives a bid request (step 2) and extracts information to retrieve a user profile from Aerospike (steps 3-4).

As the bidder can only bid with Lineitems with the status of ACTIVE, which can be changed at any time, it is important to only retrieve ACTIVE Lineitems from the collection associated with the user profile. (Step 5-6)

From the retrieved and active Lineitems, the bidder “chooses the best” item to bid with; this is an in-code algorithm proprietary to the DSP. (Step 7)

A bid response is created using the information in the bid request and the Lineitem and returned to the ad exchange (Steps 8-10).

The bidder must return a bid response within 100 milliseconds of the bid request.

The code

The companion code to this blog demonstrates the following:

• Mock data generation of UserProfiles, campaigns, Lineitems, and creatives

• Bidding simulation where UserProfiles and Linitems are retrieved as part of the bid process

• Secondary index queries on the UserProfiles

The complete code example is on GitHub.

Mock data generation

Generating mock data for UserProfile and Lineitem and creating mock eligibility relationships between them creates a set of data for the bidding simulation and secondary index queries.

The UserProfile entities simulate real user demographics, interests, and location information, while the Lineitem entities simulate campaign execution plans.

Relationships between UserProfile and Lineitem entities are also generated to create a more realistic environment reflecting potential eligibility scenarios. For instance, a user from a particular demographic and geographical location might be more likely to purchase certain types of line items.

RTB bidding simulation

The MockBidder class simulates the behavior of an RTB bidder. When a bid request is received, it matches the requests to a user profile, selects the “best” line item, and formulates a bid response. In the real world, this must be done in under 100 milliseconds.

Retrieving UserProfile

Retrieving the user profile is easy using a get operation using the profile's primary key. The retrieval latency is typically 0.5 to 1.1 milliseconds, averaging at 0.85 milliseconds; it remains predictable regardless of the number of profiles. (Numbers come from a large AdTech company using Aerospike in production).

public UserProfile fetchUser(String userId) {
        Record record = client.get(null, new Key(NAMESPACE, "profiles", userId));
        if (record == null) {
            return null;
        }
        List<?> activityList = record.getList("activity");
        List<ActivityEvent> activityEvents = activityList.stream()
                .map(map -> ActivityEvent.fromMap((Map<String, Object>)map))
                .collect(Collectors.toList());
        List<?> purchasesList = record.getList("purchases");
        List<Purchase> purchases = purchasesList.stream().map(map -> Purchase
                .fromMap((Map<String, Object>) map))
                .collect(Collectors.toList());

        UserProfile user = new UserProfile(record.getString("id"), new Date(record.getLong("createdAt")),
                new Date(record.getLong("updatedAt")),
                      Demographics.fromMap(record.getMap("demographics")),
                      (List<String>) record.getList("interests"),
                      Location.fromMap(record.getMap("location")), activityEvents,
                                purchases, (List<String>) record.getList("lineitemIds"));
        return user;
    }

• Line 2 uses the get operation to read a user profile record by its primary key.

• Lines 3 through 21 manufacture a Java UserProfile class using the data from the Aerospike record.

Reading Lineitems with Aerospike Expressions

A user profile contains a list of eligible Lineitem IDs limited to approximately 50. These Lineitems closely match the user’s interests and are eligible for bidding.

The bidder retrieves these Lineitems using a “batch read” with a filter that returns only active line items. The Lineitem to bid with is selected from the list using a proprietary DSP algorithm coded in the bidder.

1:public List<Lineitem> activeLineitems(List<String> ids) {
 2:    Key[] keys = ids.stream().map(id -> new Key(NAMESPACE, "lineitems",
 3:        id)).toArray(Key[]::new);
 4:    Expression filter = Exp.build(Exp.eq(Exp.stringBin("status"),
 5:        Exp.val(LineitemStatus.ACTIVE.toString())));
 6:    BatchPolicy batchPolicy = new BatchPolicy();
 7:    batchPolicy.filterExp = filter;
 8:    Record[] records = client.get(batchPolicy, keys);
 9:    List<Lineitem> lineitems = Arrays.stream(records)
10:        .filter(record -> record != null).map(record -> {
11:        Lineitem lineitem = new Lineitem(record.getString("id"),
12:                record.getString("campaignId"),
13:                record.getString("name"), new Date(record.getLong("startDate")),
14:                new Date(record.getLong("endDate")), record.getInt("budget"),
15:                Audience.fromMap(record.getMap("audience")),
16:                LineitemStatus.valueOf(record.getString("status")),
17:                (List<String>) (record.getList("creatives")));
18:        return lineitem;
19:    }).collect(Collectors.toList());
20:    return lineitems;
21:}

• Line 4 builds the ACTIVE filter using Aerospike Expressions

• Lines 5-7 create the batch policy, including the filter

• Line 8 executes a batch read using the filter

• Lines 9-17 create Java POJO objects from the returned records

Secondary index queries

Secondary index queries are not part of bid request processing but are used to query the user profile set.

Querying on a field in a document

One example is audience segment creation, where a query on the UserProfile set could return users who match a location segment in a specific city.

The location bin is a map containing a document structure (think JSON object); one of the map's properties is the city.

public void queryUsersByLocation(String city) {
        Statement stmt = new Statement();
        stmt.setNamespace(NAMESPACE);
        stmt.setSetName("profiles");
        stmt.setIndexName("location_index");

        // filter by the city
        stmt.setFilter(Filter.contains("location", IndexCollectionType.MAPVALUES, city));

        Log.info("Querying profiles in city: " + city);
        RecordSet rs = client.query(null, stmt);

        AtomicInteger recordCount = new AtomicInteger(0);

        rs.forEach((KeyRecord record) -> {
            recordCount.incrementAndGet();
        });
        Log.info("...found " + recordCount.get() + " matching records");
        rs.close();
    }

• Line 8 sets the filter on the location bin. Note that the filter type is: contains and IndexCollectionType.MAPVALUES

Querying for values in a List

Another example is a query to determine users that have a particular interest. Interests are stored in the interests bin as a list of strings.

public void queryUsersByInterest(String interest) {
        Statement stmt = new Statement();
        stmt.setNamespace(NAMESPACE);
        stmt.setSetName("profiles");
        stmt.setIndexName("interests_index");

        // filter by the interest value
        stmt.setFilter(Filter.contains("interests", IndexCollectionType.LIST,
                      interest));

        Log.info("Querying profiles with interest: " + interest);
        RecordSet rs = client.query(null, stmt);

        AtomicInteger recordCount = new AtomicInteger(0);

        rs.forEach((KeyRecord record) -> {
            recordCount.incrementAndGet();
        });
        Log.info("...found " + recordCount.get() + " matching records");
        rs.close();
    }

• Line 7 sets the filter on the collection of interests; note that the filter type is: contains and IndexCollectionType.LIST

• Line 11 executes the query

• Lines 15-17 process the results

Range query example

A final example is a range query on the UserProfile createdAt date.

public void queryUsersByCreatedDate(Date startDate, Date endDate) {
        Statement stmt = new Statement();
        stmt.setNamespace(NAMESPACE);
        stmt.setSetName("profiles");
        stmt.setIndexName("creation_index");

        // Set the date range for the query
        stmt.setFilter(Filter.range("createdAt", startDate.getTime(), endDate.getTime()));

        Log.info("Querying profiles created between " + startDate + " and " + endDate);
        RecordSet rs = client.query(null, stmt);
        AtomicInteger recordCount = new AtomicInteger(0);

        rs.forEach((KeyRecord record) -> {
            recordCount.incrementAndGet();
        });
        Log.info("...found " + recordCount.get() + " matching records");
        rs.close();
    }

• Line 8 sets the range filter Filter.range(…)

• Line 11 executes the query.

• Lines 14-16 process the results

Achieving real-time performance and efficiency in buy-side AdTech

Aerospike's exceptional performance, scalability, and reliability make it the ideal database for RTB in AdTech. It provides a predictable latency of approximately 1 ms or less, regardless of the number of user profiles, Lineitems, audience segments, or creatives, at a hosting cost that is usually 20% of the equivalent solution implemented with other technologies. It also removes the cognitive load (pain) of managing, balancing, and reindexing clusters of database servers.

By leveraging Aerospike, DSPs can enhance their bidding strategies, improve ad targeting, and ultimately drive better results for advertisers. As the digital advertising landscape continues to evolve, Aerospike stands out as a powerful tool for navigating the complexities of RTB and achieving success in the competitive world of AdTech.