How Dataminr scales real-time AI and generative analysis with Aerospike

Dataminr’s ingest volume increased far faster than its legacy DynamoDB and Redis pipeline could evolve. Sudden spikes in global activity produced sustained traffic surges that exceeded the system’s processing capacity. The previous design struggled to absorb this growth at real-time speeds, causing backpressure in downstream enrichment steps and instability during peak demand.

The enrichment pipeline relies on more than fifty AI models, each requiring rapid reads and writes as they add metadata and pass updated signals forward. The existing data layer struggled to support this high rate of small, sequential operations. As a result, coordination between models slowed down, causing latency spikes and delaying the reduction of raw content, which must be shrunk by more than 95 percent before alert generation.

Dataminr reached the limits of its Redis-based in-memory system as data volume and model complexity surged. The platform needed to process massive amounts of messages every second and add millions of labels every second at ultra-low latency to maintain real-time event detection. These in-memory constraints prevented Dataminr from scaling to larger message payloads, richer metadata, and higher throughput during periods of hyper growth.

Parts of the enrichment and search workflow still relied on earlier cloud native data services, such as DynamoDB for persistent storage and Redis for caching. This created a two-database pattern that added operational overhead, increased cost, and made it difficult to maintain low latency at scale. These systems were effective in earlier stages of the company but were not designed for the high throughput, metadata-driven access patterns required as Dataminr expanded globally and added more AI and ML workloads.

Managing real-time workloads across multiple regions and availability zones (AZs) amplified data movement costs and scaling challenges. Dataminr needed a more cost-efficient and predictable foundation as it expanded into new markets, including EU deployments focused on AI and ML search use cases.

Aerospike now serves as the central data layer powering predictive AI models and larger generative AI systems. Predictive models provide language detection, geodata, entity extraction, OCR, and logo detection. Larger neural networks and LLMs refine events and produce dynamic briefs. Aerospike supports both stages by supplying reliable sub-millisecond access to enrichment metadata and model outputs.

Dataminr simplified its architecture by replacing earlier cloud data services, including DynamoDB and Redis, with Aerospike. This removed operational friction, improved consistency, and consolidated real-time data access into a single platform. Aerospike delivers the throughput required to process billions of signals and millions of label updates per second.

As the content acquisition system normalizes, deduplicates, segments, and transcribes global inputs, Aerospike handles the rapid write rate and persistent scale required for downstream analysis. This supports Dataminr’s goal of enriching data while reducing raw volume by more than 95 percent before alert generation.

Aerospike allows Dataminr to increase the scope and size of messages processed during enrichment. The pipeline can now store larger representations, richer labels, and more metadata per signal, enabling broader detection across new modalities and emerging data patterns.

Aerospike’s architecture minimizes unnecessary data transfer and supports operational features such as rack awareness and flexible cluster sizing. This improves cost efficiency for multi-region workloads and supports Dataminr’s expansion into the EU with consistent performance.

Aerospike powers the enrichment path that must operate within strict real-time windows. Faster processing helps Dataminr deliver alerts significantly ahead of traditional media, with documented cases of warnings delivered more than an hour before major outlets.

Dataminr now handles billions of signals per day across text, image, video, audio, and sensor modalities. Aerospike ensures stable, predictable performance even during extreme surges in global activity.

Dataminr reduces raw input volume by more than 95 percent before generating alerts. This aggressive reduction pipeline, supported by Aerospike’s high throughput, ensures only the most relevant signals reach downstream predictive and generative AI models.

Aerospike’s low-latency writes allow Dataminr to add millions of labels per second, supporting metadata-driven detection, improved filtering, and high-quality model inputs. This capability is essential to preserving Dataminr’s identity as a real-time AI platform.

With rapid back and forth between models, Dataminr enhances its detection quality. Predictive AI enriches content, while Dataminr’s generative system, ReGenAI, uses large language models to create continuously updating event briefs. Aerospike also enables Dataminr to process larger and more complex messages, expanding the range of data types and events the platform can detect.

By consolidating earlier databases into Aerospike and reducing multi-AZ data movement, Dataminr improved cost control while scaling globally. The architecture now supports rapid growth without proportional increases in infrastructure cost.