What is operational AI? The data layer behind real-time decisions

When a payment app approves a charge, a retailer re-ranks the products on a page, or a support system picks the next ticket someone sees, there's a model making a decision inside a running business process. Every one of those decisions has a deadline. The model has to read what's true about the account, cart, or customer right then, act on it, and store what happened before the next decision on that record shows up.

Quick answer

Operational AI is artificial intelligence that makes or triggers business decisions inside live processes, within a fixed time budget, using data that is current when the decision runs. Examples include fraud scoring at checkout, dynamic pricing, search ranking, and AI agents that update records in production systems. In comparison, analytical AI produces forecasts, reports, and business intelligence that people act on later, while operational AI sends its output into the process itself.

An operational AI system keeps up in production only if its data path handles six problems:

• Training-serving skew: The data the model reads when it's serving has to match the data it learned from, or accuracy drops in production.

• Freshness: The data has to be recent enough for the decision at hand.

• Tail latency: The slowest lookups still have to fit inside the decision's time budget.

• Consistency: When an earlier decision changes a record, the next decision has to see that write.

• Reliability: When a node, drive, or region fails, the system has to keep working or fail safely.

• Cost at scale: The state the system reads and writes has to stay affordable as entities and history grow.

Agentic AI runs into the same six problems with a different kind of data: the checkpoints, session state, and memory they build up as they work.

What is operational AI, and how is it different from analytical AI?

Operational AI is artificial intelligence that runs inside a live business process and makes or triggers a decision while the work is still happening. Training produces the model. Operational AI is about inference: the model rates something that is happening right now, and the business acts on that rating within a strict time limit. 

Some vendors and analysts use the term differently, and some call the same idea operational intelligence. An AI system fits it when three conditions hold:

• It acts in the process. The output approves, declines, routes, prices, or updates something, and the result is used immediately. 

• It has a time budget. The decision has to finish before a customer, counterparty, or downstream system moves on. At card checkout, that's a fraction of a second, because the customer is waiting.

• It reads current state. Some of the inputs are seconds old, such as the last swipe on a card or the item someone just added to a cart.

Stateful decisions such as fraud scoring also write back the outcome, which means the decision and its result become inputs to the next decision on the same account, cart, or device. A search-ranking model may not write anything back.

Analytical AI, which covers most analytics and business intelligence work, turns data into actionable insights for a person to act on, usually hours or days later. The table compares the two. The boundaries aren't strict, though: Some analytics workloads run in real time, and operational systems read history too.

Let’s look at a card payment as an example. Say a fraud model uses a feature that counts how many times the card was used in the last 10 minutes. If that counter doesn’t keep up with the event stream, the model sees a quiet card even though it was used five times in that window. The model is doing the right thing, but the score is wrong because its inputs describe the account as it looked several minutes ago.

That's why the definition gives data as much weight as the model. Reading current state and writing back outcomes are both data operations, and how they behave depends on live traffic, which a fixed test set doesn't have.

There's a quick test for whether an example belongs here: If the decision has a deadline, and its inputs change between one decision and the next, treat it as operational AI and plan the data layer for it from the beginning. If neither is true, batch pipelines and analytical tools usually do the job for less money.

How does operational AI relate to MLOps, AIOps, real-time ML, and feature stores?

Operational AI is a business outcome: AI that makes live decisions. It's a narrower idea than enterprise AI, which covers every way a company uses artificial intelligence. 

However, components that are actually operational AI aren’t always called that. A team running fraud scoring might say it runs real-time machine learning (ML), an online feature store, and an MLOps pipeline, rather than operational AI. Here are some commonly used AI terms and how they fit with operational AI. 

Is operational AI the same as AIOps?

No. AIOps applies AI to running IT systems, using data such as logs, metrics, and alerts. Operational AI applies AI to business decisions, such as approving payments or pricing orders. They overlap when one of these AIOps solutions acts on its own, such as an agent that fixes a problem instead of just flagging it. At that point, the IT team is running operational AI on IT data.

What is the difference between operational AI and MLOps?

MLOps is how a team builds, deploys, and maintains models. Operational AI is what those models do once they're making live decisions. A company can practice MLOps on models that run only in nightly batch jobs, which isn’t operational AI. An operational AI system still needs MLOps, though, so it retrains and redeploys its models safely.

MLOps tooling covers the model lifecycle and the feature store covers part of the data path, but neither one covers everything a live decision needs, such as recording the outcome so the next decision can read it. Instead of relying on a broad marketing label, evaluate products by the specific functions they provide. 

What happens inside an operational AI decision?

Stateful operational AI decisions all run the same set of actions in a loop, no matter the model or the industry. There are six steps, and the last one feeds the first, so every decision depends on how well the previous decision on that record got recorded. (Stateless decisions, such as some ranking requests, skip the write-back steps.)

1. State: These are the operational records behind the decision, such as account balances, recent transactions, device history, or inventory counts.

2. Context: This is the slice of that state this particular decision needs, pulled together inside the time budget. For a card payment, that could be the card's recent transactions, a merchant risk score, and a device fingerprint.

3. Decision: A model, a rules engine, an agent, or some mix of them looks at the context and produces a score or a choice.

4. Action: The system approves, declines, routes, prices, or replies.

5. Consequence: The action changes something in the business. A payment clears, an account gets flagged, or an order ships from a different warehouse.

6. Updated state: The consequence gets written back where the next decision on that account, cart, or device can read it.

What components does an operational AI data stack need?

Every step in the loop maps to a part of the stack, and each part has one job it has to do well under production load.

How much time does each step get?

The time budget gets split across every step. Here's one way a team might divide 100 milliseconds. The numbers show how the shares add up, and your own split will depend on the model, the network, and how many lookups you make.

This makes two assumptions:

1. The write-back is synchronous and replicated before the decision returns, so the next decision on that record reads the new state even if a node fails. A fire-and-forget write, or one the application sends without waiting for confirmation, takes less time, but if the node that took it fails before the replica write is stored, the next decision reads the old state. 

2. The retry reserve is the size of a full context read, because a retry can't finish in time if it's smaller than the operation it's repeating. 

Inference time varies a lot from model to model, and this example assumes a compact fraud or pricing model.

In feature-serving systems, a lot of the time goes to context assembly, particularly when network round trips to the store are involved. 

Context assembly and write-back are also the two steps that offline model evaluation doesn’t run under live conditions. A data scientist's notebook, such as a Jupyter notebook, loads a training set that's already assembled, and it doesn't record outcomes for the next prediction. Integration and replay tests check a model’s logic, but they can't tell you how it behaves under production load, so it might fail even if it scores well in testing. 

When you're scoping a new use case, write down the decision's time budget first and give each of the six steps a share before you pick tools. If a step has no owner or no budget, that's where the system will miss its deadline once traffic grows.

What are common operational AI use cases?

The same six-step loop shows up across business operations in every industry. What changes from one use case to the next is the time budget and the data each decision reads, and grouping use cases by decision type, instead of by industry, makes those requirements easier to compare.

Payment authorization shows how tight fraud authorization can get. Authorizing a payment takes 380 milliseconds on average^[1], and fraud checks are one of several services running inside that window. Checkout.com publishes average latency but not tail latency, so use its figure only as a rough estimate of the time involved.

Agents already make some of these decisions in production. In LangChain's 2026 State of Agent Engineering survey^[2], customer service was the most common primary use case for agents, at 26.5% of respondents.

Which decisions need sub-second data?

A decision needs sub-second data when its inputs change faster than the decision window, and a stale input would change the outcome. Card velocity counters, shopping cart contents, and inventory at one warehouse are examples. Industrial maintenance decisions can usually manage with data that's seconds or minutes old, because a bearing doesn't wear out between two sensor readings.

Across the use cases in the table, time budgets and data requirements vary more than the kinds of models do. That's why the data layer is usually where one project's architecture ends up different from another's.

When an organization is picking its first project, look for a decision where fresher or more complete data would visibly change the outcome, and where you can measure that outcome. Fraud loss, conversion rate, and on-time delivery are easier to measure than most model metrics, which makes them a more obvious business benefit. 

Why do so many operational AI problems come from data?

Once a model's deployed, its weights stay fixed between retraining runs, while the data it reads changes with every event. Model serving and its dependencies can still change, but the inputs change constantly. So a lot of production problems turn up in the data path: inputs that don't match training, inputs that arrive late, lookups that run slowly, and state that grows faster than the budget.

Practitioner surveys say the same thing. In the State of Production ML 2025 survey^[3] of 135 self-selected practitioners, the most common production challenge was monitoring and observability, at 42%. Access to relevant training data came next at 37%, and building production-grade data pipelines and inconsistency between training and experimentation environments each came in at 34%. Two of the top three, data access and data pipelines, are about data. Google engineers made a similar point in Hidden Technical Debt in Machine Learning Systems^[4], which found that data dependencies in ML systems build up technical debt the same way code dependencies do, and they're harder to detect.

What should I change when model quality is stable but inference latency gets worse?

Start with the data path, because a model's compute time rarely changes by itself. Measure how long context assembly takes per decision, count how many separate lookups each decision makes, and watch the p99 latency of the store behind those lookups as traffic grows. If the store's tail latency is climbing, a faster model won't help you. You need a data layer that holds predictable latency at your request rate, or a design that makes fewer lookups per decision.

Should a model read from a database layer or get its data in the request?

Passing data in the request works when the caller already has everything the decision needs. Most operational decisions need more than that: account history, counters across lots of events, and the outcomes of other decisions on the same record. Those are stored in a data layer that every caller shares, and that's also where each decision's outcome gets recorded so the next caller sees it.

So the model and the data path need separate owners and separate budgets, and engineering time should follow that split. Expect to spend more of it on skew, freshness, latency, consistency, reliability, and cost in the data path than on picking a model, and set up monitoring for all six before releasing it to production. 

What is training-serving skew, and why does it happen?

Training-serving skew is a mismatch between the data a model learned from and the data it reads in production, plus the accuracy drop that comes with it. It usually doesn't generate any error messages. The model keeps returning confident scores while its decisions get worse, because what it sees in production differs from what it learned from.

Google's Rules of Machine Learning^[5] lists three causes: the training and serving pipelines handle data differently, the data changes between training and serving, or there's a feedback loop between the model and the system using it. It recommends monitoring skew explicitly to look for it. 

Why does a model perform well offline but poorly in production?

In an operational AI system, a lot of skew comes from the serving path, where the model's inputs get assembled under time pressure. The table lists the usual sources and how to check for each one.

The last two rows depend on how the data layer behaves under load. The Feast tuning guide^[6], for example, notes that eventually consistent DynamoDB reads are faster and cheaper, and it advises turning on consistent reads only when you need read-after-write consistency. That's a fine default in plenty of situations, but for a fraud counter that has to include the payment made a second ago, it's a source of skew. 

The three timing problems in the table each need their own fix: stale materialized features need faster pipelines, late events need watermarking or correction logic, and stale reads need a stronger read setting. All of them show up as a served value that differs from what the model would have seen in training.

Does a feature store prevent training-serving skew?

A feature store removes only one source of skew. Sharing one feature definition between training and serving, and generating point-in-time correct training sets^[7], gets rid of the skew that comes from separate code paths. But a feature store can't make a late event show up on time, stop serving code from filling in a default, or change the consistency guarantees of the database underneath it. 

To train as you serve, Google’s Rules of Machine Learning guide recommends saving the features used at serving time and piping them into the logs you train on. The guide reports that YouTube’s home page made this change and saw significant quality improvements and simpler code, though it doesn’t specify what was measured or by how much.

In an operational AI system, much of that skew comes down to data timing and consistency, which makes the data layer responsible for model quality. Three checks catch a lot of it in production: 

• Log served feature values for at least a sample of decisions

• Track the share of decisions that used a default value

• Confirm that the reads the model depends on are consistent with the latest write

How fresh does data need to be for operational AI?

Feature freshness is the age of a feature value when a decision reads it. Every feature needs its own freshness requirement. A fraud model can handle a merchant risk score that's a few hours old, but it can't handle a transaction counter that's a few minutes old. A score updated daily and the same score updated hourly act like two different features, and the model should be trained on the one it'll see in production.

Features tend to fall into three freshness tiers, and each tier needs a different way of computing and serving the data.

Stale data is hard to spot. Google's Rules of Machine Learning^[5] describes a Google Play table that went stale for six months, and refreshing that one table raised the install rate by 2%. Nothing broke, and the model kept running the whole time, but its performance slowly decayed. 

How fresh do features need to be for real-time fraud detection?

Features that count recent activity on a card, account, or device need events from the last few seconds, because fraud often comes in quick bursts of transactions. Slower-moving features, such as merchant history or account age, can be hours or days old without changing the decision.

How do I set freshness requirements for features used in online inference?

1. Estimate what age costs you. For each feature, ask how much the decision changes if the value is a minute, an hour, or a day old. 

2. Set a maximum age per feature. Record it next to the feature definition, and store a timestamp with every feature value.

3. Measure age when it's read. Track the distribution of feature ages at read time, including the slowest percentiles. Feast, for example, offers a feature freshness metric^[6] per feature view, with a sample alert when a view hasn't been refreshed in more than an hour.

4. Pick a fallback. Decide ahead of time what happens when a feature is past its limit: Switch to a more conservative rule, send the case to review, or score without that feature using a model trained for that case.

5. Treat staleness as an incident. A stale feature hits every decision that reads it, so give it the same alerting you'd give a service outage.

Tighter freshness costs more processing and more writes, which makes freshness a budget decision as well as a quality one. A per-feature freshness table gives the data team a spec to build pipelines against, and it keeps teams from paying for streaming on features a nightly job would serve just as well.

Why does tail latency get worse when a decision makes many lookups?

Tail latency is the latency of the slowest requests, usually measured at the 99th or 99.9th percentile (p99 or p99.9). An operational AI decision often reads lots of keys or feature groups at once, and it isn't ready until the slowest of those reads comes back. The more lookups a decision makes, the better the odds that one of them lands in the slow tail.

If each lookup has an independent 1% chance of being slow, the math is simple. The chance that a decision hits at least one slow lookup is 1 − 0.99^N, where N is the number of lookups. (In practice, lookups often share networks, servers, and queues, so slowdowns can be correlated, and production results can come out better or worse than this model predicts.)

Aerospike's guide to p99 latency works through the same compounding for parallel calls. Feature retrieval works the same way. The Feast performance tuning guide^[6] notes that every feature view in a request triggers a separate online store read, and for services that span five to 15 or more feature views, that fan-out accounts for most of the latency.

What is the difference between low average latency and predictable low latency?

Average latency tells you about a typical request, and predictable latency is about the slow ones. In one example, one leaf request finished in 1 ms at the median and 10 ms at p99, a tenfold difference^[8]. When a decision makes lots of reads, the slow end of that distribution determines whether you make the deadline. 

How do I budget p99 latency across feature retrieval, model inference, and post-processing?

1. Start from the deadline. Take the decision's time budget and subtract the fixed costs, such as network hops and request handling.

2. Give each stage a p99 budget. Set 99th-percentile targets for context assembly, inference, and post-processing. Adding up the per-stage p99 values is a planning heuristic and won't guarantee the end-to-end p99, so check the full latency distribution under realistic load.

3. Turn the context budget into a per-lookup target. If a decision makes 10 lookups in parallel and the context stage has to meet its target 99% of the time, each lookup has to make it about 99.9% of the time, assuming the delays are independent. As a starting point, plan for the store's p99.9 to fit inside the context budget.

4. Test at production load. Tail latency grows with concurrency and data size, so measure at peak request rate with a realistic number of records.

Here are some fixes.

• Cut the number of round trips per decision by storing data that's read together in the same record, or by precomputing feature vectors; that takes store reads from one per feature view down to one per entity. 

• Batch whatever reads are left, so lookups across several feature views or entities go out as one request instead of separate round trips. Aerospike supports batch reads, which send many key lookups in one request instead of one request per key. 

• When a request runs long, send a backup: Dean and Barroso report that hedging after 10 ms cut the p99.9 latency of a 1,000-key read across 100 servers from 1,800 ms to 74 ms, and it added only 2% more requests. 

• Pick a data layer whose p99.9 holds steady at your throughput and dataset size.

When comparing products, ask for p99 and p99.9 latency at your request rate, record count, and read/write mix, and count how many lookups each of your decisions makes.

Do I need a feature store, or can I serve features from a cache or database?

You need a feature store when several models share features, when training sets have to be point-in-time correct, or when streaming features have to stay consistent between training and serving. If you've got one or two models and features computed in batch, you can usually serve them from existing systems, such as a database you already run. Either way, the online store underneath sets the latency, consistency, and cost of every lookup.

What is a feature store?

A feature store is a system for defining machine learning features once and serving them consistently for both training and inference. It may consist of an offline store for building historical training sets and an online store for serving features at low latency in production. 

How is a feature store different from a cache or a database?

Each layer handles a different part of the serving problem, and the table shows where the lines fall.

A cache cuts backend load and speeds up typical reads, but it has a limit for operational AI. But a cache doesn’t fix tail latency^[8] unless the application's whole working set fits in the cache. For fraud scoring or personalization, the working set is every active card, account, or user, so a cache sized for the hottest keys leaves every other lookup waiting on the slower store behind it.

Should I build or buy a feature store?

Plenty of teams still build their own. In the same State of Production ML survey^[3], most feature store users relied on custom in-house tools. The vendor market is consolidating, too: Databricks announced in August 2025^[9] that it would acquire Tecton, a real-time feature store. These are the criteria that usually decide it:

• Latency budget: The p99 and p99.9 the online store has to hold at your request rate

• Ecosystem fit: How well the tool works with the AI platform, training setup, and data warehouse you already have

• Operating burden: Who runs the pipelines, orchestration, and online store that an open-source feature store depends on

• Cost at scale: How storage and compute grow with entity count, feature width, replication, and retention

• Portability: How hard it'd be to move your features if the vendor or platform changes

Which databases work well as an online feature store for large datasets and predictable latency?

The Feast performance tuning guide^[6] calls the online store the biggest factor in feature retrieval latency, and it compares the stores Feast supports. The table reproduces that comparison in alphabetical order.

Median latency is the only figure Feast lists for every store, and it says little about decisions that make lots of lookups. Two stores that both read in under 1 ms at the median can be far apart at p99.9. Feast's medians also aren't a head-to-head benchmark, so measure p99 and p99.9 for each candidate on the same workload, record size, concurrency, and configuration. Feast's general rule recommends Redis when the p99 budget is under 10 ms, noting that the Redis online store needs in-memory capacity for the dataset. Eviction and TTL settings help reduce that. 

When every feature value has to stay available, though, memory cost climbs with entity counts and feature history, and that's the situation Aerospike was designed for. In Aerospike's patented Hybrid Memory Architecture, the primary index stays in memory while record data is stored on SSD, so you get sub-millisecond reads without holding every feature value in DRAM. Aerospike works as the online store under a feature store like Feast, or behind custom feature code. It isn't a feature registry. Teams do build offline training sets from Aerospike data through its Spark connector.

The feature store and the online store are separate decisions. The feature store choice is about definitions, training-serving consistency, and how your team works. The online store choice is about latency, consistency, and cost at your data size. Making them separately keeps the second one from defaulting to whatever ships with the first tool, and that matters most once the dataset outgrows memory.

How can strong consistency and low latency coexist for real-time decisions?

"Strong consistency" covers more than one guarantee. Linearizability, the strictest, means every read sees the latest committed write to a record, as if all operations happened in one real-time order. Session consistency is weaker: a client sees its own writes in order but might not see the newest write from another client. Both require coordination between replicas, so many data stores also offer a faster, eventually consistent read that can return an older value. Some databases make you pick one at configuration time. 

You see the tradeoff in everyday configuration. Several widely used databases, DynamoDB among them, default to eventually consistent reads because they're faster and cheaper, and that works fine for features that change slowly. It's the wrong setting for a decision that has to see a payment, reservation, or agent action from a second ago.

In Aerospike, you can choose per namespace and per read, and a data layer can keep consistent reads cheap enough to fit inside a decision's time budget.

When does eventual consistency cause wrong decisions?

Stale reads cause wrong decisions when two decisions affect the same record close together. A consistent read isn't enough by itself, though. Two requests can both read the current value and then both write, so read-check-write sequences also need atomic conditional writes, compare-and-set operations, or transactions. Common cases:

• Counters and limits: Two payments on the same card both pass a velocity check because neither one sees the other.

• Inventory: Two orders reserve the last unit in the same warehouse.

• Actions that have to happen only once: A retry triggers a refund or shipment a second time because the record of the first attempt wasn't visible yet. Idempotency keys or deduplication records prevent this.

• Shared agent state: One agent acts on a ticket or order that another agent already changed.

Features that change slowly, such as a customer's long-term purchase history, rarely need consistent reads.

What does strong consistency at low latency look like in production?

PayPal's risk platform does both at scale. PayPal describes running risk checks at login, account updates, wallet changes, purchases, transfers, and withdrawals, where each interaction changes the state the next decision reads. Its real-time data layer runs on Aerospike with strong c

onsistency and Cross Datacenter Replication, on more than 2,400 nodes in two regions, and it holds P95 latency under 1 ms and P99 under 4 ms. After the change, PayPal raised the share of fraud calculations finished inside its decision window from 98.5% to 99.95%.

The guarantee itself is specific. In strong consistency mode, all writes to one record are applied i

n order, none are reordered or skipped, and each read uses session consistency or linearizability. Under session consistency, a client never goes backward; it sees each record's versions in increasing order. 

Since Aerospike Database 8, the same mode also supports distributed ACID transactions with strict serializability across multiple records, so a read-check-write sequence that spans an account record and a ledger record commits as one unit. Aerospike co-founder Multi-record transactions are somewhat slower^[10] than single-record operations, because each change gets tracked in a transaction monitor. You configure strong consistency per namespace, and it's available in Enterprise Edition and Aerospike Cloud. 

Jepsen, an independent project that tests distributed databases for consistency bugs, analyzed the mode in 2018 on Aerospike 3.99. It found per-key linearizable operations^[11] and confirmed two issues Aerospike had already documented. Database 8's transactions are built on that strong consistency mode.

Strong consistency also doesn't mean giving up availability during routine failures. All partitions stay available when fewer nodes are down than the replication factor. In a network split, a majority sub-cluster that holds either copy of a partition keeps serving it, and data becomes unavailable only when a sub-cluster can't establish that it holds the current copy. Aerospike's default mode, called AP (availability and partition tolerance), keeps taking writes on both sides of a split and gives up consistency to do it. With two copies and session consistency, strong consistency mode performs the same as AP mode when there's no split going on.

What should I ask a database vendor that claims both consistency and low latency?

• Which operations are linearizable, single-record or multi-record?

• What happens to reads and writes during a node failure or network partition?

• What are the p99 and p99.9 latencies for consistent reads, compared with relaxed reads, at your throughput?

• Does cross-region replication preserve the guarantee, or is it asynchronous?

• Has an independent party tested the consistency claims?

Consistency is part of whether a decision is correct, and you mostly pay for it in problems and in the extra work of a fully linearizable read. Sort each record type into "has to show the latest write" or "can lag," and you'll pay for consistency only where a stale read would lead to a wrong action.

What happens to an operational AI decision when the data layer fails or slows down?

A decision can be fast, fresh, consistent, and affordable and still fail if the data layer can't answer when the request comes in. For fraud, payments, and other decisions that can't wait, reliability matters as much as speed, and you have to design the failure behavior ahead of time: which failures the data layer handles, and what the decision does when it can't. If a fraud system is unavailable for even a few seconds during a failover, every payment in that window goes to a fallback path.

Some of these behaviors are database properties the application can't add on later. Aerospike's consistency mode applies writes to replicas immediately by default, which is what lets the next decision read the new state after a node failure. Its strong consistency guide recommends mapping racks to separate availability zones, because correlated failures across zones are much less likely than failures inside one rack.

What should a decision do when its data is late or unavailable?

Every decision needs a fallback picked before launch. Otherwise, someone ends up picking one under pressure in the middle of an incident. PayPal describes the cost of missing its decision window as a forced choice between allowing a potentially fraudulent transaction and interrupting a legitimate one. Common fallbacks:

• A conservative rule: Decline or challenge above an amount or risk threshold, and approve below it.

• A cached or batch score: Use the last known score and flag the decision for review.

• Deferral: Queue the decision when the business can wait, as with a payout. A card authorization can't wait.

• Fail open or fail closed, by action type: A refund might fail closed, meaning it's blocked until the check can run, while a low-value purchase fails open, meaning it goes through without the check.

Which way a decision fails is a business choice, and recording which path each decision took makes that choice auditable. Before launch, have your operations team take a node down under production-like load and measure how many decisions miss their window and which fallback they used.

What does operational AI cost at scale?

Four numbers account for most of the infrastructure cost of an operational AI data layer: 

• How many entities it tracks

• How many bytes each entity's record holds

• The replication factor

• How long data is kept

Where those bytes are stored, DRAM or SSD, determines cost as each number goes up. DRAM costs more per gigabyte than SSD, so if a design keeps every byte in memory, its most expensive line item grows with every new customer, feature, or day of history.

How much memory does an online feature store need for a billion entities?

Here's a worked example. Assume 1 billion entities, 1 KB (1,000 bytes) of feature data per entity, and a replication factor of 2, with every figure in decimal units. The sizing for Aerospike's patented Hybrid Memory Architecture comes from its capacity planning documentation, which puts primary index memory at 64 bytes × replication factor × number of records.

The two designs grow differently. If the feature record goes from 1 KB to 2 KB, the in-memory design needs at least 4 TB of DRAM. The hybrid design still needs about 128 GB, because index size depends on how many records there are, and record size doesn't change it. Aerospike's hardware planning guide gives a rule of thumb for the primary index alone: Each GiB of memory indexes about 16 million records. 

Aerospike Enterprise Edition can also keep the primary index on flash in an All Flash configuration, which cuts DRAM further for very large record counts. Both designs also need room for storage overhead, secondary or set indexes, buffers, and operating headroom, so treat these numbers as minimums. 

On SSD, Aerospike's defragmentation guidance says that at default settings, devices shouldn't be more than 50% full, which puts 2 TB of record data on about 4 TB of device capacity. Each record also carries storage overhead on top of its feature bytes.

Why do RAM-only stores get expensive?

In a RAM-only store, every increase in entity count, feature width, replication, or retention lands on DRAM. The Feast tuning guide lists this as Redis' main tradeoff^[6] as an online store: it needs in-memory capacity for the dataset. Teams can cap memory with time-to-live settings that expire old feature data. That works when old values are useless, but doesn't help when the model needs long history.

PayPal ran into this limit in its fraud platform. PayPal's case study says scaling purely in RAM had become economically inefficient as data volumes grew. After moving to Aerospike's patented Hybrid Memory Architecture, PayPal projected hardware costs dropping from $12.5 million to $3.5 million, a 72% reduction, and separately reported more than an 80% reduction in total cost of ownership over five years. The tradeoff with the hybrid design is that record reads go to SSD, so latency depends on drive performance and configuration. PayPal's data layer still holds P99 latency under 4 ms on that design.

What other costs should you consider? 

• Replication and regions: A second region running its own cluster holds another full copy at its own replication factor, so two regions at replication factor 2 store four copies of every record. Cross-region replication also adds network transfer cost, and each region needs spare capacity to take over the other's traffic during a regional failover.

• Retention: Storage grows with how much history each feature keeps, and agent checkpoints bring their own retention needs.

• Write volume: Tighter freshness means more frequent writes, which adds processing and wears out storage faster.

• Pricing model: Per-request pricing, which Feast lists as a trade-off for DynamoDB, makes spiky traffic harder to budget than provisioned capacity.

• People: Engineers who run pipelines, orchestration, and the online store are an ongoing operational cost and part of total cost of ownership, especially with open-source feature stores that rely on upstream tooling.

How steep the cost curve is matters more than where it starts, because operational AI datasets grow with every customer and every new feature. Before you choose a data layer, model DRAM, SSD, and replica count at your current entity count and at three times that, and ask each vendor how much DRAM it needs per billion records at your record size and replication factor.

How should I manage AI agent state in production?

AI agent state is operational data. Agents read it before each step and write to it after each step, for thousands of sessions at once, under the same latency limits as any other live decision. If checkpoints and memory get treated like application logs, stores behind them lose state on restart, grow without limit, or slow down as tables fill up.

Agentic AI is now a production workload. In LangChain's State of Agent Engineering survey, 57.3% of respondents had agents in production^[2], latency was the second most-cited barrier at 20%, and among enterprises with 2,000 or more employees, security was the second-biggest concern at 24.9%. How agent state is stored affects both latency and security.

How do I keep AI agent checkpoint and session state from growing without limit?

Checkpoint growth tracks the number of steps, and there are more steps than messages. In one LangGraph deployment on Azure Cosmos DB, 16,000 conversations with 36,542 messages produced 282,758 checkpoint records^[12] and about 30 GB of storage. That's roughly 7.7 checkpoint records per message, because LangGraph writes a checkpoint at every step of graph execution, and it works out to about 106 KB per checkpoint record and about 1.9 MB per conversation. A 106 KB write at every graph step is bigger than a typical key-value update, so write latency, write volume, and storage wear all grow with every step an agent takes, on top of total storage. Checkpoint size varies a lot with graph structure, message length, tool use, and retention, so measure your own rate before you size anything.

The LangGraph persistence documentation acknowledges the problem. Checkpoints pile up over long conversations^[13], increasing latency and storage costs, so the documentation recommends pruning old checkpoints or setting a retention policy. Built-in time-to-live (TTL) support depends on the deployment. The standalone PostgresSaver has no built-in TTL^[14] because TTL is a feature of LangGraph's Agent Server, so self-hosted deployments rely on scheduled cleanup jobs. Retention support varies by saver and version, so check the documentation for whichever saver you deploy.

1. Decide what has to be resumable. Keep the latest checkpoint for active threads, and keep full history only where audit or replay needs it.

2. Set retention per state type. Use record-level TTL where the store supports it and scheduled pruning where it doesn't.

3. Shrink each checkpoint. Trim or summarize message history so each snapshot carries less.

4. Keep live state apart from archives. Move closed threads to cheaper storage if you need them for analysis.

5. Watch growth per thread. Track rows and bytes per thread plus checkpoint write latency, and alert when either one climbs.

Which checkpointer or database should I use for LangGraph agents in production?

In-memory savers are for testing. The LangGraph documentation says MemorySaver and InMemorySaver keep checkpoints in RAM and lose all of them when the process restarts, and it points to persistent checkpointers such as PostgresSaver for production. Other than durability, these are the criteria that matter:

• Write latency per step: A checkpoint gets written at every step, so the store's write latency is added to every step the agent takes.

• Native retention: Record-level TTL handled by the database means you don't need cleanup jobs competing with live traffic.

• Tenant isolation: State for different users and customers has to stay separate.

• Consistency: When several workers or agents access the same thread, each one needs to see the others' latest writes.

Key-value databases with server-side expiration handle retention natively. In Aerospike, each record can carry a time to live (TTL), and the server removes expired records, so the application doesn't need cleanup jobs. 

There’s another nuance for checkpoints. By default, every write restarts the record's TTL, so a checkpoint rewritten at every step expires a set time after the agent's last step instead of after the thread started. A write setting keeps the original expiration if you want retention to count from creation. Add low-latency writes and strong consistency for state that several agents share, and the checkpoint workload has many reads and writes per second, each under a latency limit. You still need to evaluate framework integration, checkpoint semantics, and recovery behavior for any store, because low latency and TTL alone don't make a good checkpointer.

Isn't agent memory just chat history?

Chat history is one input to memory. Production memory also needs an owner, a lifetime, a record of where each item came from, and a way to delete it.

Can attackers poison an AI agent's persisted memory?

Yes. In a 2025 paper, researchers at Princeton showed that AI agents can be manipulated through injected fake memories^[15], and that agents defending against prompt injection still failed to stop memory injection. Because the poisoned entry is stored, it influences decisions in later sessions until somebody finds it. 

That makes persisted memory a security boundary. Validating what gets written, recording which user and session wrote each item, isolating tenants, and requiring explicit confirmation before high-risk actions such as moving funds are part of designing the data layer and the agent.

A growth estimate before launch heads off a lot of these. Multiply expected conversations by steps per conversation and bytes per checkpoint, then pick a store and retention policy that keep that number flat over time instead of climbing every week.

When is operational AI not worth the cost?

Real-time decisioning pays off when a decision's inputs change faster than the decision gets made, and when a fresher decision is measurably worth more than it costs to serve. When neither is true, scoring in batch and storing the results is cheaper, simpler, and just as accurate for that decision.

Batch is still the dominant inference mode. In the State of Production ML survey, the biggest group of respondents said fewer than one in ten of their models run real-time inference. Google's Rules of Machine Learning describes the alternative: you can apply a model live^[5], or precompute its results offline and store them in a table. An example is classifying web pages ahead of time while classifying chat messages live.

Do I need a real-time decisioning system if offers are delivered later anyway?

Usually not. If an offer goes out by email hours after the behavior that triggered it, scores refreshed on a schedule handle the decision cheaper. Real-time decisioning is worth the expense when the offer shows up during the session and depends on what the customer just did, such as the item they added to their cart a few seconds ago.

How do I decide which models need real-time inference versus batch scoring?

1. List each decision and when it's used. Note whether a person or a system is waiting on it.

2. Precompute wherever you can. If the decision can be made before it's needed, score it in batch and store the result.

3. Try fresher data on a subset. Serve a sample of traffic with fresher features and measure what happens to the business metric.

4. Move only what showed a lift. Many systems end up hybrid, with batch scores plus a few real-time features, such as a velocity counter added to a nightly risk score.

5. Recheck as the business changes. A new channel or a new type of fraud can change the answer.

In any AI implementation, real-time is a choice you make per decision, and often per feature. Treating it that way keeps infrastructure spending tied to results you've measured.

How should I monitor and govern operational AI in production?

Monitor the whole decision, from the data it read to the outcome it produced, and look past the model's accuracy. Accuracy gets measured against labels, and labels for fraud, churn, or credit decisions often show up weeks after the decision. You can measure the data path on every decision, so problems with freshness, defaults, or latency appear before the labels do.

Monitoring was the most common production challenge in the State of Production ML survey, and most respondents who used a monitoring tool had built it themselves. Agents show a similar split. In LangChain's survey, 89% of organizations had some form of observability for their agents^[2], while 52.4% ran offline evaluations and 37.3% ran online evaluations. NIST's report on monitoring deployed AI systems identifies a challenge: Confirming whether an AI system's predictions are correct can be slow, expensive, or impossible^[16]. Human reviewers may also struggle to keep pace as companies deploy updates more quickly.

What should I monitor when database performance starts dropping?

Watch tail latency per operation at p99 and p99.9, along with timeout and retry rates, because averages make it harder to see individual slow requests. Track how many decisions fell back to a default value because a lookup timed out, because that's where a latency problem affects model quality. 

Look for background work too. Compaction and garbage collection^[8] can also cause periodic latency spikes, including garbage collection inside SSDs. Storage engines built for flash deal with this using large sequential writes, spare device capacity, and background compaction. Aerospike, for example, appends writes into large write blocks and reclaims stale space with a continuous defragmentation process instead of doing small random overwrites.

How much autonomy should an operational AI system have?

Autonomy can go up in stages, and the system should earn each stage with measured results at the one before it:

1. Recommend: The system scores or suggests, and a person decides.

2. Act with review: The system acts, and people review a sample plus every high-risk case.

3. Act within limits: The system acts on its own within set bounds, such as transaction amounts, customer segments, or action types.

4. Act with monitoring and rollback: The system acts broadly, with alerting and a tested way to reverse its actions.

Moving up a stage should depend on error rates measured on actual edge cases, monitoring that covers the new scope, and a tested rollback path. 

Accountability also comes down to records. Every automated decision should be traceable to the inputs it read, the model and rule versions it used, the action it took, and the outcome, which audits and incident reviews need. The NIST AI Risk Management Framework^[17] offers a voluntary structure for that AI governance work.

Monitoring the data path gives you a warning signal weeks before outcome labels arrive. If you add input, data-layer, and decision metrics in the first release, you catch problems while they're still small and show auditors why each decision was made.

How do I get started with operational AI?

For a first AI deployment, start with one decision that has a clear deadline and a measurable business outcome, and design the data path for that decision before you choose tools. One well-instrumented decision shows the organization its latency, freshness, and consistency limits, and those limits carry over to the next use case.

1. Pick one decision and its deadline. Write down the decision, who or what is waiting on it, and the time budget.

2. Map the loop. Trace state, context, decision, action, consequence, and updated state for that decision, and give each step an owner.

3. Set per-feature requirements. For each input, record its freshness limit and whether it has to show the latest write.

4. Count lookups and set tail targets. Turn the time budget into p99 and p99.9 targets for the data layer.

5. Size the data at today's scale and at three times today's scale. Include record count, record size, replication, and retention.

6. Pilot with review and logging. Run the first release with human review on a sample, log served feature values from day one, and test what happens to decisions when a node goes down under load.

7. Measure the business result. Compare fraud loss, conversion, delivery time, or operational efficiency measures such as manual review volume against the process you have now.

How should a company evaluate data infrastructure providers for implementing operational AI?

Evaluate providers against the workload map from the steps above, with your own numbers plugged in. Use these factors: 

• Tail latency: p99 and p99.9 for reads and writes at your request rate, record count, and read/write mix

• Consistency: which operations are linearizable, what happens during failures, and whether an independent party has tested the claims

• Cost curve: DRAM, storage, and node count at one and three times your current data size

• Retention: native record-level expiration for features and agent state

• Operations: replication across zones and regions, behavior during node, drive, and region failures, upgrades without downtime, and the staff needed to run it.

• Ecosystem fit: support as an online store for your feature store, and as a state store for your agent framework

Teams that start this way come out with a requirements document, which makes the infrastructure decision faster and easier to defend.