How infrastructure unpredictability affects engineering productivity

Every senior engineer remembers joining a new team, spending a few weeks reading the codebase, looking at the deployment topology, and asking questions in meetings. And at some point you stop, look up, and ask: Why is it like this?

Why is there a Redis cluster sitting in front of a database that is already supposed to be fast? Why does this service write to two stores and then reconcile them with a nightly job? Why is there a queue between two components that, looking at the data flow, do not need to be decoupled? Why is there a manual operational step in the deploy that someone added a comment to in 2021 saying "remove this when X is fixed," and X has not been fixed?

The answer, almost always, is the same. None of this was designed. It accumulated.

Each piece was a rational response to a specific problem. A latency issue here. An incident there. A workload that did not quite fit the original assumptions. Each fix made sense at the time. And then, over years, the workarounds became the architecture.

This is what happens when systems supporting a team behave unpredictably. Understanding where it comes from, how it compounds, and what it costs is the first step toward doing something about it.

How accretion happens

A team builds a system that works. But as it scales and ages, its behavior becomes less predictable in some subtle way: a database where latency drifts as data accumulates, a service where throughput degrades when one of its nodes is replaced, a queue that stops draining cleanly under certain workload mixes. The team does not have time to fix the root cause. Often they could not fix it even if they had time, because the unpredictability is a property of the underlying system rather than the application code.

So they compensate.

They add a cache to make the slow path tolerable. They add a queue so the system can handle bursts. They add a retry mechanism for the times when something did not work. They add a fallback path for when the retry fails. They split a workload that is too large to handle predictably into smaller pieces that are. They introduce a second datastore that is better at one specific task the original doesn’t do well. They precompute figures that used to be computed on demand. They denormalize data that used to be normalized. They add a manual operational step that someone has to remember to run.

Each individual decision is defensible. A reviewer looking at any one pull request would approve it.

But over a few years, the cumulative effect is a system that is harder to understand. The cache layer has its own problems. The queue backs up in ways that mask the original problem. The retry mechanism interacts with the rate limiter in a way nobody understands. The second datastore needs to be kept consistent with the first, which created a new class of bug. The manual step gets missed once, and now there is an alerting rule whose only job is to detect when someone forgot.

The team that built this is not negligent. They are often the opposite: careful, thoughtful, and responsive to real production incidents. But they were building defenses against unpredictability, and over time, the defenses became the system.

Why each workaround makes sense

The instinct is to blame the team, but the team is almost never the problem.

Imagine an engineer on call. Last Tuesday, the system did something unexpected at 3 a.m. A latency spike caused a downstream service to time out, which caused a cascade of retries, which made the original problem worse. They spent two hours figuring out what happened. 

They wrote a postmortem that more than likely recommended defensive measures such as adding circuit breakers, capping the retry budget, adding a queue in front of the component that timed out and an alerting rule, and documenting a runbook. These are all sensible. They address the failure that just occurred. The system will be slightly more robust against that specific problem. 

But the failure happened because of an underlying property of the system: its tendency to behave unpredictably under certain conditions. The defensive measure does not fix it; it just adds a layer that masks a symptom.

Next month, a different combination of conditions produces a different symptom, with another postmortem and another defensive measure. 

The problem is that the incidents arrive from a place the team cannot reach: the underlying behavior of the systems they build on. And the only tool the team has is to keep adding compensating layers.

Why workarounds do not come back out

Once a workaround is in place, removing it is harder than adding it.

The cache added to make a slow path tolerable now has processes that depend on it. Their code assumes the cache exists. Some of them have built their own logic on top of it: they may read from it, rely on its consistency model, or embed its parameters in their own performance assumptions. Removing the cache means changing all of those things, renegotiating contracts with other teams, migration, and convincing leadership that this is worth doing, even though "remove a cache" does not sound like a project that affects customer value.

So the cache stays. Maybe the underlying system has improved, maybe users don’t use it the same way,  maybe the cache now causes more problems than it solves,  but removing it costs more than leaving it.

This is true of every workaround. Once it is present, it does not come out easily. The system accumulates them the way coral accumulates structure: Each piece glues to the last, and after enough time, you have something nobody designed, but everyone has to navigate.

The worst part is that the original problem, the unpredictability that started all of this, is still there. The workarounds just hid it by adding a new layer of complexity that is itself unpredictable.

How complexity compounds through fan-out

There is a further mathematical aspect to this problem. 

Consider a fairly normal user-facing request: the page load when someone opens an application. Behind that user-visible action, the application may perform an auth check, an entitlement lookup, an offer query, a personalization call, a profile fetch, and one or two feature-flag evaluations. Ten subsystem accesses are routine. Real-world fan-outs are often higher.

Now suppose each of those subsystems is 99% fast, meaning it meets its latency target 99 out of every 100 calls. How often does the user actually get a fast experience?

If the ten subsystem calls were independent, the probability that all ten are fast on a given request is 0.99 multiplied by itself ten times. , or 0.904. So 9.6% of user-visible interactions have at least one slow subsystem call, even though every subsystem in isolation looks fine on its dashboard.

The 1%  that was acceptable at the subsystem level just became almost 10% of the user experience.

With a more realistic fan-out of 20 subsystem calls, that 0.99 becomes 0.818, so 18% of user requests have a slow operation. 

If subsystems are at 99.9% fast instead of 99%, the picture improves but does not disappear: with a 20-call fan-out, approximately 2%  of user interactions still reach it. 

Compounding gets worse in practice because real-world subsystem calls are rarely independent. Multiple subsystems often share a network path, a backend node, a cache layer, or a database connection pool. When that shared resource encounters a problem, several subsystems degrade together. A user request then sees multiple slow operations stacked rather than one. Correlated tail latency turns a 9.6% problem into something disruptive at peak load.

Why p99 misleads

Most teams monitor p99 latency, the 99th percentile, as a proxy for system health. If p99 is acceptable, the system is declared healthy.

But p99 doesn’t show what happens at p99.99. And it is p99.99 and beyond that determines the user experience as fan-out compounds. A subsystem with a tight p99 of 5 milliseconds and a p99.99 of 6 milliseconds is different from one with the same p99 but a p99.99 of 200 milliseconds. The first is predictable, while the second has a long, fat tail that shows up to users during fan-out.

The only solution is to demand more from the subsystems themselves, not just on average, but in the tail. That is where fan-out math stops compounding against the team.

The cost on senior engineers

The cost of all this falls disproportionately on senior engineers. Ask them what they spent their week on, and the answer increasingly is not "building the next thing." They’re fixing problems. They were busy all week, but they didn’t ship much. The three forms of the problem

The problem shows up in three ways that compound.

1. Interruption.

A senior engineer is in the middle of designing something and instead is pulled into a production incident. Two hours later, the incident is resolved, and the engineer goes back to what they were doing.

But they do not really return to where they were. The mental state they were in is gone. Reconstructing it takes another hour, sometimes longer, if they get back to it that day at all.

The work they were pulled into was not shallow, either. Diagnosing an unfamiliar production problem requires loading the architecture of the system, the specifics of recent changes, the behavior of dependencies, the metrics and traces and log timelines, and the history of similar problems. It requires focus, working memory, judgment, and systems thinking. It is the same cognitive effort that goes into designing a complex system,  except instead of being directed at what they’re building, it is being directed at figuring out why something else is misbehaving.

So they pay the cost twice: once to load the production-debugging context, and again later to reload the design context they were in before the interruption. A senior engineer who loses three focused hours per week to this loses more than 150 hours per year. 

2. Context switching.

Subtle, hard-to-explain production behavior requires deep system knowledge to investigate. A junior engineer can handle a clear, well-understood failure. They cannot be expected to untangle a multi-component interaction that nobody on the team has seen before. So the work escalates. For any individual incident, this is the system working correctly. The problem is the volume of escalation, and what it does to the people receiving it month after month.

3. Defensive engineering. When a senior engineer has been burned enough times by unpredictable behavior, they write code differently, adding timeouts, retries, fallbacks, and monitoring. They argue in design reviews for more conservative choices.

Some of this is wisdom; they have learned what can go wrong, and they are protecting against it. But at some point, the balance tips. They are no longer designing for the product they want to build. They are designing around the infrastructure behaviors they are worried about. The code is full of defensive measures against things that might misbehave. New features take longer because every design conversation carries a long list of what-ifs about how the system might fail.

This is the moment that should concern engineering leadership, because it is invisible. The engineer is shipping. The reviews look fine. But velocity has dropped, complexity is growing, and the underlying reason — that the engineer has lost trust in the systems beneath them — is rarely named out loud.

Why this falls on the busiest people 

Junior engineers handle routine work: feature development, clearly defined bugs, and predictable on-call shifts. They are not the ones being pulled into unusual production problems, because they do not yet have the context to resolve them. So unusual problems land on the people who have been at the company longest, who know the most history, and who know how things work.

These are the people who are most expensive in opportunity-cost terms. When a senior engineer spends two hours on a debugging session, the cost is not two hours of their salary, but whatever they would have built in those two hours, plus the hour needed to recover context, plus the compound effect on everything they work on for the rest of the week.

These are also the people who are hardest to replace. An organization can hire more engineers, but it cannot hire engineers with five years of system knowledge. 

What predictable infrastructure looks like

Consider two clusters handling the same sustained workload: 50,000 reads, 40,000 writes, 6,000 updates, and 4,000 deletes per second, run over ten hours. The metric is p99.99 read latency. 

One cluster is Aerospike on four nodes; the other is Cassandra on six. A node failure is deliberately injected at hour four.

The Cassandra cluster stays available throughout, with no outage or errors, but the behavior changes. At hour four, the latency profile becomes more variable. It never returns to where it was before the failure. A system can be available and still become less predictable. 

The Aerospike cluster, on fewer nodes handling the same workload, holds a flat band all the way through, before, during, and after the same disruption.

A second comparison makes a different point. Two clusters run a 6 TB workload over time: Aerospike on eight nodes, ScyllaDB on twelve. 

The ScyllaDB cluster is still running. By most conventional health metrics, it looks fine, with no alerts, no errors, and no outage. But the throughput, the maximum amount of work the cluster sustains, is shrinking. Teams discover this drift when they hit a peak load they used to handle six months ago and find they no longer can.

The Aerospike line is flat. There is variation, but the envelope stays predictable so an engineer can plan against it. 

The point of these comparisons is not that one system is faster than another. The point is the trend line. Predictability is about whether the metric lets the team expect what next week looks like.

Predictability reduces cognitive load

Engineering productivity is connected to cognitive load, how much engineers have to hold in their heads to safely make a change.

When systems are predictable, engineers reason about them. They know what to expect, make changes confidently, and diagnose problems faster, because the system makes sense. 

When systems are unpredictable, every change feels riskier. Will this still work at peak traffic? What happens when this table grows? What if a node fails during this deploy? What if cache hit rates drop?

Each question is reasonable, but requires attention. And focused, unbroken attention from experienced engineers is one of the most expensive resources in any engineering organization. Organizations should not want that resource spent brooding about the infrastructure.

This is also why "hire more engineers" is not the answer. Faster developers do not help if production systems make every change feel dangerous. An organization can provide a team with the best tools, the best AI assistance, and the best processes in the world, but if the underlying system behaves unpredictably, that gets spent on caution rather than construction.

How to break the cycle

There is no one answer. But there are practices that help, especially adopted before accretion becomes unmanageable.

• Treat workarounds as temporary, even when they are not. When adding a defensive measure, document what would have to be true to remove it. If the condition is "we replaced the database," at least it is written down. The worst workarounds are the ones where there is no record of why they exist or when they could go.

• Audit layers periodically. Every few years, look at the compensating infrastructure and ask which are still solving the problem they were added to solve. Some will be solving a problem that has since been fixed elsewhere, or that has changed, or that no longer matters. Those are candidates for removal, even if removal is painful.

• Distinguish defensive complexity from essential complexity. Some complexity in any large system is necessary, but some is defensive, compensating for something that misbehaves.

• Measure the problem. Most organizations do not talk about this openly. Senior engineers know it is happening, but they may not feel they have permission to say "I spent the week on production issues" without it sounding like an excuse. Ask explicitly in reviews: How many hours went to scheduled work, and how many to incident response, debugging, and defensive engineering?

• Protect deep-work time.  If engineering leadership can carve out time where senior engineers are off the escalation path, they can do the jobs they’re supposed to do. This is difficult, because the next incident might need them. But the cost of routing every incident to the most experienced engineers is higher than it appears.

• Address the unpredictability where it actually happens. This is the hardest recommendation, but it is the one that breaks the cycle. If the underlying systems an application depends on behave unpredictably, no amount of compensating layers fixes the problem. The workarounds keep accumulating. The only sustainable answer is to choose foundations whose behavior is predictable enough that compensation is rarely required. That choice is usually made years upstream of where the symptoms appear.

The basic issue

Workarounds become architecture because they are rational at every step. There is no time when someone makes a bad decision. The trajectory is the sum of many small, sensible choices made by capable engineers responding to real problems.

This is why the problem is so persistent, and why "just don't add complexity" is not useful advice. Teams add complexity because something beneath them is misbehaving and they need to ship.

If foundations are predictable, the team builds on top of them. If the foundations are not, the team ends up building around the foundations rather than on them. 

Predictable behavior is the difference between a system that grows by design and a system that grows by accretion. Teams that get to build by design tend to be the ones who got the foundations right.