One CDN Engineer Spent a Year Unlearning Cache Invariants Nobody Documented
Every CDN operator learns cache invalidation early. It's the first thing you study, the first thing you debug, the first thing that breaks in production. But the rules you learn—the invariants you assume are carved in stone—might not exist. One engineer spent a year discovering that the cache coherency model he trusted was never documented because nobody had ever verified it against the actual behavior of 768 servers pretending to be one.
The Invariant That Never Existed
Cache invalidation is famously one of the two hard things in computer science. Every distributed systems textbook covers it. Every CDN vendor claims to solve it. But the real-world behavior of purge pipelines, TTL expiration, and edge node consistency is often a black box. The documented invariants—like "a purge is fully propagated within 30 seconds" or "stale objects are never served after invalidation"—are frequently aspirational, not operational.
One engineer, let's call him Alex, joined a major CDN after graduating from Georgia Tech. He had studied distributed systems, read the classic papers, and built toy caches in coursework. He thought he understood the fundamentals. Then his first month on the job: a broken purge pipeline caused stale assets to linger for hours during a major traffic spike. The senior engineers told him to trust the docs. The docs described a system that no longer existed.
The original architecture had been designed around 2015, when the CDN ran on a few hundred servers in a handful of regions. By 2025, the fleet had grown to thousands of nodes across dozens of edge locations, with complex hierarchical caching layers. The documentation had been updated piecemeal, but the invariants—the guarantees about coherency—had never been revalidated. They were inherited assumptions, passed down like folklore.
Alex decided to spend his spare cycles reverse-engineering the actual behavior. He logged every cache miss, every stale hit, every purge acknowledgment, and every latency anomaly. He built a dashboard that compared the documented purge propagation time against real measurements. The gap was not small. The median propagation time was close to the documented 30 seconds, but the tail stretched to over 15 minutes. And that was only for objects that were fully invalidated—some objects never received the purge at all.
A Career Built on Wrong Assumptions
Alex's background was typical for a systems engineer at a large CDN. He had interned at cloud providers, built microservices in Go, and read the standard literature. But his education had not prepared him for the gap between textbook cache coherency and production reality. He had assumed that purges were atomic and total, like a transaction. They were not.
His first month involved a postmortem for an incident where a critical JavaScript bundle was served stale for over an hour. The root cause was a purge that had been acknowledged by the control plane but never delivered to a subset of edge nodes. The documented invariant said purges were retried until acknowledged. But the retry logic had a bug: it would stop retrying after three failures, assuming the target node was offline. The node was online, but its purge queue was full. The object sat in cache, serving old versions, until its TTL expired naturally.
The senior engineers who told him to trust the docs were not malicious. They had been at the company for years, and the docs had worked for them—mostly. But they had never tested the edge cases. They had never instrumented every node. They had never asked whether the invariants still held after the last major architecture overhaul. Alex began to realize that the entire team was operating on shared assumptions that had never been validated against empirical data.
Over the next year, he made it his mission to catalog every discrepancy. He wrote scripts to send synthetic purges and measure propagation. He correlated logs from the control plane, the edge nodes, and the origin servers. He found that roughly 40% of invalidation events were incomplete in some dimension: either not propagated to all nodes, not applied to all variants (e.g., gzip vs. brotli), or not acknowledged within the documented window. No single team owned the full picture—the control plane team assumed the edge team handled retries, and the edge team assumed the control plane guaranteed delivery.
What 768 Servers Actually Do
PlanetScale's recent post on making 768 servers look like one resonated deeply with Alex. The post describes the illusion of a single database that is actually a distributed cluster. CDNs sell the same illusion: a global cache that behaves like a single, coherent store. But the reality is that cache coherency under load is a messy, probabilistic affair.
In Alex's environment, the CDN consisted of multiple tiers: a global load balancer, regional caching layers, and edge nodes closest to users. A single object could be cached in dozens of locations simultaneously. When a purge was issued, the control plane sent invalidation messages to every node that might hold the object. But nodes could be slow to process the message, or the message could be lost in transit, or the node could be temporarily partitioned. The system was designed for eventual consistency, but the documentation promised strong consistency within a bounded time.
The gap between promise and reality was especially stark during traffic spikes. Under normal load, purges propagated within seconds. But during a flash crowd, when thousands of requests per second hit the edge, purge queues would back up. Nodes would prioritize serving requests over processing invalidations. Stale objects would linger for minutes, sometimes hours. The documented invariant—"stale objects are never served after invalidation"—was simply false under those conditions.
Alex found that the illusion of a single coherent cache was maintained by a combination of aggressive TTLs (most objects had TTLs of 5–15 minutes), client-side retries, and a dash of luck. The system worked well enough for most use cases, but it was not the reliable foundation that the documentation implied. Teams that relied on cache invalidation for correctness—like those serving dynamically generated content—would occasionally pay the price.
The Unlearning Process
Alex's unlearning process was systematic. He started by instrumenting every edge node to log purge receipt and application timestamps. He set up a continuous test that issued purges for a known set of objects and measured how long it took for every node to stop serving the old version. The results were a distribution, not a single number.
He then categorized the failure modes. Some nodes never received the purge at all—the control plane had marked them as offline, even though they were serving traffic. Some nodes received the purge but did not apply it because the object was not in cache at that moment (it had been evicted), so the purge was discarded; later, when the object was re-fetched from origin, it could be served stale if the origin itself returned a cached response. Some nodes applied the purge but then immediately re-cached the stale version due to a race condition with a concurrent request.
He logged every miss and every stale hit for three months. The data showed that roughly 40% of invalidation events were incomplete in at least one of these dimensions. The team was shocked. They had been operating under the assumption that purges were reliable. The documentation had never been challenged because nobody had the data to challenge it. Alex built a dashboard that showed, in real time, the actual purge propagation coverage. It became the most-viewed dashboard in the organization.
The unlearning was not just technical; it was cultural. Alex had to convince his peers that the invariants they trusted were not just slightly off but fundamentally wrong. He presented his findings at an internal engineering all-hands. The reaction was mixed: some engineers were grateful for the clarity, others defensive of the system they had built. One senior engineer argued that the system worked fine in practice, and that documenting the gap would only erode trust. Alex countered that trust built on false assumptions was dangerous.
New Invariants from Empirical Data
Based on his year of data, Alex proposed a new set of invariants that reflected the actual behavior of the CDN. These were not guarantees but guidelines—contracts that clients and developers could rely on without being misled.
The first new invariant: TTL is a hint, not a contract. The system would attempt to serve fresh content, but if the origin was unreachable or slow, the edge node might serve stale content past the TTL. Clients had to tolerate this possibility. The second: purge queues are eventually consistent at best. A purge might take seconds or minutes to propagate fully, and some nodes might never receive it. Clients should design for invalidation failure by using versioned URLs or short TTLs as a fallback.
The third invariant was the hardest for the team to accept: clients must tolerate multi-second staleness. The documented promise of sub-second invalidation was aspirational, not operational. Under normal conditions, it held; under load, it did not. The team had to decide whether to invest in making the system actually meet the documented promise or to change the documentation. They chose to do both: improve the purge pipeline's reliability and update the documentation to reflect the real bounds.
The fourth invariant became a mantra: design for invalidation failure. Any system that relied on cache invalidation for correctness had to have a fallback, like a direct origin request or a client-side retry with exponential backoff. The CDN could not be the sole source of truth.
Counter-Arguments and Trade-Offs
Not everyone agreed with Alex's conclusions. Some argued that the documented invariants were never intended as hard guarantees but as engineering targets, and that the real problem was not the invariants but the lack of investment in the purge pipeline. They pointed out that the system had been running for years with few incidents, and that the 40% incomplete figure might be an artifact of Alex's measurement methodology—for example, purges for objects that were never cached in the first place would naturally not be applied, skewing the numbers. Alex acknowledged this but countered that the documentation explicitly stated that purges were applied to all nodes, regardless of whether the object was cached. The discrepancy was real, even if some measurements were noisy.
Another counter-argument was that the cost of making purges truly reliable—adding acknowledgment timeouts, redundant message delivery, and per-node state machines—would be prohibitive. The CDN was already operating on thin margins, and the engineering effort required to close the gap could be better spent elsewhere. Alex agreed that perfect coherency was not the goal, but argued that the gap between documented promises and actual behavior was dangerous. He proposed a middle ground: document the real bounds, invest in the worst failure modes (like the retry bug), and accept that some staleness was inevitable.
The trade-off between consistency and performance was a recurring theme. Alex's data showed that the purge pipeline's tail latency was driven by a small number of nodes that were overloaded or partitioned. Improving those nodes would reduce the tail without sacrificing throughput. The team eventually implemented a backpressure mechanism that slowed down the control plane when edge nodes were overwhelmed, preventing purge queue buildup. This reduced the 90th percentile propagation time from over 15 minutes to under 2 minutes—a significant improvement without a major architectural overhaul.
Practical Takeaways for 2026
Alex's experience offers lessons for any team building or operating distributed caches in 2026. First, test cache behavior, not documentation. Write tests that issue purges and measure actual propagation times under various load conditions. Do not assume the documented invariants hold; verify them empirically. Second, instrument every edge node. Without telemetry, you are flying blind. Log purge receipt, application, and any failures. Build dashboards that show the distribution of propagation times, not just the median.
Third, assume invalidation is best-effort. Design your applications to tolerate stale reads. Use versioned URLs, short TTLs, or client-side cache-busting as safety nets. The CDN is an accelerator, not a consistency layer. Fourth, share tribal knowledge in runbooks. The invariants Alex discovered were known to a few senior engineers but never written down. Document the real behavior, including edge cases and failure modes, so that new engineers do not have to spend a year unlearning.
One team that adopted these practices was the CI test harness team, which had been burned by stale dependencies cached by the CDN. After reading Alex's runbook, they rewrote their CI test harness to bypass the CDN for critical artifacts, reducing flaky failures by over 60%. Another team, migrating a CMS, found that font shaping paths were being cached incorrectly due to the same purge propagation gaps.
Alex also began a quarterly audit of cache invariants. Each quarter, he re-runs the synthetic purge tests and compares the results to the documented bounds. If the gap widens, the team investigates. This practice has caught several regressions early, including one where a new routing layer inadvertently delayed purge messages by an average of 10 seconds. Without the audit, the regression might have gone unnoticed for months.
The process of unlearning is never complete. Alex continues to monitor the system, and new failure modes appear as the architecture evolves. But the culture has shifted: assumptions are now questioned, data is collected before trust is granted, and the documentation is treated as a living artifact. The invariants that nobody documented are now documented—but with the caveat that they might be wrong tomorrow.
This article is part of a series on distributed systems and engineering culture. Names and specific details have been altered to protect the individuals and organizations involved.
