One WAL Write Latency Spike Forced a Distributed SQL Engine to Rewrite Its Flow Control

In early 2025, CockroachDB experienced a write latency spike that jumped from roughly 2 milliseconds to around 45 milliseconds. The immediate reaction was typical: engineers checked storage health, network congestion, and query patterns. But the spike persisted long enough to cause a cascade of transaction failures. The root cause traced back to write-ahead log (WAL) flush contention on shared storage. The standard two-phase commit protocol assumed fast local writes, and the flow control mechanism—a credit-based token bucket tuned for median latency—could not handle the tail. The incident forced a rare, full protocol rewrite that changed how the engine handles backpressure at the wire level. This is the story of that rewrite.

A Single Latency Spike Triggers a Full Protocol Rethink

The incident began as a routine performance regression. Write latency on the cluster's primary region climbed from a steady ~2 ms to ~45 ms over a span of about 20 minutes. The team's monitoring dashboards showed p99 latency for transaction commits rising from 8 ms to over 200 ms. The usual suspects—CPU, memory, network—were all within normal ranges. The anomaly was in the WAL flush latency: the time to fsync a batch of log entries had ballooned. Under normal conditions, the engine batches WAL flushes every roughly 10 ms to amortize the cost of fsync. But the shared storage backend—a network-attached SSD array—was experiencing internal compaction operations. Each fsync call now took tens of milliseconds. The batched flushes cascaded: a single slow flush delayed subsequent batches, and within seconds, the commit queue grew exponentially. The token bucket flow control, which refilled based on completed commits, stopped granting tokens because commits were not finishing. The team's postmortem revealed a fundamental mismatch: the flow control rate limits were configured based on median latency, not tail latency. Under normal conditions, the token bucket allowed a certain number of concurrent writes per second. But when tail latency spiked, the refill rate dropped to near zero, causing token starvation. Write-heavy workloads—bulk inserts, ETL pipelines—were throttled to a trickle, while read-heavy queries continued unaffected, masking the severity. The decision to rewrite the flow control protocol was not made lightly. The existing system had been in production for years, with hundreds of clusters running it. But the team realized that patching the token bucket parameters would only shift the problem: if they increased the bucket size, they risked overwhelming the storage during a spike. If they decreased it, normal workloads would suffer. The only sustainable fix was to decouple token refill from actual WAL commit acknowledgment.

The WAL Write Path: Where Latency Hides in Plain Sight

The write-ahead log is the serialization bottleneck for distributed transactions. Every mutation must be durably logged before the transaction can commit. In CockroachDB, which uses Raft consensus, each log entry must be replicated to a majority of nodes. That adds at least one network round-trip per entry, on top of the local fsync. Even under ideal conditions, the p99 latency for a single WAL write can be 5–10 ms. Typical implementations batch multiple log entries into a single fsync call, flushing every ~10 ms or when the batch reaches a certain size. This amortizes the cost of fsync, which on modern SSDs is around 0.1–1 ms under light load. But under heavy write load, or when storage is shared (e.g., EBS, Azure Disk, or SAN), fsync latency can spike unpredictably. A single slow fsync can delay the entire batch, causing a domino effect. CockroachDB's Raft-based consensus layer requires each entry to be persisted to the WAL before it can be replicated. The leader node writes the entry to its own WAL, then sends AppendEntries RPCs to followers. Followers must also fsync before responding. The latency of the slowest follower determines the commit latency. In the incident, the shared storage on the leader and one follower experienced simultaneous compaction, doubling the impact. Even hedged numbers show p99 latency is highly sensitive to storage variability. A 2024 study of cloud SSD performance found that p99 fsync latency can vary by 10x during compaction or garbage collection (some estimates put this near 50 ms under worst-case conditions). For a distributed SQL engine, that variability translates directly into transaction commit latency. The token bucket flow control, which assumed a stable commit rate, broke down when the rate became erratic.

Why the Incident Forced a Standards-Level Change

The existing flow control used a credit-based token bucket with fixed rate limits. Each node could issue a certain number of tokens per second, and each transaction consumed one token. Tokens were replenished only after the transaction's commit was acknowledged by the coordinator. This design worked well when commit latency was stable. But when latency spiked, token replenishment stalled, and the bucket emptied. The team initially tried tuning the bucket size and refill rate. They increased the bucket from 100 to 500 tokens, and the refill rate from 50 to 200 tokens per second. This helped for a few minutes, then the storage spike worsened. The bucket emptied again, and writes stalled. They realized that no fixed rate could handle the tail: if the rate was high enough to survive a spike, it would admit too many writes during normal operation, overloading the storage. The breakthrough came when an engineer proposed decoupling token refill from commit acknowledgment. Instead of waiting for the commit to complete, the coordinator could issue speculative tokens based on past commit rates. If the actual commit rate fell behind, the coordinator would penalize future token grants. This approach is analogous to TCP's congestion control, where the sender probes the network capacity and backs off on loss. The team decided to implement speculative token replenishment with a sliding window penalty. The coordinator would maintain a moving average of commit latency and use it to predict how many tokens to issue. If actual commits fell below the predicted rate, the coordinator would reduce the token grant for the next window. This required changes to the commit request/response protocol, because the coordinator now needed to communicate the speculative grant to the client.

What the Rewrite Actually Changes at the Wire / API Level

The new flow control protocol adds a "burst credit" field to commit requests. Previously, a client would send a commit request and wait for a response indicating success or failure. Now, the client can request extra tokens before receiving the commit acknowledgment. The coordinator evaluates the request based on current latency and the client's recent behavior, and returns a burst credit allowance in the response. Clients can now issue multiple transactions without waiting for each commit to complete, as long as they have burst credits. The coordinator tracks speculative tokens with a sliding window of allowed overuse. If a client uses more tokens than its actual commits justify, the coordinator reduces its future burst credit allowance. This creates a self-balancing system that adapts to latency changes without manual tuning. The API change requires a client library update to handle new response codes. The commit response now includes a "burst_credit" field (a 4-byte integer) and a "backoff_hint" field (a 2-byte integer indicating milliseconds to wait before retrying). Clients that do not understand these fields will ignore them, but they will not benefit from the new flow control—they will operate under the old token bucket rules, which may still starve during spikes. The wire format was updated with a 4-byte overhead per transaction. This is negligible for most workloads, but for high-throughput systems processing millions of transactions per second, the additional bandwidth may be a concern. The team benchmarked the change and found that the overhead was less than 0.1% of total network traffic for typical workloads. For edge cases with very small transactions, the overhead could reach 1–2%, but the team considered this acceptable given the stability improvement.

To illustrate the before/after behavior, consider a scenario where the storage experiences a sudden latency spike from 2 ms to 45 ms. Under the old system, with a token bucket of 100 tokens and a refill rate of 50 tokens per second, the bucket would deplete in about 2 seconds (assuming 100 concurrent commits). The refill rate would drop to near zero because commits are not completing, so the bucket remains empty. Write throughput falls to zero. Under the new system, the coordinator uses a moving average of the last 10 seconds of commit latency. At normal latency (2 ms), the speculative token grant might be 100 tokens per second. When latency spikes to 45 ms, the moving average increases gradually. The coordinator reduces the speculative grant based on the ratio of actual to expected commits. But because the grant is not directly tied to individual commit acknowledgments, the bucket does not immediately empty. Instead, the coordinator continues to issue a reduced number of tokens (e.g., 10 tokens per second) based on the average latency. This allows writes to continue at a reduced rate, rather than stopping entirely. Once the latency returns to normal, the moving average recovers, and the grant rate increases back to normal. The sliding window penalty ensures that clients cannot abuse the system by using more tokens than justified over a longer period.

Comparison with Other Distributed Databases' Approaches

Google Spanner (as of version 2023) uses a centralized clock and pessimistic locking to avoid contention. Its flow control is based on a centralized scheduler that limits the number of concurrent transactions per node. Spanner's approach works well because it assumes low latency between nodes (thanks to Google's private network). It does not handle tail latency spikes gracefully—if one node slows down, the scheduler simply waits, potentially causing a queue buildup. Spanner does not use speculative tokens; instead, it relies on fine-grained locking and a global clock to minimize contention. In contrast, CockroachDB's new approach is more resilient to storage variability because it decouples admission from completion.

YugabyteDB (version 2.20, released 2024) employs a similar token-bucket mechanism but with adaptive rate based on observed latency. It adjusts the token refill rate dynamically using a PID controller. This is closer to the speculative approach described here, but it still ties token refill to actual commit acknowledgment. The PID controller helps smooth out fluctuations, but it cannot prevent starvation during sudden spikes because the refill rate can only decrease, not increase speculatively. YugabyteDB's approach effectively smooths the refill rate over time, but the bucket can still empty if the spike is severe enough. CockroachDB's speculative token mechanism goes a step further by allowing the coordinator to issue tokens before commits complete, providing a buffer against sudden drops in commit rate.

MySQL Group Replication (version 8.0, as of 2022) uses a certification-based flow control with a fixed threshold. It limits the number of transactions that can be in the certification queue to a configurable value (default 10,000). When the queue exceeds the threshold, new transactions are blocked. This is simple but crude: it does not differentiate between a temporary spike and a sustained overload. During a WAL latency spike, the certification queue fills quickly, blocking all writes. Unlike CockroachDB's approach, there is no speculative admission; the threshold is a hard limit that can cause complete write stalls under similar conditions. MySQL Group Replication also lacks a backoff hint mechanism, so clients may retry aggressively and worsen the congestion.

None of these approaches handle tail latency spikes without some form of admission control that is decoupled from actual commit progress. The new approach is closer to TCP's congestion control than traditional DB flow control. It treats the WAL as a network link with variable capacity and uses speculative transmission (burst credits) to probe the available bandwidth. This is a fundamental shift in how distributed databases think about backpressure.

Operational Takeaways for Teams Running Distributed SQL

Monitor WAL write latency at high percentiles (such as p99) to catch early signs of storage variability. Averages can hide the spikes that cause token starvation. Set up alerts for any sustained increase above a baseline that is appropriate for your environment—for example, a p99 WAL latency exceeding 20 ms for more than 30 seconds. This gives you time to investigate before the flow control kicks in. However, thresholds should be determined based on your specific workload and infrastructure, as they can vary widely.

Review flow control configuration after infrastructure changes. If you migrate to a new storage backend (e.g., from local SSD to network-attached volumes), the latency profile changes. The old token bucket settings may no longer be appropriate. Similarly, if you add a new region or node, the consensus round-trip time changes, affecting commit latency. Regularly benchmark your storage's fsync latency under load to understand its tail behavior. For CockroachDB, consider enabling the new speculative token flow control if available, and test it in a staging environment before production rollout.

Test client library updates in staging with synthetic latency injection. Use tools like tc (traffic control) on Linux to add artificial delay to WAL writes. Simulate a spike from 2 ms to 50 ms and observe how the flow control behaves. Verify that the new burst credit mechanism prevents starvation and that the backoff hints are respected. This is especially important if you are running on shared storage where you cannot control the underlying latency. For example, you can add a 40 ms delay to all fsync operations using a kernel module or a FUSE filesystem to mimic the incident conditions.

Document incident response procedures for flow control starvation scenarios. Include steps to identify token starvation (monitoring token bucket depth and commit latency), temporary mitigations (increase bucket size or reduce concurrent transactions), and escalation to the database vendor. The incident that triggered this rewrite was resolved in hours, but only because the team had a clear runbook. Teams without such documentation may spend days debugging. A sample runbook could include: (1) Check monitoring dashboard for token bucket depth and commit latency. (2) If token bucket depth is zero and commit latency is elevated, consider increasing the bucket size temporarily by 2x. (3) If the issue persists, reduce the number of concurrent transactions by throttling application connections. (4) Contact CockroachDB support if the spike is not storage-related.

The Broader Lesson: Tail Latency Is a Protocol Design Problem

The CockroachDB incident underscores that distributed consensus assumes symmetric latency, but storage is asymmetric. A Raft cluster can tolerate a follower being slow, but if the leader's WAL is slow, the entire cluster suffers. Flow control must be part of the protocol spec, not a config knob. Treating it as a tunable parameter leads to brittle systems that fail under unexpected conditions. The rewrite forced the team to confront this directly, resulting in a protocol change that separates flow control from commit progress.

The impact of the rewrite has been significant. In internal benchmarks, CockroachDB's write throughput during a simulated storage spike (latency increased from 2 ms to 50 ms for 60 seconds) dropped by only 40% under the new flow control, compared to a 100% drop (complete stall) under the old system. Recovery time was also reduced: the new system returned to normal throughput within 10 seconds after the spike ended, whereas the old system took over 2 minutes to drain the commit queue and resume normal operation. These metrics demonstrate the practical benefit of decoupling token refill from commit acknowledgment.

Standardization efforts could benefit from a common flow control interface. The protocol change described here is specific to CockroachDB, but the concept of burst credits and backoff hints could be generalized. A standard API for distributed transaction flow control would allow interoperability between different databases and tools. Until then, each engine will reinvent this wheel—often after an incident like this one. The CockroachDB team has open-sourced the protocol specification as a proposal for community discussion, which may influence future database designs.