The KV Cache Ceiling: Diagnosing Throughput Collapse in SGLang Inference

Introduction

In the high-stakes world of large language model inference, few moments are as disorienting as watching throughput crater after a promising start. Message 3893 captures precisely such a moment. The assistant had just celebrated a peak throughput of 1,317 tokens per second with 121 concurrent requests, enabled by a carefully tuned hierarchical cache configuration. Minutes later, throughput had collapsed to 606 tok/s with only 34 active requests and 115 stuck in queue. This message is the diagnostic pivot — a moment where the assistant steps back from tweaking knobs and performs a deep analysis of why the numbers don't add up.

The message is remarkable not for what it builds, but for what it unbuilds: a misconception about how hierarchical caching works in practice. The assistant had invested significant effort in enabling SGLang's --enable-hierarchical-cache with 48GB of host memory per rank, believing it would dramatically increase concurrent decoding capacity. This message reveals why that belief was wrong, and in doing so, provides a masterclass in reasoning about memory-bound inference bottlenecks.

The Setup: A Server Under Load

The context leading into this message is essential. The team was running a large-scale inference pipeline to generate synthetic training data for EAGLE-3 speculative decoding. The server hosted the Kimi-K2.5 model (a DeepSeek-v2 architecture variant with Multi-head Latent Attention) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using tensor parallelism. The server had been configured with --mem-fraction-static 0.88 after an earlier OOM crash at 0.93, yielding 159,277 GPU KV cache tokens. Hierarchical cache was enabled with 48GB of host DRAM per rank, using a write-through policy and kernel-based I/O backend.

The inference client (run_inference.py) was processing the B2_opencodeinstruct partition, resuming from 1,374 already-completed samples out of ~14,714 total. It used 150 short-concurrency and 32 long-concurrency workers, with max tokens of 10,240 and 16,384 respectively. The server had been humming along at 989–1,317 tok/s with token usage around 0.07–0.11 — seemingly plenty of headroom.

Then the numbers changed.

The Observation: A Puzzling Drop

The assistant checks the server log and sees:

[2026-02-24 12:03:07 TP0] Decode batch, #running-req: 35, #token: 150158, token usage: 0.94, cuda graph: True, gen throughput (token/s): 606.88, #queue-req: 115

Token usage is at 0.94 — the GPU KV cache is nearly full. Only 35 requests are actively decoding while 115 are queued. Throughput has halved. The assistant's first reaction is a telling "Hmm" — the opening word of the message, signaling genuine surprise and the beginning of analytical re-evaluation.

The initial hypothesis is that the hierarchical cache isn't helping as much as expected. The assistant had allocated 48GB of host memory per rank specifically to increase the effective KV cache pool. But the numbers suggest something is fundamentally different from what was anticipated.

The Analysis: Why Hierarchical Cache Doesn't Help Active Decodes

This is the intellectual core of the message. The assistant works through the reasoning step by step:

"The issue is that hicache helps with evicted radix cache entries (prefix reuse), but for actively generating requests the KV must be on GPU. So hicache doesn't actually increase the number of concurrent decode requests — it only helps with prefix caching."

This is a crucial insight. SGLang's hierarchical cache architecture works by offloading evicted KV cache entries from GPU to host memory. When a new request shares a prefix with an evicted entry, the prefix can be reloaded from host memory rather than recomputed. This is valuable for prefix caching — where many requests share common system prompts or conversation histories.

But for active decoding — the phase where the model generates tokens one at a time — the KV cache for each actively generating sequence must reside on the GPU. Every decoding step reads the entire KV cache for the sequence, and reading from host memory over PCIe would be far too slow for the tight latency budget of autoregressive decoding. The hierarchical cache is a victim cache for evicted entries, not an extension of the active working set.

The assistant then quantifies the constraint:

"With 159K tokens and 4K avg per request, max ~40 concurrent decodes."

This is a back-of-the-envelope calculation that reveals the fundamental bottleneck. 159,277 GPU tokens divided by ~4,000 tokens per request equals roughly 40 concurrent requests. The observed 34–35 active requests is right at this ceiling. The system is not underperforming — it's operating at its memory-imposed limit.

The assistant further validates this by noting:

"The server generating 600 tok/s with 35 concurrent requests × ~17 tok/s per request is actually correct."

At ~17 tokens per second per request (a typical decode rate for large models), 35 concurrent requests yields ~595 tok/s — exactly what's observed. The system is behaving correctly; the expectations were wrong.

The Assumptions Under Scrutiny

This message reveals several assumptions that turned out to be incorrect:

Assumption 1: Hierarchical cache increases concurrent decode capacity. This was the most significant misconception. The assistant had invested heavily in enabling and tuning hicache — researching the codebase, calculating per-rank host memory budgets, verifying allocation — all based on the implicit assumption that more total KV cache (GPU + host) would translate to more concurrent decodes. The analysis shows this is false for actively generating sequences.

Assumption 2: The observed headroom (token usage 0.07–0.11) would persist. Earlier in the run, token usage was very low because the system was still warming up — requests were in prefill phase, not yet accumulating KV cache entries. The assistant may have extrapolated from early throughput numbers (989–1,317 tok/s) without accounting for the transient nature of the warm-up period.

Assumption 3: More tokens means more throughput. The assistant had increased max_total_num_tokens from 116K to 159K (and briefly to 231K before the OOM crash) expecting proportional throughput gains. But the analysis reveals that token capacity only matters up to the point where it can accommodate the active decode set — beyond that, additional tokens don't help because they can't be used for active generation.

The Knowledge Required to Understand This Message

To fully grasp the assistant's reasoning, one needs:

  1. Understanding of KV cache mechanics in transformer inference. The key-value cache grows linearly with sequence length and batch size. Each token in a sequence requires ~137KB of KV cache (for this model at this precision), and the total GPU memory available for KV cache is determined by mem_fraction_static.
  2. Knowledge of SGLang's hierarchical cache architecture. The hicache system uses host DRAM as a spill destination for evicted radix cache entries. It's designed for prefix reuse scenarios, not for expanding the active decode set.
  3. Awareness of the autoregressive decode bottleneck. During token generation, each step reads the full KV cache for the sequence. This is a memory-bandwidth-bound operation that cannot tolerate remote memory access latencies.
  4. Understanding of the specific model characteristics. The Kimi-K2.5 model uses MLA (Multi-head Latent Attention), which has a compressed KV cache representation but still requires significant memory per token.
  5. The server configuration context. Knowing that mem_fraction_static=0.88 yields 159K tokens, that tp-size 8 distributes the model across 8 GPUs, and that each GPU has 48GB of HBM memory.

The Knowledge Created by This Message

This message produces several valuable pieces of knowledge:

  1. A validated mental model of hierarchical cache limitations. The insight that hicache helps with prefix reuse but not active decode capacity is a general principle applicable to any system using victim caches for LLM inference.
  2. A quantified bottleneck analysis. The calculation of 159K tokens ÷ 4K avg = ~40 concurrent decodes provides a concrete method for estimating maximum concurrency from KV cache capacity.
  3. A diagnostic pattern. The observation that throughput drops when token usage approaches 1.0, combined with a growing queue and stable per-request decode rate, is a reliable signature of KV cache saturation.
  4. A prioritized list of solutions. The assistant identifies three levers: more GPUs (infeasible), shorter sequences (undesirable), and KV cache quantization (promising). The subsequent exploration of --kv-cache-dtype fp8_e4m3 represents the actionable output of this analysis.

The Thinking Process: A Window into Diagnostic Reasoning

The message structure reveals the assistant's cognitive process. It begins with raw observation (the log line), moves to hypothesis testing (why isn't hicache helping?), develops a refined model (hicache is for eviction, not active sets), validates against observed numbers (35 reqs × 17 tok/s = 595 tok/s), and finally pivots to actionable next steps (exploring KV cache quantization).

The most impressive aspect is the willingness to challenge the previous investment. The assistant had spent multiple messages researching, calculating, and deploying the hierarchical cache configuration. Admitting that it doesn't solve the core problem requires intellectual honesty and a commitment to understanding over optimization.

The final action — checking available --kv-cache-dtype options — is a natural consequence of the analysis. If GPU KV cache capacity is the bottleneck, and adding more memory (via hicache) doesn't help active decodes, then the remaining lever is to use the existing memory more efficiently. FP8 KV cache halves the memory per token, potentially doubling the concurrent decode capacity from ~40 to ~80, which would roughly double throughput.

Conclusion

Message 3893 is a textbook example of diagnostic reasoning in systems engineering. It demonstrates the critical skill of distinguishing between what a system can do (offload evicted cache entries) and what the workload needs (active KV cache for concurrent decodes). The hierarchical cache was not a mistake — it genuinely helps with prefix reuse and reduces recomputation — but it was not the bottleneck the assistant needed to solve.

The message also illustrates the importance of validating assumptions against first principles. The assistant could have continued tweaking hicache parameters, trying different ratios and sizes, chasing marginal gains. Instead, it stopped, thought through the memory model, and identified the true constraint. This kind of analytical rigor — the willingness to say "my model was wrong, let me rebuild it" — is what separates effective debugging from random exploration.

The KV cache ceiling identified in this message would drive the next phase of optimization: exploring FP8 quantization, and eventually, a fundamental rethinking of the EAGLE-3 speculative decoding pipeline that could bypass the bottleneck entirely by generating fewer tokens per request.