The Three Levers of Throughput: Diagnosing and Solving a KV Cache Bottleneck in SGLang

Introduction

In the middle of a high-stakes inference pipeline — generating 88,000 training samples for an EAGLE-3 speculative decoding model — the assistant sent a concise status report that reads more like a victory lap than a technical update. Message <msg id=3909> is a four-bullet summary of a server optimization effort that nearly tripled throughput:

Server is running with all levers pulled: - --mem-fraction-static 0.90 — more GPU KV space (with 3.5GB headroom, safe from OOM) - --kv-cache-dtype fp8_e4m3 — halves KV memory per token → 376K max tokens (3.2x baseline) - --enable-hierarchical-cache --hicache-size 48 — 48GB/rank host RAM overflow for prefix caching - NCCL tuning + --num-continuous-decode-steps 4

>

Result: 134-150 concurrent requests, 1300-1450 tok/s gen throughput. B2 (4K+ avg tokens) is running at ~0.3 req/s vs previous 0.1 req/s — roughly 3x faster. Shorter datasets should see even bigger gains since they fit more concurrent requests.

On its surface, this message is a simple progress update. But beneath the bullet points lies a rich story of iterative diagnosis, a subtle misunderstanding about how hierarchical caching works, and a series of engineering tradeoffs that transformed a pipeline from painfully slow to genuinely productive. This article unpacks that story.

The Bottleneck That Wasn't Where Expected

The optimization journey began not with a clean plan, but with a puzzle. Earlier in the session, the assistant had deployed the SGLang inference server with a hierarchical cache (--enable-hierarchical-cache --hicache-size 48) and a memory fraction of 0.88, expecting this to dramatically increase throughput. The hierarchical cache uses host RAM (48GB per GPU rank) as overflow storage for evicted KV cache entries, enabling better prefix reuse across requests.

The initial results were encouraging: the server reached 989 tok/s with 48 concurrent requests and plenty of headroom. But as the pipeline ran longer with the B2 dataset — which has an average sequence length of 4,000+ tokens — the picture changed. Throughput dropped to 606 tok/s, with only 34-35 requests actively decoding and 115 queued. Token usage sat at 0.92-0.94, indicating the GPU KV cache was nearly full.

This was the critical diagnostic moment, captured in <msg id=3893>. The assistant realized a subtle but crucial fact: the hierarchical cache helps with evicted radix cache entries (prefix reuse), but for actively generating requests, the KV data must live on GPU memory. The hicache does not increase the number of concurrent decode slots — it only helps when new requests share prefixes with previously completed ones. For a workload where each request has a unique, long prompt, prefix reuse is minimal, and the hicache provides little benefit during steady-state decode.

This insight reframed the entire problem. The real bottleneck was not memory efficiency or scheduling — it was raw GPU KV cache capacity. With 159,000 GPU token slots and 4,000 tokens per request, the server could fit at most ~40 concurrent decodes. Each of those decodes generated tokens at roughly 17 tok/s, so the theoretical maximum throughput was around 680 tok/s — exactly what the server was achieving.

The FP8 Epiphany

Once the root cause was clear, the solution space narrowed. The assistant enumerated three options: add more GPUs (not possible), shorten sequences (not desirable), or quantize the KV cache. SGLang supported --kv-cache-dtype with options including fp8_e4m3, fp8_e5m2, and even fp4_e2m1. For a model using Multi-head Latent Attention (MLA), the KV cache is already stored as compressed latent representations, but further quantizing from bf16 to fp8 would halve the memory per token, effectively doubling capacity.

The assistant chose fp8_e4m3 — a 8-bit floating point format with 4 exponent bits and 3 mantissa bits, offering a good balance of dynamic range and precision. This was paired with an increased memory fraction of 0.90 (up from 0.85 in the baseline), exploiting the fact that fp8 KV cache leaves more GPU memory free for other allocations.

The server restart with these settings produced a dramatic result: max_total_num_tokens=376029 — a 3.2× increase over the baseline of 116,000 tokens. With 3.5GB of headroom per GPU after CUDA graph capture, the configuration was stable and safe from out-of-memory errors.

The Results and Their Interpretation

Message <msg id=3909> reports the outcome: 134-150 concurrent requests, 1300-1450 tok/s generation throughput, and a B2 completion rate of ~0.3 req/s — triple the previous 0.1 req/s. The assistant also provides a comparison table in the preceding message <msg id=3908> that traces the optimization path:

| Config | GPU KV tokens | Concurrent reqs | Gen tok/s | Effective tok/s | |--------|:------------:|:---------------:|:---------:|:---------------:| | Baseline (0.85, bf16 KV) | 116K | 35-50 | 600-850 | ~400-800 | | +hicache (0.88, bf16 KV) | 159K | 50-120 | 600-1300 | ~600-1200 | | +FP8 KV (0.90, fp8_e4m3, hicache) | 376K | 134-150 | 1300-1450 | ~1100-1400 |

The table reveals that the hierarchical cache did provide some benefit — increasing GPU KV tokens from 116K to 159K by allowing a higher memory fraction (0.88 vs 0.85) — but the real leap came from fp8 quantization, which nearly tripled capacity again to 376K.

However, the message also contains an implicit tension. The server reports 1300-1450 tok/s generation throughput, but the effective throughput — measured by completed requests per second multiplied by average tokens per request — is only ~1100-1400 tok/s. The gap between generation throughput and effective throughput reflects the overhead of prefill, scheduling, and the fact that not all generated tokens are immediately written to the output file. The assistant acknowledges this nuance by reporting both metrics.

Assumptions and Their Risks

The message makes several assumptions worth examining. First, it assumes that fp8 quantization of the MLA KV cache does not degrade output quality. For a training data generation pipeline, quality degradation could propagate into the EAGLE-3 drafter model, potentially producing a weaker speculative decoder. The assistant does not mention any validation of output quality after the switch — this is a pragmatic risk taken to meet throughput targets.

Second, the message assumes that 3.5GB of headroom per GPU is sufficient for transient allocations during prefill and CUDA graph operations. This is a reasonable heuristic based on prior experience, but it is not guaranteed. If a future request triggers an unusually large prefill allocation, the server could OOM.

Third, the prediction that "shorter datasets should see even bigger gains" is logically sound — shorter sequences mean more requests fit in the KV cache simultaneously — but it remains untested at the time of the message. The assistant is extrapolating from the B2 workload to B1, B6, and B7 without empirical confirmation.

Knowledge Created and Consumed

To fully understand this message, a reader needs knowledge of: SGLang's server architecture and its KV cache management; the difference between radix caching, hierarchical caching, and active decode KV storage; the memory characteristics of MLA-based models; and the practical meaning of fp8 quantization for inference throughput.

The message itself creates several pieces of output knowledge: a validated configuration for running Kimi-K2.5 on 8×RTX PRO 6000 GPUs with optimal throughput; the empirical finding that fp8 KV cache provides a 3.2× capacity increase over bf16 for this model; the insight that hierarchical cache is not a substitute for GPU KV capacity when active decode slots are the bottleneck; and a benchmark baseline (1300-1450 tok/s) against which future optimizations can be measured.

Conclusion

Message <msg id=3909> is deceptively simple. It reads as a status update, but it represents the culmination of a rigorous diagnostic process: forming a hypothesis (hicache will help), testing it empirically, discovering it was wrong for the active-decode workload, tracing the real bottleneck to GPU KV capacity, identifying the correct lever (fp8 quantization), deploying it, and measuring the result. The message distills this entire journey into four bullet points and a few numbers. For the engineer who lived through the debugging, those numbers tell a story of 3.5GB headroom margins, 376K token slots, and a pipeline that finally runs fast enough to generate 88,000 training samples in hours instead of days.