Diagnosing the KV Cache Bottleneck: A Detective Story in LLM Inference Optimization

Introduction

In the middle of a large-scale data generation pipeline for training an EAGLE-3 speculative decoding drafter, a performance problem emerged. The user observed that SGLang, the inference engine serving the Kimi-K2.5 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, was delivering only around 500 tokens per second (tps) — far below the expected 4–5× improvement the user anticipated. Message [msg 3834] captures the moment the assistant synthesized scattered log data into a coherent diagnosis: the KV cache was saturated, starving the scheduler of memory to admit new requests. This article dissects that single message, exploring the reasoning, assumptions, knowledge, and decisions that shaped it.

The Context: A Pipeline Under Pressure

By the time message [msg 3834] was written, the conversation had already traversed a long arc. The team had set up an ML environment on Ubuntu 24.04 with eight GPUs, resolved numerous build issues with flash-attn and CUDA toolkits, and deployed the GLM-5-NVFP4 / Kimi-K2.5 model using SGLang. They had pivoted from vLLM to SGLang after discovering that vLLM's EAGLE-3 integration with Multi-Head Latent Attention (MLA) yielded only a ~15% acceptance rate, making speculative decoding worthless. They had then developed a custom hidden state extraction pipeline, trained an EAGLE-3 drafter from scratch, and were now generating synthetic training data at scale — 88,000 prompts across multiple categories (B1_glaive, B2_opencodeinstruct, etc.).

The inference pipeline used SGLang's /generate endpoint with raw token IDs, bypassing the OpenAI-compatible chat completions API to avoid parsing ambiguity with the thinking and response special tokens. The server was configured with 150 concurrent request slots, but the throughput was disappointing.

What the Message Actually Says

The assistant begins with a moment of insight:

Now I see the problem clearly. Look at the pattern:

>

- #running-req: 46-56 — only ~50 concurrent requests running despite 150 concurrency and 100+ in queue - token usage: 0.85-0.96 — KV cache is nearly full! - Prefills are rare — the scheduler can't admit new requests because the KV cache is saturated

>

The bottleneck is KV cache capacity. With --mem-fraction-static 0.85, the server can hold ~116K tokens total. With 50 requests averaging ~2000 tokens each, that's already 100K tokens. The scheduler can't prefill queued requests because there's no room.

The assistant then notes a critical shift in the data: B2_opencodeinstruct is now running with an average of 4,116 tokens per response — much longer than the earlier B1_glaive category. Long sequences fill the KV cache faster, reducing the number of concurrent requests that can fit.

The message then lists three options for increasing throughput:

  1. Increase --mem-fraction-static to 0.88 or 0.90
  2. Reduce max_new_tokens (but this is undesirable)
  3. Use chunked prefill to overlap prefill with decode Finally, the assistant queries the server's live configuration via curl to confirm the current settings, revealing mem_fraction_static: 0.85, max_total_num_tokens: 116171, chunked_prefill_size: 8192, and num_continuous_decode_steps: 4.

The Reasoning Process: From Raw Metrics to Root Cause

The assistant's reasoning in this message is a textbook example of diagnostic inference. It begins with three observed facts extracted from the SGLang logs (gathered in [msg 3832] and [msg 3833]):

  1. Running requests are capped at ~50, even though the concurrency limit is 150 and there are 94–101 requests queued.
  2. KV cache token usage is 0.85–0.96, meaning the cache is 85–96% full.
  3. Prefill batches are rare — the scheduler rarely admits new requests. The assistant then connects these facts through a causal chain: if the KV cache is nearly full, the scheduler cannot allocate space for new request prefills. Even though there are 100+ requests waiting in the queue, they cannot begin processing because there is no room in the KV cache to store their key-value tensors. The running requests must finish (or at least generate enough tokens to free cache space) before new ones can start. This creates a throughput ceiling: the server generates tokens at 800–890 tok/s for the ~50 concurrent requests, but the effective throughput is lower because the pipeline is gated by cache availability. The assistant also notes the transition from B1_glaive to B2_opencodeinstruct. The earlier category had shorter responses (~1,500 tokens on average), allowing more concurrent requests. The new category averages 4,116 tokens per response, which means each request occupies more KV cache slots for longer. This is a crucial observation — the bottleneck is not static but changes with the data distribution.

Assumptions Embedded in the Diagnosis

The assistant makes several assumptions in this message, some of which are later refined or challenged:

Assumption 1: Average token length is ~2,000. The assistant states "With 50 requests averaging ~2000 tokens each." However, the log data from [msg 3833] shows avg_comp=4116tok for B2_opencodeinstruct. The 2,000-token estimate appears to be a rough mental calculation that underestimates the actual length. This is a minor error — the true situation is even worse than the assistant's estimate, because longer sequences consume more KV cache per request.

Assumption 2: KV cache capacity is the sole bottleneck. The assistant frames the problem entirely around KV cache saturation. While this is correct as far as it goes, subsequent messages ([msg 3836] and [msg 3837]) reveal that increasing mem_fraction_static provides only marginal gains (3.5–12% more tokens). The assistant later discovers that the client-side throughput measurement of 0.1 req/s is misleading — B2 is actually running at 0.5 req/s with shorter responses. The bottleneck is more nuanced than pure KV cache capacity.

Assumption 3: The linear scaling of KV cache with mem_fraction_static. In [msg 3836], the assistant computes new_tokens = 116171 * new_frac / 0.85, assuming a linear relationship. This is a reasonable approximation but ignores that mem_fraction_static is a fraction of free memory after model weights are loaded, not a fraction of total memory. The actual relationship is slightly different, though the approximation is close enough for diagnostic purposes.

Assumption 4: The user's "500 tps" refers to generation throughput. The assistant's initial investigation in [msg 3832] shows server-side gen throughput of 780–890 tok/s, not 500. The assistant implicitly assumes the user is measuring effective throughput (including queueing and prefill overhead) rather than raw generation speed. This turns out to be correct — the client-side measurement of 0.1 req/s at 4,116 tokens each yields ~411 tok/s effective, which aligns with the user's ~500 tps observation.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message [msg 3834], one needs:

  1. SGLang architecture knowledge: Understanding that SGLang uses a continuous batching scheduler that maintains a KV cache in GPU memory. The scheduler decides when to admit new requests based on available cache space. The mem_fraction_static parameter controls what fraction of GPU memory (after model weights) is reserved for the KV cache. The max_total_num_tokens is the total number of token slots available across all sequences.
  2. KV cache mechanics: For transformer-based LLMs, each token in a sequence generates key and value tensors that must be stored for the duration of the generation. With Multi-Head Latent Attention (MLA) as used in Kimi-K2.5, the KV cache is compressed but still consumes significant memory. The cache is shared across all concurrent requests — when it fills up, no new requests can be prefilled.
  3. The inference pipeline context: The assistant had recently rewritten run_inference.py to use SGLang's /generate endpoint with raw token IDs, bypassing the OpenAI API. This change was motivated by a reasoning capture bug where --reasoning-parser wasn't configured, causing thinking content to be embedded in message.content with reasoning_content: null. The new approach pre-tokenizes prompts via apply_chat_template and receives the model's exact token sequence.
  4. The data generation pipeline: The team is generating synthetic training data for EAGLE-3 across multiple categories. B1_glaive (short function-calling tasks) had completed, and B2_opencodeinstruct (longer coding tasks) was now running. The shift in average response length from ~1,500 to ~4,100 tokens dramatically changes the KV cache dynamics.
  5. GPU memory constraints: Each of the eight RTX PRO 6000 GPUs has 97,887 MiB of VRAM. The model weights consume approximately 68.4 GB (547 GB total / 8 GPUs). At mem_fraction_static=0.85, the KV cache gets ~14.8 GB per GPU, which translates to ~116K total token slots.

Output Knowledge Created by This Message

The message produces several concrete outputs:

  1. A clear diagnosis: The bottleneck is KV cache capacity, not generation speed. The server can generate tokens at 800–890 tok/s, but it cannot sustain that throughput because it runs out of cache space and cannot admit new requests.
  2. Specific config parameters: The assistant retrieves the live server configuration, showing mem_fraction_static: 0.85, max_total_num_tokens: 116171, chunked_prefill_size: 8192, and num_continuous_decode_steps: 4. This gives a precise baseline for optimization.
  3. Three optimization levers: The assistant identifies three knobs to turn — increasing mem_fraction_static for more KV cache space, reducing max_new_tokens (undesirable), or using chunked prefill to overlap prefill with decode.
  4. A data-driven observation: The shift from B1 to B2 categories changes the average response length from ~1,500 to ~4,100 tokens, which fundamentally alters the KV cache pressure. This insight informs the subsequent optimization strategy.

The Broader Significance: A Pivot Point in the Pipeline

Message [msg 3834] is a pivot point in the conversation. It triggers a chain of optimization attempts documented in subsequent messages: trying --mem-fraction-static 0.93 (which OOMs), considering --kv-cache-dtype fp8_e4m3 (rejected for quality concerns), and eventually settling on --enable-hierarchical-cache with --hicache-size 48 (which initially overflows host RAM but is later tuned to work). The final configuration achieves 159K GPU tokens and ~930–1350 tok/s throughput — roughly 2–3× improvement over the initial baseline.

The message also reveals a subtle but important truth about LLM inference optimization: the bottleneck is often not where it first appears. The user saw 500 tps and assumed the server was slow. The assistant discovered the server was actually generating at 800+ tok/s — the bottleneck was in the scheduler, not the decoder. This distinction between generation throughput and effective throughput is critical for anyone operating large-scale inference systems.

Mistakes and Refinements

The assistant makes two notable errors in this message:

  1. Underestimating average token length: The estimate of ~2,000 tokens per request is roughly half the actual average of 4,116 tokens. This error propagates into the mental model of how many requests can fit in the KV cache. With 116K total slots and 4,116 tokens per request, only ~28 requests could fit if every request ran to completion simultaneously — far fewer than the ~50 observed. The discrepancy is explained by requests at different stages of completion (some just starting, some nearly finished), but the underestimate makes the problem seem less severe than it actually is.
  2. Overlooking the client-side measurement: The assistant accepts the user's framing of "500 tps" without immediately questioning the measurement methodology. It takes until [msg 3837] to check the client-side logs and discover that B2 is actually running at 0.5 req/s with 1,560 avg tokens — a different dataset than the one showing 4,116 tokens. The assistant initially conflates two different data streams. These errors are corrected in subsequent messages and do not undermine the core diagnosis. The KV cache bottleneck remains the primary constraint; the errors only affect the quantitative estimates.

Conclusion

Message [msg 3834] exemplifies the kind of diagnostic reasoning that separates effective LLM operations from guesswork. The assistant takes three sparse observations — low running requests, high cache usage, rare prefills — and synthesizes them into a coherent causal model. It identifies the KV cache as the bottleneck, quantifies the constraint, and proposes concrete optimization paths. The message is not the final word on the problem — subsequent messages refine the analysis and test solutions — but it is the moment the problem was correctly framed. In any optimization effort, framing is half the battle.