The Hierarchical Cache Pivot: Diagnosing and Solving SGLang's KV Cache Bottleneck

Introduction

In the course of a large-scale synthetic data generation pipeline for training an EAGLE-3 speculative decoding drafter, a critical performance bottleneck emerged. The SGLang inference server, tasked with generating responses from the Kimi-K2.5 model across 88,000 prompts, was delivering far below its potential throughput. The user observed roughly 500 tokens per second and asserted the system should be capable of 4–5× that rate. What followed was a multi-message diagnostic journey culminating in message [msg 3841], where the assistant pivots from incremental VRAM tuning to a fundamentally different architectural approach: SGLang's hierarchical KV cache. This message represents the decisive moment where the assistant commits to a high-impact optimization strategy, performs the quantitative analysis to validate it, and begins the implementation — all while making visible the reasoning process behind the choice.

The Context: A KV Cache Wall

The story leading up to message [msg 3841] begins several messages earlier. The user's initial complaint at [msg 3831] — "Look at sglang metrics, currently only doing 500tps, what seems to be the bottleneck? Should be 4-5x faster" — triggered a rapid investigation. The assistant examined the server logs ([msg 3832]) and discovered that the actual generation throughput was 780–890 tok/s, not 500, but that only 47–56 concurrent requests were running despite a concurrency setting of 150 and over 90 requests sitting in the queue.

The root cause became clear in [msg 3834]: the KV cache was saturated. With --mem-fraction-static 0.85, the server could hold approximately 116,171 total tokens in its KV cache across all 8 GPUs. With each request averaging 2,000–4,000 tokens (the B2_opencodeinstruct dataset averaged 4,116 tokens per response), only about 28–50 requests could fit simultaneously. The scheduler simply could not admit new prefills because there was no room in the KV cache. The server was generating tokens at a healthy rate for the requests it could serve, but most of the 150 concurrent requests were stuck waiting for existing ones to finish and free up cache space.

The assistant explored incremental fixes. Increasing --mem-fraction-static from 0.85 to 0.95 would yield only about 12% more tokens ([msg 3836]). The user's suggestion at [msg 3839] — "Consider ways to increase context to maximum with max perf (ram offload IFF that will likely make things faster)" — framed the problem correctly: the bottleneck was context capacity, and the solution might involve offloading to host RAM if it would actually improve performance rather than degrade it.

Message 3841: The Two Approaches

Message [msg 3841] opens with the assistant laying out two candidate strategies:

Two promising approaches:

>

1. --enable-hierarchical-cache with --hicache-size — spills KV cache to host RAM. We have ~449GB RAM, model takes maybe 60GB in RAM for loading, leaving ~380GB free. This could dramatically increase effective KV capacity.

>

2. --cpu-offload-gb — offloads model weights to CPU RAM, freeing more GPU VRAM for KV cache. But this would slow down inference.

>

Option 1 is the winner — keep model weights on GPU for fast inference, but overflow KV cache to host RAM. MLA's compressed KV (small per token) means CPU↔GPU transfer should be fast relative to the decode step.

This is the critical decision point. The assistant evaluates two mechanisms that SGLang provides for breaking through the VRAM wall, and the reasoning is sharp. Option 2 — offloading model weights to CPU — would indeed free GPU memory for KV cache, but at a terrible cost: every inference step would require fetching model weights from host RAM over the PCIe bus, which is orders of magnitude slower than GPU memory. The assistant correctly identifies that this would slow down inference, not speed it up.

Option 1, the hierarchical cache, is more elegant. It keeps the model weights on GPU where they belong (fast inference), but allows the KV cache — which grows linearly with sequence length and batch size — to overflow into host RAM. The key insight is that MLA (Multi-head Latent Attention), the attention mechanism used by DeepSeek-derived models like Kimi-K2.5, has a compressed KV representation. Instead of storing full key-value vectors for every attention head, MLA projects them into a low-dimensional latent space (kv_lora_rank=512). This means each token's KV cache entry is small — the assistant estimates about 68.6 KB per token across all 61 layers, or about 8.6 KB per GPU with tensor parallelism of 8. Small KV entries mean that transferring them between GPU and CPU over PCIe should be fast relative to the decode computation itself, making the hierarchical cache viable.

The Quantitative Analysis

Having chosen the hierarchical cache, the assistant immediately moves to validate the approach with numbers. The bash command embedded in the message is a Python calculation that attempts to compute the expected benefit:

# Current: 116171 max tokens at 0.85 mem fraction
# MLA KV cache per token: kv_lora_rank=512, rope part, 61 layers
# In bf16: each token per layer stores ~(512 + rope) * 2 bytes 
# rope_head_dim for MLA = 64 (q_rope_head_dim from config)
# KV per token per layer = (512 + 64) * 2 = 1152 bytes
# Total per token = 1152 * 61 = 70272 bytes = 68.6 KB
# With TP=8: per-GPU = 70272 / 8 = 8784 bytes per token

The assistant works through two approaches to sizing the KV cache. First, it tries a bottom-up calculation based on the model architecture: kv_lora_rank=512, rope dimension of 64, bf16 storage (2 bytes per value), 61 layers, and tensor parallelism of 8. This yields about 8,784 bytes per token per GPU. Then it cross-checks this against the empirical observation that 116,171 tokens fit in approximately 14.8 GB of GPU KV cache space, yielding about 136,793 bytes per token per GPU — a number that seems inconsistent with the architectural estimate.

This inconsistency is visible in the assistant's own thinking within the code comments:

# Actually let me just compute from known values:
# 116171 tokens at 0.85 mem_fraction
# Free VRAM after model: approx 97887 - (92137 - 0.85*97887) MiB?
# Simpler: KV cache size = 116171 tokens * bytes_per_token
# The GPU has about 14.8 GB for KV (from 5.1GB free + 0.85 fraction)

The assistant is wrestling with uncertainty about how SGLang's mem_fraction_static parameter actually works — is it a fraction of total GPU memory, or a fraction of free memory after model loading? The code shows multiple abandoned approaches to this calculation. This is a realistic example of the kind of ambiguity that arises when working with complex inference engines: the documentation may be unclear, and the only reliable method is to derive values from observed behavior.

Despite the uncertainty, the assistant settles on a pragmatic approach: use the empirical ratio of 116,171 tokens at the current 0.85 fraction as a baseline, and scale proportionally. With 300 GB of host RAM allocated to the hierarchical cache, distributed across 8 GPUs, the calculation yields approximately 294,352 additional tokens, for a total of about 410,523 tokens. At 4K average tokens per request, this would support about 103 concurrent requests — roughly double the current capacity.

What This Message Achieves

Message [msg 3841] is the decisive analytical step in a multi-stage optimization process. It accomplishes several things:

  1. Identifies the correct bottleneck: While earlier messages had identified KV cache saturation as the problem, this message frames it in terms of the fundamental VRAM constraint and evaluates the available architectural solutions.
  2. Selects the optimal strategy: The assistant correctly rejects CPU offloading of model weights (which would harm performance) and selects hierarchical cache (which keeps fast inference while expanding capacity).
  3. Performs feasibility analysis: The quantitative estimates show that 300 GB of host RAM could roughly double the effective KV cache capacity, supporting ~100 concurrent requests instead of ~50.
  4. Reveals the reasoning process: The code comments show the assistant working through ambiguous parameter semantics, making assumptions, and triangulating between architectural calculations and empirical observations.
  5. Identifies a key architectural advantage: The insight that MLA's compressed KV representation makes hierarchical cache particularly viable — small per-token data means fast CPU↔GPU transfers — is a non-trivial observation that draws on knowledge of both the model architecture and the inference engine's capabilities.

Assumptions and Potential Mistakes

Several assumptions are visible in this message, some of which could be incorrect:

Assumption about hicache scaling: The assistant assumes that adding hierarchical cache will allow the scheduler to run more concurrent requests, and that decode throughput will scale roughly linearly with the number of concurrent requests. This is a reasonable assumption for a well-batched inference server, but it may not hold if the hierarchical cache introduces latency overhead from PCIe transfers, or if the scheduler's policy doesn't efficiently utilize the expanded capacity.

Assumption about host RAM availability: The assistant estimates ~380 GB free RAM after accounting for model loading (~60 GB). This assumes the model's memory footprint in RAM is similar to its GPU footprint, which may not be accurate. The model might require additional memory for intermediate buffers, Python overhead, or other processes running on the machine.

Uncertainty about mem_fraction_static semantics: The code shows the assistant struggling to determine whether mem_fraction_static is a fraction of total GPU memory or a fraction of free memory after model loading. The final calculation uses a proportional scaling approach that sidesteps this ambiguity, but the underlying uncertainty could affect the accuracy of the estimates.

The syntax error: The bash command at the end of the message fails with zsh:42: unmatched " — a quoting error in the complex nested Python command. This is a practical failure: the assistant's inline calculation didn't execute successfully. The actual computation would need to be retried (as indeed happens in the following message [msg 3842], where the calculation succeeds with a simpler command). This error is instructive: it shows the assistant working in real time, making mistakes, and needing to iterate.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces:

The Thinking Process

What makes this message particularly valuable is the visibility of the assistant's reasoning process. The code comments reveal a mind working through a complex problem with multiple unknowns:

  1. Problem framing: "Two promising approaches" — the assistant immediately structures the solution space.
  2. Evaluation: Option 2 (CPU offload) is rejected because it would "slow down inference" — a clear cost-benefit judgment.
  3. Architectural insight: "MLA's compressed KV (small per token) means CPU↔GPU transfer should be fast relative to the decode step" — this is the key insight that makes the hierarchical cache viable.
  4. Quantitative validation: The Python code attempts to compute expected benefits from first principles (model architecture) and from empirical data (observed token counts at current settings).
  5. Uncertainty management: The code shows multiple abandoned approaches to computing KV cache size, with comments like "Wait, that is not how it works" and "Actually let me just compute from known values." This is the assistant calibrating its understanding against available data.
  6. Practical failure: The syntax error at the end shows that even well-reasoned plans can stumble on implementation details. The assistant doesn't get the answer in this message — it has to retry in the next message.

Conclusion

Message [msg 3841] is a pivotal moment in a larger optimization narrative. It represents the transition from incremental tuning (bumping mem_fraction_static from 0.85 to 0.88 or 0.93, yielding marginal gains) to a fundamentally different architectural approach (spilling KV cache to host RAM via hierarchical cache, potentially doubling throughput). The assistant's reasoning is clear, the quantitative analysis is thorough (even if it doesn't execute successfully), and the decision framework is sound.

The message also illustrates a deeper truth about large-scale ML inference optimization: the most impactful changes often come not from squeezing a few more percent out of existing resources, but from rethinking the resource allocation strategy entirely. The KV cache was full because GPU VRAM is finite and expensive. Host RAM is abundant and cheap. The hierarchical cache exploits this asymmetry, using the GPU for what it does best (fast computation) and the host for what it does adequately (storage of infrequently accessed data). It's a textbook application of the principle that the right architecture for a system depends on the relative costs and capabilities of its components.

The following messages ([msg 3842], [msg 3843]) would execute the successful calculation and begin implementation. The chunk summary for segment 28 confirms that the hierarchical cache approach was ultimately successful, yielding 159K GPU tokens and ~930–1350 tok/s throughput — roughly 2–3× improvement over the initial baseline. Message [msg 3841] is where that success was conceived and justified.