Scaling Throughput by Expanding KV Cache: The Hierarchical Cache Pivot in SGLang Inference

Introduction

In the high-stakes world of large language model inference, throughput is king. When generating tens of thousands of training samples for speculative decoding — specifically EAGLE-3 drafter training data for the Kimi-K2.5 model — every token per second counts. Message 3843 in this opencode session captures a pivotal analytical moment: the assistant has identified the root cause of a throughput bottleneck and is now calculating the viability of a specific solution. The message is a bridge between diagnosis and action, translating a performance problem into a concrete engineering proposal backed by quantitative reasoning.

The Context: A Bottleneck Identified

The conversation leading up to this message had been a deep dive into SGLang server performance. The team was running inference on a massive dataset (88K prompts across multiple categories like B1_glaive and B2_opencodeinstruct) to generate responses for EAGLE-3 training. The user had noticed the server was only achieving around 500 tokens per second and asked pointedly: "Look at sglang metrics, currently only doing 500tps, what seems to be the bottleneck? Should be 4-5x faster" ([msg 3831]).

The assistant's investigation in preceding messages revealed a clear pattern. The server's decode batch logs showed #running-req: 46-56 despite a concurrency setting of 150, with 94-101 requests sitting in the queue. The token usage was hovering at 0.85-0.96 — the KV cache was nearly saturated. With --mem-fraction-static 0.85, the server could hold approximately 116,171 total tokens. Given that B2_opencodeinstruct responses averaged 4,116 tokens each, only about 28 concurrent requests could fit at steady state. The scheduler was blocking new prefills because there was simply no room in the KV cache ([msg 3834]).

The user then cut through the analysis with a direct instruction: "Consider ways to increase context to maximum with max perf (ram offload IFF that will likely make things faster)" ([msg 3839]). This set the stage for the assistant to explore the hierarchical cache option — SGLang's mechanism for spilling KV cache entries to host RAM when GPU memory is exhausted.

The Message: Analytical Reasoning and Feasibility Check

Message 3843 opens with the assistant synthesizing the quantitative case for hierarchical cache:

With 300GB hicache, we'd go from 116K → 410K total tokens, supporting ~100 concurrent requests at 4K avg length. That should roughly double the effective throughput.

This single sentence encapsulates the entire value proposition. The assistant has already done the mental math: current capacity is 116K tokens, 300GB of host RAM would add approximately 294K additional tokens (based on the per-token memory footprint calculated in the previous message), bringing the total to 410K tokens. At 4K average response length, that translates to roughly 100 concurrent requests — nearly double the ~50 that were fitting before.

The assistant then adds a secondary consideration:

But actually, the real win is simpler — we should also try just increasing --mem-fraction-static to 0.92-0.93, AND increase --max-running-requests to let more short requests in. But hicache is the bigger lever.

This is a revealing moment of engineering judgment. The assistant acknowledges that there are multiple levers to pull. Increasing mem_fraction_static would allocate more GPU memory to the KV cache at the expense of headroom, potentially allowing more tokens to be stored on-device. Increasing max_running_requests would allow the scheduler to admit more concurrent requests. But the assistant correctly identifies hierarchical cache as "the bigger lever" because it taps into a much larger memory pool — 408GB of available host RAM versus the ~5GB of free GPU VRAM per card.

The message then executes a pragmatic feasibility check:

Let me also check if hicache actually works well in practice on this setup, and if the --hicache-ratio approach is simpler:

The bash command free -g reveals the system has 449GB total RAM with 408GB available. This confirms that dedicating 300GB to hierarchical cache is feasible without starving the operating system or other processes.

The Thinking Process: From Observation to Quantified Proposal

What makes this message particularly interesting is the thinking process visible in its structure. The assistant is not just reporting numbers — it is reasoning through trade-offs in real time.

The chain of reasoning proceeds as follows:

Bottleneck confirmed: KV cache is full at 116K tokens, limiting concurrent requests to ~50 despite 150 concurrency setting.
Solution identified: Hierarchical cache can spill KV entries to host RAM, dramatically increasing effective capacity.
Quantified projection: Using the bytes-per-token calculation from the previous message (~136,793 bytes per token per GPU), 300GB of host RAM translates to ~294K additional tokens. Total capacity becomes ~410K tokens.
Concurrent request estimate: At 4K average response length, 410K tokens supports ~100 concurrent requests. At 2K average (shorter responses), it supports ~205.
Feasibility verification: The system has 408GB available RAM, so 300GB allocation is realistic.
Alternative consideration: The assistant doesn't stop at hicache — it also notes that increasing mem_fraction_static and max_running_requests could help, particularly for shorter requests that don't benefit as much from the hicache overhead. This multi-faceted thinking shows a mature engineering approach. The assistant doesn't latch onto a single solution but instead evaluates a portfolio of options, ranking them by expected impact.

Assumptions and Their Implications

Several assumptions underpin the assistant's analysis, and understanding them is crucial for evaluating the proposal's validity.

Assumption 1: Linear scaling of throughput with KV capacity. The assistant projects that doubling concurrent requests from ~50 to ~100 will roughly double effective throughput. This assumes the GPU's compute capacity is not already saturated — that the bottleneck is purely memory capacity, not compute. If the GPU decoders are already running near peak utilization, adding more concurrent requests would increase latency per request without improving aggregate throughput. The server logs showing ~850 tok/s gen throughput with only 50 requests suggest there is headroom, but this is not explicitly verified.

Assumption 2: Hierarchical cache overhead is negligible. Spilling KV cache to host RAM and fetching it back during decode introduces latency. For MLA (Multi-head Latent Attention), the compressed KV representation is relatively small per token (~68.6 KB total across all layers, or ~8.6 KB per GPU with TP=8), so the PCIe transfer cost might be manageable. However, the assistant does not model this overhead. If hicache introduces even a few milliseconds of latency per decode step, the throughput gains could be significantly reduced.

Assumption 3: 300GB is a safe allocation. The system has 408GB available, but the operating system, file system cache, and other processes need memory too. The assistant implicitly assumes that 300GB leaves sufficient headroom (108GB) for the OS and other workloads. This is reasonable but not explicitly validated against the system's other memory consumers.

Assumption 4: The per-token memory calculation is accurate. The bytes-per-token figure of ~136,793 was derived from approximate values (14.8 GB current KV size, 116,171 tokens). If the actual per-token footprint differs, the capacity projections would shift accordingly.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several concepts:

KV cache: The key-value cache stores attention keys and values from previous tokens, enabling the model to attend to the full context without recomputation. It is the primary memory consumer during autoregressive decoding.
SGLang's memory management: SGLang uses --mem-fraction-static to control what fraction of remaining GPU memory (after model weights are loaded) is allocated to the KV cache. The max_total_num_tokens parameter directly limits how many tokens can be cached.
Hierarchical cache (hicache): A SGLang feature that spills KV cache entries from GPU VRAM to host DRAM when GPU memory is exhausted, fetching them back on demand during decode steps. This effectively extends the KV cache capacity at the cost of PCIe transfer latency.
MLA (Multi-head Latent Attention): The attention mechanism used by DeepSeek-family models (including Kimi-K2.5). MLA compresses the KV cache into a low-rank latent space (kv_lora_rank=512), making each token's cache footprint significantly smaller than standard multi-head attention. This compression is what makes hierarchical cache particularly attractive — the per-token transfer cost is low.
Tensor parallelism (TP=8): The model is sharded across 8 GPUs, so each GPU stores only 1/8 of the KV cache. This means the per-GPU memory footprint per token is correspondingly smaller.
EAGLE-3 training pipeline: The broader context is generating training data for EAGLE-3, a speculative decoding framework that uses a lightweight draft model to predict multiple tokens per forward pass. The training data consists of input-output pairs where the output includes the model's reasoning trace (enclosed in thinking and response tokens).

Output Knowledge Created

This message produces several concrete outputs:

A quantified capacity projection: 300GB hicache → 410K total tokens → ~100 concurrent requests at 4K avg length.
A throughput estimate: "roughly double the effective throughput" — from the current ~412-789 tok/s (depending on dataset) to potentially ~800-1,500 tok/s.
A prioritized action plan: (a) Enable hierarchical cache with ~300GB host RAM allocation, (b) also increase mem_fraction_static to 0.92-0.93, (c) increase max_running_requests.
A feasibility confirmation: The system has 408GB available RAM, making the 300GB hicache allocation realistic.
A secondary insight: The assistant identifies that hicache is "the bigger lever" compared to simply increasing GPU memory fraction, correctly recognizing that the GPU VRAM is the binding constraint and host RAM is the abundant resource.

What Happens Next

The subsequent messages in the session (which form the broader segment 28) show the actual implementation of these ideas. The team experimented with --mem-fraction-static 0.93 (which caused an OOM), --kv-cache-dtype fp8_e4m3 (rejected by the user as potentially degrading quality), and --enable-hierarchical-cache with --hicache-size 48 (48GB per rank, which initially caused issues with host RAM overflow). The winning configuration settled at mem_fraction_static=0.88 + bf16 KV + hicache=48GB, yielding 159K GPU tokens and approximately 930-1,350 tok/s throughput — a roughly 2-3x improvement over the initial 600 tok/s baseline.

This trajectory shows that the assistant's analysis in message 3843 was directionally correct but the implementation required iteration. The 300GB hicache target was scaled back to 48GB per rank (384GB total across 8 ranks), and the final configuration combined multiple levers rather than relying on hicache alone.

Conclusion

Message 3843 exemplifies the kind of analytical reasoning that separates effective engineering from trial-and-error. Rather than blindly tweaking parameters, the assistant first diagnosed the bottleneck (KV cache saturation), quantified the constraint (116K token capacity), calculated the impact of a proposed solution (300GB hicache → 410K tokens → ~100 concurrent requests), verified feasibility (408GB available RAM), and acknowledged complementary approaches (mem_fraction_static increase, max_running_requests adjustment).

The message also reveals the assistant's understanding of the system's architecture — from the compressed MLA KV cache format that makes hicache viable, to the tensor parallelism that distributes the cache across GPUs, to the scheduler dynamics that determine how requests flow through the system. This holistic understanding allows the assistant to propose solutions that address the root cause rather than treating symptoms.

In the end, the actual deployment achieved a 2-3x throughput improvement — not quite the 4-5x the user initially hoped for, but a substantial gain nonetheless. And the analytical framework established in this message — identify the bottleneck, quantify the constraint, calculate the solution's impact, verify feasibility — is a reusable pattern that applies far beyond this specific inference optimization task.