The KV Cache Calculus: Computing the Impact of Hierarchical Memory for SGLang Inference

In the middle of an intense debugging session focused on SGLang inference throughput, the assistant fires off a single SSH command that runs a compact Python calculation. The message, <msg id=3842>, is deceptively simple — a few lines of arithmetic that compute bytes per token, extrapolate cache capacity, and project concurrent request limits. But beneath this brevity lies a critical decision point: whether to invest in SGLang's hierarchical cache feature to spill KV cache to host RAM, and whether that investment will yield the 4–5× throughput improvement the user expects.

The Bottleneck Revealed

To understand why this message exists, we must trace the reasoning chain that led to it. Earlier in the conversation, the user had asked the assistant to investigate why the SGLang server was achieving only ~500 tokens per second (tps) when it "should be 4-5× faster" ([msg 3831]). The assistant examined server logs and found that the gen throughput was actually 780–890 tok/s at the server level, but the effective throughput was much lower because only 47–56 requests ran concurrently despite a concurrency setting of 150 and over 100 requests sitting in the queue ([msg 3833]).

The root cause was clear: KV cache saturation. With --mem-fraction-static 0.85, the server could hold approximately 116,171 tokens total in GPU memory. With each request averaging 2,000–4,000 tokens (depending on the dataset category), only about 28–58 requests could fit simultaneously. The scheduler was blocked from admitting new requests because there was simply no room to store their key-value cache entries.

The assistant first explored whether simply increasing --mem-fraction-static would help ([msg 3836]). The calculation showed diminishing returns: moving from 0.85 to 0.95 would yield only ~12% more token capacity — far short of the 4–5× improvement the user wanted. The KV cache was already consuming most available GPU memory after the model weights (approximately 68.4 GB for the model across 8 GPUs, leaving ~14.8 GB per GPU for KV cache).

Discovering Hierarchical Cache

The user then explicitly suggested exploring RAM offload: "Consider ways to increase context to maximum with max perf (ram offload IFF that will likely make things faster)" ([msg 3839]). The assistant checked SGLang's available flags and discovered --enable-hierarchical-cache with --hicache-size — a feature that spills KV cache entries to host RAM when GPU memory is exhausted ([msg 3840]). With approximately 449 GB of host RAM available and only ~60 GB consumed by the model during loading, roughly 380 GB could be repurposed for KV cache overflow.

This led to a preliminary analysis in <msg id=3841>, where the assistant sketched the rough math: if 300 GB of host RAM were allocated to hierarchical cache, and if the current GPU KV cache of ~14.8 GB held 116K tokens, then the additional capacity would be approximately (300 GB / 8 GPUs) / (14.8 GB / 116K tokens) ≈ 294K additional tokens. This would bring total capacity to ~410K tokens — enough for ~102 concurrent requests at 4K average token length.

The Subject Message: Precision Through Calculation

The subject message <msg id=3842> is the assistant's attempt to validate this rough estimate with a precise computation. The command is:

ssh root@10.1.230.174 'python3 -c "
current_tokens = 116171
current_kv_gb = 14.8
bpt = (current_kv_gb * 1024**3) / current_tokens
print(f\"Bytes per token per GPU: {bpt:.0f}\")
hicache_gb = 300
extra_per_gpu = (hicache_gb * 1024**3) / 8
extra_tokens = extra_per_gpu / bpt
total = current_tokens + extra_tokens
print(f\"Extra tokens from {hicache_gb}GB hicache: {extra_tokens:.0f}\")
print(f\"Total tokens: {total:.0f}\")
print(f\"At 4K avg: {total/4000:.0f} concurrent req\")
print(f\"At 2K avg: {total/2000:.0f} concurrent req\")
"'

The output confirms the estimate:

Bytes per token per GPU: 136793
Extra tokens from 300GB hicache: 294352
Total tokens: 410523
At 4K avg: 103 concurrent req
At 2K avg: 205 concurrent req

The Reasoning Behind the Numbers

Every constant in this calculation encodes a prior discovery. The current_tokens = 116171 value comes from the server's max_total_num_tokens field, retrieved via the SGLang server info API in <msg id=3834>. The current_kv_gb = 14.8 is an approximation derived from the GPU memory analysis in <msg id=3835> and <msg id=3836>: with 97,887 MiB total VRAM per GPU, approximately 92,137 MiB used, and the model weights consuming roughly 68.4 GB (547 GB / 8 GPUs), the remaining ~14.8 GB is allocated to KV cache.

The calculation bpt = (current_kv_gb * 1024**3) / current_tokens yields 136,793 bytes per token per GPU — approximately 134 KB. This is a critical number because it encodes the memory footprint of the MLA (Multi-head Latent Attention) KV cache for the Kimi-K2.5 model. With 61 layers, kv_lora_rank=512, and a rope dimension of 64, each token requires storing compressed key-value pairs across all layers. The 134 KB figure is consistent with the MLA architecture's design: the compressed KV representation is significantly smaller than standard multi-head attention would require, but still substantial when multiplied across thousands of tokens and eight GPU replicas.

The choice of hicache_gb = 300 is deliberate — it's a conservative allocation that leaves headroom in the 449 GB of host RAM for the operating system, file system cache, and other processes. The division by 8 accounts for tensor parallelism (TP=8), distributing the host memory evenly across the eight GPUs.

What This Calculation Actually Tells Us

The output reveals that hierarchical cache could increase effective KV capacity by 253% (from 116K to 410K tokens). At 4K average token length, this translates to approximately 103 concurrent requests — roughly double the current ~50. At 2K average token length (typical for the B1_glaive dataset), the server could handle 205 concurrent requests, exceeding the 150 concurrency target.

But these numbers come with implicit assumptions that the assistant does not explicitly state:

  1. Linear scaling of throughput with concurrent requests: The calculation assumes that doubling concurrent requests will double throughput. This is optimistic — at higher concurrency, scheduling overhead, memory bandwidth contention, and the cost of transferring KV cache pages between CPU and GPU via PCIe could reduce the effective throughput gain.
  2. Negligible hicache overhead: The calculation does not account for the latency of fetching KV cache entries from host RAM. SGLang's hierarchical cache uses a write-back or write-through policy, meaning that when a token's KV cache is needed for decoding, it may need to be fetched from CPU memory. If the PCIe bandwidth becomes a bottleneck, the decode step could slow down, partially offsetting the gains from higher concurrency.
  3. The 14.8 GB figure is approximate: The assistant derived this from nvidia-smi output showing ~92 GB used out of ~97 GB, combined with the mem_fraction_static=0.85 setting. The actual split between model weights, activation memory, and KV cache is not precisely known without instrumenting the SGLang memory allocator.
  4. 300 GB is an arbitrary choice: The assistant picked 300 GB as a reasonable allocation, but the optimal value depends on how much host RAM is truly available after accounting for the OS, the model's CPU-side footprint, and the dataset processing pipeline running concurrently.

The Decision Point

Despite these caveats, the calculation provides a compelling quantitative basis for the decision to enable hierarchical cache. The alternative — increasing mem_fraction_static to 0.95 — would yield only 129,838 tokens (an 11.8% improvement). Hierarchical cache offers a 253% improvement. The choice is clear.

The assistant's reasoning, visible across the sequence of messages, follows a classic debugging pattern: measure the bottleneck, evaluate marginal improvements, discover a high-impact alternative, and validate with quantitative analysis. The subject message is the validation step — the moment where a hypothesis (hierarchical cache will dramatically increase capacity) is tested against the numbers before committing to implementation.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. Bytes per token per GPU for MLA KV cache: 136,793 bytes (~134 KB) for the Kimi-K2.5 model with TP=8. This is a model-specific constant that could be used for capacity planning in future deployments.
  2. Projected capacity with 300 GB hicache: 410,523 total tokens, supporting ~103 concurrent requests at 4K average length or ~205 at 2K average length.
  3. Confirmation that hierarchical cache is the right approach: The 253% capacity increase justifies the complexity of enabling and tuning the feature.
  4. A template for similar calculations: The Python snippet is reusable — change current_tokens, current_kv_gb, and hicache_gb to evaluate different configurations or models.

The Broader Context

This message sits at a pivotal moment in the session. The assistant has spent hours debugging EAGLE-3 speculative decoding, fixing hidden state extraction bugs, and tuning SGLang performance. The user's frustration is evident — the inference pipeline is running at a fraction of expected speed, and the 28+ hour ETA for dataset generation is unacceptable. The hierarchical cache calculation represents a potential breakthrough: if it works as projected, the generation pipeline could complete in hours rather than days.

The next messages in the conversation (not shown in the context) would involve actually enabling hierarchical cache, tuning the hicache-size parameter, restarting the server, and measuring the real-world throughput improvement. The calculation in <msg id=3842> is the foundation for all of that work — the quantitative justification that the effort is worthwhile.

Conclusion

The subject message <msg id=3842> is a masterclass in targeted quantitative reasoning. In six lines of Python, the assistant transforms a vague hypothesis ("RAM offload might help") into a concrete projection ("300 GB hicache → 410K tokens → 103 concurrent requests"). It bridges the gap between the observed symptom (low throughput) and the proposed solution (hierarchical cache), providing the numerical confidence needed to proceed with a complex configuration change.

The calculation is not perfect — it makes simplifying assumptions about linear scaling and ignores overhead — but it is good enough for decision-making. In the high-velocity environment of ML infrastructure debugging, perfect precision is less valuable than timely, directional accuracy. The assistant correctly prioritizes getting a "good enough" answer over a perfectly accurate one, enabling the team to move forward with confidence.