Learning from Failure: The Corrected Hierarchical Cache Launch

In the middle of an intensive inference pipeline optimization session, a single assistant message ([msg 3867]) represents a pivotal moment of learning and correction. The message is deceptively simple — a single bash command launching an SGLang server with a hierarchical cache configuration — but it arrives only after a costly failure, a deep dive into source code, and careful recalculation. This article examines that message in detail: why it was written, the reasoning behind its decisions, the assumptions that had to be corrected, and the knowledge it both required and produced.

The Bottleneck That Started It All

The context for this message stretches back several rounds. The assistant had been running a large-scale inference pipeline to generate training data for an EAGLE-3 speculative decoding drafter, using the Kimi-K2.5 model served by SGLang across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The pipeline was processing datasets like B2_opencodeinstruct, but throughput was severely constrained. As the assistant diagnosed in [msg 3840], the KV cache was the bottleneck: at a maximum of approximately 116,000 tokens, only about 50 concurrent requests could fit given the average 4,000-token sequence length. With 150 concurrent requests being fired at once, over 100 were sitting in queue, waiting for KV cache slots to free up. The effective throughput was far below what the hardware could deliver.

The user's directive was unambiguous: "Use all levers" ([msg 3848]). The assistant needed to dramatically increase KV cache capacity, and the most promising mechanism was SGLang's hierarchical cache (--enable-hierarchical-cache), which spills KV cache entries from GPU VRAM into host RAM when GPU memory is exhausted. With 449 GB of host RAM available, the potential was enormous.

The First Attempt and Its Failure

The assistant's first attempt, in [msg 3850], was aggressive: launch with --hicache-size 300, intending to allocate 300 GB of host RAM for KV cache overflow. The server command was issued with mem-fraction-static 0.93, pushing GPU memory utilization to its limit, and the hierarchical cache enabled with write_through policy and kernel I/O backend.

The result was catastrophic. As the assistant discovered in [msg 3854], each tensor-parallelism (TP) rank independently tried to allocate 300 GB. With 8 TP ranks, that meant 2.4 TB of host memory requested — far exceeding the 449 GB available. The server hung, consumed 439 GB of RAM, and left GPUs in a stuck state requiring a Proxmox host-level kill to reset. The assistant had made a critical assumption: that --hicache-size specified a total pool shared across all ranks, when in fact it was a per-rank allocation.

The Diagnostic Pivot

What follows in messages [msg 3854] through [msg 3866] is a textbook example of systematic debugging. The assistant first confirmed the OOM by checking free -g and pgrep. Then it killed the stuck processes, but residual CUDA contexts remained on GPUs 0 and 2, requiring a Proxmox host-level intervention (pct exec 129) to fully reset. Once the system was clean — 445 GB available, all GPUs at 0 MiB — the assistant turned to understanding the root cause.

The critical diagnostic step was reading the SGLang source code. In [msg 3862], the assistant grepped through hiradix_cache.py and memory_pool_host.py to understand how hicache_size was interpreted. The key finding, articulated in [msg 3866], was that host_size > 0 maps to int(host_size * 1e9 // self.size_per_token) — it's in gigabytes, and it's per-rank. Each TP worker creates its own host memory pool. The code does check psutil.virtual_memory().available, but each rank checks independently without coordination, so 8 ranks each requesting 300 GB will collectively OOM.

This is a subtle design issue in SGLang: the parameter name --hicache-size suggests a total system-wide allocation, but the implementation treats it as per-rank. The assistant had to read the actual source to discover this mismatch between documentation intuition and implementation reality.

The Calculation

With the per-rank semantics confirmed, the assistant calculated the safe allocation. The available host RAM was 445 GB. The model loading overhead was estimated at ~40 GB. A safety margin of ~10 GB was added. That left approximately 395 GB for hierarchical cache. Dividing by 8 ranks yielded roughly 49 GB per rank. The assistant rounded down to 48 GB for a clean number with margin.

The assistant also explored an alternative: --hicache-ratio, which sizes the host pool as a multiple of the device pool. With mem-fraction-static 0.93, the device pool was estimated at ~127,000 tokens. At 136,793 bytes per token per GPU, that's about 17.4 GB per rank. A ratio of 2.5 would yield ~44 GB per rank, and a ratio of 3.0 would yield ~52 GB per rank. Either would work, but the assistant chose --hicache-size 48 for explicit control.

The Subject Message: A Corrected Launch

Message [msg 3867] is the culmination of this diagnostic journey. The assistant launches the server with --hicache-size 48, having learned the hard way that this is a per-rank parameter. The command is otherwise identical to the failed attempt: the same NCCL environment variables for optimized inter-GPU communication (NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS), the same model path, TP size of 8, mem-fraction-static 0.93, num-continuous-decode-steps 4, disable-custom-all-reduce, and the hierarchical cache enabled with write_through policy and kernel I/O backend.

The comment in the message — "48GB per rank × 8 = 384GB total, fits in 445GB with margin" — reveals the corrected mental model. The assistant now understands that the 384 GB total allocation (48 × 8) leaves approximately 61 GB of headroom (445 - 384), which accounts for the ~40 GB model loading overhead plus a comfortable 21 GB buffer. This is a conservative but safe configuration.

Input Knowledge Required

To understand this message, several pieces of prior knowledge are essential. First, the reader must know that SGLang uses tensor parallelism (TP), where the model is sharded across multiple GPUs and each rank runs its own process. Second, the hierarchical cache mechanism must be understood: it allows KV cache entries to overflow from GPU VRAM to host RAM, trading latency for capacity. Third, the relationship between KV cache size and concurrent request capacity must be clear — each request consumes KV cache proportional to its sequence length, so more KV cache means more concurrent requests. Fourth, the specific memory characteristics of this setup matter: the MLA (Multi-head Latent Attention) architecture of Kimi-K2.5 produces a compressed KV representation of approximately 136,793 bytes per token per GPU, which determines how many tokens fit in a given memory budget.

Output Knowledge Created

This message produces several important outputs. Most immediately, it creates a running SGLang server with dramatically expanded KV cache capacity — from ~116,000 tokens (GPU-only) to an estimated ~410,000 tokens (GPU + host). This translates from ~28 concurrent requests at 4K average length to approximately 100 concurrent requests, potentially tripling throughput. The message also validates the corrected understanding of --hicache-size semantics, which becomes operational knowledge for future deployments. Finally, it establishes a methodology for sizing hierarchical cache allocations: measure available RAM, subtract model overhead and safety margin, divide by TP rank count, and round down.

The Thinking Process

The reasoning visible in the surrounding messages reveals a sophisticated diagnostic process. The assistant initially assumed --hicache-size was a total allocation, a natural interpretation given the parameter name. When the OOM occurred, the assistant didn't just retry with a smaller number — it investigated why the parameter behaved unexpectedly. It read the SGLang source code to understand the implementation, confirmed the per-rank semantics, and only then recalculated. This is a pattern of deep debugging: not treating the symptom (OOM) but understanding the root cause (parameter semantics), then applying the corrected understanding.

The assistant also demonstrated resourcefulness by exploring the alternative --hicache-ratio parameter, calculating equivalent values, and choosing the more explicit --hicache-size for precise control. The decision to use 48 GB rather than 49 GB (the calculated maximum) shows a conservative engineering judgment — round down for safety margin.

Conclusion

Message [msg 3867] is far more than a simple server launch command. It is the product of a complete learning cycle: hypothesis (300 GB total), failure (OOM), diagnosis (source code analysis), corrected hypothesis (48 GB per rank), and validated execution. The message captures the moment when the assistant's understanding of SGLang's hierarchical cache aligns with reality, and it marks the transition from debugging to productive inference. In the broader narrative of this coding session, it represents a turning point where KV cache capacity was finally unlocked, enabling the throughput optimization that would follow.