The Per-Rank Trap: Debugging SGLang's Hierarchical Cache Configuration

Introduction

In the course of deploying a large-scale inference pipeline for synthetic data generation, a seemingly simple configuration parameter—--hicache-size 300—triggered a cascade of failures that consumed nearly twenty messages of debugging. The subject message, <msg id=3866>, represents the breakthrough moment: the assistant finally understood why the parameter was causing out-of-memory (OOM) crashes and, more importantly, how to correctly size it. This article examines that single message in depth, unpacking the reasoning, assumptions, mistakes, and knowledge that led to this critical insight.

The Context: A Bottleneck Discovered

The broader session (Segment 28) was focused on optimizing an inference server running the Kimi-K2.5 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The server was generating synthetic training data for an EAGLE-3 speculative decoding drafter, but throughput was severely limited. The bottleneck was clear: the KV cache could only hold ~116K tokens, which at an average of 4K tokens per request meant only about 28-50 concurrent requests could be served. The rest sat in queue.

The user had directed the assistant to "use all levers" to increase effective KV cache capacity. Three approaches were available: increasing --mem-fraction-static to allocate more GPU VRAM to the KV cache, switching to FP8 KV cache dtype (rejected by the user as quality-degrading), and enabling SGLang's hierarchical cache (--enable-hierarchical-cache) to spill KV cache entries to host RAM.

The hierarchical cache option was the most promising. With 449GB of host RAM available, the assistant initially tried --hicache-size 300, reasoning that 300GB of host memory for KV cache overflow would dramatically increase capacity. The server promptly crashed, consuming 439GB of RAM and leaving GPU processes in a defunct state.

The Subject Message: A Forensic Analysis

The message <msg id=3866> opens with a crystallized insight:

Key finding: host_size > 0 means int(host_size * 1e9 // self.size_per_token). It's in GB, and it's per-rank (each TP worker creates its own pool). The code checks psutil.virtual_memory().available but each rank checks independently — they don't coordinate, so 8 ranks each trying 300GB will OOM.

This single paragraph encapsulates the result of a multi-step debugging process that involved reading SGLang source code, examining memory pool initialization logic, and running calculations. Let us break down what this finding reveals.

First, the assistant has traced the parameter through the codebase. The expression int(host_size * 1e9 // self.size_per_token) shows that host_size (the value passed to --hicache-size) is interpreted as a value in gigabytes (multiplied by 1e9 to convert to bytes), then divided by the size of a single KV cache token to determine how many tokens of host cache to allocate.

Second, and more critically, the assistant has confirmed that this allocation happens per-rank. With tensor parallelism (TP) set to 8, eight separate worker processes each create their own host memory pool. Each pool independently attempts to allocate the full --hicache-size amount. When the assistant initially passed --hicache-size 300, each of the 8 ranks tried to allocate 300GB, for a total of 2.4TB—far exceeding the 449GB of physical RAM.

Third, the assistant identifies a design flaw: the code calls psutil.virtual_memory().available to check available memory, but each rank performs this check independently without any coordination mechanism. Rank 0 might see 449GB available and proceed to allocate 300GB. Rank 1 then sees only ~149GB remaining (after rank 0's allocation) but still attempts to allocate 300GB. The result is either an immediate OOM or, more likely, a slow death as the system runs out of memory and the kernel's OOM killer terminates processes.

The Reasoning Process: From Crash to Insight

The path to this understanding was not immediate. Looking at the preceding messages, we can trace the reasoning:

  1. Initial attempt (<msg id=3850>): The assistant launches with --hicache-size 300, assuming the parameter represents total host memory to be shared across ranks.
  2. First suspicion (<msg id=3854>): After the server hangs, the assistant notices memory usage at 439GB and writes: "That's a problem — each TP rank is trying to allocate 300GB. 8 × 300GB = 2.4TB, far more than the 449GB total RAM." This is the first articulation of the per-rank hypothesis, but it's still a guess.
  3. Cleanup and investigation (<msg id=3855-3862>): The assistant kills processes, resets GPUs (via the Proxmox host, since GPU reset is unavailable inside the container), and recovers to 445GB free RAM. Then it begins calculating: "Available: ~445GB. Model loading overhead: ~40GB. Safety margin: ~25GB. Remaining for hicache: ~380GB / 8 ranks = 47GB per rank."
  4. Code reading (<msg id=3862-3865>): Rather than relying on guesswork, the assistant reads the SGLang source code directly—hiradix_cache.py, host_memory_pool.py, memory_pool_host.py, and server_args.py. It traces how hicache_size flows through the initialization code.
  5. The breakthrough (<msg id=3866>): Having read the code, the assistant now knows the parameter is per-rank and can explain the exact mechanism: int(host_size * 1e9 // self.size_per_token). This progression from guess to hypothesis to confirmed understanding is a textbook example of systematic debugging. The assistant did not simply accept its initial hypothesis (that --hicache-size was total memory); it validated the hypothesis by reading the source code and then corrected its mental model.

The Decision: Sizing the Cache

With the per-rank semantics confirmed, the assistant calculates a safe value:

Safe per-rank size: 445GB available, ~40GB for model loading, ~10GB reserved = ~395GB / 8 = ~49GB per rank. Let me use --hicache-size 48.

The calculation is conservative: 445GB available, minus 40GB for model loading overhead (the model weights need to be loaded into host memory before being distributed to GPUs), minus 10GB safety reserve, divided by 8 ranks. Rounding down to 48GB per rank provides a comfortable margin.

But the assistant does not stop there. It also explores --hicache-ratio as an alternative:

Actually, let me also check --hicache-ratio as an alternative — it's based on the device pool size, so it auto-scales.

This is a sophisticated consideration. The --hicache-ratio parameter specifies the host cache size as a multiplier of the GPU device pool size. If the device pool holds, say, 17.4GB of KV cache per rank, a ratio of 2.5 would allocate 43.5GB of host cache per rank. The advantage of the ratio approach is that it automatically adjusts if the device pool size changes (e.g., if --mem-fraction-static is modified).

The assistant runs a Python calculation to compare the two approaches:

hicache-ratio 2.5: ~44 GB per rank, ~348 GB total
hicache-ratio 3.0: ~52 GB per rank, ~418 GB total
Total tokens at ratio 2.5: ~444K = ~111 concurrent reqs at 4K avg
Total tokens at ratio 3.0: ~508K = ~127 concurrent reqs at 4K avg

This calculation translates the abstract "GB of host cache" into a concrete operational metric: concurrent requests at average sequence length. Going from ~116K tokens (no host cache) to ~444K tokens (ratio 2.5) would increase concurrent capacity from ~28 to ~111 requests at 4K average length—a nearly 4× improvement.

Assumptions and Potential Pitfalls

Several assumptions underpin the assistant's analysis:

Assumption 1: The device pool size at mem_fraction_static=0.93 is ~127K tokens. This is an estimate based on extrapolating from the known value at 0.85 (116K tokens). The actual device pool size depends on the model's memory footprint, which can vary. If the model uses more memory than expected, the device pool shrinks, and the host ratio allocates less host cache.

Assumption 2: The bytes-per-token calculation (136,793 bytes) remains valid at the higher memory fraction. This value was computed at mem_fraction_static=0.85. At 0.93, the device pool has more tokens but the per-token size should remain constant (it's determined by the model architecture: kv_lora_rank, rope dimensions, number of layers, and tensor parallelism). This assumption is safe.

Assumption 3: The hierarchical cache actually works correctly on this hardware. The assistant had not yet successfully run the server with hierarchical cache enabled. The OOM crash from the initial attempt meant no functional test had been completed. The code path for hierarchical cache on SM120 GPUs (the Blackwell architecture) might have undiscovered bugs. Indeed, the segment summary later reveals that the winning config settled on mem_fraction_static=0.88 + bf16 KV + hicache=48GB, suggesting further tuning was needed.

Assumption 4: 10GB of safety margin is sufficient. With 449GB of RAM, the model loading overhead of ~40GB is an estimate. If the model requires more host memory during loading (e.g., for temporary buffers, intermediate allocations, or Python overhead), the safety margin might be insufficient.

Potential mistake: The assistant initially used --hicache-size 300 without first checking the code. This was the root cause of the OOM. The parameter documentation (from --help) did not clarify that it was per-rank, and the assistant assumed it was a total system-wide value. Reading the source code earlier would have saved multiple rounds of debugging. However, this is a forgivable error—the parameter's semantics are not obvious, and discovering them required code inspection.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. SGLang architecture knowledge: Understanding that SGLang uses tensor parallelism (TP), where a model is sharded across multiple GPUs, each running a separate process (rank). Each rank has its own memory pools.
  2. KV cache fundamentals: Knowledge that transformer inference caches key-value tensors from previous tokens to avoid recomputation, and that this cache is the primary consumer of GPU memory during long-sequence generation.
  3. Hierarchical cache concept: Understanding that SGLang's hierarchical cache spills KV data from GPU VRAM to host RAM when GPU memory is exhausted, using a radix-tree-based caching system.
  4. Memory sizing arithmetic: The ability to compute per-token KV cache size from model parameters (kv_lora_rank, rope dimensions, number of layers, tensor parallelism degree, and byte width).
  5. The broader pipeline context: This optimization was in service of generating synthetic training data for an EAGLE-3 speculative decoding drafter, where throughput directly impacts how quickly training data can be collected.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Documented parameter semantics: The definitive understanding that --hicache-size is per-rank in GB, not total system memory. This is not clearly documented in SGLang's help output.
  2. A sizing methodology: A repeatable process for calculating safe hierarchical cache sizes: measure available RAM, subtract model loading overhead and safety margin, divide by TP rank count.
  3. A validated alternative: The --hicache-ratio parameter as an auto-scaling alternative that adjusts to device pool size changes.
  4. Concrete numbers for this specific deployment: 48GB per rank (or ratio 2.5-3.0) yields ~444K-508K total KV tokens, supporting ~111-127 concurrent requests at 4K average length.
  5. A calibration of expected throughput improvement: From ~116K tokens (baseline) to ~444K+ tokens (with hierarchical cache), representing a ~3.8-4.4× increase in concurrent request capacity.

The Thinking Process: A Window into Debugging Methodology

What makes this message particularly instructive is the visible thinking process. The assistant does not simply present a conclusion; it walks through the reasoning:

Conclusion

Message <msg id=3866> is a turning point in a larger optimization effort. It transforms a catastrophic failure (OOM crash with 439GB consumed) into a calibrated understanding of SGLang's hierarchical cache configuration. The assistant learned that --hicache-size is a per-rank parameter in gigabytes, that TP ranks do not coordinate memory allocation, and that safe sizing requires dividing available host memory by the number of ranks after accounting for model loading overhead.

The broader lesson extends beyond this specific deployment: when configuring distributed inference systems, parameters that appear to be global may in fact be per-worker. Reading the source code—not just the help text—is essential for understanding true semantics. And when a configuration causes a catastrophic failure, the path forward is not to guess a smaller number but to understand why the failure occurred and compute the correct value from first principles.

This message also illustrates the value of exploring alternatives. By considering --hicache-ratio alongside --hicache-size, the assistant equipped itself with a more robust configuration option that automatically scales with device pool size—a hedge against future changes to memory fraction or model architecture.

In the end, the assistant settled on mem_fraction_static=0.88 + bf16 KV + hicache=48GB as the winning configuration, yielding 159K GPU tokens and ~930-1350 tok/s throughput—a 2-3× improvement over the initial 600 tok/s baseline. That success was built on the foundation laid in this single message of forensic code analysis and careful arithmetic.