Pulling All Levers: The Moment SGLang's Hierarchical Cache Was Unleashed

In the course of a marathon optimization session for deploying the Kimi-K2.5 model on an 8-GPU RTX PRO 6000 Blackwell system, message [msg 3850] represents a critical inflection point — the moment when the assistant, after a long chain of diagnostics, decided to stop tuning individual parameters and instead throw every available optimization at the server simultaneously. The message is deceptively simple: a single nohup command that launches SGLang with a battery of flags. But behind that command lies hours of careful bottleneck analysis, a deep understanding of the SGLang serving stack, and a calculated gamble on a relatively new feature called the hierarchical KV cache.

The Context That Produced This Message

To understand why this message was written, one must trace the reasoning chain that led to it. The session had been wrestling with a throughput problem for some time. The user had deployed a large-scale inference pipeline to generate synthetic training data for the EAGLE-3 speculative decoding system, and the pipeline was running far slower than expected. The user's initial complaint was blunt: "Look at sglang metrics, currently only doing 500tps, what seems to be the bottleneck? Should be 4-5x faster" ([msg 3831]).

The assistant's investigation revealed a more nuanced picture. The server-side generation throughput was actually 780-890 tokens per second — not 500 — but the effective throughput was being throttled by a different constraint. The server logs showed a consistent pattern: only 46-56 requests were running concurrently despite a concurrency setting of 150, while 94-101 requests sat in the queue. The KV cache utilization was hovering at 85-96%, meaning the scheduler could not admit new requests because there was simply no room for their key-value tensors ([msg 3833]).

This was the core bottleneck. With --mem-fraction-static 0.85, the server could hold approximately 116,171 tokens total in GPU KV cache. With each request averaging 2,000-4,000 tokens, only about 28-58 concurrent requests could fit. The remaining 100+ requests were stuck in the queue, waiting for existing requests to complete and free up KV cache slots.

The Diagnostic Journey

The assistant explored multiple avenues before arriving at the command in [msg 3850]. First, it considered simply increasing --mem-fraction-static to 0.90 or 0.93, but calculations showed this would only yield a 6-12% improvement in token capacity — far short of the 4-5x the user wanted ([msg 3836]). The fundamental issue was that GPU VRAM was the scarce resource, and the model weights consumed the majority of it.

The assistant then discovered SGLang's hierarchical cache feature (--enable-hierarchical-cache), which spills KV cache entries to host RAM when GPU memory is full. With 408GB of available host RAM, this could theoretically increase KV cache capacity by 3-4x. The calculations in [msg 3842] showed that 300GB of hierarchical cache could add ~294,000 additional tokens, bringing total capacity to ~410,000 tokens — enough for ~103 concurrent requests at 4K average length.

The user's directive "Use all levers" ([msg 3848]) was the final catalyst. Rather than incrementally testing each optimization, the assistant decided to combine everything: increase GPU memory fraction, enable hierarchical cache with generous host memory allocation, and tune NCCL communication parameters for maximum throughput.

The Command: A Technical Deconstruction

The message launches a single, carefully constructed bash command via SSH:

nohup bash -c "source ~/ml-env/bin/activate && \
NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 \
python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --trust-remote-code \
  --tp-size 8 \
  --mem-fraction-static 0.93 \
  --host 0.0.0.0 --port 8000 \
  --num-continuous-decode-steps 4 \
  --disable-custom-all-reduce \
  --enable-hierarchical-cache \
  --hicache-size 300 \
  --hicache-write-policy write_through \
  --hicache-io-backend kernel \
  --log-level info"

Each flag represents a deliberate decision:

--mem-fraction-static 0.93: This allocates 93% of available GPU memory (after model weights) to the KV cache. At 0.85, the server had ~14.8GB per GPU for KV. At 0.93, this would increase to approximately 127,000 tokens — a 9.4% improvement. The assistant had previously calculated this would yield only marginal gains ([msg 3836]), but combined with hierarchical cache, every bit of GPU-side KV capacity helps reduce host RAM transfers.

--enable-hierarchical-cache --hicache-size 300: This is the primary lever. The hierarchical cache uses host RAM as a secondary tier for KV cache entries. When GPU memory is full, the least recently used KV pages are evicted to host RAM. On subsequent access, they are fetched back to GPU. The 300GB allocation was based on the assistant's calculation that 408GB of host RAM was available ([msg 3843]), leaving a safety margin for the operating system and other processes.

--hicache-write-policy write_through: This policy writes KV cache entries to both GPU and host RAM simultaneously. The alternative, write_back, would write only to GPU and evict to host only when GPU memory is full. write_through ensures that evicted pages are immediately available in host RAM, avoiding a write-back penalty on eviction. This is a reasonable choice when host RAM bandwidth is not the bottleneck.

--hicache-io-backend kernel: This selects the kernel-based I/O backend for host RAM transfers. The alternative direct backend uses DMA, which may have higher latency but lower CPU overhead. The kernel backend is typically more mature and stable, making it the safer choice for a production deployment.

--num-continuous-decode-steps 4: This controls how many decode steps the server performs before checking for new prefill requests. Higher values improve throughput by reducing scheduler overhead but increase latency for queued requests. The assistant had identified that the server was spending too much time switching between prefill and decode ([msg 3833]), and this setting helps batch decode operations more efficiently.

NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512: These NCCL environment variables configure the inter-GPU communication backend. LL (Low Latency) protocol with Ring algorithm is standard for NVLink-connected GPUs. NCCL_P2P_LEVEL=SYS enables peer-to-peer transfers across the system fabric. The channel count, buffer size, and thread count are tuned for the specific GPU topology.

--disable-custom-all-reduce: This disables SGLang's custom all-reduce kernel, falling back to NCCL's implementation. This is a notable choice — custom all-reduce is typically faster on homogeneous GPU setups. The assistant may have disabled it due to compatibility issues with the hierarchical cache or the specific GPU architecture (SM120 Blackwell).

Assumptions and Potential Mistakes

The message embodies several assumptions that deserve scrutiny:

Assumption 1: Hierarchical cache will not become a bottleneck. The assistant assumes that spilling KV cache to host RAM and fetching it back on demand will be fast enough to not degrade throughput. This is not guaranteed. If the PCIe bandwidth between GPU and host is saturated, or if the kernel I/O backend introduces high latency, the hierarchical cache could actually reduce throughput by adding transfer overhead to every decode step. The assistant's calculation of 136,793 bytes per token per GPU ([msg 3842]) suggests each token requires ~137KB of KV cache data. At 4K tokens per request, that's ~560MB per request. Moving this data across PCIe for each cache miss could be expensive.

Assumption 2: 300GB is a safe allocation. The assistant checked free -g and saw 408GB available ([msg 3843]), then allocated 300GB for hicache. This leaves 108GB for the OS, other processes, and the model weights loaded in host RAM during initialization. If the model's host-side memory footprint is larger than expected, or if other processes consume memory, the system could OOM. The assistant's earlier calculation of model weights (~68.4GB per GPU, ~547GB total) suggests the model is loaded in GPU VRAM, but SGLang also maintains a host-side copy for initialization and potential CPU offloading.

Assumption 3: mem_fraction_static=0.93 will not OOM. The assistant had previously calculated that 0.93 would yield ~127,000 tokens ([msg 3836]), but this was based on a linear extrapolation from the 0.85 baseline. In practice, the relationship between memory fraction and token capacity depends on fragmentation, alignment, and other memory management factors. The chunk summary reveals that this config actually did OOM ([chunk 28.0]), forcing a retreat to 0.88.

Assumption 4: All NCCL optimizations are compatible. The combination of NCCL_PROTO=LL, NCCL_ALGO=Ring, and NCCL_P2P_LEVEL=SYS with --disable-custom-all-reduce may not be optimal for 8-way tensor parallelism on Blackwell GPUs. The LL protocol and Ring algorithm are well-tested on NVLink-connected GPUs, but the Blackwell RTX PRO 6000's interconnect topology may differ from datacenter GPUs like H100 or A100.

Assumption 5: The user wants maximum throughput at any cost. The user said "Use all levers" ([msg 3848]), which the assistant interpreted as "maximize throughput regardless of other concerns." However, the user had previously rejected --kv-cache-dtype fp8_e4m3 due to quality concerns ([chunk 28.0]), suggesting there are constraints. The assistant's decision to push mem_fraction_static to 0.93 — which later OOM'd — may have been overly aggressive.

Input Knowledge Required

To fully understand this message, one needs:

  1. SGLang architecture knowledge: Understanding what mem_fraction_static, num_continuous_decode_steps, hierarchical cache, and the various NCCL parameters do requires familiarity with SGLang's serving model — how it manages KV cache, schedules requests, and coordinates across GPUs.
  2. NCCL and GPU communication: The NCCL environment variables control how GPUs communicate during tensor-parallel inference. NCCL_PROTO=LL selects the low-latency protocol, NCCL_ALGO=Ring selects the ring all-reduce algorithm, and NCCL_P2P_LEVEL=SYS enables system-level peer-to-peer transfers.
  3. KV cache mechanics: Understanding that KV cache is the primary memory consumer in LLM serving, that it scales linearly with sequence length and batch size, and that it determines how many concurrent requests can be served.
  4. Hardware topology: The 8-GPU RTX PRO 6000 Blackwell system with its NVLink interconnect, PCIe topology, and host RAM capacity are all relevant to understanding why certain parameters were chosen.
  5. The Kimi-K2.5 model architecture: The model uses Multi-head Latent Attention (MLA), which has a compressed KV cache representation (kv_lora_rank=512). This makes the hierarchical cache more feasible because each token's KV data is smaller than in standard multi-head attention.

Output Knowledge Created

This message produces:

  1. A running SGLang server with hierarchical cache enabled, serving the Kimi-K2.5 model on 8 GPUs. This server will be used to generate synthetic training data for the EAGLE-3 speculative decoding system.
  2. A benchmarkable configuration that can be evaluated against the previous baseline. The assistant will later measure throughput and compare it to the 780-890 tok/s baseline.
  3. A potential failure mode: If the configuration OOMs or performs poorly, the assistant learns which combination of parameters caused the failure. The chunk summary reveals that mem_fraction_static=0.93 did OOM, and the final working config was mem_fraction_static=0.88 + bf16 KV + hicache=48GB — a significantly more conservative setup.
  4. Diagnostic data: The server logs will reveal whether hierarchical cache improves or degrades throughput, how often cache misses occur, and what the effective token capacity becomes.

The Thinking Process Visible in This Message

The message's reasoning is compressed into a single line: "Good, all clean. Now let me launch with everything maxed out." But the thinking process is visible in the structure of the command itself.

The assistant is thinking in terms of system bottlenecks. It has identified the KV cache as the primary constraint, and it is attacking that constraint from multiple angles simultaneously: increasing GPU-side capacity (mem_fraction_static), adding host-side capacity (hicache), and optimizing inter-GPU communication (NCCL) to ensure that the larger batch sizes enabled by more KV cache actually translate to higher throughput.

The choice to combine all optimizations in a single launch, rather than testing them incrementally, reflects a risk/reward calculation. The user has expressed urgency ("Should be 4-5x faster") and has authorized aggressive optimization ("Use all levers"). The assistant judges that the cost of a failed launch (restarting the server) is low compared to the time saved by not iterating through individual parameter combinations.

The NCCL environment variables reveal a secondary concern: inter-GPU communication overhead. With 8 GPUs running tensor parallelism, every decode step requires all-reduce operations across all GPUs. If the hierarchical cache introduces additional GPU-to-CPU transfers, the NCCL configuration must be efficient enough to not become the new bottleneck.

The --disable-custom-all-reduce flag is particularly interesting. SGLang's custom all-reduce is typically faster than NCCL's because it uses a specialized implementation optimized for the SGLang runtime. By disabling it, the assistant is implicitly trusting NCCL's implementation more. This may be because the custom all-reduce has compatibility issues with hierarchical cache, or because the assistant has prior experience with NCCL being more stable on this hardware.

The Aftermath

The chunk summary tells us what happened next: the 0.93 configuration OOM'd, and the assistant retreated to mem_fraction_static=0.88 with hicache-size 48 (not 300). The final working configuration was significantly more conservative than the "everything maxed out" approach in this message. This is a classic pattern in systems optimization: the aggressive first attempt fails, and the optimal configuration is found through iterative refinement.

But the message is not a failure — it's a necessary experiment. Without testing the boundaries, the assistant would not know where they lie. The OOM at 0.93 provides concrete data: the model's memory overhead at startup exceeds the 7% headroom left by 0.93. The failure of the 300GB hicache allocation (the chunk summary mentions "48GB/rank host RAM overflow") reveals that the hierarchical cache's memory management has per-rank limits that must be respected.

In the end, the system achieved 930-1350 tok/s — roughly 2-3x improvement over the initial baseline. The hierarchical cache, even at the reduced 48GB allocation, was the key enabler. Message [msg 3850] represents the moment the assistant committed to that strategy, even if the exact parameters needed adjustment.