The Final Lever: Combining FP8 KV Cache, Hierarchical Cache, and Aggressive Memory Fraction in SGLang

Introduction

In the middle of a high-stakes inference pipeline optimization session for the Kimi-K2.5 model running on 8× RTX PRO 6000 Blackwell GPUs, message <msg id=3900> represents a pivotal moment: the launch of an SGLang server with every available optimization lever pulled simultaneously. After a long chain of trial-and-error debugging—spanning OOM crashes, throughput collapses, zombie GPU processes, and a fundamental realization about the limits of hierarchical caching—the assistant issues a single, decisive bash command. This message is not just a server restart; it is the culmination of a deep diagnostic journey and a calculated bet on the combination of three techniques: aggressive GPU memory allocation (--mem-fraction-static 0.90), KV cache quantization (--kv-cache-dtype fp8_e4m3), and host-memory hierarchical cache overflow (--enable-hierarchical-cache with --hicache-size 48).

The Message

The assistant writes:

Clean. Now launch with all levers — FP8 KV cache + higher mem fraction + hicache:

``bash ssh root@10.1.230.174 'nohup bash -c "source ~/ml-env/bin/activate && \ NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 \ python3 -m sglang.launch_server \ --model-path /shared/kimi-k2.5-int4 \ --trust-remote-code \ --tp-size 8 \ --mem-fraction-static 0.90 \ --kv-cache-dtype fp8_e4m3 \ --host 0.0.0.0 --port 8000 \ --num-continuous-decode-steps 4 \ --disable-custom-all-reduce \ --enable-hierarchical-cache \ --hicache-size 48 \ --hicache-write-policy write_through \ --hicache-io-backend kernel \ --log-level info" > /data/eagle3/synth_100k/logs/sglang_inference.log 2>&1 &' echo "Server starting: mem=0.90 + fp8_kv + hicache=48GB/rank..." ``

The message is terse—just a command and a confirmation echo—but it carries enormous weight. Every flag in this invocation was the result of a painful debugging process that unfolded over the preceding dozen messages.

Why This Message Was Written: The Bottleneck Diagnosis

To understand why this message exists, one must trace the reasoning chain that led to it. The assistant had been tuning an SGLang inference server for the Kimi-K2.5 model, a massive Mixture-of-Experts architecture with Multi-head Latent Attention (MLA). The goal was to generate synthetic training data for EAGLE-3 speculative decoding, requiring high-throughput inference across 88,000 prompts.

The journey began with a baseline configuration using --mem-fraction-static 0.85, which yielded approximately 600 tok/s throughput. The assistant then attempted to increase capacity by raising the memory fraction to 0.93 (see <msg id=3872>), which boosted max_total_num_tokens from 116K to 231K—nearly doubling the GPU KV cache capacity. However, this configuration OOM'd during prefill of a large batch, leaving only 0.82 GB of free GPU memory after CUDA graph capture ([msg 3879]). The server crashed, and the assistant had to kill zombie processes and clean up.

The next attempt used --mem-fraction-static 0.88 combined with --enable-hierarchical-cache and --hicache-size 48 ([msg 3881]). This worked: the server achieved 989–1317 tok/s initially, with 159K GPU tokens and 48 GB of host memory overflow per rank. But within minutes, throughput collapsed to 600 tok/s as the KV cache saturated at 0.92–0.94 token usage ([msg 3893]). The assistant realized a critical insight:

"The issue is that hicache helps with evicted radix cache entries (prefix reuse), but for actively generating requests the KV must be on GPU. So hicache doesn't actually increase the number of concurrent decode requests — it only helps with prefix caching."

This was the key diagnostic breakthrough. Hierarchical cache (hicache) spills evicted KV cache entries to host memory, which is useful for prefix caching and request resumption, but it does not increase the number of concurrent decoding slots. Active requests hold their KV cache on the GPU, and once the GPU KV cache is full, new requests must wait in the queue. With 159K GPU tokens and ~4,000 tokens per request, the server could only sustain ~40 concurrent decodes. At ~15 tok/s per request, that yields ~600 tok/s—exactly what was observed.

The assistant then identified the real solution: KV cache quantization. For MLA models, the KV cache is already a compressed latent representation, but further quantizing from BF16 to FP8 (fp8_e4m3) would roughly double the effective capacity to ~318K tokens, enabling ~80 concurrent requests and pushing throughput toward 1,200 tok/s or higher ([msg 3893]).

How Decisions Were Made

The decision in <msg id=3900> was the product of a deliberate three-variable optimization problem:

Variable 1: Memory fraction. The assistant had empirical data for three values:

0.85 → safe but low capacity (116K tokens, ~600 tok/s baseline)
0.88 → moderate capacity (159K tokens, worked with hicache but saturated)
0.93 → OOM'd during prefill The choice of 0.90 was a calculated compromise. It was higher than 0.88 (more KV capacity) but lower than 0.93 (leaving headroom for transient prefill allocations). The assistant's earlier analysis showed that 0.88 left ~5.5 GB free per GPU after CUDA graph capture ([msg 3884]). With 0.90, the free margin would shrink but hopefully remain above the OOM threshold. Variable 2: KV cache dtype. The assistant queried SGLang's help to discover available KV cache dtypes ([msg 3893]): auto, fp8_e5m2, fp8_e4m3, bf16, bfloat16, and fp4_e2m1. The choice of fp8_e4m3 (E4M3 format, 4 exponent bits and 3 mantissa bits) was the standard FP8 quantization for deep learning—it offers better numerical fidelity than E5M2 for activations and is widely supported on NVIDIA Blackwell GPUs. FP4 was available but likely deemed too lossy for quality-sensitive training data generation. Variable 3: Hierarchical cache. The assistant kept hicache enabled with 48 GB per rank despite its limitations for concurrent decode. This was a conservative choice: even though hicache doesn't directly increase decode slots, it provides value for prefix caching, request resumption after eviction, and handling bursty traffic patterns. The write_through policy and kernel I/O backend were retained from the previous working configuration. The NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.) were carried forward from earlier tuning ([msg 3881]) and represent a separate optimization axis for inter-GPU communication. These settings use Low Latency protocol with Ring algorithm and system-level P2P, with tuned buffer sizes and thread counts.

Assumptions and Risks

The message makes several assumptions, some explicit and some implicit:

Assumption 1: FP8 quantization does not degrade quality for training data generation. This is the most significant assumption. The assistant is generating synthetic data that will be used to train an EAGLE-3 drafter model. If FP8 KV cache introduces quantization noise that biases the generated tokens, the downstream training could be compromised. The assistant implicitly trusts that fp8_e4m3 is safe for this use case, likely based on the model's MLA architecture which already compresses the KV cache into a latent space—further quantization of the latent representation may have minimal impact.

Assumption 2: mem_fraction_static=0.90 will not OOM. The assistant is extrapolating from two data points: 0.88 worked (5.5 GB free) and 0.93 failed (0.82 GB free, then OOM). The threshold for OOM is somewhere between these values, and 0.90 is an educated guess. The assistant does not know the exact memory consumption of the largest prefill batch that the inference pipeline will generate.

Assumption 3: The NCCL environment variables are still optimal. These settings were tuned for a different memory configuration and may not be optimal with FP8 KV cache (which changes the memory access patterns). The assistant reuses them without re-validation.

Assumption 4: The server will start successfully on the first attempt. The preceding messages show multiple failed starts due to zombie processes holding GPU memory (<msg id=3895-3899>). The assistant cleaned up all GPUs (confirmed by nvidia-smi showing 0 MiB on all devices) and assumes the cleanup is complete.

Potential mistake: Overlooking the interaction between FP8 KV cache and hicache. The assistant is combining two memory optimization techniques that operate at different levels. FP8 quantization reduces the size of GPU KV cache entries, while hicache manages eviction to host memory. If FP8 doubles GPU capacity, the hicache eviction policy may behave differently—fewer evictions, different eviction patterns. The assistant does not analyze this interaction.

Input Knowledge Required

To understand this message, one needs knowledge of:

SGLang server architecture: The role of --mem-fraction-static in allocating GPU memory between model weights and KV cache, and how max_total_num_tokens is derived from available memory.
KV cache mechanics in transformer inference: How the KV cache limits concurrent decode requests, and why prefix caching (radix cache) differs from active decode capacity.
Hierarchical cache in SGLang: How --enable-hierarchical-cache works with --hicache-size to spill evicted cache entries to host memory, and its limitations for active requests.
FP8 quantization formats: The difference between E4M3 and E5M2, and why E4M3 is preferred for KV cache quantization.
NCCL tuning: The meaning of NCCL_PROTO, NCCL_ALGO, NCCL_P2P_LEVEL, and related environment variables for multi-GPU communication.
The Kimi-K2.5 model architecture: That it uses MLA (Multi-head Latent Attention) which compresses the KV cache into a latent space, making FP8 quantization potentially safer than for standard attention.
The broader project context: That this inference server is generating synthetic training data for EAGLE-3 speculative decoding, explaining why throughput matters and why quality degradation from quantization is a concern.

Output Knowledge Created

This message produces several forms of knowledge:

A new server configuration that combines three optimization techniques. The assistant will soon learn whether this configuration works (server starts successfully), whether it OOMs, and what throughput it achieves.
An empirical data point for the interaction between FP8 KV cache and hierarchical cache on MLA models with 8× Blackwell GPUs. This is a relatively rare hardware/software combination, and the results would be valuable for others deploying similar setups.
A validated hypothesis about the root cause of throughput saturation. If FP8 KV cache doubles concurrent decode capacity and throughput rises to ~1,200 tok/s, it confirms that GPU KV cache capacity was the primary bottleneck. If throughput does not improve significantly, it suggests other bottlenecks (e.g., CPU-side scheduling, network bandwidth, or the NCCL configuration).

The Thinking Process Visible in Reasoning

The assistant's reasoning is visible across the context messages leading to <msg id=3900>. The chain of thought follows a clear diagnostic pattern:

Observation: Throughput drops to ~600 tok/s with hicache enabled.
Measurement: Token usage is 0.92–0.94, running requests are ~35, queue is ~115.
Hypothesis: The GPU KV cache is saturated, limiting concurrent decodes.
Analysis: Hicache does not help with active decode capacity—it only helps with evicted entries for prefix reuse.
Calculation: 159K tokens / 4K avg per request ≈ 40 concurrent requests. 40 × 15 tok/s ≈ 600 tok/s. The math checks out.
Solution identification: KV cache quantization (FP8) would double capacity.
Tool exploration: Query SGLang's help to discover available KV cache dtypes.
Decision: Combine FP8 KV cache with a slightly higher memory fraction (0.90) and keep hicache for its secondary benefits.
Execution: Clean up zombie processes, verify GPU memory is free, launch the combined configuration. This is textbook systems debugging: observe → measure → hypothesize → calculate → verify → identify root cause → apply targeted fix. The assistant resists the temptation to throw more GPUs at the problem or to accept the 600 tok/s ceiling, instead drilling down to the fundamental memory constraint.

Conclusion

Message <msg id=3900> is a deceptively simple command that encapsulates an entire debugging arc. It represents the moment when a system engineer, after methodically eliminating alternative explanations, converges on the root cause and applies a targeted fix. The combination of FP8 KV cache, aggressive memory fraction, and hierarchical cache is not random—it is the product of empirical observation, quantitative analysis, and careful risk assessment. Whether the server starts successfully and delivers the hoped-for throughput is uncertain, but the reasoning that produced this message is a model of disciplined systems optimization.

The message also illustrates a broader lesson about inference optimization: that "obvious" solutions (like hierarchical cache for memory pressure) can address the wrong bottleneck, and that real progress requires understanding the fundamental constraints of the hardware-software stack. In this case, the constraint was not total memory capacity but the GPU KV cache's role as an active working set that cannot be swapped out during decode. The FP8 quantization lever was the correct tool because it directly addressed this constraint at the right level of abstraction.