The Moment of Proof: Validating FP8 KV Cache Optimization in a Large-Scale Inference Pipeline

Introduction

In the sprawling, multi-session effort to train an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model, few messages carry the weight of a single measurement that validates an entire optimization hypothesis. Message <msg id=3905> is precisely such a moment. After rounds of debugging, server crashes, memory tuning, and architectural pivots, the assistant issues a simple command — wait three minutes, then sample the server's throughput — and receives back numbers that confirm the most impactful optimization of the entire pipeline: FP8 KV cache quantization had tripled the effective inference capacity.

This article examines that message in detail: the reasoning that led to it, the decisions embedded in its timing and measurement strategy, the assumptions it validated and those it left unresolved, and the knowledge it produced for the broader project.

The Message in Full

The message consists of a single bash command executed over SSH on the remote inference server, preceded by a brief explanatory comment:

Now let me wait 3 minutes for things to warm up and measure: [bash] sleep 180 && ssh root@10.1.230.174 'echo "=== SERVER ==="; tail -5 /data/eagle3/synth_100k/logs/sglang_inference.log | grep throughput; echo; echo "=== INFERENCE ==="; tail -10 /data/eagle3/synth_100k/logs/inference_all.log; echo; echo "=== B2 COUNT ==="; wc -l /data/eagle3/synth_100k/prepared/B2_opencodeinstruct/raw_responses.jsonl 2>/dev/null'

The output is laconic but revelatory:

=== SERVER === [2026-02-24 12:23:41 TP0] Decode batch, #running-req: 150, #token: 229786, token usage: 0.61, cuda graph: True, gen throughput (token/s): 1400.64, #queue-req: 0 [2026-02-24 12:23:45 TP0] Decode batch, #running-req: 150, #token: 235786, token usage: 0.63, cuda graph: True, gen throughput (token/s): 1444.22, #queue-req: 0

=== INFERENCE ===

=== B2 COUNT === 1485 /data/eagle3/synth_100k/prepared/B2_opencodeinstruct/raw_responses.jsonl

On its surface, this is a routine status check. But to understand why this message matters, one must trace the chain of reasoning that led to it.

The Bottleneck That Defined a Day

The story begins roughly twenty messages earlier, at <msg id=3893>. The assistant had deployed SGLang with hierarchical cache (--enable-hierarchical-cache --hicache-size 48) and a memory fraction of 0.88, achieving max_total_num_tokens=159277 on GPU. This was already a significant improvement over the baseline 116K tokens. Yet when the inference pipeline ran, throughput settled at only ~600 tok/s with just 34-35 concurrent requests and a queue of 115+. Token usage was 0.92-0.94 — the KV cache was saturated.

This prompted a critical diagnostic insight. The assistant realized that hierarchical cache (hicache) only helps with evicted radix cache entries — it enables prefix reuse across requests whose prefixes have been pushed out of GPU memory. But for actively generating requests, the KV cache must reside on GPU. Hicache does not increase the number of concurrent decode slots. The true bottleneck was GPU KV cache capacity, pure and simple.

With 159K tokens of GPU KV cache and an average sequence length of ~4K tokens, the server could fit at most ~40 concurrent long-running requests. At ~17 tok/s per request (a typical decode rate for large models), that caps total throughput at roughly 680 tok/s — exactly what was observed.

The assistant enumerated three possible solutions: more GPUs (infeasible), shorter sequences (undesirable for training data quality), or KV cache quantization. The third option — --kv-cache-dtype fp8_e4m3 — promised to roughly double effective KV capacity by halving the memory footprint of each cached token.

The Leap of Faith

FP8 KV cache for MLA (Multi-head Latent Attention) models is not an obvious win. The Kimi-K2.5 model uses MLA, where the KV cache is already a compressed latent representation rather than raw key/value tensors. Quantizing an already-compressed representation risks quality degradation. The assistant acknowledged this implicitly by checking the available options (fp8_e5m2, fp8_e4m3, fp4_e2m1) and choosing fp8_e4m3 — the higher-precision 8-bit format — as a conservative bet.

The server restart at <msg id=3900> combined three levers simultaneously:

Higher memory fraction: --mem-fraction-static 0.90 (up from 0.88, but below the 0.93 that had OOM'd earlier)
FP8 KV cache: --kv-cache-dtype fp8_e4m3
Hierarchical cache retained: --enable-hierarchical-cache --hicache-size 48 This was a coordinated bet. The assistant could not know in advance whether FP8 quantization would work correctly with MLA, whether it would interact badly with hicache, or whether the 0.90 memory fraction would leave enough headroom for prefill allocations. The server took several minutes to initialize, capture CUDA graphs, and become healthy. Only then could the inference pipeline be restarted at <msg id=3904>.

Why Wait Three Minutes?

The three-minute wait in <msg id=3905> is not arbitrary. It reflects a deep understanding of the system's dynamics. When the inference pipeline starts, it launches 150 concurrent short-concurrency requests (the --short-concurrency 150 flag). These requests flood the server, which must prefill each one — a memory-intensive operation that populates the KV cache. The server then transitions to decode mode, generating tokens for all active requests simultaneously.

Three minutes is enough time for:

The initial prefill burst to complete
The server to enter steady-state decode
The KV cache to reach its equilibrium occupancy
The throughput measurement to stabilize A shorter wait might capture transient behavior (the initial burst of prefills looks very different from steady-state decode). A longer wait risks wasting time if something went wrong. Three minutes is the sweet spot — long enough for warm-up, short enough for rapid iteration.

The Results: Validation at Scale

The output tells a remarkable story. The two server log lines show:

150 running requests: The concurrency limit has been hit. Every decode slot is filled. Previously, only 34-35 requests could run simultaneously.
229,786-235,786 tokens in cache: Token usage at 0.61-0.63, meaning the KV cache is only 61-63% utilized. There is headroom — more requests could be added.
1,400-1,444 tok/s generation throughput: More than double the ~600 tok/s from the bf16 configuration. The throughput is now limited by the server's compute capacity, not by KV cache starvation.
Queue at 0: All 150 requests are being served immediately. No backlog. The pipeline is keeping up.
B2 count at 1,485: In the roughly 3 minutes since the pipeline started, approximately 77 completions were generated (the count was 1,408 at <msg id=3891>). At ~0.43 requests per second, this aligns with the measured throughput: 1,400 tok/s ÷ 4,100 avg tokens per completion ≈ 0.34 completions per second, with some overhead for prefill and scheduling. The FP8 KV cache had increased max_total_num_tokens from 159,277 to 376,029 — a 2.36× improvement. Combined with the higher memory fraction, the effective decode capacity had expanded from ~40 concurrent requests to ~150. The bottleneck had shifted from memory capacity to something else — likely compute or memory bandwidth.

The Unresolved Thread: Empty Inference Log

One detail stands out: the inference log (inference_all.log) is empty. This is a recurring issue throughout the session. The run_inference.py script appears to buffer its output or write to stderr rather than stdout. The assistant does not address this in the message — it simply notes the B2 count as the ground truth for progress. This is a pragmatic decision: the raw response count is the real metric, and the log is a secondary concern. But it hints at a deeper issue with the inference pipeline's observability that would need to be addressed separately.

Assumptions Embedded in the Measurement

The message makes several assumptions worth examining:

Three minutes is sufficient for warm-up: This assumes the server has finished CUDA graph capture and the hicache buffers are populated. If hicache requires more time to fill its 48GB per rank, the throughput might improve further. But the token usage of 0.61 suggests the GPU cache is not yet saturated, so hicache spillover is minimal.
The throughput is representative: The measurement captures two consecutive decode batches at 2-second intervals. This is a snapshot, not a long-term average. However, the consistency between the two samples (1,400 and 1,444 tok/s) suggests stability.
FP8 quantization does not degrade quality: The assistant does not verify that the generated responses are still valid training data. This is an implicit assumption — that FP8 KV cache for MLA preserves enough fidelity for EAGLE-3 training. The user would later reject this assumption (as noted in the chunk summary), preferring bf16 KV cache for quality reasons.
The B2 count reflects pipeline health: With only 1,485 of 14,714 prompts completed, the pipeline is still in its early stages. The throughput could degrade as the KV cache fills and hicache eviction begins.

Knowledge Created

This message produces several pieces of actionable knowledge:

FP8 KV cache works with MLA on SM120: This is a non-trivial finding. The combination of --kv-cache-dtype fp8_e4m3 with the Kimi-K2.5 model on Blackwell GPUs is validated at scale.
The throughput ceiling is now ~1,400 tok/s: With 150 concurrent requests and 0.61 token usage, the bottleneck is no longer KV cache capacity. Further optimization would need to target compute or memory bandwidth.
The inference pipeline is producing data at ~0.43 req/s: At this rate, completing the remaining ~13,200 B2 prompts would take approximately 8.5 hours. This is acceptable but not fast — the dataset size capping introduced later (--max-tokens-per-dataset 10000000) would become necessary.
The server can sustain 150 concurrent decodes without queue buildup: The queue of 0 indicates that the prefill pipeline is keeping pace with decode, and the scheduler is not overwhelmed.

Conclusion

Message <msg id=3905> is a quiet triumph in a session defined by struggle. It represents the payoff of a carefully reasoned optimization chain: diagnose the bottleneck (KV cache capacity), identify the solution (FP8 quantization), deploy it with appropriate safety margins (0.90 memory fraction, conservative FP8 format), and measure the outcome at the right moment (after warm-up, in steady state). The numbers speak for themselves — 1,400 tok/s, 150 concurrent requests, zero queue — and they validate the entire approach.

But the message also contains the seeds of future work. The empty inference log, the unverified quality of FP8-quantized outputs, and the long tail of remaining prompts would all demand attention in subsequent messages. For this moment, however, the assistant has earned the right to a simple satisfied observation: the optimization worked.