The Moment of Reckoning: Benchmarking a Custom EAGLE-3 Drafter That Wasn't Ready

In the long and winding journey to accelerate inference on the Kimi-K2.5 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 3216] marks a quiet but devastating inflection point. After hours of patching model files, resolving API incompatibilities, building training pipelines, and running 10,000 samples through hidden state extraction, the assistant finally launches a benchmark of its custom-trained EAGLE-3 speculative decoding drafter. The results are delivered in three terse lines:

Prompt: Explain the theory of general relativity in detail...
  Tokens: 256 completion, 18 prompt
  Time: 6.26s, TPOT: 24.5ms, TPS: 40.9 tok/s

Prompt: Write a Python function to implement merge sort wi...
  Tokens: 256 completion, 20 prompt
  Time: 6.27s, TPOT: 24.5ms, TPS: 40.8 tok/s

Prompt: What are the main causes and effects of climate ch...
  Tokens: 256 completion, 19 prompt
  Time: 6.28s, TPOT: 24.5ms, TPS: 40.8 tok/s

At 40.8 tokens per second, this is barely two-thirds of the base SGLang performance of 63.6 tok/s, and less than half of what vLLM achieved at 82.5 tok/s. The message itself is sparse — just a benchmark invocation and its output — but its implications cascade through the subsequent conversation. It is the moment when a carefully constructed hypothesis meets the unforgiving reality of empirical measurement.

The Context: A Long Road to This Benchmark

To understand why this message was written, one must trace the arc of the preceding segment. The assistant had been engaged in a multi-phase optimization campaign for the Kimi-K2.5 model, a 547GB INT4-quantized MoE architecture running on eight GPUs. After profiling revealed that AllReduce communication was the dominant bottleneck at 51.5% of decode time ([msg 3193] area), the assistant pivoted to speculative decoding as a software-only optimization path that could bypass the communication overhead entirely.

The EAGLE-3 approach was chosen: a lightweight draft model that predicts multiple future tokens in parallel, which the target model then verifies. If the drafter's predictions are accepted at a high rate (typically 60-80%), the effective throughput increases because multiple tokens are generated per forward pass of the large model.

The assistant had already tested the AQ-MedAI EAGLE-3 drafter ([msg 3207] through [msg 3210]), which achieved a ~42% acceptance rate — better than vLLM's broken 15% but far below the 70%+ needed for a speedup. The single-stream throughput was 62.9 tok/s, essentially identical to the base SGLang rate of 63.6 tok/s. At high concurrency, speculative decoding was actually worse: 849 tok/s vs 2,370 tok/s for the base server, because SGLang automatically limited max_running_requests to 48 in speculative mode versus 2048 for base.

The assistant's reasoning at that point was clear: the AQ-MedAI drafter was trained for Kimi-K2, not Kimi-K2.5. The architecture differences between these model versions explained the mediocre acceptance rate. The obvious next step was to test the custom drafter that had been painstakingly trained on 10,000 samples of actual Kimi-K2.5 inference data ([msg 3210]). If the training data matched the target model, surely the acceptance rate would improve.

The Benchmark Execution

Message [msg 3216] begins with the assistant confirming the server is ready: "Server is ready with our custom drafter. Let me benchmark." This line reveals the expectation — the assistant believes the custom drafter is loaded and operational, and the benchmark is a routine verification step.

The benchmark script (/tmp/sglang_bench.py) is the same one used throughout this session. It runs three single-stream prompts, each generating 256 completion tokens. The prompts are standard: explain general relativity, implement merge sort, describe climate change causes and effects. The script measures time-to-first-token (TTFT) and tokens-per-second (TPS), reporting both.

The results are consistent across all three prompts: approximately 6.26-6.28 seconds for 256 tokens, yielding 40.8-40.9 tok/s. The consistency itself is telling — it's not a fluke or a transient slowdown. The drafter is systematically slower than both the base server and the AQ-MedAI drafter.

What the Message Doesn't Say (But Subsequent Messages Reveal)

The message cuts off at "=== Mul..." — the multi-stream benchmark results are not shown in this message. But the subsequent messages ([msg 3217], [msg 3218]) fill in the critical missing data. The assistant checks the acceptance logs:

accept len: 1.00, accept rate: 0.25

An acceptance length of 1.00 means the drafter is accepting exactly zero draft tokens per step — the "1.00" represents only the original token that was always going to be generated. The acceptance rate of 25% (1/4) is the worst possible: it means the drafter proposes four tokens and all four are rejected every time. As the assistant later puts it ([msg 3218]): "this is essentially random — our drafter is completely broken."

This is the devastating truth behind the 40.8 tok/s number. The custom drafter, trained on 10,000 samples of Kimi-K2.5 data, is performing worse than the generic AQ-MedAI drafter trained for a different model. The speculative decoding overhead is pure cost with zero benefit.

The Root Cause: Hidden State Misalignment

The assistant's analysis in [msg 3218] identifies the likely root cause: "the hidden state extraction pipeline between the quantized INT4 model and the EAGLE-3 drafter has alignment issues — the drafter was trained on FP16/BF16 hidden states but the serving model is INT4 quantized, and the hidden states differ."

This is a subtle but critical insight. The EAGLE-3 training pipeline extracts hidden states from the target model during inference, then trains the drafter to predict future hidden states. But if the training extraction used FP16/BF16 model weights while the serving inference uses INT4 quantized weights, the hidden state distributions differ. The drafter learned to predict patterns from one distribution but is being asked to predict patterns from another. The result is essentially random predictions.

This assumption — that hidden states from a quantized model are interchangeable with those from the full-precision model — was the critical error. The assistant had built an elaborate training pipeline ([msg 3213] through [msg 3215]), patched model files, resolved API incompatibilities, and run thousands of samples through extraction, all without verifying that the hidden states used for training matched those produced during inference.

The Broader Implications

Message [msg 3216] is the empirical verdict on a multi-hour engineering effort. The complete comparison table that emerges in [msg 3218] tells the story:

| Config | Single-stream | Peak throughput | Accept rate | |--------|--------------|-----------------|-------------| | vLLM baseline | 82.5 tok/s | 1,536 tok/s | N/A | | SGLang base (CG) | 63.6 tok/s | 2,370 tok/s | N/A | | SGLang + AQ-MedAI EAGLE-3 | 62.9 tok/s | 849 tok/s | ~42% | | SGLang + Custom K2.5 EAGLE-3 | 40.8 tok/s | 597 tok/s | 25% (broken) |

The conclusion is stark: EAGLE-3 speculative decoding provides no benefit on this hardware configuration. The base SGLang server with CUDA graphs achieves the best throughput (2,370 tok/s at C=128), and the single-stream leader remains vLLM at 82.5 tok/s.

The Thinking Process Visible in the Message

Though the message itself is brief — just a benchmark command and its output — the thinking process is visible in what is not said. The assistant does not immediately declare the custom drafter broken. It runs the benchmark, observes the numbers, and moves on to check the acceptance logs in the next message. This is methodical empirical science: first collect the data, then interpret it.

The assistant's decision to benchmark single-stream first (rather than jumping to high-concurrency tests) is also telling. Single-stream performance isolates the drafter's effectiveness from batch-size effects. If the drafter can't accelerate a single request, it certainly won't help under load.

Input and Output Knowledge

To understand this message, the reader needs to know: the EAGLE-3 speculative decoding architecture, the difference between single-stream and throughput benchmarks, the concept of acceptance rate, the context of previous AQ-MedAI drafter testing, and the fact that the custom drafter was trained on 10,000 samples of Kimi-K2.5 inference data.

The message creates new knowledge: the custom K2.5-trained EAGLE-3 drafter achieves only 40.8 tok/s single-stream, significantly worse than both the base server and the AQ-MedAI drafter. This knowledge forces a strategic pivot: the assistant kills the speculative decoding server and restores the base SGLang configuration ([msg 3219]), abandoning the EAGLE-3 approach for this hardware.

Conclusion

Message [msg 3216] is a masterclass in the scientific method applied to ML engineering. A hypothesis is formed (the custom drafter will outperform the generic one), an experiment is designed (benchmark three prompts), data is collected (40.8 tok/s), and the hypothesis is rejected. The message's brevity belies its significance: it represents the culmination of hours of work and the moment of reckoning when that work is measured against reality. The assistant's response — to accept the data, diagnose the root cause, and pivot — is the essence of effective engineering.