The Moment of Truth: Testing a Custom EAGLE-3 Drafter on Kimi-K2.5

In the high-stakes world of large language model inference optimization, few moments are as tense as the first test of a custom-trained speculative decoding drafter. Message [msg 3212] captures precisely such a moment — a brief config inspection that precedes the launch of a custom EAGLE-3 drafter trained over the course of an entire session on the Kimi-K2.5 model. The message is deceptively simple: the assistant reads a JSON configuration file to verify compatibility before launching the drafter. But this single action sits at the convergence of days of work — building a training pipeline, generating synthetic data, extracting hidden states, finetuning a draft model, and debugging API incompatibilities — all leading to this critical experiment.

The Road to This Message

To understand why message [msg 3212] was written, one must trace the arc of the preceding session. The assistant had been engaged in a comprehensive effort to deploy the Kimi-K2.5 INT4 model across 8 Blackwell GPUs, using SGLang as the inference engine. After extensive profiling (see [segment 19]), the dominant bottleneck was identified: AllReduce communication consumed 51.5% of decode time. This led the assistant to investigate speculative decoding as a software-only optimization path that could bypass the communication bottleneck by reducing the number of decoding steps.

The assistant explored several speculative decoding approaches before settling on EAGLE-3, a sophisticated draft-then-verify scheme that uses a lightweight transformer to predict multiple future tokens in parallel. After ruling out n-gram speculation as slower ([segment 20]), the assistant built a complete EAGLE-3 training pipeline from scratch — writing scripts for hidden state extraction, synthetic data generation, and finetuning. This pipeline was tested end-to-end on 10 samples, scaled to 1,000 samples, and finally run on a full 10K sample dataset ([segment 22]).

The first attempt to deploy EAGLE-3 used vLLM, but the integration proved problematic. The acceptance rate was only ~15%, yielding a net throughput of 0.66x compared to baseline — meaning speculative decoding was actually slower than regular inference ([segment 23]). The assistant then pivoted to SGLang, which loaded the model in 22 seconds but initially appeared to deadlock on the SM120 architecture. This deadlock turned out to be a false alarm: the server was actually loading the 547GB model, which simply took 5–10 minutes ([chunk 24.0]).

With SGLang base running, the assistant benchmarked it and found impressive results: 63.6 tok/s single-stream and a peak throughput of 2,370 tok/s at concurrency 128, significantly outperforming vLLM's peak of 1,536 tok/s. The assistant then tested the AQ-MedAI EAGLE-3 drafter — a publicly available draft model trained for Kimi-K2 (not K2.5). The results were underwhelming: ~42% acceptance rate with no speedup over baseline, largely because SGLang automatically limited max_running_requests to 48 in speculative mode, crippling throughput.

This brings us to message [msg 3212]. The assistant had just finished analyzing the AQ-MedAI results and updating the todo list. The logical next step was to test the custom EAGLE-3 drafter that had been painstakingly trained on 10K samples of Kimi-K2.5 data. But before launching a multi-GPU server that would take 10 minutes to load, the assistant needed to verify one thing: was the drafter's configuration compatible with SGLang's EAGLE-3 implementation?

Anatomy of the Message

The message reads:

Let me check our trained drafter's config to make sure it's compatible, then launch with it: [bash] ssh root@10.1.230.174 'cat /data/eagle3/output_10k/4/config.json | python3 -m json.tool | head -30' { "architectures": [ "LlamaForCausalLMEagle3" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 163584, "draft_vocab_size": 32000, "dtype": "bfloat16", "eagle_config": { "eagle_aux_hidden_state_layer_ids": [ 2, 30, 58 ], "use_aux_hidden_state": true }, "eos_token_id": 163585, "head_dim": 128, "hidden_act": "silu", "hidden_size": 7168, "initializer_range"...

The assistant's reasoning is explicit: "Let me check our trained drafter's config to make sure it's compatible, then launch with it." This reveals a methodical approach — rather than blindly launching and waiting 10 minutes for a server that might crash, the assistant performs a quick sanity check. The bash command reads the config file and pipes it through python3 -m json.tool for pretty-printing, then limits output to the first 30 lines with head -30. This is a lightweight operation that returns results in seconds.

The config output reveals several critical details about the drafter architecture. The "architectures" field shows "LlamaForCausalLMEagle3" — this is the expected architecture for SGLang's EAGLE-3 implementation, confirming compatibility at the model class level. The "dtype" of "bfloat16" indicates the drafter was trained in bfloat16 precision, which is standard for modern GPU training. The "hidden_size": 7168 and "head_dim": 128 are key architectural parameters that must match the target model's configuration — and they do, since Kimi-K2.5 uses a similar transformer backbone.

The "eagle_config" section is particularly important. It specifies "eagle_aux_hidden_state_layer_ids": [2, 30, 58], meaning the drafter was trained to accept auxiliary hidden states from layers 2, 30, and 58 of the target model. The "use_aux_hidden_state": true flag confirms this feature is enabled. This is a distinctive aspect of the EAGLE-3 architecture: instead of using only the final layer's hidden states (as in EAGLE-2), EAGLE-3 can incorporate intermediate representations from multiple layers, potentially providing richer conditioning for the draft prediction.

The "draft_vocab_size": 32000 indicates the drafter's output vocabulary size, which is smaller than the target model's full vocabulary (which has bos_token_id at 163584 and eos_token_id at 163585, suggesting a vocabulary of over 160K tokens). This is expected — EAGLE-3 drafters typically use a reduced vocabulary for efficiency, mapping draft tokens back to the full vocabulary during verification.

What the Config Does Not Reveal

While the config confirms structural compatibility, it cannot reveal the most critical factor: the quality of the trained drafter. The config says nothing about the training data, the number of training steps, the loss curves, or — most importantly — whether the hidden states from the INT4 quantized target model align with the bfloat16 hidden states the drafter was trained on.

This hidden state alignment issue would prove to be the fatal flaw. The training pipeline had extracted hidden states from the model running in bfloat16/fp16 mode, but the serving configuration used INT4 quantization (via the --model-path /shared/kimi-k2.5-int4 flag). Quantization compresses the model weights and can subtly alter the hidden state distributions, especially in the intermediate layers that EAGLE-3 relies on. The drafter had learned to predict tokens conditioned on one distribution of hidden states, but at inference time it received a different distribution.

The Outcome

The subsequent messages reveal the disappointing result. In [msg 3216], the assistant benchmarks the custom drafter and finds single-stream throughput of only 40.8 tok/s — significantly worse than the SGLang baseline of 63.6 tok/s. In [msg 3217], the acceptance rate logs show accept len: 1.00, accept rate: 0.25 — meaning the drafter accepted exactly zero draft tokens per step (accept len of 1.00 means only the original token was kept, and 0.25 = 1/4 is the rate of accepting 0 out of 4 draft tokens, which is the worst possible outcome). The assistant's analysis in [msg 3218] is blunt: "Our custom drafter is completely broken."

The assistant then synthesizes the complete comparison:

| Config | Single-stream tok/s | C=128 tok/s | Accept rate | |--------|---------------------|-------------|-------------| | vLLM baseline | 82.5 | 1,536 | N/A | | SGLang base (CG) | 63.6 | 2,370 | N/A | | SGLang + AQ-MedAI EAGLE-3 | 62.9 | 849 | ~42% | | SGLang + Custom K2.5 EAGLE-3 | 40.8 | 597 | 25% (broken) |

The conclusion is stark: "EAGLE-3 on this model is fundamentally not working well." The assistant identifies the root cause as the hidden state mismatch between the INT4 quantized model and the BF16-trained drafter. This is a critical insight — speculative decoding pipelines must ensure that the hidden states used during training exactly match those produced during inference, or the drafter will fail to predict accurately.

Assumptions and Their Consequences

Message [msg 3212] reveals several assumptions, some explicit and some implicit. The explicit assumption is that config compatibility is the primary concern — "make sure it's compatible, then launch with it." The assistant correctly verifies that the architecture name, hidden size, head dimension, and EAGLE-3 configuration are structurally compatible with SGLang's expectations.

The implicit assumption is more subtle: that a drafter trained on 10K samples of Kimi-K2.5 data would perform at least as well as a publicly available drafter trained on the older Kimi-K2 model. This assumption proved incorrect. The custom drafter performed worse than AQ-MedAI (25% vs 42% acceptance), despite being trained on the correct model's data. The hidden state distribution shift from quantization overwhelmed any benefit from using K2.5-specific training data.

There is also an assumption about the training pipeline itself — that the hidden state extraction, data generation, and finetuning steps were correctly implemented. The assistant had tested the pipeline on 10 samples and 1,000 samples, but the leap to 10K samples may have introduced subtle issues. The training loss may have been monitored, but the ultimate test — acceptance rate on the actual inference server — could only be measured after the full deployment.

Input Knowledge Required

To fully understand message [msg 3212], one needs knowledge of several domains. First, the EAGLE-3 speculative decoding architecture: how draft models use hidden states from the target model to predict multiple future tokens, and how the verification step ensures correctness. Second, the SGLang server architecture: how it loads models across 8 GPUs with tensor parallelism, how it captures CUDA graphs for optimization, and how it manages speculative decoding workers. Third, the Kimi-K2.5 model architecture: its use of DeepSeekV2-derived components, its INT4 quantization, and its large vocabulary. Fourth, the training pipeline that produced this drafter: the hidden state extraction script, the synthetic data generation from actual model outputs, and the finetuning process using the speculators library.

Output Knowledge Created

Message [msg 3212] itself produces limited output — it confirms that the drafter config is structurally compatible with SGLang's EAGLE-3 implementation. The architecture name LlamaForCausalLMEagle3 matches SGLang's expected model class. The hidden size of 7168 and head dimension of 128 are consistent with the target model. The EAGLE-3 configuration specifies three auxiliary hidden state layers (2, 30, 58), which is a valid configuration.

But the true knowledge created by this message is what it enables: the launch and benchmarking of the custom drafter, which ultimately reveals that EAGLE-3 speculative decoding provides no benefit on this hardware configuration. This negative result is valuable — it saves future effort that might have been spent on further EAGLE-3 tuning, and it redirects focus to other optimization strategies (in this case, tuning SGLang's NCCL communication settings to match vLLM's single-stream performance).

The Thinking Process

The assistant's thinking in this message is visible in the brief but deliberate reasoning: "Let me check our trained drafter's config to make sure it's compatible, then launch with it." This reveals a two-step decision process. First, verify structural compatibility — if the config doesn't match SGLang's expectations, the server will crash after a 10-minute load time. Second, launch and test — the real validation can only happen through benchmarking.

The choice of head -30 is also telling. The assistant doesn't need to see the entire config — the first 30 lines contain the critical fields: architecture, hidden size, head dimension, EAGLE-3 configuration, and vocabulary parameters. The remaining lines (attention heads, layer counts, normalization parameters) are either standard values or can be inferred. This is efficient information foraging — get just enough data to make the go/no-go decision.

Conclusion

Message [msg 3212] is a small but pivotal moment in a larger narrative of ML engineering. It represents the bridge between building a speculative decoding pipeline and validating its real-world performance. The config check itself is routine — a simple file read and pretty-print — but it carries the weight of an entire session's work. The custom EAGLE-3 drafter, trained over hours of computation, is about to face its moment of truth. And as the subsequent messages reveal, that truth is disappointing: the drafter is broken, and EAGLE-3 speculative decoding offers no benefit on this model and hardware.

The lesson is a familiar one in ML engineering: the gap between training and inference is full of hidden pitfalls. Hidden state distributions shift under quantization. Training data quality matters more than quantity. And sometimes, the best optimization is not the most sophisticated one — in this case, the assistant would pivot back to tuning SGLang's base performance with NCCL environment variables, achieving better results without the complexity of speculative decoding.