Chunk 27.0

## Summary This chunk resolved the critical EAGLE-3 hidden state concatenation bug that had been blocking all draft model deployment. The root cause was traced to a simple but devastating flag mismatch: the server was started with `--speculative-algorithm EAGLE` instead of `EAGLE3`. The `is_eagle3()` check is strict — only `EAGLE3` triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With `EAGLE`, the draft model received 7168-dim final-layer-only states instead of the expected 21504-dim concatenated states, causing the `fc` fusion layer to be silently bypassed and all trained weights to be useless. After restarting with the correct flag, hidden states correctly arrive as 21504-dim and the draft model's predictions are actually being accepted (accept_len ~2.1, up from 1.0). Benchmarking the fix showed the best EAGLE-3 config achieves 82.3 tok/s (CUDA graphs + 5 draft tokens) vs the 90 tok/s non-speculative baseline — still ~9% slower. The accept_len of ~2.1 is insufficient to overcome speculation overhead; the EAGLE-3 paper's scaling curve suggests more training data is the primary lever. The AQ-MedAI drafter was also tested with the correct `EAGLE3` flag and performed slightly worse (50.5 tok/s), confirming that our custom K2.5-trained drafter is better but still data-limited. The major parallel effort was scaling up the training dataset by 10×. Ten parallel agents searched for agentic coding, reasoning, and general chat datasets. Ten datasets were selected and prepared totaling 88,088 samples (4,800 tokenized Kimi-native + 83,288 prompts needing inference). An inference pipeline was launched on the baseline SGLang server at ~830 tok/s throughput, processing all 83K prompts through Kimi-K2.5 to regenerate responses matching the target model's token distribution. This is expected to take ~24-55 hours. A live progress monitor script was created, and the full pipeline plan was documented in `train_plan_v4.md`.

From a One-Character Bug to a 10× Data Pipeline: The EAGLE-3 Speculative Decoding Odyssey 2875 words

Message Articles

Subagent Sessions