Chunk 32.0

In this chunk, the user discovered and corrected a critical misunderstanding about the EAGLE-3 hidden state wiring. The previous "fix" (adding embedding capture with `layer_id=-1`) was actually wrong — the training data had never captured the embedding output. The HS dump patch captured at layers 3, 31, 59 (outputs of layers 2, 30, 58), and `standardize_data_v1` used `cat([layer3_out, layer31_out, layer59_out])`. The original config `[2, 30, 58]` was correct all along. After reverting the config, the accept rate jumped from ~19% to ~47%, confirming the fix. The user then added profiling instrumentation to the eagle worker and discovered that the target model verify forward consumes 95%+ of the cycle time (21-28ms), while the draft model is negligible (<5%). NCCL tuning (`NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS`) proved critical — reducing verify time by ~27%. Testing step counts from 1 to 10 revealed that 2 steps (3 draft tokens) is optimal, achieving 94 tok/s — beating the 88.8 tok/s baseline by ~5.9%. More steps add verify overhead that outweighs marginal acceptance gains, while fewer steps don't produce enough accepted tokens per cycle. The user also compared against AQ-MedAI's Kimi-K2-Instruct-eagle3 model, noting their drafter uses the same architecture but was trained on 38x more data (1.4M vs 37K samples), achieving accept lengths of 3.2-3.5 vs our ~2.1. This suggests more training data is the highest-leverage remaining improvement. The chunk demonstrates a systematic, profiling-driven optimization approach — measuring exact per-phase timing, identifying the true bottleneck (target verify, not draft model), and finding the optimal configuration through empirical sweep rather than guessing.

From Wrong Fix to Optimal Performance: The EAGLE-3 Debugging and Optimization Odyssey 2140 words

Message Articles