Chunk 33.0

In this chunk, the user discovered that the EAGLE-3 speculation setup that previously showed 94 tok/s was actually not reproducible — the current stable baseline is 82-83 tok/s, and EAGLE-3 2-step speculation is delivering only 59-61 tok/s (27% worse than baseline). The root cause was identified: the verify step runs in extend mode without CUDA graphs, costing ~30ms per cycle regardless of attention mode (prefill or decode), compared to ~12ms for a single-token decode with CUDA graphs. The user attempted multiple fixes to propagate NCCL tuning env vars to spawn worker processes (engine.py patch, scheduler.py patch, sitecustomize.py), but none resolved the 30ms verify time, confirming this is the real cost of running 3-token verify through the 1T MoE model on 8 PCIe GPUs. The user then analyzed the math for making EAGLE-3 viable: with 30ms verify cycles, break-even requires accept_len 2.46 (vs current 2.0), and 150 tok/s would require 78% conditional accuracy. They downloaded and inspected the AQ-MedAI K2 drafter from HuggingFace, confirming its architecture is identical to theirs (same hidden_size=7168, intermediate_size=18432, attention heads, fc projection dimensions, draft_vocab_size=32000), making it a drop-in compatible initialization for K2.5 fine-tuning. The session concluded with writing a comprehensive `eagle-k2finetune-game-plan.md` document covering three approaches: fine-tuning AQ-MedAI's drafter with existing 37K K2.5 samples, scaling training data to 200K+ samples, and a direct plug-in probe to measure hidden state similarity between K2 and K2.5. NCCL tuning vars were permanently persisted in `/usr/lib/python3.12/sitecustomize.py` to survive reboots.

The 30ms Wall: How EAGLE-3 Speculative Decoding Collapsed on 8 PCIe GPUs 2209 words

Message Articles