Segment 33
In this sub-session, the user discovered that the previously reported 94 tok/s EAGLE-3 speculation was not reproducible, with the current baseline at 82-83 tok/s and EAGLE-3 at 59-61 tok/s (27% worse). Root cause was identified as the verify step running without CUDA graphs, costing ~30ms per cycle regardless of attention mode. Multiple attempts to propagate NCCL tuning env vars to worker processes failed, confirming the 30ms verify time is inherent. The user then analyzed the break-even math for EAGLE-3 viability, downloaded and inspected the AQ-MedAI K2 drafter from HuggingFace confirming identical architecture, and wrote a comprehensive `eagle-k2finetune-game-plan.md` document covering fine-tuning approaches. NCCL tuning vars were permanently persisted in `/usr/lib/python3.12/sitecustomize.py` to survive reboots.
The 30ms Wall: How EAGLE-3 Speculative Decoding Collapsed and Forced a Strategic Pivot