Segment 32

In this sub-session, the user discovered that the previous attempt to fix EAGLE-3 hidden state wiring by adding embedding capture was actually wrong — the training data had never captured the embedding output, and the original layer config [2,30,58] was correct. After reverting, accept rate jumped from ~19% to ~47%. The user then added profiling instrumentation to the eagle worker, revealing that target model verify forward consumes 95%+ of cycle time. NCCL tuning (LL protocol, Ring algorithm, SYS P2P level) reduced verify time by ~27%. A sweep of step counts from 1 to 10 found that 2 steps (3 draft tokens) is optimal, achieving 94 tok/s — beating the 88.8 tok/s baseline by ~5.9%. The user also compared against AQ-MedAI's drafter trained on 38x more data, confirming that scaling training data is the highest-leverage remaining improvement.

Fix EAGLE-3 hidden state wiringAdd profiling instrumentation to eagle workerProfile target verify bottleneckTune NCCL settings for speculationSweep step counts for optimal configurationCompare against AQ-MedAI drafterIdentify training data as leverage point

From Broken Wiring to Breakthrough: The Systematic Optimization of EAGLE-3 Speculative Decoding on Blackwell GPUs 3685 words

Chunks