Chunk 31.0

In this chunk, the user investigated and debugged poor EAGLE-3 speculative decoding performance. The SGLang server was running with 16 draft tokens but achieving only ~56.8 tok/s (vs 90.0 baseline) with an accept length of ~1.6. Investigation revealed two root causes: first, `--speculative-num-steps 1` was silently overriding `--speculative-num-draft-tokens 16` to just 2 draft tokens due to a SGLang constraint when `topk=1`. After fixing to `--speculative-num-steps 15`, performance worsened to 46.7 tok/s with accept length still only ~1.9, indicating the draft model wasn't predicting well despite 74.7% training accuracy. The user then wrote a standalone test to isolate the draft model from SGLang, discovering a critical wiring mismatch: the training pipeline used `cat([embed_output, layer3, layer31])` as input to the fc layer (taking the first 3 of 4 hidden states), but SGLang was passing `cat([layer3, layer31, layer59])` (the 3 auxiliary hidden states captured by the target model). With the correct input format, the standalone test achieved 76.9% accuracy matching training metrics. The user modified `deepseek_v2.py` in SGLang to capture the embedding output when `layer_id=-1` is specified, updated the draft model config from `[2, 30, 58]` to `[-1, 2, 30]`, and restarted the server. After the fix, performance improved slightly to 54.8 tok/s with accept length ~1.8 out of 6 draft tokens, still far below the 90.0 baseline, suggesting additional issues remain with the inference pipeline beyond the input format mismatch.

The Hidden State Wiring Bug: Debugging EAGLE-3 Speculative Decoding from 56.8 tok/s to Root Cause 3312 words

Message Articles

Subagent Sessions