Segment 26
In this sub-session, the user questioned whether the EAGLE-3 draft model is data-limited with only ~21M unique tokens from 10K samples. The assistant analyzed the training run (epoch 4 of 5) showing diminishing returns and discussed options to generate more data or run extended grokking training. The user chose to benchmark the current checkpoint first. However, when launching SGLang with the newly trained EAGLE-3 checkpoint, the server exhibited the same broken behavior as the previous round: accept_len ~1.00 and accept_rate ~0.20, meaning zero draft tokens were accepted. Debugging revealed two critical issues: (1) a weight key name mismatch where the speculators library saves the decoder layer as `layers.0.*` but SGLang expects `midlayer.*`, causing trained weights to be silently dropped; and (2) more fundamentally, the hidden states passed to the draft model are 7168-dimensional (single layer) instead of the expected 21504-dimensional concatenation of three auxiliary layer hidden states. The `fc` fusion layer is bypassed because the shape check `hidden_states.shape[-1] != embeds.shape[-1]` evaluates to `7168 != 7168` → False. The root cause is that `eagle_use_aux_hidden_state` is not properly activated for the KimiK25 model, so the target model's `capture_aux_hidden_states` mechanism does not produce multi-layer hidden states. This explains why both the old vLLM-trained drafter and the new SGLang-trained drafter exhibit identical zero-acceptance behavior.