The Algorithm Name That Broke Speculative Decoding: How a Single Flag Caused Zero-Acceptance in EAGLE-3

Introduction

In the world of speculative decoding for large language models, few debugging experiences are as disheartening as watching a carefully trained draft model produce zero useful predictions at inference time. After investing days in building a complete EAGLE-3 training pipeline — synthetic data generation, hidden state extraction, vocabulary mapping, and multi-epoch training — the moment of truth arrives: the benchmark runs, and the acceptance rate is exactly 0.20. With five draft tokens being proposed per step, this means only the mandatory base token survives verification. Every single draft token is rejected. The model is producing predictions no better than random noise.

This article chronicles the debugging odyssey documented in Segment 26 of an opencode coding session — a concentrated investigation spanning nearly 100 messages across a single sub-session. What begins as a strategic discussion about data scaling and grokking transforms into a systematic root-cause analysis that traverses weight key naming conventions, vocabulary mapping tensor formats, hidden state dimensionality, and ultimately lands on a single character difference in a command-line flag: --speculative-algorithm EAGLE versus --speculative-algorithm EAGLE3.

The story is a masterclass in ML debugging methodology, demonstrating how the most elusive bugs often hide not in neural network architectures or custom CUDA kernels, but in the humble configuration flags and delegation chains that connect components developed by different teams. It also illustrates the critical importance of dynamic instrumentation — adding a single debug print to observe runtime behavior — when static code analysis reaches its limits.

The Strategic Crossroads: Data Scaling vs. Grokking

The segment opens with the user raising a prescient concern at [msg 3486]: "Don't we just have ~10-20M tokens of data for a 1B model? Do we want to get maybe 5-10x more data?" This question strikes at the heart of the EAGLE-3 training effort. The draft model — a 1.2-billion-parameter transformer with a single decoder layer — has been trained on approximately 10,000 samples of hidden state data extracted from the Kimi-K2.5 target model, totaling roughly 21 million unique tokens.

The assistant's analysis at [msg 3492] is a model of pragmatic research synthesis. Drawing on the EAGLE-3 paper's scaling laws — which show clear gains up to 8× more data without saturation — the assistant quantifies the exact data situation: 10K samples, 21M tokens, 924 GB on disk, with only 1.8 TB free on /data. The paper's best results used approximately 532,000 samples from ShareGPT and UltraChat combined. The assistant's own validation metrics tell a worrying story: loss plateauing at ~6.13, step-0 accuracy stuck at ~74.5%, with diminishing returns across epochs 1 through 4.

The practical constraints are stark. Scaling to 5× more data would require approximately 4.6 TB of hidden state storage — far exceeding the available 1.8 TB. The assistant proposes three options: using SpecForge's online training mode (generating hidden states on-the-fly without disk storage), extracting in batches with deletion between rounds, or reducing hidden state precision.

But then the user interjects with a radically different proposal at [msg 3493]: "If we want to try to go for grokking?" This six-word question reframes the entire problem. Grokking — the phenomenon where a neural network suddenly generalizes after being trained far past the point of memorization on a small dataset — offers an alternative path. Instead of investing 10–20 hours generating more data, what if the team simply trained the existing 10,000 samples for 50–200 epochs with a constant learning rate?

The assistant's response at [msg 3498] lays out a concrete "Grokking Strategy": use a constant learning rate (the cosine scheduler had decayed to near-zero), restart from the best checkpoint with --lr 1e-5, train for 100 epochs (~35 hours), and keep AdamW's weight decay as a regularizer. The assistant is intellectually honest about the risks: grokking is best documented for algorithmic tasks like modular arithmetic, not for distribution-matching problems like draft model training. The honest assessment is that if the acceptance rate is below 30%, "something is fundamentally broken again" — not a data scaling issue at all.

The user chooses the pragmatic middle path at [msg 3507]: "Benchmark first." This decision — to measure before committing to either data scaling or grokking — proves prescient. The benchmark will reveal that the draft model is fundamentally broken at inference time, and no amount of additional training or data could fix it.

The Server Launch Saga

Before the benchmark can run, the assistant must launch SGLang with the newly trained EAGLE-3 checkpoint. The launch command at [msg 3511] is dense with meaning, carrying the weight of dozens of previous debugging sessions across multiple segments:

NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS NCCL_MAX_NCHANNELS=16 \
NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 \
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 --trust-remote-code --tp-size 8 \
  --mem-fraction-static 0.85 --host 0.0.0.0 --port 8000 \
  --num-continuous-decode-steps 4 --disable-custom-all-reduce \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path /data/eagle3/output_10k_sglang/4 \
  --speculative-num-draft-tokens 5 --speculative-eagle-topk 4 \
  --speculative-num-steps 3

Every flag has a history. The NCCL environment variables are tuned for the Blackwell GPU architecture's PCIe-only topology. The --disable-custom-all-reduce flag avoids a known hang with SGLang's custom allreduce on more than two PCIe-connected GPUs. The SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN environment variable permits the draft model's shorter max position embeddings (131072) to coexist with the target model's longer context (262144).

But the server doesn't start. The assistant's health check loop — 60 iterations of 10-second polls — times out without ever seeing the server become ready. The log file shows weight loading at 97–100% completion, but the server never responds to requests. The assistant suspects a hang during CUDA graph compilation and eventually restarts with --disable-cuda-graph. After a 320-second wait, the server finally comes online.

This 10-minute delay is more than an inconvenience — it's a diagnostic signal. The server's struggle to start foreshadows deeper integration issues between the trained draft model and SGLang's inference engine.

The Moment of Truth: 24.8 tok/s

When the benchmark finally runs at [msg 3535], the results are devastating: approximately 24.8 tokens per second. The baseline without speculation is 90 tok/s. The draft model is actively slowing inference by a factor of 3.6×. The assistant immediately checks the server logs and finds the telltale metrics at [msg 3536]: accept len: 1.00, accept rate: 0.20. With five draft tokens being proposed per step, an acceptance rate of exactly 0.20 means only the mandatory base token (the one that always passes verification) is kept — zero draft tokens are ever accepted.

What makes this particularly baffling is that it's identical to the behavior of the old vLLM-trained drafter from the previous round. The assistant had retrained the draft model from scratch using SGLang-extracted hidden states specifically to eliminate the suspected cross-framework distribution mismatch. Yet the behavior is indistinguishable. This rules out the hidden-state-mismatch hypothesis and points to a deeper, more fundamental issue.

Phase 1: The Weight Key Name Mismatch

The assistant pivots to investigating how SGLang loads the checkpoint. By dumping the checkpoint's weight keys and comparing them against SGLang's LlamaForCausalLMEagle3 model implementation, the assistant discovers a critical mismatch at [msg 3539]: the speculators training library saves the decoder layer under the key prefix layers.0.* (e.g., layers.0.self_attn.q_proj.weight), but SGLang's EAGLE-3 model expects the same weights under the prefix midlayer.* (e.g., model.midlayer.self_attn.qkv_proj.weight).

Because SGLang's load_weights method tries to match checkpoint keys to model parameter names — first directly, then with a model. prefix — the layers.0.* keys fail to match model.midlayer.*, causing the decoder layer weights to be silently dropped. The draft model runs with randomly initialized decoder weights, producing garbage predictions.

The fix is elegant: a simple Python script that renames layers.0.* to midlayer.* in the safetensors file. The assistant applies the fix at [msg 3543], restarts the server, and re-runs the benchmark. The result at [msg 3552]: acceptance rate improves from 0.20 to 0.21 — a marginal change that confirms the weights are now loading, but the model is still producing essentially random predictions.

This is a critical clue. If the decoder weights are now correctly loaded from the trained checkpoint, why is the acceptance rate still zero? The answer must lie in the inputs to the draft model, not its weights.## Phase 2: The Hidden State Dimensionality Mismatch

The assistant's attention shifts from the output side of the draft model (token mapping, lm_head) to the input side — specifically, what hidden states the draft model actually receives during inference. This pivot is guided by a deep understanding of the EAGLE-3 architecture.

EAGLE-3 draft models do not receive just the final layer's hidden state from the target model. Instead, they receive a concatenation of hidden states from multiple intermediate layers — typically three auxiliary layers whose outputs are fused together to provide richer feature information. For the Kimi-K2.5 model, each layer produces a 7168-dimensional hidden state, so the concatenated input should be 3 × 7168 = 21,504 dimensions. The draft model then uses an fc (fusion) layer — a learned linear projection — to compress this 21,504-dimensional input down to 7,168 dimensions before feeding it into the transformer blocks.

The critical code path in SGLang's llama_eagle3.py is:

hidden_states = forward_batch.spec_info.hidden_states
if hidden_states.shape[-1] != embeds.shape[-1]:
    hidden_states = self.fc(hidden_states)

If the hidden states are 21,504-dimensional (three concatenated layers), the shape check 21504 != 7168 evaluates to True, and the fc layer projects them down. But if the hidden states are only 7,168-dimensional (a single layer), the check evaluates to 7168 != 7168 — False — and the fc layer is silently bypassed.

The assistant adds a targeted debug print to the draft model's forward method at [msg 3579], injecting diagnostic code that would print the shape, dtype, mean, and standard deviation of the hidden states on the first invocation. The server is restarted with this instrumentation, and after sending a test request, the debug output arrives at [msg 3593]:

[EAGLE3-DEBUG] hidden_states shape=torch.Size([21, 7168]), dtype=torch.bfloat16

This single line of output is the smoking gun. The hidden states are 7,168-dimensional — a single layer's output — not the expected 21,504-dimensional concatenation of three auxiliary layers. The fc fusion layer is never being applied because the shape check silently passes without triggering the projection.

The draft model is receiving impoverished single-layer features at inference time, despite being trained on rich multi-layer features. No amount of training data or model capacity could overcome this fundamental input mismatch.

The Red Herring: d2t Vocabulary Mapping

In the midst of this investigation, the assistant chases a compelling but ultimately incorrect hypothesis about the d2t vocabulary mapping tensor. The d2t tensor maps draft token IDs (32,000 vocabulary) to target token IDs (163,840 vocabulary). The assistant notices that SGLang's code computes hot_token_id = d2t + torch.arange(32000), which implies d2t should store diffs (offsets), not absolute IDs. The checkpoint appears to store absolute IDs. A fix is written, tested, and applied at [msg 3568].

But then the assistant pauses and reconsiders at [msg 3569]. Looking more carefully at the data, many early draft tokens map to target token 0 — which seems suspicious. The assistant traces back to the original t2d boolean mask used to build the mapping and discovers the truth: the d2t tensor was already in the correct offset format. The first 20 target token IDs happen to be [0, 1, 2, ..., 19], which are identical to the draft token IDs [0, 1, 2, ..., 19], making the offset zero. The value 787 at index 1000 is correct because draft token 1000 maps to target token 1786, giving an offset of 1786 - 1000 = 786... actually 787, which checks out.

The assistant had broken a correctly formatted tensor. The immediate revert at [msg 3572] restores the correct values from the source file, but the damage to confidence is real. This episode illustrates a classic debugging pitfall: confirmation bias. Having formed the hypothesis that the d2t format was wrong, the assistant interpreted the evidence (zeros in the first entries) as supporting that hypothesis, without considering the alternative explanation (that zeros were the correct offset for those entries).

The User's Critical Question

At [msg 3582], the user asks a question that proves pivotal: "Also btw weren't we training from scratch but with frozen embeddings etc?" This question reframes the debugging from "what is SGLang doing wrong?" to "what did we actually train, and does the inference code know about it?" It prompts the assistant to verify the entire weight loading chain one more time.

The assistant traces through SGLang's EAGLE worker code and confirms that set_embed(embed) replaces only the embedding with the target model's embedding while preserving the trained lm_head. The hot_token_id mapping is correctly computed from the d2t offsets. The weight loading chain is verified to be correct. But the hidden states remain 7168-dimensional.

This verification is important because it eliminates another class of hypotheses and narrows the search to the single remaining question: why aren't the auxiliary hidden states being captured and concatenated?

The Algorithm Name That Broke Everything

The assistant now has a clear picture of the full code path. The target model's capture_aux_hidden_states mechanism is supposed to capture hidden states from three intermediate layers (2, 30, 58) during the forward pass, concatenate them into a 21,504-dimensional vector, and pass them to the draft model. But the debug output shows only 7,168-dimensional final-layer hidden states arriving.

The assistant launches a series of subagent tasks to trace every link in the chain. One subagent reads the eagle_worker.py code and discovers the critical enum definition at [msg 3604]:

class SpeculativeAlgorithm(Enum):
    EAGLE = auto()
    EAGLE3 = auto()
    
    def is_eagle3(self) -> bool:
        return self == SpeculativeAlgorithm.EAGLE3

The is_eagle3() method is strict — it only returns True for the EAGLE3 enum value, not EAGLE. The server was launched with --speculative-algorithm EAGLE, which sets the enum to SpeculativeAlgorithm.EAGLE. Every code path that checks is_eagle3() — including the critical block in model_runner.py that calls set_eagle3_layers_to_capture on the target model — evaluates to False.

This means:

[EAGLE3-DEBUG] hidden_states shape=torch.Size([21, 21504]), dtype=torch.bfloat16

The hidden states are now 21,504-dimensional — three layers concatenated. The fc fusion layer will now be applied. The draft model will receive the same rich multi-layer features it was trained on.

Why This Bug Was So Elusive

The EAGLE vs EAGLE3 flag difference is a textbook example of a "silent configuration bug" — a bug that doesn't produce an error message, doesn't crash the server, and doesn't log a warning. The server starts up normally, loads all weights correctly, and begins processing requests. The only symptom is that the speculative decoding system produces zero useful predictions.

Several factors made this bug particularly difficult to find:

  1. The flag names are semantically similar. Both EAGLE and EAGLE3 are valid speculative decoding algorithms supported by SGLang. A user who knows that EAGLE-3 is the latest version of the EAGLE family might reasonably assume that --speculative-algorithm EAGLE would select the most advanced variant, or that EAGLE is a shorthand that encompasses all EAGLE variants.
  2. The documentation is ambiguous. The SGLang documentation and server help text list EAGLE and EAGLE3 as separate options without clearly explaining the difference. The EAGLE option exists for backward compatibility with EAGLE-1 and EAGLE-2 models, but it doesn't enable the auxiliary hidden state capture mechanism that EAGLE-3 requires.
  3. The failure mode is silent. When is_eagle3() returns False, the code doesn't raise an error or log a warning. It simply doesn't set up the auxiliary hidden state capture. The target model runs its normal forward pass, returning only final-layer hidden states. The draft model receives these single-layer states and silently bypasses its fc fusion layer because the shape check 7168 != 7168 evaluates to False. Everything appears to work — just with zero acceptance.
  4. The bug affected both training runs equally. Both the old vLLM-trained drafter and the new SGLang-trained drafter were benchmarked with the same --speculative-algorithm EAGLE flag, so both exhibited identical zero-acceptance behavior. This made it appear that the problem was fundamental to the training data or model architecture, rather than a configuration issue.

Implications and Next Steps

The discovery that the server was launched with the wrong speculative algorithm flag has profound implications for the entire EAGLE-3 deployment effort:

  1. The draft model may actually work. With the correct EAGLE3 flag, the auxiliary hidden state capture mechanism should now be activated, and the draft model should receive the 21,504-dimensional concatenated features it was trained on. The acceptance rate may be significantly higher than the 0.20 observed during debugging.
  2. The training data may be sufficient. The data scaling discussion that opened this segment — "Don't we just have ~10-20M tokens of data for a 1B model?" — may turn out to be moot. If the acceptance rate is reasonable with the correct flag, the 10,000 samples may be enough to achieve useful speedups, and the grokking strategy may not be necessary.
  3. The weight key fix was necessary. Even though the algorithm flag was the primary bug, the weight key name mismatch between speculators and SGLang would have caused the decoder weights to be silently dropped. Both fixes are required for correct operation.
  4. CUDA graph support remains a challenge. The server is running with --disable-cuda-graph because SGLang hangs during CUDA graph capture with EAGLE-3 on this model. This limits throughput significantly. Enabling CUDA graphs for EAGLE-3 on the KimiK25 architecture remains an open problem.

Conclusion

This segment of the coding session is a masterclass in debugging complex ML systems. It demonstrates several crucial principles:

Measure before investing. The decision to benchmark before committing to data scaling or grokking saved days of wasted computation. The benchmark revealed that the model was fundamentally broken at inference time — a problem that more data or more epochs could not solve.

Dynamic instrumentation breaks through analysis paralysis. The assistant spent dozens of messages reasoning about code paths, reading source files, checking configurations, and verifying tensor formats. Each analysis was logically sound, yet the problem persisted. The debug print — a single line of Python that printed the hidden states shape at runtime — broke through this ceiling by providing direct empirical evidence about the system's behavior.

Check the interfaces between components. The weight key name mismatch between speculators and SGLang, and the algorithm flag mismatch between the launch command and the code's expectations, are both interface failures. When components developed by different teams interact, naming conventions and configuration parameters must be explicitly verified.

Question your own conclusions. The assistant's willingness to reconsider the d2t fix — to go back and check whether the very first step of the pipeline was correct — demonstrates the intellectual humility that distinguishes effective debugging from mere guesswork.

The most subtle bugs hide in configuration, not code. The root cause of the entire zero-acceptance saga — spanning two training runs, two inference engines, and dozens of hours of debugging — was a single missing character in a command-line flag. The neural network architecture was correct. The training pipeline was correct. The weight loading was correct. The vocabulary mapping was correct. But the server was told to use EAGLE instead of EAGLE3, and the entire speculative decoding system collapsed into uselessness.

In the end, the 21 million tokens were enough — if only the server had been started with the right flag.## References

[1] Segment 26, Chunk 0 — "The Data Scaling Paradox: When 21 Million Tokens Wasn't the Problem" (analyzer summary and article)

[2] Segment 26, Chunk 1 — "The Debugging Odyssey: Tracing an EAGLE-3 Zero-Acceptance Bug Through Weight Names, Token Mappings, and Hidden State Dimensions" (analyzer summary and article)

[3] [msg 3486] — User raises data scaling question: "Don't we just have ~10-20M tokens of data for a 1B model?"

[4] [msg 3492] — Assistant analyzes data scaling situation, references EAGLE-3 paper scaling laws

[5] [msg 3493] — User proposes grokking strategy

[6] [msg 3498] — Assistant outlines concrete grokking plan with constant LR

[7] [msg 3507] — User chooses to benchmark first before committing to data scaling or grokking

[8] [msg 3511] — SGLang server launch command with EAGLE-3

[9] [msg 3535] — Benchmark results: 24.8 tok/s, accept_len 1.00, accept_rate 0.20

[10] [msg 3536] — Assistant discovers zero acceptance rate identical to old vLLM-trained drafter

[11] [msg 3537] — Assistant begins investigating weight loading

[12] [msg 3539] — Discovery of weight key name mismatch: layers.0.* vs midlayer.*

[13] [msg 3543] — Weight key rename fix applied

[14] [msg 3552] — After fix: acceptance rate improves from 0.20 to 0.21 (still broken)

[15] [msg 3555] — Assistant verifies weights are loaded correctly

[16] [msg 3556] — Assistant pivots to investigating hidden state format

[17] [msg 3572] — d2t tensor incorrectly modified, then reverted

[18] [msg 3579] — Debug print added to llama_eagle3.py to observe hidden states at runtime

[19] [msg 3582] — User asks critical question about training from scratch with frozen embeddings

[20] [msg 3593] — Debug output reveals hidden_states shape=[21, 7168] instead of expected [21, 21504]

[21] [msg 3595] — Draft model config confirmed to have use_aux_hidden_state: true with layer IDs [2, 30, 58]

[22] [msg 3604] — Root cause discovered: is_eagle3() returns False for --speculative-algorithm EAGLE

[23] [msg 3607] — Server restarted with --speculative-algorithm EAGLE3

[24] [msg 3609] — Debug output confirms hidden_states shape=[21, 21504] after fix