The Moment of Reckoning: Debugging a Zero-Acceptance EAGLE-3 Draft Model

"24.8 tok/s — that's terrible. Much worse than our 90 tok/s baseline without speculation, and even worse than the 27 tok/s we got without CUDA graphs initially."

In the high-stakes world of speculative decoding for large language models, few moments are as deflating as watching a carefully trained draft model produce worse throughput than no speculation at all. This is precisely the situation captured in message 3536 of an opencode coding session, where an AI assistant discovers that its newly trained EAGLE-3 draft model — the product of an elaborate multi-day pipeline spanning data generation, hidden state extraction, and full fine-tuning — is delivering a paltry 24.8 tokens per second. The baseline without any speculation achieves 90 tok/s. The draft model is actively hurting performance.

This message is a turning point in the session. It represents the moment of discovery that something is fundamentally broken, and it sets off an intensive debugging chain that will consume the next several dozen messages. Understanding why this message matters requires unpacking the context, the reasoning, the assumptions, and the diagnostic methodology on display.

The Context: An Ambitious EAGLE-3 Pipeline

To appreciate what's at stake in message 3536, we need to understand the broader project. The session involves deploying the Kimi-K2.5 model — a massive 1.2T-parameter Mixture-of-Experts architecture — on an 8-GPU machine with RTX PRO 6000 Blackwell GPUs. The team has been working to implement EAGLE-3 speculative decoding, a technique where a smaller "draft" model proposes token sequences that the large "target" model then verifies in parallel. If the draft model is accurate enough, this can yield substantial speedups.

The journey to this message has been arduous. Earlier segments of the session (segments 21-25) covered: fixing API incompatibilities between the speculators library and vLLM 0.16, patching custom model workers for the Kimi-K2.5 architecture, extracting hidden states from the target model, training an initial EAGLE-3 drafter, discovering that the vLLM-trained drafter achieved zero acceptance rate, pivoting to SGLang as the inference engine, building custom kernel modules for the Blackwell architecture, tuning SGLang single-stream performance to 90 tok/s, developing a server-side hidden state extraction patch, extracting 10K training samples via SGLang, and training a brand-new EAGLE-3 drafter from scratch using those SGLang-extracted features.

The current segment (segment 26) began with the user raising a critical question about data scaling: with only ~21M tokens of unique training data (10K samples) for a 1.2B-parameter draft model, was the model severely data-limited? The assistant analyzed the training run (epoch 4 of 5, ~74% complete) and observed diminishing returns — validation loss plateauing at ~6.13, step-0 accuracy at ~74.5%. Drawing on the EAGLE-3 paper's scaling laws and the concept of grokking, the assistant discussed two paths: generating 5-10× more data or running a 100-epoch grokking continuation. The user opted to benchmark the current checkpoint first.

That benchmark is what message 3536 reports — and the results are devastating.

The Message: Raw Data and Immediate Diagnosis

The message begins with the assistant's stark assessment: 24.8 tok/s. To drive home how bad this is, the assistant immediately contextualizes it against two baselines: the 90 tok/s achieved without any speculation, and the 27 tok/s seen in earlier runs without CUDA graphs. The speculation is not just failing to help — it's making things worse by a factor of 3.6×.

But the assistant doesn't stop at the headline number. The very next action is diagnostic: checking the acceptance rate from the server logs. The assistant runs a grep command to extract lines containing "accept", "draft", "spec.rate", or "avg.len" from the SGLang server log file. The results are damning:

accept len: 1.00, accept rate: 0.20

These two numbers tell the entire story. With speculative-num-draft-tokens set to 5, an accept rate of 0.20 means exactly 1 out of every 5 draft tokens is accepted — which is precisely the base token that comes from the verification pass itself. In other words, zero draft tokens are being accepted. The draft model's predictions are no better than random noise, and the verification pass is rejecting every single one of them.

An accept length of 1.00 confirms this: on average, only one token (the verified base token) survives each speculative step. The entire speculative decoding apparatus is running at full computational cost — generating 5 draft tokens, running the full target model verification — for zero benefit.

The Reasoning: Connecting the Dots

What makes this message particularly interesting is the assistant's reasoning process. The assistant immediately recognizes that this is exactly the same behavior as the old vLLM-trained drafter, which also showed accept len: ~1.00, accept rate: ~0.20. This observation is crucial because it eliminates several potential explanations:

1. It's not a hidden-state-mismatch issue between training and inference. The new drafter was trained on SGLang-extracted hidden states and is being evaluated on SGLang — the training/inference gap should be minimal.
2. It's not a data quality issue specific to the vLLM extraction pipeline. The same failure mode appears with both extraction methods.
3. It's not a training convergence issue. The model achieved 74.5% step-0 accuracy on the validation set, which should translate to meaningful draft token acceptance. The assistant's reasoning is: "The drafter trained on SGLang hidden states is just as broken as the vLLM one. This is NOT a hidden-state-mismatch issue. The model is producing essentially random predictions at inference time despite achieving 74.5% step-0 accuracy on the validation set." This leads to the hypothesis that the problem is a weight loading issue — the trained weights are being silently dropped or misaligned when SGLang loads the checkpoint. The assistant pivots to investigating the weight key names, which will consume the next several messages (3537-3570+).

Assumptions and Their Consequences

Several assumptions are visible in this message and its immediate aftermath:

Assumption 1: The training was successful. The assistant assumed that because the validation metrics looked reasonable (loss ~6.13, step-0 accuracy ~74.5%), the model had learned something useful. The zero acceptance rate disproves this assumption — either the validation metrics are misleading, or the model's learned knowledge is not being applied correctly at inference time.

Assumption 2: The weight loading was correct. The assistant initially assumed that SGLang's weight loading logic would correctly map the checkpoint's parameter names to the model's expected parameter names. As we'll see in subsequent messages, this assumption was wrong in two ways: first, the layers.0.* prefix didn't match SGLang's expected midlayer.* prefix; second, the d2t tensor stored absolute target token IDs instead of the diffs that SGLang expected.

Assumption 3: The new training would fix the old problem. The entire point of retraining with SGLang-extracted hidden states was to eliminate the suspected mismatch between vLLM's hidden state format and what the draft model expected. When the same failure mode appeared, it forced a deeper investigation.

The Diagnostic Methodology

Message 3536 showcases a systematic diagnostic approach that's worth examining:

1. Establish baselines. The assistant immediately compares against the 90 tok/s no-speculation baseline and the 27 tok/s no-CUDA-graph baseline. This quantifies the severity of the problem.
2. Check the right metrics. Rather than just looking at throughput, the assistant checks acceptance rate and accept length — the root-cause metrics for speculative decoding performance.
3. Recognize patterns. The assistant instantly recognizes that accept rate: 0.20 with 5 draft tokens means zero acceptance, and connects this to the identical failure mode from the previous attempt.
4. Formulate hypotheses. The assistant generates a specific hypothesis (weight loading mismatch) and proceeds to test it in the following messages.

Input and Output Knowledge

To fully understand this message, the reader needs to know:

• Speculative decoding basics: How draft models propose tokens and target models verify them, and why acceptance rate is the critical performance metric.
• EAGLE-3 architecture: The draft model uses a single transformer layer with a fusion layer (fc) that concatenates hidden states from multiple layers of the target model.
• The project history: Previous attempts with vLLM failed with the same symptoms, leading to the pivot to SGLang and the retraining effort.
• SGLang's logging format: The accept len and accept rate fields in the decode batch log lines. The message creates new knowledge: the definitive finding that the newly trained EAGLE-3 drafter achieves zero acceptance on SGLang, ruling out several hypotheses and pointing toward a weight loading or architectural mismatch issue.

Why This Message Matters

Message 3536 is the critical inflection point where the assistant realizes that the entire EAGLE-3 training pipeline — days of work spanning data generation, hidden state extraction, model training, and inference integration — has produced a model that is functionally useless at inference time. The 24.8 tok/s throughput is not just disappointing; it's a signal that something fundamental is wrong.

The message also demonstrates a key principle of debugging complex ML systems: when a model performs well on validation metrics but fails completely at inference, the problem is almost certainly in the inference pipeline, not the training. The assistant's immediate pivot to investigating weight loading — rather than questioning the training data or hyperparameters — reflects this understanding.

In the messages that follow, the assistant will discover two critical bugs: a weight key name mismatch (layers.0.* vs midlayer.*) and a d2t tensor format mismatch (absolute IDs vs diffs). But even after fixing both, the acceptance rate barely improves — leading to the deeper realization that the hidden states passed to the draft model are 7168-dimensional (single-layer) instead of the expected 21504-dimensional (three-layer concatenated) features. That final discovery, building on the foundation laid in this message, will reveal that the eagle_use_aux_hidden_state mechanism is not properly activated for the KimiK25 model, explaining why both the old and new drafters exhibit identical zero-acceptance behavior.

Message 3536 is thus the moment when the assistant stops assuming the pipeline works and starts systematically proving where it breaks. It's a masterclass in diagnostic reasoning for ML engineering — and a reminder that in complex systems, the most carefully laid plans can fail in unexpected ways.