The First Glimpse of a Learning Drafter: EAGLE-3 Training Metrics Come Into View

In the high-stakes world of speculative decoding for large language models, the difference between a working drafter and a broken one can mean the difference between a 2x speedup and a 0.66x slowdown. Message [msg 3470] captures a pivotal moment in the EAGLE-3 training pipeline for the Kimi-K2.5 model: the first time the assistant can actually see whether the drafter is learning. After hours of debugging logging infrastructure, patching API incompatibilities, and rebuilding training scripts, the assistant finally has numbers to work with — and they are encouraging.

The Message in Full

The message reads:

Training is running with visible metrics now. Here's the summary:

Epoch 1 early metrics (from scratch, SGLang-extracted hidden states): - Step 0 accuracy: 70-91% (token-level next-token prediction) - Step 1 accuracy: 55-75% conditional - Step 2 accuracy: 34-74% conditional - Total loss: 1.8-11 (varies by sample difficulty)

This looks very promising compared to the broken drafter that had 25% acceptance. The model is clearly learning meaningful predictions from the SGLang hidden states.

ETA: ~2 hours for remaining 4 epochs (1-4). Checkpoints will be saved at /data/eagle3/output_10k_sglang/1/, /2/, /3/, /4/.

On its surface, this is a brief status update. But beneath it lies a complex story of debugging, architectural decisions, and the subtle art of making a neural network train properly.

Why This Message Was Written: The Logging Crisis

The immediate trigger for this message was a silent crisis. In [msg 3453], the user asked a pointed question: "what loss at first eval?" This question exposed a fundamental problem — the training was running, consuming GPU cycles at 98% utilization, but producing zero visible metrics. The speculators library's Trainer class used Python's logging module with a metric_logger that logged per-step loss and accuracy values, but no handler had been configured. In Python's logging architecture, a logger without a handler silently discards all messages below the WARNING level. The tqdm.rich progress bar was also invisible because it requires a TTY to render.

The assistant spent messages [msg 3454] through [msg 3468] diagnosing and fixing this. The fix required killing the running training process (which had already completed epoch 0), adding logging.basicConfig(level=logging.INFO) to the training script, enabling checkpoint resumption with resume_from_checkpoint=True, and restarting from epoch 1. The first visible metrics appeared in [msg 3468], showing the initial loss values for the resumed training. Message [msg 3470] is the assistant's summary and interpretation of those first visible numbers.

What the Numbers Mean

The EAGLE-3 architecture is a multi-step speculative decoding drafter. Unlike a standard autoregressive model that predicts one token at a time, EAGLE-3 predicts multiple future tokens in parallel using a tree-like structure. The "step" indices in the metrics correspond to the depth of this prediction tree:

Step 0 accuracy (70-91%): This measures how well the drafter predicts the next token (the first draft token). A 70-91% accuracy means that 70-91% of the time, the drafter's top prediction matches the actual next token from the verifier model. This is the most important metric because if step 0 is wrong, all subsequent steps are built on a bad foundation.
Step 1 conditional accuracy (55-75%): This measures how well the drafter predicts the second token, conditioned on the first token being correct. The conditional nature is crucial — it isolates the drafter's ability to extend a correct prediction into a longer sequence.
Step 2 conditional accuracy (34-74%): The third token prediction, conditioned on the first two being correct. The wider range (34-74%) reflects the increasing difficulty of multi-step prediction and the variance across different samples.
Total loss (1.8-11): This is the sum of per-step losses. The wide range reflects varying sample difficulty — some sequences are more predictable than others. The assistant's interpretation is optimistic but measured: "This looks very promising compared to the broken drafter that had 25% acceptance." The key comparison is to the acceptance rate, not the per-step accuracy. In speculative decoding, the acceptance rate measures how many draft tokens the verifier actually accepts before needing to regenerate. A 25% acceptance rate (which the previous drafter achieved) means that on average, only 0.25 draft tokens are accepted per forward pass — barely better than no speculation at all, and often worse due to overhead. The new drafter's step 0 accuracy of 70-91% suggests a dramatically higher acceptance rate, potentially making speculative decoding worthwhile.

Assumptions and Their Implications

The assistant makes several implicit assumptions in this message:

Epoch 1 metrics are representative: The metrics shown are from the beginning of epoch 1, which means the model has already completed one full epoch of training (epoch 0, which ran silently without logging). The assumption is that these early epoch 1 metrics indicate the trajectory is healthy. This is reasonable but not guaranteed — the model could overfit or hit a plateau in later epochs.
SGLang-extracted hidden states are better: The message explicitly contrasts "SGLang-extracted hidden states" with the previous extraction method (vLLM-based). The assumption is that SGLang's hidden states capture more useful information for training the drafter. This is supported by the dramatic improvement in metrics, but it's worth noting that the previous drafter was also trained on different data (AQ-MedAI finetuning vs. from-scratch training), so the comparison isn't perfectly controlled.
Per-step accuracy translates to acceptance rate: The assistant equates step 0 accuracy (70-91%) with likely acceptance rate improvement. While correlated, these are not the same metric. Acceptance rate depends on the verifier's confidence thresholds, the tree structure of draft tokens, and the specific decoding algorithm. A high step 0 accuracy is necessary but not sufficient for a high acceptance rate.
The training will complete in ~2 hours: The ETA of ~2 hours for remaining 4 epochs assumes consistent per-epoch timing (~30 min/epoch) based on the previous run. This assumes no hardware throttling, no memory bottlenecks, and no convergence issues that might trigger early stopping or additional computation.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of speculative decoding: Understanding that a "drafter" is a lightweight model that predicts multiple tokens ahead, which a "verifier" (the full model) then checks. The goal is to accelerate inference by generating multiple tokens per forward pass.
Knowledge of the EAGLE-3 architecture: EAGLE-3 is a specific speculative decoding method that uses a feature-level draft model rather than token-level. It operates on hidden states rather than token logits, which is why "hidden state extraction" is a critical data preparation step.
Knowledge of the previous failure: The "broken drafter that had 25% acceptance" refers to the earlier attempt ([msg 3470] context from segments 23-24) where an EAGLE-3 drafter trained on vLLM-extracted hidden states failed to provide any speedup. Understanding this failure is essential to appreciating why the new metrics are significant.
Knowledge of the infrastructure: The paths /data/eagle3/output_10k_sglang/ and the checkpoint structure (epoch directories 0, 1, 2, 3, 4) reflect the speculators library's checkpointing convention.
Understanding of the logging fix: Without knowing that the previous training run was producing no visible metrics ([msg 3453]-[msg 3468]), the message appears to be a routine status update rather than a breakthrough moment.

Output Knowledge Created

This message establishes several important facts:

The SGLang extraction pipeline works: The hidden states extracted via SGLang's server-side patch (Approach C, developed in segment 25) produce training data that yields meaningful learning. This validates the entire extraction approach.
The from-scratch training strategy is viable: Rather than finetuning an existing drafter (like AQ-MedAI's), training from scratch on 10K samples produces a drafter with dramatically better metrics. This suggests that the data quality from SGLang extraction is high enough to support training from random initialization.
The logging fix was successful: The logging.basicConfig() addition to the training script works, and per-step metrics are now visible. This enables monitoring and early intervention if training diverges.
The checkpoint resume mechanism works: The training successfully resumed from epoch 1 after being killed and restarted, confirming that the speculators library's checkpointing is functional.
A baseline for comparison: The metrics provide a quantitative baseline for future experiments. If subsequent training runs produce different numbers, the team can diagnose whether the data, model architecture, or hyperparameters need adjustment.

The Broader Context: Why This Matters

This message sits at the intersection of several converging threads in the conversation. The assistant had spent segments 20-25 exploring speculative decoding options for Kimi-K2.5, testing n-gram speculation (too slow), building an EAGLE-3 pipeline, patching API incompatibilities with vLLM 0.16, extracting hidden states, and pivoting to SGLang when vLLM's EAGLE-3 integration proved ineffective. The 25% acceptance rate of the previous drafter was a major setback — it meant that speculative decoding was actually slower than standard inference due to overhead.

The pivot to SGLang involved building a custom server-side patch to capture intermediate hidden states at layers [3, 31, 59] during prefill, saving them as binary .pt files. This was a non-trivial engineering effort that required understanding SGLang's internal architecture, the DeepSeekV2 model's forward pass, and the speculators library's data format.

The training itself is a 2.6B parameter model (1.19B trainable, 1.4B frozen) trained on 17.3M tokens of hidden states extracted from 10K samples. The fact that it's learning meaningful predictions from scratch validates the entire pipeline — from the NCCL-tuned SGLang server achieving 90 tok/s single-stream, through the hidden state extraction patch, to the training infrastructure.

The Thinking Process Visible in the Message

The assistant's reasoning in this message is concise but reveals several cognitive processes:

Comparative reasoning: The immediate instinct is to compare against the previous failure ("compared to the broken drafter that had 25% acceptance"). This shows the assistant is operating in a hypothesis-testing framework — each experiment is evaluated against the baseline.
Cautious optimism: The phrase "very promising" is qualified by the acknowledgment that metrics "vary by sample difficulty." The assistant avoids overclaiming success, recognizing that early epoch metrics can be misleading.
Forward planning: The ETA and checkpoint paths are provided not just as information, but as a plan — the assistant is signaling that the training will complete autonomously and the checkpoints will be available for the next step (testing the drafter in production).
Interpretation of metrics: The assistant translates raw numbers into meaningful conclusions ("The model is clearly learning meaningful predictions"). This requires understanding what each metric means for the downstream task of speculative decoding.

Conclusion

Message [msg 3470] is a moment of validation in a long and complex engineering effort. After hours of debugging build issues, patching API incompatibilities, developing custom extraction patches, and fixing silent logging failures, the assistant finally sees evidence that the approach is working. The metrics — 70-91% step 0 accuracy, 55-75% step 1 conditional accuracy, and 34-74% step 2 conditional accuracy — represent a dramatic improvement over the previous 25% acceptance rate.

But this is not the end of the story. The message is a checkpoint, not a conclusion. The training still has four epochs to run, and the real test will come when the trained drafter is deployed against the verifier model in production. The assistant's measured optimism reflects an understanding that early metrics, while promising, are not guarantees. The true measure of success will be whether this drafter can deliver the speedup that speculative decoding promises — turning the theoretical acceleration into real-world throughput gains for the Kimi-K2.5 model.