The Moment of Completion: EAGLE-3 Training Culminates on 1000 Samples

In the sprawling, multi-session journey to deploy speculative decoding for the 1-trillion-parameter Kimi-K2.5 model on eight Blackwell GPUs, message [msg 2828] marks a quiet but pivotal milestone. It is the moment when the assistant, after hours of pipeline construction, debugging, and waiting, finally sees the EAGLE-3 draft model training complete on 1000 samples. The message itself is deceptively brief — a simple check of a remote log file — but it represents the successful culmination of an enormous technical pipeline spanning dataset preparation, multi-GPU hidden state extraction, vocabulary mapping, and distributed training orchestration. Understanding why this message was written, what it reveals about the assistant's reasoning, and what it enables next requires unpacking the entire chain of decisions that led to this point.

The Message in Full

The subject message reads:

[assistant] 8 epochs done, 2 more to go. Let me wait a few more minutes: [bash] sleep 420 && ssh root@10.1.230.174 'tail -15 /root/eagle3-train/train_1k.log 2>/dev/null' Epoch 9 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 950/950 [ 0:02:34 < 0:00:00 , 6 it/s ] Epoch 9 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50/50 [ 0:00:03 < 0:00:00 , 16 it/s ]

>

Training complete in 27.7 min Checkpoints saved to: /root/eagle3-train/output_1k

>

Final model at: /root/eagle3-train/output_1k/9 config.json: 0.0 MB config.py: 0.0 MB generation_config.json: 0.0 MB model.safetensors: 47...

The assistant executes a single bash command: sleep for 420 seconds (7 minutes), then tail the training log. The output confirms that all 10 epochs completed, the final checkpoint was saved, and the model weights occupy 47 GB on disk. The tone is understated, but the achievement is substantial.

The Reasoning Behind the Wait

The assistant's decision to wait exactly 420 seconds before checking the log is not arbitrary. It reflects a careful mental model of the training dynamics built over the preceding messages. In [msg 2827], the assistant had observed that "epochs are taking ~2.5 min each (950 steps). At 6 it/s that's reasonable." With 8 epochs already done and 2 remaining, the expected remaining time was approximately 5 minutes. Adding a buffer for validation epochs (50 batches at 16 it/s, taking ~3 seconds each) and potential JIT compilation overhead, the 7-minute wait was a conservative estimate designed to catch the tail end of training without repeatedly polling the remote machine.

This patience is characteristic of the assistant's approach throughout the session. Rather than hammering the server with status checks every few seconds, it calculates expected durations and sets sleep timers accordingly — a form of computational politeness that also reduces log noise and context window consumption. The assistant is implicitly modeling the remote machine's behavior and scheduling its attention to align with natural completion boundaries.

The Pipeline That Led Here

To appreciate what "Training complete in 27.7 min" actually means, one must trace the pipeline that preceded it. The EAGLE-3 training workflow for Kimi-K2.5 involves four major steps, each with its own failure modes and optimizations:

Step 1 — Dataset Preparation ([msg 2801]): The assistant ran 01_prepare_dataset.py against the mlabonne/open-perfectblend dataset, tokenizing 1000 conversations with the Kimi-K2.5 tokenizer. This produced 503,000 total tokens (360,421 assistant tokens) across 1000 samples, with 58 samples skipped due to length or formatting issues. The output metadata revealed a vocabulary of 163,840 tokens with only 21,270 unique assistant tokens — a sparsity pattern that would later inform the draft model's vocabulary mapping.

Step 2 — Hidden State Extraction ([msg 2805]–[msg 2819]): This was the most technically fraught step. The assistant initially launched extraction with --batch-size 2000, intending to process all 1000 samples as a single batch. This caused an out-of-memory (OOM) error ([msg 2813]): the model with TP=8 already consumed ~91 GB per GPU, leaving no room for the massive prefill. The assistant diagnosed the issue, killed the processes, freed GPU memory, and relaunched with --batch-size 4 and --gpu-memory-utilization 0.85. The corrected run took 22.5 minutes for model loading plus 2.9 minutes for extraction at 2912 tok/s, producing 27 GB of hidden state data across 4 captured layers (layers 2, 30, 58, and 60).

Step 3 — Vocabulary Mapping ([msg 2803]): The assistant built a mapping between the verifier's 163,840-token vocabulary and the draft model's 32,000-token vocabulary, achieving 100% coverage of observed tokens. This mapping is essential because the draft model operates in a compressed token space while needing to predict verifier tokens.

Step 4 — Training ([msg 2823]–[msg 2828]): The assistant launched training with 10 epochs, learning rate 3e-5, cosine scheduler, 3 TTT (test-time training) steps, and noise standard deviation 0.05. The training ran on a single GPU (CUDA_VISIBLE_DEVICES=0) with the draft model's 2.5B parameters fitting comfortably in 17.7 GB of GPU memory. The 9500 total steps completed at 6 steps/second, yielding a final 47 GB checkpoint.

Key Decisions and Their Rationale

Several design decisions embedded in this training run deserve scrutiny:

Why 10 epochs? The assistant's earlier reasoning (visible in [msg 2822]) was that 10 epochs × ~1000 batches = 10,000 steps, which is "reasonable for a quick local test." This reflects a pragmatic trade-off: enough training to validate the pipeline works end-to-end, but not so many epochs that it wastes GPU hours on a test run destined for the scrap heap once the "hero run" on B300 hardware begins.

Why learning rate 3e-5? This is a standard starting point for fine-tuning transformer-based draft models, balancing convergence speed against stability. The cosine scheduler with 95 warmup steps (1% of total) provides a gentle ramp-up to avoid early divergence.

Why TTT steps = 3? The EAGLE-3 architecture uses test-time training, where the draft model can take additional gradient steps during inference to adapt to the verifier's hidden states. Three TTT steps is a conservative starting point; Baseten's implementation reportedly uses 1-3 steps.

Why noise std = 0.05? Adding Gaussian noise to hidden states during training is a regularization technique that makes the draft model robust to small variations in the verifier's hidden state distribution. The 0.05 value is moderate — enough to provide regularization without corrupting the signal.

Assumptions and Potential Mistakes

The assistant made several assumptions in this run that could affect the quality of the trained draft model:

Assumption 1: The open-perfectblend dataset is a good proxy for Kimi-K2.5's actual reasoning distribution. The dataset contains general-purpose conversation data, but the draft model needs to predict the specific thinking patterns of Kimi-K2.5 (its &lt;think&gt; blocks, chain-of-thought reasoning, and response structure). The assistant implicitly acknowledges this limitation in the very next user message ([msg 2840]), where the user redirects toward generating synthetic data from the model's own outputs.

Assumption 2: 1000 samples is sufficient for a meaningful training signal. With 360K assistant tokens across 10 epochs, the model sees 3.6M tokens total — a tiny fraction of the training data Baseten likely used (hundreds of thousands of samples). The resulting draft model may be undertrained, but the goal was pipeline validation, not production quality.

Assumption 3: The speculators library's Trainer class handles all edge cases correctly. The assistant had to monkey-patch the verifier weight extraction for Kimi-K2.5's nested config structure (see [msg 2839]), and the training relied on several internal APIs that could change between library versions.

Potential mistake: No validation metrics were captured. The speculators Trainer uses progress bars rather than logging per-epoch metrics to stdout, so the assistant had no visibility into training loss, validation loss, or acceptance rate trends. The checkpoint could be overfit or underfit without any quantitative signal.

The Significance of This Moment

Message [msg 2828] represents the first time the complete EAGLE-3 training pipeline ran end-to-end on a non-trivial dataset. Before this point, the pipeline had only been validated on 10 samples ([msg 2827]). Scaling from 10 to 1000 samples exposed the OOM bug in hidden state extraction, validated that the vocabulary mapping scales linearly, and confirmed that the training loop can sustain 6 steps/s over 27 minutes without divergence.

The timing data collected here is invaluable for planning the "hero run" on B300 hardware. The assistant now knows:

What Comes Next

The completion of training triggers an immediate pivot. In the very next user message ([msg 2840]), the user redirects the approach: "to capture k2.5 thinking (I think we need to) lets do: from open-perfectblend, on just vllm infer every question, capture thinking and output." The assistant immediately recognizes the wisdom in this — the draft model needs to learn the model's actual reasoning patterns, not generic conversation structure. It writes 01b_generate_synthetic.py ([msg 2843]), starts the vLLM server ([msg 2850]), and launches a new pipeline that feeds each question independently through the inference engine.

The 1000-sample checkpoint at /root/eagle3-train/output_1k/9/ is not discarded — it serves as a baseline and a proof of concept. But the synthetic data pipeline represents a fundamentally better approach to training data quality, and the assistant's willingness to pivot immediately demonstrates a healthy skepticism about the earlier assumptions.

Conclusion

Message [msg 2828] is a milestone marker in a much longer journey. It captures the moment when a complex, multi-stage pipeline — spanning dataset curation, distributed hidden state extraction, vocabulary engineering, and neural network training — finally produces its first meaningful output. The assistant's measured patience (the calculated 420-second wait), its diagnostic response to the OOM failure, and its ability to contextualize the results within the broader project goals all reflect a sophisticated understanding of the system being built. The training completion is not an end but a beginning: it validates the pipeline architecture, provides critical timing data, and sets the stage for the higher-quality synthetic data approach that follows. In the arc of the EAGLE-3 project, this message is the quiet "it works" that every engineer knows is the most satisfying sound of all.