The Moment It All Came Together: EAGLE-3 Training Begins on Kimi-K2.5

A Milestone Reached After Days of Pipeline Construction

In the sprawling, multi-day effort to deploy speculative decoding for the 1-trillion-parameter Kimi-K2.5 model on 8x Blackwell GPUs, message [msg 2983] represents a quiet but monumental turning point. After a marathon of infrastructure setup, synthetic data generation, hidden state extraction, and pipeline debugging, the assistant finally reports that EAGLE-3 training has started. The message is deceptively brief—a few bullet points and a bash command—but it encapsulates the successful completion of an extraordinarily complex pipeline that had consumed the better part of a week.

The Message Itself

The assistant writes:

Training started successfully: - 13 weight tensors loaded from AQ-MedAI checkpoint, 2 skipped (t2d, d2t) - 2,594.7M total params, 1,190.9M trainable - 9,000 train / 1,000 val files - 45,000 total steps (9,000 × 5 epochs) - 450 warmup steps

It then issues a bash command to check progress after a two-minute delay, and the returned log output confirms that weight loading, dataset preparation, and the training loop initialization all completed without error.

The Long Road to This Point

To understand why this message matters, one must appreciate the pipeline that preceded it. The assistant had been working for days across multiple segments (see [segment 18] through [segment 23]) to build a complete EAGLE-3 speculative decoding system for Kimi-K2.5, a model based on the DeepSeek V3 architecture with Multi-head Latent Attention (MLA). The pipeline involved:

  1. Synthetic data generation: Running the Kimi-K2.5 model itself to produce 10,000 reasoning traces, with careful handling of the thinking/ response token wrappers that the model uses to separate internal reasoning from final answers. The generation script had to be fixed multiple times to properly extract the reasoning field from the vLLM server responses.
  2. Hidden state extraction: Loading the full 547GB model across 8 GPUs with tensor parallelism and extracting the internal hidden states from layers [2, 30, 58, 60] for each of the 10,000 samples. This process took approximately 91 minutes (including 24.5 minutes for model loading) and produced 828 GB of training data at a throughput of 3,165 tokens per second.
  3. Vocab mapping: Building the mapping between the verifier's 163,840-token vocabulary and the drafter's 32,000-token vocabulary, since EAGLE-3 uses a smaller vocabulary for efficiency.
  4. Checkpoint acquisition: Obtaining the AQ-MedAI pre-trained drafter checkpoint as a starting point for fine-tuning, rather than training from scratch. Each of these steps had its own failure modes and required careful debugging. The hidden state extraction alone was a 1.5-hour process that the assistant monitored obsessively, checking progress every 10-20 minutes to ensure it hadn't stalled or crashed ([msg 2970] through [msg 2977]).## What the Training Log Reveals The log output that follows the assistant's message provides rich detail about the training configuration and the state of the model: Weight loading: The assistant loaded 13 weight tensors from the AQ-MedAI checkpoint while skipping 2 (t2d and d2t, the token-to-drafter and drafter-to-token embedding mappings). This is correct behavior—the vocab mapping tensors are specific to the verifier-drafter pair and must be recomputed from the actual vocab mapping data rather than loaded from a pre-trained checkpoint. The assistant's code handled this gracefully with a skip mechanism. Parameter counts: The model has 2,594.7M total parameters, of which 1,190.9M are trainable and 1,403.8M are frozen. This split reflects the EAGLE-3 architecture: the drafter's transformer layers are fine-tuned while the embedding and projection layers inherited from the verifier are kept frozen. The roughly 46% trainable / 54% frozen ratio is typical for EAGLE-3 fine-tuning, where the goal is to adapt the drafter to predict the verifier's hidden states without changing the fundamental representation space. Dataset split: 9,000 training files and 1,000 validation files, each file corresponding to one of the 10,000 synthetic samples. The 90/10 split is standard practice, though the assistant had configured a --val-ratio 0.1 flag to control this. Training schedule: 45,000 total steps (9,000 batches × 5 epochs) with 450 warmup steps (1% of total). The warmup ratio of 0.01 and cosine scheduler were explicitly configured via command-line flags (--scheduler cosine --warmup-ratio 0.01), reflecting a deliberate choice to use a learning rate schedule that starts low, ramps up, and then decays smoothly—a common approach for fine-tuning large models to avoid catastrophic forgetting in early steps.

Design Decisions and Their Rationale

Several implicit assumptions and decisions are baked into this training run:

Why fine-tune from AQ-MedAI rather than train from scratch? The AQ-MedAI checkpoint provides a pre-trained drafter that already understands the general structure of hidden state prediction. Fine-tuning from this base requires only 1.19B trainable parameters and 5 epochs, whereas training from scratch would require orders of magnitude more data and compute. This is a pragmatic engineering decision that leverages transfer learning.

Why 5 epochs on only 10,000 samples? The synthetic data is high-quality (generated by the target model itself) but limited in diversity. Training for multiple epochs on a small dataset risks overfitting, but the assistant presumably judged that the quality of the data and the regularization from the frozen weights would mitigate this. The --noise-std 0.05 flag (adding Gaussian noise during training) provides additional regularization.

Why batch size of 1 (implied by 9,000 batches for 9,000 files)? The training appears to process one sample per batch. This is typical for EAGLE-3 training where each sample has a variable sequence length and the hidden states are pre-computed. The --num-workers 4 flag enables parallel data loading to compensate.

What This Message Achieves

This message is the culmination of the entire EAGLE-3 pipeline. It represents the transition from data preparation to actual model training—the point at which all the infrastructure, scripts, and data finally converge into a running training loop. The assistant's tone is matter-of-fact, but the context makes clear that this is a hard-won victory.

The message also serves as a checkpoint for the human user: it provides visibility into the training configuration and confirms that the pipeline is executing correctly. The subsequent log output validates that weights loaded properly, the dataset was found and split correctly, and the training loop initialized without errors.

The Broader Context

This training run was destined to reveal a fundamental limitation: when the trained drafter was later tested with vLLM's EAGLE-3 integration, it achieved only ~15% acceptance rate and 0.66× throughput—worse than no speculation at all ([chunk 23.0]). This led to a pivot to SGLang, which had first-class EAGLE-3 support but then encountered SM120 deadlock issues. The training itself was not the problem; the issue was in the inference engine's integration with MLA attention.

But at this moment, captured in message [msg 2983], none of that was known. The training was starting successfully, the pipeline was working end-to-end, and the assistant had accomplished something genuinely impressive: building a complete EAGLE-3 training pipeline from scratch for a 1T-parameter model on a novel GPU architecture. The message is a testament to the complexity of modern ML engineering and the satisfaction of seeing a multi-stage pipeline finally converge.