The Training Launch: Scaling EAGLE-3 from 10 Samples to 1,000

On the surface, message 2823 appears to be a routine command: a bash invocation that launches a Python training script on a remote server, dispatched to run in the background with nohup. But this single line represents the culmination of an intense, multi-hour engineering effort to build a complete EAGLE-3 speculative decoding training pipeline for the Kimi-K2.5 model — a 1-trillion-parameter Mixture-of-Experts architecture deployed across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The message is the moment when all the pieces finally click into place, and the assistant takes the leap from validating the pipeline on a toy dataset of 10 samples to running it at meaningful scale on 1,000 samples.

The Message

ssh root@10.1.230.174 'CUDA_VISIBLE_DEVICES=0 nohup /root/ml-env/bin/python3 /root/eagle3-train/04_train.py \
    --verifier-path /shared/kimi-k2.5-int4 \
    --data-dir /root/eagle3-train/data_1k/hidden_states/rows_0-2000 \
    --vocab-mapping-dir /root/eagle3-train/data_1k/vocab_mapping \
    --output-dir /root/eagle3-train/output_1k \
    --epochs 10 \
    --lr 3e-5 \
    --max-seq-len 2048 \
    --ttt-steps 3 \
    --val-ratio 0.05 \
    --noise-std 0.05 \
    --scheduler cosine \
    --num-workers 4 \
    > /root/eagle3-train/train_1k.log 2>&1 &
echo "PID: $!"'
PID: 269976

Why This Message Was Written: The Culmination of a Pipeline

To understand the significance of this message, one must trace the chain of events that led to it. The assistant had been working for hours — across multiple segments of the conversation — to set up EAGLE-3 training for the Kimi-K2.5 model. EAGLE-3 is a speculative decoding technique that trains a lightweight "draft" model to predict the hidden states of the large "verifier" model, enabling faster autoregressive generation through speculation.

The journey had been arduous. The assistant had to:

  1. Explore the speculators library's training API — understanding how Eagle3SpeculatorConfig, Eagle3DraftModel, and the built-in Trainer class work together, and monkey-patching verifier weight extraction for Kimi-K2.5's nested config structure.
  2. Write and rewrite 04_train.py — the training script itself, which had to properly instantiate the speculator components, handle data loading from extracted hidden states, and manage the training loop.
  3. Validate on 10 samples — a critical sanity check that the pipeline actually runs end-to-end. This completed in about 1 minute for 3 epochs, confirming the basic mechanics were correct.
  4. Scale up data preparation — running 01_prepare_dataset.py to download and tokenize 1,000 samples from the mlabonne/open-perfectblend dataset, producing 503,000 total tokens (360,421 assistant tokens).
  5. Build vocabulary mapping — creating the t2d (target-to-draft) and d2t (draft-to-target) mappings that bridge the full 163,840-token vocabulary of Kimi-K2.5 to the 32,000-token draft vocabulary.
  6. Extract hidden states at scale — the most time-consuming step: loading the 547 GB Kimi-K2.5 INT4 model across all 8 GPUs (22.5 minutes), then extracting hidden states from layers 2, 30, 58, and 60 for all 1,000 samples (2.9 minutes at 2,912 tokens/second). This produced 27 GB of hidden state data.
  7. Free GPUs and copy the updated script — cleaning up after the extraction process and ensuring the latest version of 04_train.py was on the remote machine. Message 2823 is the next logical step: actually run the training. After all the preparation, the assistant is finally ready to train the draft model on the full 1,000-sample dataset. The message is a launch command — a signal that the pipeline is complete and the assistant is moving from preparation to execution.## How Decisions Were Made Every parameter in this command reflects a deliberate decision, informed by the assistant's deep understanding of the model architecture, hardware constraints, and training dynamics. CUDA_VISIBLE_DEVICES=0: This is a critical choice. The assistant explicitly restricts training to a single GPU (GPU 0), despite having 8 GPUs available. This decision stems from the nature of EAGLE-3 draft model training: the draft model is tiny (1 layer, 7168 hidden size) compared to the verifier (60 layers, Mixture-of-Experts). Training such a small model on a single GPU is more efficient than coordinating across multiple GPUs, which would add communication overhead. The other 7 GPUs remain free for other tasks — a practical consideration in a shared multi-tenant environment. --epochs 10: The assistant chose 10 epochs based on the reasoning that with ~1,000 batches per epoch (due to sequence packing at max-seq-len 2048), 10 epochs yields ~10,000 training steps. This is a reasonable starting point for a local test run — enough to see convergence trends without overcommitting compute time. --lr 3e-5: A conservative learning rate, typical for fine-tuning transformer-based models. The assistant likely chose this based on standard practices for speculative decoding training, where the draft model needs to learn the mapping from verifier hidden states to token predictions without destabilizing the learned representations. --ttt-steps 3: This parameter controls the number of "text-to-text" (teacher-forcing) steps during training. A value of 3 is moderate, balancing between learning from the verifier's hidden states and learning to generate autonomously. --val-ratio 0.05: 5% of the 1,000 samples (50 samples) are held out for validation. This is a small but reasonable validation set for monitoring overfitting. --noise-std 0.05: Adding Gaussian noise with standard deviation 0.05 to the hidden states during training. This is a regularization technique that improves robustness — the draft model must learn to work with slightly perturbed inputs, mimicking the distribution shift it will encounter during actual speculative decoding. --scheduler cosine: A cosine learning rate schedule, which gradually reduces the learning rate following a cosine curve. This is a well-established choice for transformer training, providing a smooth decay that helps convergence. --num-workers 4: Four data-loading worker processes, balancing I/O throughput against memory consumption. With 27 GB of hidden state data on disk, parallel loading is beneficial.

Assumptions Made

The message rests on several assumptions, some explicit and some implicit:

  1. The training script is correct and complete. The assistant had just rewritten 04_train.py to use the speculators library's proper API, but the 1,000-sample run would be the first real test at scale. There was an implicit assumption that the script would handle the larger dataset without issues — no memory leaks, no data-loading bugs, no numerical instabilities.
  2. The hidden state data is correctly formatted. The extraction process produced 27 GB of data across 1,000 files. The assistant assumed the data format matched what 04_train.py expected, including the correct tensor shapes, layer ordering, and metadata.
  3. The vocabulary mapping is correct. The t2d and d2t mappings were built from token frequency statistics on the same dataset. The assistant assumed these mappings would generalize correctly — that the 21,270 unique assistant tokens seen in the 1,000 samples are representative of the full distribution.
  4. GPU 0 has sufficient memory. The draft model is small (1 layer), but training also requires loading the verifier model's configuration and potentially some verifier weights for loss computation. The assistant assumed that a single RTX PRO 6000 Blackwell GPU (96 GB) would be sufficient.
  5. The background process will complete without intervention. By using nohup and redirecting output to a log file, the assistant assumed the training would run to completion without requiring interactive monitoring or debugging.

Input Knowledge Required

To understand this message fully, one needs knowledge spanning several domains:

Output Knowledge Created

This message produces several forms of knowledge:

  1. A trained EAGLE-3 draft model checkpoint — saved to /root/eagle3-train/output_1k/, containing the draft model weights in a vLLM-compatible format. This checkpoint can be loaded into vLLM for speculative decoding inference.
  2. Training metrics and logs — captured in /root/eagle3-train/train_1k.log, providing per-step loss values, validation metrics, and timing information. These metrics inform future hyperparameter tuning.
  3. Validation of the pipeline at scale — successfully running on 1,000 samples confirms that the entire data preparation, extraction, and training pipeline works end-to-end. This is a significant milestone, as many subtle bugs only manifest at larger scales.
  4. Baseline performance data — the training time (later revealed to be 27.7 minutes for 10 epochs at 6 steps/second) establishes a baseline for future optimization efforts.

The Thinking Process Behind the Message

The assistant's reasoning is visible in the surrounding messages. In [msg 2822], the assistant explicitly thinks through the hyperparameter choices:

"For a 1000-sample training run, let me think about the hyperparameters: - 1000 samples, with packed sequences at max_seq_len=2048, each batch packs ~1-4 samples - With batch_size=1 (packing, dataloader batches), we get ~1000 batches per epoch - 10 epochs = 10,000 steps, which is reasonable for a quick local test - LR = 3e-5 with cosine schedule"

This reveals the assistant's mental model: it's thinking in terms of steps and epochs, balancing training thoroughness against time. The "quick local test" framing is telling — the assistant views this as a validation run, not a production training session. The goal is to verify the pipeline works at scale and produce a usable checkpoint, not to achieve state-of-the-art results.

The choice of CUDA_VISIBLE_DEVICES=0 is also significant. The assistant had just freed all 8 GPUs from the extraction process. Rather than using all available resources, it deliberately constrains training to one GPU. This reflects an understanding that the draft model is small enough that multi-GPU training would add complexity without benefit, and that leaving GPUs free allows other work (like the synthetic data generation that follows in the next chunk).

Mistakes and Incorrect Assumptions

The most notable assumption that proved incorrect was the batch size during hidden state extraction. In [msg 2809], the assistant launched extraction with --batch-size 2000, attempting to process all 1,000 samples as a single batch. This caused an OOM error ([msg 2813]) because the prefill of 503,000 tokens at once exceeded GPU memory. The assistant had to kill the process and re-run with --batch-size 4 ([msg 2816]), which succeeded.

This mistake stemmed from a misunderstanding of the --batch-size parameter: the assistant initially interpreted it as a token-level batch size (matching the extraction rate of ~2,900 tok/s), but it actually controlled the number of samples processed per batch. With samples up to 2,048 tokens each, a batch of 2,000 samples could require prefilling up to 4 million tokens — far exceeding the available 96 GB per GPU after loading the 547 GB model across 8 GPUs.

This error was a valuable learning moment. It demonstrated that even with careful planning, parameter interpretation can lead to resource exhaustion. The assistant's response was methodical: kill the process, free GPU memory, adjust the parameter based on the known working configuration (batch_size=4 worked for 10 samples), and re-launch. This systematic debugging approach is characteristic of the entire session.

Conclusion

Message 2823 is a deceptively simple command that represents the culmination of hours of engineering work. It is the moment when a complex, multi-stage pipeline — spanning data preparation, vocabulary construction, large-model hidden state extraction, and training script development — finally comes together to produce a tangible result. The careful parameter choices reflect deep understanding of the model architecture, hardware constraints, and training dynamics. And the message's placement in the conversation — immediately after scaling from 10 to 1,000 samples — marks a critical transition from validation to production-scale execution. In the broader narrative of the session, this message is the pivot point: the point at which the assistant proves the EAGLE-3 pipeline works and can turn attention to improving training data quality, which becomes the focus of the next chunk.