Chunk 30.0

## Summary This chunk completed the EAGLE-3 training pipeline from data preparation through deployment. After a VM crash and disk migration, the hidden state extraction was restarted and completed successfully, producing 37,312 samples (87.8M tokens, ~4.6 TB) with zero errors. Training was launched on 4 GPUs using torchrun with TTT=5, batch_size=8, and max_seq_len=8192, achieving ~100% GPU utilization at 350-400W power draw. The training completed 5 epochs in ~10.8 hours, converging to a final validation accuracy of 74.7% (full_acc_0) and an estimated acceptance length of ~2.95 tokens — a significant improvement over the previous 10K drafter's 2.1. Key technical challenges were addressed along the way: fixing the Triton shared-memory OOM at seq_len=16384 by reducing to 12288 (then ultimately using batch_size=8 packing at 8192), correcting the SGLang server argument names for speculative decoding (`--speculative-num-draft-tokens` instead of `--num-speculative-tokens`, and requiring `--speculative-num-steps`), and applying the weight key fix (`layers.0` → `midlayer`) for SGLang compatibility. The drafter checkpoint at `/data/eagle3/output_100k_sglang/4/` was prepared with vLLM-compatible config. The chunk concluded with the SGLang server deployed with EAGLE3 speculation at 16 draft tokens, ready for benchmarking. The overarching theme was the transition from data generation and extraction to training and deployment, with a focus on maximizing GPU utilization through proper batch sizing and packing, and preparing for the inference speedup that amortizes PCIe communication costs through deeper speculative decoding.

The Long March: From Data to Deployment in the EAGLE-3 Training Pipeline 2504 words

Message Articles