Segment 23
In this sub-session, the assistant completed the entire EAGLE-3 training pipeline: fixing the synthetic data generation script to properly wrap reasoning with think tokens, running a successful 10K inference run in ~5.3 hours, extracting 828 GB of hidden states at 3,165 tok/s, and finetuning from the AQ-MedAI checkpoint in 2.6 hours. However, testing with vLLM EAGLE-3 revealed a fundamental integration issue with MLA attention—both the trained drafter and the pre-trained baseline achieved only ~15% acceptance rate, resulting in 0.66x throughput (worse than no speculation). The user directed pivoting to SGLang, which has first-class EAGLE-3 support. The assistant built sgl-kernel for SM120 (48 minutes) and verified it works; SGLang loads the 547GB model in just 22 seconds (vs 25 minutes in vLLM), but the server processes deadlock after weight loading with zero CPU/GPU utilization. Debugging is ongoing with verbose logging and CUDA graph disabling to resolve this SM120 compatibility issue.
The EAGLE-3 Integration Odyssey: From Training Triumph to Framework Pivot on Blackwell GPUs