Segment 24
In this sub-session, the assistant first resolved the apparent SGLang hang on SM120 by discovering the server was actually loading the model for 5-10 minutes. After benchmarking base SGLang with CUDA graphs, it achieved 63.6 tok/s single-stream and 2,370 tok/s peak throughput, significantly outperforming vLLM's peak but lagging in single-stream latency. The assistant then patched kimi_k25.py to add EAGLE-3 delegation methods and tested both the AQ-MedAI drafter (42% acceptance, no speedup) and the custom K2.5-trained drafter (25% acceptance, effectively broken), concluding that EAGLE-3 speculative decoding provided no benefit on this hardware due to low acceptance rates and SGLang's automatic max_running_requests limit for speculative mode. The assistant then pivoted to tuning SGLang's single-stream performance by applying NCCL environment variables and attention backend settings, while simultaneously planning a new EAGLE-3 training pipeline using SGLang-based hidden state extraction to ensure alignment. The user directed to focus on matching vLLM's single-stream speed and retrain the EAGLE-3 drafter with 15K samples using SGLang extraction. The tuned SGLang server began loading, and the data pipeline for 15K samples was being prepared.
The EAGLE-3 Verdict and the Strategic Pivot: SGLang Performance Tuning on 8× Blackwell GPUs