Chunk 24.0

In this chunk, the assistant resolved the apparent SGLang hang on SM120 by discovering that the server was actually running—it just took ~5–10 minutes to load the 547GB model. After benchmarking base SGLang with CUDA graphs enabled, the assistant achieved 63.6 tok/s single-stream and 2,370 tok/s peak throughput (C=128), significantly outperforming vLLM's peak of 1,536 tok/s but lagging in single-stream latency (63.6 vs 82.5 tok/s). The assistant then patched `kimi_k25.py` to add the three EAGLE-3 delegation methods (`set_eagle3_layers_to_capture`, `get_embed_and_head`, `set_embed_and_head`) and successfully launched SGLang with both the AQ-MedAI EAGLE-3 drafter (accept rate ~42%, no speedup) and the custom K2.5-trained drafter (accept rate 25%, effectively broken). Analysis showed that EAGLE-3 speculative decoding provided no benefit on this hardware due to low acceptance rates and SGLang's automatic `max_running_requests=48` limit for speculative mode. The assistant then pivoted to tuning SGLang's single-stream performance by applying the same NCCL environment variables vLLM used (`NCCL_PROTO=LL`, `NCCL_ALGO=Ring`, `NCCL_P2P_LEVEL=SYS`, `NCCL_MAX_NCHANNELS=16`, `NCCL_BUFFSIZE=16777216`, `NCCL_NTHREADS=512`) and launching with `--attention-backend flashinfer --num-continuous-decode-steps 4 --disable-custom-all-reduce`. Simultaneously, the assistant began planning a new EAGLE-3 training pipeline using SGLang-based hidden state extraction to ensure alignment between training data and inference hidden states. The user then directed to focus on tuning SGLang to match vLLM's single-stream speed and to retrain the EAGLE-3 drafter with 15K samples using SGLang extraction (reusing the existing inference data but re-extracting hidden states). The tuned SGLang server is currently loading, and the data pipeline for 15K samples is being prepared.

The EAGLE-3 Verdict and the Pivot to Performance Tuning: A Comprehensive Account of SGLang Optimization on Blackwell GPUs 2876 words

Message Articles

Subagent Sessions