Segment 25
The assistant tuned SGLang's single-stream performance to 90.0 tok/s (surpassing vLLM's 82.5 tok/s) by applying NCCL environment variables and --num-continuous-decode-steps 4, using triton attention after flashinfer caused hangs on SM120. For the EAGLE-3 pipeline, a non-invasive server-side patch was developed to capture intermediate hidden states at layers [3,31,59] during prefill, saving them as .pt files to /dev/shm/. The full 10K-sample extraction completed successfully, producing 17.3M tokens (924 GB) in speculators v1 format, and the old vLLM-extracted hidden states were deleted. Training a new EAGLE-3 drafter from scratch (not finetuned from AQ-MedAI) began with 32K draft vocab and 5 epochs, showing healthy metrics (~74% step 0 accuracy, ~64% step 1 conditional accuracy, ~56% step 2 conditional accuracy) across the first three epochs — dramatically better than the previous broken drafter's 25% acceptance rate. Logging issues were fixed and loss/accuracy charts were generated.
From 90 Tok/s to EAGLE-3: Building a Complete Speculative Decoding Pipeline for Kimi-K2.5