Chunk 25.0

## Summary The assistant successfully tuned SGLang's single-stream performance to **90.0 tok/s**, surpassing vLLM's 82.5 tok/s by applying NCCL environment variables (`NCCL_PROTO=LL`, `NCCL_ALGO=Ring`, etc.) and `--num-continuous-decode-steps 4`. The initial attempt with `--attention-backend flashinfer` caused the server to hang on SM120, so the working configuration used triton attention (the default for DeepSeek on SM120) combined with NCCL tuning. This benchmark was validated before proceeding to the EAGLE-3 pipeline. For hidden state extraction, the assistant developed a **non-invasive server-side patch** (Approach C) that captures intermediate hidden states at layers [3, 31, 59] during prefill and saves them as binary `.pt` files to `/dev/shm/`. The server was launched with `--disable-cuda-graph` and `--disable-radix-cache` to ensure correct extraction. The full 10K-sample extraction completed successfully, producing 17.3M tokens of hidden states (924 GB) in the speculators v1 format with zero errors. The old vLLM-extracted hidden states (828 GB) were deleted to free space. Training the new EAGLE-3 drafter **from scratch** (not finetuned from AQ-MedAI) began with 32K draft vocab, 5 epochs, and the SGLang-extracted hidden states. Initial logging issues were fixed by adding a logging handler, and training resumed from epoch 1. Visible metrics show healthy learning: ~74% step 0 accuracy, ~64% step 1 conditional accuracy, and ~56% step 2 conditional accuracy across the first three epochs — dramatically better than the previous broken drafter's 25% acceptance rate. Loss/accuracy charts were generated and saved to `./train-progress/`.

From 90 Tok/s to EAGLE-3: Engineering a Speculative Decoding Pipeline Across a Trillion-Parameter Model 4107 words

Message Articles