Chunk 37.0

This session focused on parallel throughput benchmarking and the failed attempt to implement dynamic speculation disable. The assistant ran a comprehensive comparison between the EAGLE-3 speculative decoding server and the baseline (no speculation) server using coding/agentic prompts. The critical finding was that **the baseline strictly outperformed EAGLE-3 in total throughput at every concurrency level**, saturating at ~773 tok/s compared to EAGLE-3's ~354 tok/s. This definitively showed that EAGLE-3's value is limited to marginal per-request latency gains at very low concurrency (C=1), and it becomes a significant liability for throughput under load, with the gap widening to over 2x at high concurrency. The assistant then attempted to implement a dynamic speculation disable mechanism to automatically switch between modes based on server load. This was attempted on the standard non-overlap `EAGLEWorker` (v1) path, but ran into fundamental issues with deeply coupled batch state management (e.g., `out_cache_loc` pre-allocated for draft token dimensions, CUDA graph shape expectations). The effort was abandoned in favor of investigating the `spec_v2` overlap path (`EAGLEWorkerV2`), which has a cleaner separation of concerns. However, spec_v2 requires `topk=1`, which significantly reduces the draft tree size from 16 tokens to 3 tokens. The session concluded with the assistant starting a server with `topk=1` and `SGLANG_ENABLE_SPEC_V2=True` to test the viability of this path for potential future dynamic disable. The user also asked about a related PR (#15623) for overlapped constrained decoding, which was found to be not directly relevant to the current task. The benchmarks were also updated to use coding/agentic prompts to better match the EAGLE-3 drafter's training data, though this did not significantly change the results.

The Speculation That Couldn't: EAGLE-3's Throughput Collapse and the Quest for Dynamic Disable 2849 words

Message Articles

Subagent Sessions