Segment 37
This sub-session conducted a comprehensive parallel throughput comparison between the EAGLE-3 speculative decoding server and the baseline server using coding/agentic prompts. The critical finding was that the baseline strictly outperformed EAGLE-3 in total throughput at every concurrency level, saturating at ~773 tok/s compared to EAGLE-3's ~354 tok/s, definitively showing EAGLE-3's value is limited to marginal per-request latency gains at very low concurrency (C=1). The assistant then attempted to implement a dynamic speculation disable mechanism to automatically switch between modes based on server load. This was attempted on the standard non-overlap EAGLEWorker (v1) path, but ran into fundamental issues with deeply coupled batch state management (e.g., out_cache_loc pre-allocated for draft token dimensions, CUDA graph shape expectations). The effort was abandoned in favor of investigating the spec_v2 overlap path (EAGLEWorkerV2), which has a cleaner separation of concerns but requires topk=1, significantly reducing the draft tree size. The session concluded with the assistant starting a server with topk=1 and SGLANG_ENABLE_SPEC_V2=True to test the viability of this path for potential future dynamic disable.