The Verdict on Speculative Decoding: Documenting a Definitive Benchmark

In the course of an intensive optimization session for the GLM-5-NVFP4 model deployed on an 8×RTX PRO 6000 Blackwell GPU system, the assistant reached a critical inflection point. Message <msg id=5586> is deceptively simple — a single line confirming that an edit was applied to a documentation file. Yet this message represents the culmination of hours of benchmarking, debugging, and architectural analysis, and it marks the moment when a definitive verdict was recorded about the viability of EAGLE-3 speculative decoding for this deployment.

The Context: A Long Optimization Journey

To understand the significance of this message, one must trace back through the session's trajectory. The assistant had been working on deploying the Kimi K2.5 model with EAGLE-3 speculative decoding — a technique where a lightweight "drafter" model predicts multiple future tokens in parallel, allowing the main model to verify them in a single forward pass, theoretically reducing per-request latency. Earlier in the session ([msg 5573]), the assistant had successfully tuned EAGLE-3 to achieve 96.1 tok/s single-stream throughput, a net-positive result after extensive CUDA stack upgrades and Blackwell-specific optimizations.

However, a critical question remained unanswered: how does EAGLE-3 behave under concurrent load? Speculative decoding is known to be latency-optimizing at the expense of throughput — the draft generation and verification steps consume GPU cycles that could otherwise be used to serve more requests. The assistant needed to quantify this tradeoff for the specific hardware configuration.

The Benchmarking Campaign

The assistant designed a comprehensive parallel throughput benchmark comparing two server configurations: one with EAGLE-3 speculation enabled (--speculative-algorithm EAGLE3 --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16) and one pure baseline (no speculation). Both used identical settings for tensor parallelism (TP=8), attention backend (FlashInfer), and allreduce fusion. The benchmark tested concurrency levels from 1 to 250 concurrent requests, measuring total throughput in tokens per second.

A key methodological improvement was made: the assistant replaced the original encyclopedic prompts with coding and agentic tasks (code generation, debugging, SWE-bench-style problems, system design) that better matched the EAGLE-3 drafter's training data distribution ([msg 5577]). This ensured the benchmark reflected the drafter's optimal performance rather than penalizing it for out-of-distribution inputs.

The Results: Baseline Dominates

The results were stark. At C=1 (single concurrent request), EAGLE-3 achieved 80.9 tok/s compared to the baseline's 92.7 tok/s — the baseline was already 15% faster even at the lowest concurrency. As concurrency increased, the gap widened dramatically. EAGLE-3 saturated at approximately 354 tok/s, while the baseline continued scaling to 773 tok/s — a 2.2× advantage at saturation ([msg 5583], [msg 5584]).

This finding was definitive: the baseline strictly outperformed EAGLE-3 at every concurrency level for total throughput. The speculative decoding overhead — generating draft tokens, running the verify forward pass, and managing the complex batch state — consumed GPU cycles that could have been used to serve more requests directly. The only scenario where EAGLE-3 might have value was marginal per-request latency gains at very low concurrency, but even there the baseline was faster in aggregate throughput.

The Failed Dynamic Disable Attempt

Before running these final benchmarks, the assistant had attempted an ambitious engineering intervention: implementing a dynamic speculation disable mechanism that would automatically switch between EAGLE-3 and baseline modes based on server load ([msg 5562]-[msg 5573]). The idea was to use EAGLE-3 when concurrency was low (benefiting from latency reduction) and fall back to pure decode when the server was under load.

This attempt ran into fundamental architectural obstacles. The EAGLE worker's batch state management was deeply coupled with the speculative pipeline. The out_cache_loc tensor, which tracks KV cache slot assignments, was pre-allocated for draft token dimensions (e.g., 10 requests × 16 draft tokens = 160 slots). When the assistant tried to bypass speculation and run a normal decode forward, the CUDA graph runner encountered a shape mismatch: tensor of size 160 vs expected size 2 ([msg 5568]).

The assistant traced this to the scheduler's prepare_for_decode method, which for speculative batches returns early without allocating the standard 1-token-per-request out_cache_loc ([msg 5570]-[msg 5572]). The allocation is deferred to the EAGLE worker's _draft_preprocess_decode, which allocates for the full draft tree. Replicating the normal decode path would require manually resetting out_cache_loc, incrementing decode_batch_idx, kv_committed_len, and kv_allocated_len per request, updating sequence lengths, and setting input_ids — all while avoiding conflicts with the CUDA graph's shape expectations.

The assistant correctly recognized this as "complex and error-prone" ([msg 5573]) and abandoned the approach, restoring the original eagle_worker.py from backup ([msg 5574]). This was a pragmatic decision: the dynamic switching mechanism would require a fundamental refactoring of SGLang's speculative decoding architecture, far beyond a simple patch.

The Subject Message: Recording the Verdict

Message <msg id=5586> is the edit that writes these findings into the project's documentation file, eagle-fast-verify.md. The file already contained earlier optimization results (the 96.1 tok/s single-stream achievement), and the assistant had read its current state in the preceding message ([msg 5585]) to prepare the update.

The edit itself is not shown in the message — the tool call result simply confirms "Edit applied successfully." But from the surrounding context, we know the update included the full parallel throughput comparison, the key finding that baseline wins at every concurrency level, and the documentation of the failed dynamic disable attempt. The assistant's subsequent summary message ([msg 5588]) confirms that "All results documented" in eagle-fast-verify.md.

Assumptions and Lessons Learned

Several assumptions were tested and found incorrect during this process:

  1. EAGLE-3 would be beneficial at moderate concurrency. The assistant initially believed that EAGLE-3's advantage at low concurrency would extend to C=2 or C=5, where the speculative overhead might be amortized. The data showed otherwise — the baseline was superior even at C=1.
  2. Dynamic switching could be implemented as a simple patch. The deep coupling of batch state management with the speculative pipeline made this far more complex than anticipated. The out_cache_loc tensor size, CUDA graph shape expectations, and the scheduler's early-return path in prepare_for_decode all conspired to defeat the naive approach.
  3. Coding prompts would significantly change the results. The assistant hypothesized that matching prompts to the drafter's training distribution might improve EAGLE-3's acceptance rate and thus its throughput. The actual improvement was marginal (~4% at C=1), insufficient to change the overall conclusion.

Input and Output Knowledge

The input knowledge required to understand this message includes: speculative decoding architecture (draft models, verification, acceptance rates), CUDA graph execution and its shape constraints, SGLang's scheduler batch management (ScheduleBatch, out_cache_loc, prepare_for_decode), tensor parallelism and allreduce communication patterns, and the specific hardware topology (8×RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5).

The output knowledge created by this message is the definitive benchmark comparison for this specific deployment configuration. The documentation serves as a decision record: for throughput-oriented workloads on this hardware, EAGLE-3 speculative decoding is not beneficial. Future optimization efforts should focus on baseline improvements (CUDA graph tuning, attention backend optimization, memory management) rather than speculative decoding refinements.

The Thinking Process

The assistant's reasoning, visible in the preceding messages, shows a methodical approach to problem-solving. When the dynamic disable patch failed, the assistant did not simply give up — it traced the error through multiple layers of the codebase, from the runtime error message ("tensor a (160) vs b (2)") back through forward_batch_generation to prepare_for_decode and ultimately to the EAGLE worker's _draft_preprocess_decode. Each step involved reading source code, forming hypotheses, and testing them against the observed behavior.

The decision to abandon dynamic switching and instead run clean benchmarks was a strategic pivot. Rather than continuing to fight the architecture, the assistant recognized that the most valuable output would be a clean, apples-to-apples comparison that definitively answered the throughput question. This prioritization of empirical data over engineering ambition reflects a mature approach to optimization work.

Conclusion

Message <msg id=5586> is a quiet but pivotal moment in the session. It represents the transition from exploration and optimization to documentation and decision-making. The edit to eagle-fast-verify.md crystallizes hours of work into a clear finding: for this deployment, on this hardware, with this workload, EAGLE-3 speculative decoding is not the path forward for throughput. The message is a testament to the importance of rigorous benchmarking, the value of knowing when to abandon a complex approach in favor of a simpler one, and the discipline of recording results for future reference.