The Verdict That Changed Everything: EAGLE-3's Throughput Collapse and the Quest for Dynamic Speculation Disable

Introduction

In the high-stakes world of large language model serving, where every token per second translates directly to operational cost and user experience, the promise of speculative decoding is tantalizing. The idea is elegant: use a small, fast "draft" model to predict multiple candidate tokens, then have the large "target" model verify them in a single parallel forward pass. If the draft is accurate, the system generates multiple tokens per forward pass of the expensive model — a pure win. But as a months-long optimization campaign on an 8× NVIDIA RTX PRO 6000 Blackwell GPU system revealed, the gap between elegant theory and production reality can be devastatingly wide.

This article synthesizes a critical sub-session of that campaign — a multi-hour investigation spanning parallel throughput benchmarking, architectural code archaeology, and a failed attempt to implement dynamic speculation disable. The narrative arc moves from discovery (the baseline strictly outperforms EAGLE-3 at every concurrency level) through diagnosis (the overhead of draft + verify consumes compute that would be better spent processing more requests) to attempted remediation (dynamic speculation disable) and finally to architectural reckoning (the v1 EAGLEWorker's deeply coupled state makes runtime toggling nearly impossible). The session ends with a pivot to the spec_v2 overlap path, a cleaner architecture that comes with its own constraints.

The Benchmark That Killed a Hypothesis

The journey began with a rigorous parallel throughput comparison. The assistant had spent the previous segment (segment 36) achieving a breakthrough: by upgrading CUDA from 12.8 to 13, patching SGLang for SM120 (Blackwell) support, enabling FlashInfer allreduce fusion, and tuning NCCL parameters, EAGLE-3 speculative decoding had been transformed from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s in single-stream benchmarks — a 3.8% improvement over the baseline's 92.6 tok/s. This was celebrated as the "Best result!" in the optimization plan document.

But single-stream benchmarks measure only one dimension of performance. Real-world serving requires handling many concurrent requests. In messages 5435–5437, the assistant ran a comprehensive parallel throughput comparison between the EAGLE-3 speculative decoding server and a baseline server (no speculation) across concurrency levels from 1 to 250 concurrent requests, each generating up to 512 tokens. The results were unambiguous and devastating:

| Concurrency | EAGLE-3 (tok/s) | Baseline (tok/s) | Ratio | |---|---|---|---| | 1 | 77.5 | 92.6 | 0.84× | | 2 | 125.1 | 159.7 | 0.78× | | 5 | 183.2 | 292.5 | 0.63× | | 10 | 239.6 | 457.2 | 0.52× | | 30 | 299.5 | 711.0 | 0.42× | | 70 | 337.7 | 784.2 | 0.43× | | 100 | 338.8 | 761.6 | 0.44× | | 250 | 340.9 | 772.1 | 0.44× |

The baseline strictly outperformed EAGLE-3 at every single concurrency level. The gap widened dramatically with concurrency: from a 19% deficit at C=1 to over 2× at C=70+. Baseline throughput scaled 8.5× from C=1 to saturation (~773 tok/s), while EAGLE-3 managed only 4.4× scaling, plateauing at ~354 tok/s.

The assistant's analysis in [msg 5437] captured the critical insight: "EAGLE-3 speculation only helps per-request latency when there's negligible batching load. But for throughput (total tok/s across all concurrent requests), the baseline always wins because the overhead of draft + verify steps consumes compute that would be better spent processing more requests in parallel."

This was the death knell for the unconditional speculation hypothesis. The assistant also noted a telling discrepancy: the single-stream benchmark had shown EAGLE-3 at 96.1 tok/s, but the parallel benchmark at C=1 showed only 77.5 tok/s. The explanation was methodological — the single-stream test used a fixed prompt with high acceptance rate, while the parallel benchmark used diverse coding and agentic prompts, revealing that EAGLE-3's acceptance rate varies dramatically by prompt content. The single-stream victory had been a mirage.

The Prompt Alignment Investigation

Before accepting the verdict, the assistant undertook an important methodological investigation. The EAGLE-3 drafter had been trained on 37K synthetic samples from the Kimi-K2.5 model, and the assistant discovered that the original benchmark prompts (general knowledge questions like "Explain the theory of general relativity") were not representative of the coding and agentic tasks that the drafter was trained on. This raised a legitimate concern: was the benchmark unfairly penalizing EAGLE-3 by testing it on out-of-distribution prompts?

The assistant updated the benchmark prompts to better match the EAGLE-3 drafter's training data distribution, switching to coding and agentic tasks. Interestingly, this change did not significantly alter the results. The coding prompts yielded nearly identical outcomes to the encyclopedic prompts — a marginal +4% improvement at C=1 (77.5→80.9 tok/s) that did not change the overall conclusion. The baseline still dominated at all concurrency levels, confirming that the throughput gap was structural rather than a matter of prompt mismatch.

This finding itself was significant: it suggested that prompt distribution was not a meaningful factor in EAGLE-3's performance characteristics for this hardware configuration. The gap between speculation and baseline was a fundamental property of the system, not an artifact of the evaluation methodology.

The Pivot to Dynamic Speculation Disable

Faced with this evidence, the assistant could have abandoned EAGLE-3 entirely. Instead, it formulated a more nuanced strategy: dynamic speculation disable — a mechanism that would automatically turn off speculative decoding when server load exceeds a threshold, and re-enable it when load drops. This would capture the marginal latency benefit of EAGLE-3 for low-concurrency scenarios while avoiding the throughput penalty under load.

The assistant's first action was to update the optimization plan document eagle-fast-verify.md, appending the parallel throughput comparison results to the permanent record. Then, in [msg 5442], it spawned a subagent task to thoroughly explore the SGLang v0.5.9 source code and map the interaction between the EAGLE-3 speculative worker and the scheduler.

The subagent's investigation revealed a critical architectural fact: SGLang has two distinct code paths for speculative decoding. Path A is the overlap scheduling path (EAGLEWorkerV2, enabled by SGLANG_ENABLE_SPEC_V2=True), which is the default production path. Path B is the non-overlap path (EAGLEWorker), an older implementation used when disable_overlap_schedule=True. The subagent traced the full control flow from the scheduler through to the speculative worker, identifying key data structures like SpeculativeAlgorithm, ScheduleBatch, and GenerationBatchResult.

The assistant then read the actual source code — eagle_worker_v2.py at 878 lines — to understand the implementation details. The analysis revealed that the overlap path had a cleaner separation of concerns, but came with a significant constraint: it required topk=1, which reduced the draft tree from 16 tokens to just 3 tokens. This was a substantial reduction in speculative power, but the assistant reasoned that even a 3-token draft with dynamic disable would beat a 16-token draft that was always on (and always slower than baseline under load).

The Failed Attempt on the v1 Path

Despite the subagent's focus on the v2 overlap path, the assistant's server configuration was using disable_overlap_schedule=True, meaning it was actually running the standard EAGLEWorker (v1) — not EAGLEWorkerV2. This was discovered when the assistant checked the server logs and found the wrong file had been patched.

The assistant pivoted to implementing dynamic disable on the v1 EAGLEWorker path. This proved to be a journey into a deeply coupled architectural tangle. The core problem was that the EAGLE worker's batch state management was woven throughout the entire processing pipeline. The out_cache_loc tensor — which determines where in the KV cache each token's outputs are written — was pre-allocated with dimensions matching the draft token count. The CUDA graphs, which pre-record sequences of GPU operations for faster execution, expected fixed tensor shapes baked in at initialization time.

The assistant's first attempt was to clear batch.spec_info and batch.spec_algorithm before calling the target model forward in the fallback path. This seemed logical — if speculation is disabled, clear the speculative metadata. But the server crashed with a cryptic error: RuntimeError: The size of tensor a (160) must match the size of tensor b (2) at non-singleton dimension 0.

The diagnostic process was methodical. The assistant decomposed 160 as 10 × 16 — 10 concurrent requests times 16 draft tokens per request (determined by topk=4 and num_steps=2). The tensor a with size 160 was the out_cache_loc, pre-allocated for speculative dimensions. The tensor b with size 2 was the actual batch's token count for a non-speculative decode. The mismatch occurred because prepare_for_decode on the ScheduleBatch had already allocated cache slots for speculative dimensions before forward_batch_generation was called — the damage was done before the dynamic disable logic could intervene.

The assistant tried multiple fixes across several rounds of patching and testing. Each attempt produced the same error, repeated across all eight tensor-parallelism workers. The root cause was fundamental: the EAGLE v1 path was designed with the assumption that speculation is always active. The batch's seq_lens, out_cache_loc, and other state fields are sized and managed with draft token dimensions baked in. Disabling speculation mid-flight requires not just skipping the draft forward, but also resetting this state to non-speculative dimensions — which means modifying the scheduler's batch preparation logic, not just the worker's forward pass.

The assistant's analysis in [msg 5569] captured the breakthrough insight: "The problem is that out_cache_loc was already allocated by prepare_for_decode on the ScheduleBatch before forward_batch_generation is called. In the EAGLE path, prepare_for_decode allocates num_seqs * alloc_len_per_decode cache slots (10 16 = 160 for topk=4, num_steps=2). When I try to run a normal decode forward with this oversized out_cache_loc, the CUDA graph runner sees a mismatch."*

This was not a bug to be fixed — it was a fundamental architectural constraint. The v1 EAGLEWorker was not designed for dynamic mode switching, and retrofitting it would require significant refactoring of the batch preparation logic.

The Cleanup and Documentation

The abandonment of the dynamic disable approach was executed with characteristic discipline. The assistant restored the original eagle_worker.py from backup, killed all zombie processes, verified that all 8 GPUs showed 0 MiB of used memory, and started a fresh baseline server. Each step was methodical and deliberate — a ritual of transition from one phase of work to the next.

The 510-second wait for the baseline server to start was not time wasted. It was the price of a clean answer. The assistant could have rushed ahead, started the benchmark prematurely, and gotten corrupted results. Instead, it waited the full 8.5 minutes, ensuring that the baseline comparison would be clean and trustworthy.

The results of that comparison confirmed what the assistant likely suspected: the baseline achieved 92.7 tok/s at C=1 and saturated at 773 tok/s — strictly better than EAGLE-3 at every concurrency level. The assistant's response — "Excellent! Now I have both datasets with the same coding/agentic prompts. Let me compile the final comparison" — might seem incongruous with data that showed speculative decoding as a net-negative. But the enthusiasm was genuine: after the failed dynamic switching implementation, after the cryptic tensor errors, after the uncertainty of whether the old encyclopedic prompts were a fair comparison, the assistant finally had clean, apples-to-apples data. Certainty, even when it brings bad news, is valuable because it enables correct decisions.

The results were then documented in the project's eagle-fast-verify.md file, preserving the knowledge for future reference. The todo list showed all high-priority items marked as completed — the experimental phase was definitively closed.

The Pivot to spec_v2

The session summary included a "What Could Come Next" section that listed several potential directions: train a better drafter, test reduced speculation configs, use a load balancer for dynamic switching, and "Investigate SGLang's spec_v2 overlap path (may have cleaner separation for dynamic disable)."

The user's response was immediate and direct: "Look at spec_v2." This three-word message was a masterclass in effective human-AI collaboration. The user did not micromanage — they did not say "check whether spec_v2 supports topk > 1" or "implement dynamic disable on the v2 path." They gave a direction, trusting the assistant to execute the investigation with full autonomy. The three words carried an implicit message: I've read your summary, I see you gave up on this approach, but there's another angle you haven't explored. Go look at it.

The assistant's response was immediate and thorough. Beginning with [msg 5590], it started reading the relevant source files: server_args.py to understand how SGLANG_ENABLE_SPEC_V2 was configured, eagle_worker_v2.py to understand the v2 forward pass architecture, and eagle_info_v2.py to understand how the v2 draft input prepared batches for decoding.

The spec_v2 Investigation: Discovery and Constraints

The investigation revealed several critical findings. First, spec_v2 is automatically enabled for EAGLE/EAGLE3/STANDALONE algorithms when SGLANG_ENABLE_SPEC_V2 is set, but it requires topk=1. The code at line 1412 of server_args.py explicitly checks speculative_eagle_topk == 1 before enabling v2. Since the current configuration used topk=4, spec_v2 was silently disabled — there was no error message, no warning, just a quiet fallthrough to the non-overlap path.

This discovery reframed the optimization problem. The entire dynamic speculation disable effort had been focused on the v1 (non-overlap) path, which proved intractable due to state coupling. The v2 path was supposed to be the clean alternative, but it required topk=1 — a fundamentally different speculation regime. With topk=4, the draft tree produces 16 tokens. With topk=1, the draft tree collapses to a simple chain of num_steps + 1 = 3 tokens. This is a dramatic reduction in speculation breadth.

The assistant immediately pivoted to evaluating whether topk=1 was even worth pursuing. The acceptance rate with topk=1 would be much lower because there's no tree search — each step can only accept if the single draft token matches the target model's top-1 prediction. With topk=4, the tree can accept alternative branches even when the top-1 draft is wrong.

Despite these constraints, the assistant proceeded to read the EAGLEWorkerV2 code more carefully. The assessment was immediate: "This is much cleaner than v1. The v2 path takes a ModelWorkerBatch (already constructed by the scheduler), and the decode path at 674-698 is well-separated."

This single sentence encapsulates the core architectural insight that had eluded the team during the v1 investigation. The critical difference is one of responsibility separation. In v1, the EAGLE worker is deeply entangled with batch state management — it allocates cache slots, manages sequence lengths, and handles the full lifecycle of a decode step. In v2, the scheduler constructs a complete ModelWorkerBatch (including properly allocated out_cache_loc) and passes it to the worker. The worker's job is simpler: process the batch and return results.

The assistant identified two key advantages for dynamic disable:

  1. model_worker_batch is pre-built by the scheduler with out_cache_loc already allocated for speculation — This means the allocation happens in a single, well-defined location (the scheduler's prepare_for_decode), not deep inside the worker. If the worker decides to fall back to normal decode, the batch structure is already correct.
  2. self.target_worker.forward_batch_generation(model_worker_batch) is the standard TpModelWorker forward — This is the same call used for non-speculative decoding. The v2 worker delegates to the standard target worker's forward pass, which "should work with any spec_info." This is the crucial insight: because v2 uses the standard forward path, a fallback that clears or modifies spec_info before calling the forward pass would naturally produce correct non-speculative output.

Tracing the Allocation and Data Flow

The assistant then traced the scheduler flow in spec_v2 mode. The scheduler:

  1. Calls batch.get_model_worker_batch() to construct a ModelWorkerBatch
  2. Passes it to self.model_worker.forward_batch_generation(model_worker_batch) where model_worker is EAGLEWorkerV2
  3. Stores the result's next_draft_input back to batch.spec_info
  4. Updates batch.seq_lens from next_draft_input.new_seq_lens The critical detail is that batch.get_model_worker_batch() in spec_v2 mode calls draft_input.prepare_for_decode(batch) to handle the speculative batch setup. This is the key difference from v1: the batch preparation is externalized to the EagleDraftInput object, not buried inside the EAGLE worker's internal logic. A targeted grep confirmed that prepare_for_decode exists only in eagle_info_v2.py at line 82, not in any v1 file. This was the final piece of the architectural puzzle: the v2 path externalizes batch preparation to the EagleDraftInput class, creating a cleaner separation of concerns that might make dynamic speculation disable feasible. Reading the actual prepare_for_decode method revealed several important details. The method begins with batch.maybe_evict_swa() and batch.maybe_wait_verify_done(), then immediately starts computing cur_kv_lens_cpu, nxt_kv_lens_cpu, and num_needed_tokens using get_alloc_len_per_decode(). This suggests that v2's batch preparation is more explicit about memory allocation tracking, potentially offering a cleaner separation between speculative and non-speculative paths. A critical discovery came when examining the allocation strategy: the v2 prepare_for_decode allocates 2 * alloc_len_per_decode tokens per request. This is a direct consequence of the overlap scheduling strategy. In the v1 (non-overlap) path, the speculative decoding pipeline is sequential: the drafter generates draft tokens, the target model verifies them, and only then does the next decode iteration begin. The KV cache allocation can be tightly scoped to the current iteration's needs. In the v2 overlap path, however, the scheduler overlaps the verification of one batch with the preparation of the next. This means the KV cache slots for the current batch's draft tokens must remain live while the next batch is being set up — hence the 2× multiplier. Tracing get_alloc_len_per_decode to its definition in managers/utils.py revealed the full formula. The function computes max(topk * num_steps, num_draft_tokens). With the current configuration of topk=4 and num_steps=2, this gives max(8, 16) = 16. But spec_v2 requires topk=1, which would give max(2, 3) = 3, and then the prepare_for_decode method doubles it to 6 tokens per request. The comment in the code revealed the fundamental architectural difference between v1 and v2: - Spec v1: Allocates topk * num_steps tokens during draft decoding, then restores the allocation. Allocates num_draft_tokens when verifying the drafts. This two-phase allocation pattern means the KV cache allocation changes dynamically between draft generation and verification — precisely what makes dynamic disable so difficult. - Spec v2: Allocates max(topk * num_steps, num_draft_tokens) — a single allocation that covers the maximum of both phases. This is a much simpler pattern: one allocation, one shape, no dynamic resizing.

The Verify Method and Output Contract

With the allocation understood, the assistant turned to the verify method. The v2 forward_batch_generation at line 674-698 takes a ModelWorkerBatch which is already properly set up. The key call is at line 691:

batch_output = self.verify(model_worker_batch)

The verify() method returns a GenerationBatchResult with next_draft_input, accept_lens, and next_token_ids. For a no-speculation fallback, the assistant would need to produce the same output format. The verify method's sampling logic was traced:

(predict, accept_length, accept_index) = verify_input.sample(batch, logits_output, vocab_mask)
new_seq_lens = batch.seq_lens + accept_length

For a no-speculation fallback, the assistant would need to produce a result where accept_length is always 1 (accepting only the target model's own prediction) and next_draft_input is a no-op draft input that doesn't trigger further speculation.

The final piece of the investigation was understanding the output contract between the speculative worker and the scheduler. Reading _resolve_spec_overlap_token_ids in scheduler_output_processor_mixin.py revealed the exact format:

def _resolve_spec_overlap_token_ids(
    self: Scheduler, result: GenerationBatchResult, batch: ScheduleBatch
) -> List[List[int]]:
    assert result.next_token_ids.is_cpu
    assert result.accept_lens.is_cpu
    next_token_ids = result.next_token_ids.tolist()
    accept_lens = result.accept_lens.tolist()
    result.num_accepted_tokens = sum(accept_lens) - len(batch.reqs)

This function is the bridge between the speculative worker's output and the scheduler's batch state. The accept_lens tensor encodes how many draft tokens were accepted per request, and next_token_ids contains the actual accepted token IDs with padding. The function computes num_accepted_tokens = sum(accept_lens) - len(batch.reqs), which accounts for the fact that each request always accepts at least its first real token.

The PR #15623 Investigation

Alongside the main investigation, the user asked about a related pull request (#15623) concerning overlapped constrained decoding. The assistant investigated this PR but found it was not directly relevant to the current task of dynamic speculation disable for EAGLE-3. The PR addressed a different use case (constrained decoding with overlap scheduling) and did not provide a template or mechanism for the load-based speculation toggle the assistant needed. This investigation was a useful dead end — it confirmed that the dynamic disable problem had not been solved elsewhere in the SGLang codebase, and that the assistant was exploring genuinely novel territory.

Lessons and Implications

This session offers several profound lessons for ML systems engineering.

First, single-stream benchmarks can be dangerously misleading. The entire optimization campaign was guided by single-stream throughput numbers. The 3.8% improvement of EAGLE-3 over baseline at C=1 seemed promising enough to justify continued investment. But this metric was not representative of real-world deployment where multiple users query the system simultaneously. The parallel benchmark revealed that the optimization effort was optimizing for the wrong regime.

Second, speculative decoding's value proposition is fundamentally different from what is often assumed. The conventional narrative is that speculative decoding accelerates generation by processing multiple tokens per forward pass. But this session revealed a more nuanced truth: EAGLE-3 helps per-request latency at very low concurrency (where the GPU has idle capacity), but hurts total throughput under any significant load because the draft + verify overhead consumes compute that could otherwise process independent requests.

Third, architectural coupling determines what optimizations are feasible. The v1 EAGLEWorker was designed with the assumption that speculation is always active. Its batch state management — out_cache_loc, CUDA graph shapes, spec_info — is deeply intertwined with speculative dimensions. This coupling made dynamic disable nearly impossible without significant refactoring. The v2 overlap path, by contrast, had cleaner separation, but came with its own constraints. This is a lesson in software architecture: modularity and separation of concerns are not just abstract virtues — they determine what future optimizations are possible.

Fourth, the value of having multiple implementation paths. The existence of the spec_v2 overlap path (even with its topk=1 limitation) provided a viable alternative when the v1 path proved too coupled to modify. This is a testament to the value of architectural diversity in complex systems.

Fifth, the importance of benchmarking under realistic conditions. The single-stream benchmark (96.1 tok/s) suggested EAGLE-3 was a net positive, but the parallel throughput benchmark revealed it was a net negative under any real server load. This is a cautionary tale about the dangers of optimizing for the wrong metric, and a testament to the value of comprehensive, multi-dimensional evaluation.

What Could Come Next

The session summary listed several potential directions for future work:

Conclusion

This session of the conversation is a testament to the importance of rigorous benchmarking, the value of knowing when to abandon a complex approach, and the discipline of recording results for future reference. The assistant began with a hypothesis (EAGLE-3 improves throughput) and designed an experiment to test it. When the data contradicted the hypothesis, the assistant did not double down or rationalize — it accepted the finding and pivoted to a more productive direction.

The failed dynamic disable attempt, while producing no working code, generated deep architectural knowledge about SGLang's speculative decoding internals. The subsequent investigation of spec_v2, while revealing significant constraints (topk=1), opened a new path forward that had not been previously explored. In engineering, a well-documented dead end is often more valuable than a working solution whose rationale is unclear — and a promising new direction, even with constraints, is the seed of the next breakthrough.

The story of this session is ultimately about the relationship between measurement and understanding. The benchmarks provided the measurement; the architectural investigation provided the understanding. Together, they transformed the project's trajectory from "how do we make speculation work?" to "when is speculation actually useful?" — a question whose answer would reshape the entire optimization strategy.

In the end, the assistant's willingness to let data override prior beliefs — even after weeks of optimization effort including CUDA stack upgrades, kernel patches, and NCCL tuning — is the hallmark of rigorous engineering. The speculation dream died, but the lessons learned will inform every future optimization decision.

References

[1] "The Speculation That Couldn't: EAGLE-3's Throughput Collapse and the Quest for Dynamic Disable" — Chunk 0 article covering the parallel throughput benchmark and the failed v1 dynamic disable attempt.

[2] "The Verdict and the Pivot: How Benchmarking EAGLE-3 Forced a Fundamental Reassessment of Speculative Decoding Architecture" — Chunk 1 article covering the prompt alignment, cleanup, and spec_v2 investigation.

[3] Segment 37 summary — The comprehensive analyzer summary of the entire segment's work.

[4] [msg 5437] — The critical analysis message where the assistant concluded that baseline strictly outperforms EAGLE-3 at all concurrency levels.

[5] [msg 5569] — The breakthrough insight about out_cache_loc pre-allocation preventing dynamic disable on the v1 path.

[6] [msg 5588] — The session summary with the "What Could Come Next" section listing potential future directions.

[7] [msg 5589] — The user's three-word directive: "Look at spec_v2."

[8] [msg 5592] — Discovery that spec_v2 requires topk=1.

[9] [msg 5594] — Assessment that v2 is "much cleaner than v1."

[10] [msg 5596] — The grep confirming prepare_for_decode exists only in v2.

[11] [msg 5598] — Discovery of the 2× allocation multiplier in v2's prepare_for_decode.

[12] <msg id=5599-5600> — Tracing get_alloc_len_per_decode to its definition.

[13] [msg 5601] — Tracing the verify method and sampling logic.

[14] [msg 5602] — Understanding _resolve_spec_overlap_token_ids and the output contract.