Reading the Architecture: How spec_v2 Could Unlock Dynamic Speculation Disable

The Message

[assistant] Now I see the v2 flow. The scheduler:
1. Calls `batch.get_model_worker_batch()` which creates a `ModelWorkerBatch`
2. Passes it to `self.model_worker.forward_batch_generation(model_worker_batch)` where `model_worker` is `EAGLEWorkerV2`
3. The result's `next_draft_input` is stored back to `batch.spec_info`
4. `batch.seq_lens` is updated from `next_draft_input.new_seq_lens`

The key is that `batch.get_model_worker_batch()` in spec_v2 mode constructs the batch properly because `prepare_for_decode` calls `draft_input.prepare_for_decode(batch)` which handles the speculative setup.

Let me check how `EagleDraftInput.prepare_for_decode` works in v2:
[bash] ssh root@10.1.230.174 'grep -n "def prepare_for_decode" /root/sglang/python/sglang/srt/speculative/eagle_info.py'

Context: The Quest for Dynamic Speculation Disable

This message sits at a critical inflection point in a long optimization campaign. The assistant and user have been working for days to deploy a large language model (Kimi K2.5) with EAGLE-3 speculative decoding across 8 PCIe-connected Blackwell GPUs. The journey has been arduous: they upgraded CUDA to version 13, patched SGLang for SM120 support, enabled FlashInfer allreduce fusion, and finally got EAGLE-3 working as a net-positive at 96.1 tok/s for single-stream inference.

But then came the hammer blow. In the immediately preceding messages ([msg 5577], [msg 5583]), the assistant ran parallel throughput benchmarks comparing EAGLE-3 against the baseline (no speculation) server using realistic coding and agentic prompts. The results were unambiguous and damning: baseline strictly outperformed EAGLE-3 at every concurrency level. At C=1, baseline delivered 92.7 tok/s versus EAGLE-3's 80.9 tok/s — a 15% deficit. At high concurrency (C=100), the gap widened to over 2x: 773 tok/s baseline versus 354 tok/s with EAGLE-3.

This finding completely reframed the value proposition of speculative decoding. EAGLE-3 was not a universal performance win — it was a narrow tool that helped only at the lowest concurrency levels, and became a liability under load. The obvious solution was dynamic speculation disable: automatically turn off speculation when the server is busy, and turn it back on when concurrency drops.

The assistant attempted to implement this on the standard EAGLEWorker (v1, non-overlap) path ([msg 5572], [msg 5573]), but ran into fundamental issues. The v1 path deeply couples batch state management with the speculative pipeline — out_cache_loc is pre-allocated for draft token dimensions, CUDA graph shapes expect speculative layouts, and the entire decode setup is handled inside the EAGLE worker rather than in the shared prepare_for_decode method. The assistant concluded: "it's complex and error-prone" and deferred the effort.

Then the user issued a simple command: "Look at spec_v2" ([msg 5589]). This message (5595) is the assistant's response — a deep read of the alternative EAGLEWorkerV2 (overlap) path to assess whether it offers a cleaner foundation for dynamic speculation disable.

Why This Message Was Written

The message was written for a precise investigative purpose: to understand the architectural differences between the v1 and v2 speculative decoding paths in SGLang, specifically to determine whether v2's batch construction and state management would make dynamic speculation disable feasible.

The assistant had already discovered that spec_v2 requires topk=1 (found at line 2374 of server_args.py in [msg 5591]), which reduces the draft tree from 16 tokens to just 3 tokens. This is a significant performance trade-off — a smaller draft tree means fewer tokens generated per speculation step, reducing the potential speedup even when speculation is active. But the trade-off might be worth it if v2 enables clean dynamic switching.

The assistant is reading code to answer several questions:

1. How does the scheduler construct batches in spec_v2 mode?
2. Is the batch construction clean enough that we could simply bypass the EAGLE worker and call the target worker directly?
3. Does EagleDraftInput.prepare_for_decode handle the speculative setup in a way that could be conditionally skipped?

The Thinking Process Visible in the Message

The message reveals a structured, architectural mode of thinking. The assistant is tracing the control flow from the scheduler through to the model worker, building a mental model of how spec_v2 differs from v1.

The numbered list (1-4) shows the assistant reconstructing the v2 flow from code reading:

1. The scheduler calls batch.get_model_worker_batch() — this is the standard batch construction
2. It passes the batch to EAGLEWorkerV2.forward_batch_generation() — the speculative worker
3. The result's next_draft_input is stored back to batch.spec_info — state persistence across iterations
4. batch.seq_lens is updated from the draft input's new sequence lengths — maintaining correct KV cache state The key insight the assistant identifies is: "batch.get_model_worker_batch() in spec_v2 mode constructs the batch properly because prepare_for_decode calls draft_input.prepare_for_decode(batch) which handles the speculative setup." This is the crucial architectural difference from v1. In v1, the EAGLE worker itself handles all the batch setup internally, making it impossible to bypass without replicating that logic. In v2, the batch is constructed by the shared scheduler code, and the speculative setup is delegated to draft_input.prepare_for_decode() — a method that could potentially be made conditional. The message ends with a tool call to find def prepare_for_decode in eagle_info.py. This is the next step in the investigation: reading the actual implementation of this method to understand exactly what it does and whether it can be modified or bypassed.

Assumptions and Their Implications

The assistant makes several implicit assumptions in this message:

Assumption 1: The v2 flow is fundamentally cleaner than v1. This is based on the observation that batch construction is handled by the scheduler rather than inside the EAGLE worker. This appears to be correct based on the code read so far, but the assistant hasn't yet verified that draft_input.prepare_for_decode() doesn't have its own deep coupling issues.

Assumption 2: The prepare_for_decode method is the key leverage point for dynamic disable. The assistant assumes that if this method can be made conditional (skip speculative setup when speculation should be disabled), the rest of the flow will work naturally. This is a reasonable architectural assumption, but it depends on whether the batch's out_cache_loc and other state fields are compatible between speculative and non-speculative modes.

Assumption 3: The spec_v2 path with topk=1 is a viable starting point. The assistant hasn't yet tested whether topk=1 speculation provides any meaningful speedup. With only 3 draft tokens per step, the acceptance rate might be too low to justify the overhead. This assumption is implicitly challenged by the earlier finding that even 16-token speculation only provides marginal gains at low concurrency.

Input Knowledge Required

To understand this message, the reader needs knowledge of:

1. SGLang's speculative decoding architecture: The distinction between EAGLEWorker (v1, non-overlap) and EAGLEWorkerV2 (v2, overlap), and how they integrate with the scheduler.
2. The ModelWorkerBatch abstraction: This is the data structure that carries all per-batch state (input IDs, cache locations, sequence lengths) between the scheduler and the model worker. Understanding how it's constructed and consumed is essential.
3. The prepare_for_decode method: This is the method that allocates KV cache slots and sets up the batch for a decode step. In speculative mode, it must allocate slots for both the target model's tokens and the draft model's tokens.
4. The topk parameter: In EAGLE-style speculation, topk controls how many candidate tokens are generated at each draft step. With topk=4 and num_steps=2, the draft tree has 1 + 4 + 4 = 9 nodes (or 16 with the full tree). With topk=1, the tree collapses to a simple chain of 3 tokens.
5. The CUDA graph constraint: SGLang uses CUDA graphs to accelerate inference, but these graphs have fixed shapes. The speculative graph expects a certain number of draft tokens, which makes dynamic switching difficult.
6. The overlap schedule: SGLang's overlap schedule allows the target model's compute to overlap with the draft model's compute, reducing latency. Spec_v2 enables this overlap, while the standard v1 path does not.

Output Knowledge Created

This message creates several pieces of knowledge:

1. A clear architectural map of the spec_v2 flow: The four-step sequence from scheduler batch construction through to sequence length update is now documented and understood.
2. Identification of draft_input.prepare_for_decode() as the critical method: This is the leverage point where speculative setup happens. If dynamic disable can be implemented here, it would work cleanly with the v2 architecture.
3. A concrete next step: The assistant knows exactly what to read next — the prepare_for_decode method in eagle_info.py. This guides the investigation forward.
4. A comparison point with v1: The message implicitly documents why v1 was unsuitable (batch state management deeply coupled inside the EAGLE worker) and why v2 might be better (batch construction handled by shared scheduler code).

Mistakes and Incorrect Assumptions

The most significant potential mistake is the assumption that the v2 path's cleaner architecture will translate to feasible dynamic disable. The assistant hasn't yet verified that draft_input.prepare_for_decode() doesn't have its own coupling issues. It's possible that this method:

• Allocates out_cache_loc for draft token dimensions (same problem as v1)
• Modifies CUDA graph shapes in ways that can't be undone
• Has side effects on other batch state that would break non-speculative forward passes Additionally, the assistant hasn't considered the performance implications of topk=1. The earlier benchmarks showed that even with a 16-token draft tree, EAGLE-3 only provided a marginal 4% gain at C=1 (80.9 vs 92.7 tok/s baseline — actually a loss). With only 3 draft tokens, the acceptance rate would likely be lower, making the speculation overhead even harder to justify. The dynamic disable mechanism might end up being unnecessary if topk=1 speculation is never beneficial. There's also an unstated assumption that the v2 path works correctly with the current CUDA 13 + SM120 setup. The assistant has been running the v1 path throughout the optimization campaign. Switching to v2 might introduce new compatibility issues — the overlap schedule might interact poorly with the FlashInfer allreduce fusion, or the CUDA graph caching might not work correctly with the smaller draft tree.

The Broader Significance

This message represents a pivot from implementation to investigation. After the failed attempt to patch the v1 worker, the assistant is now doing careful architectural reconnaissance before writing any code. This is a mature engineering approach: understand the system's structure before attempting to modify it.

The message also reveals the tension between two competing design goals in speculative decoding systems: performance and flexibility. The v1 path was optimized for raw throughput with a fixed speculation configuration, but this optimization came at the cost of tightly coupling all the state management. The v2 path sacrifices some performance (via topk=1 and the smaller draft tree) in exchange for cleaner separation of concerns. Whether this trade-off is worth it depends on whether the cleaner architecture actually enables dynamic disable — and whether the resulting system provides any net benefit over the baseline.

The investigation continues in the next step, where the assistant will read the prepare_for_decode implementation. The outcome of this reading will determine whether dynamic speculation disable is feasible on the v2 path, or whether the team needs to consider alternative approaches like running two server instances with a load balancer.