The Pivot to spec_v2: Evaluating Overlap Scheduling for Dynamic Speculation Disable

Introduction

In the ongoing optimization of EAGLE-3 speculative decoding on an 8×GPU Blackwell system, message [msg 5612] marks a critical inflection point. After a comprehensive parallel throughput benchmark revealed that the baseline (no speculation) server strictly outperformed EAGLE-3 at every concurrency level—saturating at 773 tok/s versus EAGLE-3's 354 tok/s—the assistant was forced to reconsider the entire approach to speculative decoding. The subject message captures the moment when the assistant confirms that the spec_v2 overlap path has been successfully enabled with a reduced configuration (topk=1, num_steps=2, num_draft_tokens=3), and begins evaluating a GitHub pull request (#15623) that the user suggested might be relevant. This message is a study in parallel investigation: the assistant fetches documentation while waiting for a server to finish loading, demonstrating the multi-threaded nature of the optimization workflow.

The Context: A Journey from Benchmarking to Pivot

To understand why this message was written, one must trace the path that led to it. The session began with a definitive finding: EAGLE-3 speculative decoding, despite its impressive single-stream latency of 96.1 tok/s (a 5.9% improvement over baseline), became a net liability under parallel load. The gap widened from 15% at C=1 to over 2× at C≥30. This was documented in the eagle-fast-verify.md optimization plan and confirmed with both encyclopedic and coding/agentic prompts.

The assistant's first attempt to address this was to implement a dynamic speculation disable mechanism—a system that would automatically switch between speculative and non-speculative modes based on server load. This was attempted on the standard EAGLEWorker (v1, non-overlap) path, but ran into fundamental issues. The v1 path deeply couples batch state management with the speculative pipeline: out_cache_loc is pre-allocated for draft token dimensions, CUDA graph shapes expect speculative layouts, and the scheduler's batch construction is tightly integrated with the speculation parameters. Disabling speculation mid-flight would require unwinding all these assumptions, a task that proved intractable within the existing architecture.

The user then suggested investigating spec_v2 (message [msg 5589]), and the assistant discovered that SGLang has an alternative speculative decoding path called EAGLEWorkerV2 (the "overlap" path), enabled by setting SGLANG_ENABLE_SPEC_V2=True. This path uses a different scheduling strategy where the draft model and target model overlap their execution, potentially improving throughput. However, the assistant discovered a critical constraint: spec_v2 requires speculative_eagle_topk=1 (message [msg 5591]). With the previous configuration of topk=4 and num_steps=2, the draft tree had 16 tokens. With topk=1, it collapses to a simple chain of 3 tokens (num_steps + 1).

This was a significant reduction in speculative power. The assistant nevertheless decided to test whether spec_v2 with topk=1 was viable, starting a server in message [msg 5606] with the command:

SGLANG_ENABLE_SPEC_V2=True nohup ~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 --tp 8 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_100k_sglang/4 \
  --speculative-num-steps 2 --speculative-eagle-topk 1 \
  ...

The Subject Message: Content and Analysis

The subject message ([msg 5612]) reads:

Spec v2 is enabled: disable_overlap_schedule=False, speculative_eagle_topk=1, speculative_num_draft_tokens=3. Let me fetch that PR while waiting: [webfetch] ... [PR #15623 URL] [bash] tail -10 /data/eagle3/synth_100k/logs/eagle3_topk1_v2.log

This message contains three distinct actions, all performed in parallel within a single round:

  1. Confirmation of spec_v2 state: The assistant reports that the server has started with spec_v2 enabled (disable_overlap_schedule=False), topk=1, and 3 draft tokens. This is the first confirmation that the new configuration is actually working—the server log showed the warning "Spec v2 is enabled for eagle/eagle3 speculative decoding and overlap schedule is turned on" (message [msg 5611]).
  2. Fetching PR #15623: The user asked in message [msg 5608] whether PR #15623 was relevant. The assistant fetches the PR description to evaluate its applicability. The PR, titled "Full Support for Overlapped Constrained Decoding + Spec V2" by Ubospica, appears to extend the spec_v2 overlap path to support constrained decoding (e.g., JSON mode, regex-guided generation). The assistant is checking whether this PR provides a mechanism for dynamic speculation disable or if it's only tangentially related.
  3. Checking server log: While the PR is being fetched, the assistant also tails the server log to monitor progress. The server is still loading model shards (67-91% completion shown in the truncated output).

Why This Message Was Written: Reasoning and Motivation

The primary motivation for this message is parallel investigation under time constraints. The server startup takes approximately 90 seconds (loading 64 safetensors shards across 8 GPUs). Rather than idly waiting, the assistant uses this window to simultaneously evaluate the user's PR suggestion. This reflects a key characteristic of the optimization workflow: every second of server startup time is an opportunity for research and planning.

The deeper motivation is the pivot from v1 to v2. The assistant had just spent significant effort trying to implement dynamic speculation disable on the v1 EAGLEWorker path, only to discover fundamental architectural obstacles. The v2 path represents a fresh start—a different codebase with cleaner separation of concerns. The assistant is eager to validate that spec_v2 works at all before investing in further development. The confirmation that disable_overlap_schedule=False and the server is running with spec_v2 is the first validation step.

The fetch of PR #15623 serves a dual purpose: it addresses the user's direct question, but it also represents a search for any existing mechanism that could simplify the dynamic disable implementation. If the PR already implements a way to toggle speculation on and off, that would save days of development work.

Assumptions and Decisions

Several assumptions are embedded in this message:

  1. topk=1 is viable: The assistant assumes that reducing the draft tree from 16 tokens to 3 tokens still provides enough speculative benefit to be worth the overhead. This is a significant assumption—with topk=4, the tree could accept partial matches at any branch. With topk=1, it's a single chain: if the first draft token doesn't match the target's top-1 prediction, no tokens are accepted. The acceptance rate will likely drop substantially.
  2. spec_v2 overlap scheduling compensates: The assumption is that the overlap scheduling in v2 (where draft and target execution overlap) will improve throughput enough to offset the reduced acceptance rate from topk=1. This is untested.
  3. PR #15623 is potentially relevant: The assistant assumes the PR might contain mechanisms useful for dynamic disable, even though its title focuses on constrained decoding.
  4. The server will start successfully: The assistant assumes the configuration is valid and the server won't crash during initialization. This is not guaranteed—spec_v2 with EAGLE3 and topk=1 is an unusual configuration that may have untested edge cases.

Input Knowledge Required

To understand this message, one needs:

Output Knowledge Created

This message produces:

The Thinking Process

The assistant's reasoning is visible in the structure of the message itself. The phrase "Let me fetch that PR while waiting" reveals the parallel thinking: the assistant is consciously using the server startup time productively. This is not an idle wait—it's an opportunity to gather information that will inform the next steps.

The assistant is also implicitly answering the user's question from message [msg 5608]: "Will we need to look at https://github.com/sgl-project/sglang/pull/15623 or is there already some benefit?" The assistant doesn't answer definitively in this message—it's still gathering data. But the act of fetching the PR signals that the assistant takes the suggestion seriously and is evaluating it.

The confirmation of spec_v2 state (disable_overlap_schedule=False, speculative_eagle_topk=1, speculative_num_draft_tokens=3) shows that the assistant is methodically verifying each assumption before proceeding. This is a debugging mindset: confirm the configuration is applied correctly before measuring performance.

Conclusion

Message [msg 5612] is a small but pivotal moment in a larger optimization story. It captures the transition from a failed approach (dynamic disable on v1) to a new investigation (spec_v2 overlap path), while simultaneously evaluating a user-suggested PR for potential shortcuts. The message demonstrates the assistant's ability to parallelize investigation—using server startup time to gather documentation—and its methodical approach to validating assumptions before proceeding. Whether spec_v2 with topk=1 will prove viable, and whether PR #15623 offers any relevant mechanisms, are questions that will be answered in subsequent messages. But this message marks the moment when the investigation pivots decisively toward the overlap path, setting the stage for the next phase of optimization.