The Sweet Spot That Wasn't: Investigating topk=2 for EAGLE-3 Speculative Decoding

Introduction

In the high-stakes world of large language model deployment, every token per second counts. When you're running an 8-GPU inference server with a 547GB model (Kimi-K2.5 INT4) on PCIe-connected Blackwell GPUs, the difference between a configuration that works and one that soars can be a single parameter. This article examines a pivotal moment in an intensive optimization session — message [msg 5655] — where the assistant responds to the user's question "Worth checking topk=2?" with a crisp analysis that reveals the delicate balance between speculative decoding strategies, architectural constraints, and the relentless pursuit of throughput.

The message, quoted in full below, captures a moment of strategic decision-making that builds on hours of prior work diagnosing, patching, and benchmarking speculative decoding configurations.

Good call — topk=2 is the sweet spot to check. It gives a small tree (7 draft tokens with num_steps=2) instead of a chain (3), which should improve acceptance rate at C=1, but spec_v2 requires topk=1. Let me check if that restriction is still enforced or if we can work around it.

[bash] ssh root@10.1.230.174 'grep -n "topk.1\|topk.v2\|spec_v2.*topk\|ENABLE_SPEC_V2\|enable_spec_v2" /root/sglang/python/sglang/srt/server_args.py | head -20' 1412: if not envs.SGLANG_ENABLE_SPEC_V2.get(): 1413: envs.SGLANG_ENABLE_SPEC_V2.set(True) 1433: if not envs.SGLANG_ENABLE_SPEC_V2.get(): 1434: envs.SGLANG_ENABLE_SPEC_V2.set(True) 1788: 1.1 We will turn on FA3 on hopper unless user use spec decode with topk > 1 or page_size > 1. 1789: 1.2 Use trtllm_mha for SM100/SM103 (Blackwell B200/GB200/B300) excluding spec with topk > 1. 2371: an...

The Context: A Benchmarking Breakthrough with a Lingering Gap

To understand why this message matters, we must first understand what came before it. The assistant had just completed a grueling optimization campaign spanning multiple segments (segments 33–38 of the conversation). The journey began with EAGLE-3 speculative decoding performing worse than baseline — a net-negative 54.1 tok/s versus the baseline's 82 tok/s. Through a series of interventions including upgrading to CUDA 13, patching SGLang for SM120 (Blackwell) support, enabling FlashInfer allreduce fusion, and enabling Torch symmetric memory, the assistant transformed EAGLE-3 into a net-positive 96.1 tok/s.

But the real breakthrough came with the spec_v2 overlap path. In [msg 5648], the assistant presented a comprehensive benchmark comparing three configurations across concurrency levels from 1 to 250:

topk=1 + spec_v2 (overlap): The new configuration
topk=4 + v1: The previous EAGLE-3 configuration
Baseline (no speculation): The raw model throughput The results were striking. At high concurrency (C≥30), topk=1+v2 actually beat baseline — 759.3 tok/s versus 689.4 tok/s at C=30, a 10.1% improvement. At C=100, it was essentially tied at 775 tok/s. The overlap scheduling was hiding the speculation overhead so effectively that speculative decoding went from a liability to an asset. However, there was a persistent gap at single-stream (C=1): 86.8 tok/s versus baseline's 92.7 tok/s, a 6.4% deficit. The assistant attributed this to the simpler topk=1 chain producing only 3 draft tokens (with num_steps=2), whereas the topk=4 tree produced many more candidates. At low concurrency, where there's no queue of requests to absorb the speculation overhead, every draft token that gets rejected is pure wasted computation. This is the context into which the user's question lands: "Worth checking topk=2?"## The Reasoning Behind topk=2 The assistant's response in [msg 5655] is deceptively simple — just three sentences and a bash command — but it encodes a rich chain of reasoning that reveals deep understanding of the speculative decoding architecture. First sentence: "Good call — topk=2 is the sweet spot to check." This immediately signals that the assistant recognizes the user's intuition as valid. The assistant has been living with this benchmark data for hours and has already internalized the trade-offs. The phrase "sweet spot" is telling: topk=2 sits between the too-simple topk=1 (a linear chain of 3 draft tokens) and the too-complex topk=4 (a tree of many candidates that proved problematic with the v1 scheduler). It's the Goldilocks option. Second sentence: "It gives a small tree (7 draft tokens with num_steps=2) instead of a chain (3), which should improve acceptance rate at C=1." This is the core hypothesis. The assistant is reasoning about the fundamental geometry of speculative decoding. With topk=1 and num_steps=2, the drafter produces a chain: draft token A → draft token B → draft token C. If the target model rejects token A, the entire chain collapses. With topk=2 and the same num_steps=2, the drafter produces a small tree: at each step, it considers the two most likely tokens, creating a branching structure with up to 7 unique draft tokens (1 + 2 + 4). This tree structure is more robust — even if the target model rejects the first choice at some position, an alternative branch may still be accepted. The assistant correctly identifies that this would improve the acceptance rate at C=1, which is exactly where the benchmark showed a 6.4% gap. The reasoning is sound: more draft tokens means more opportunities for the target model to find a token it agrees with, reducing wasted computation. Third sentence: "but spec_v2 requires topk=1." This is the critical constraint. The spec_v2 overlap path — the very feature that enabled the breakthrough throughput at high concurrency — has a hard requirement: topk must be exactly 1. The assistant doesn't know why this restriction exists, only that it's enforced. This creates a fundamental tension: the configuration that fixes the C=1 gap (topk=2) is incompatible with the configuration that fixes the high-concurrency throughput (spec_v2).

The Investigation: Checking the Source

The assistant doesn't stop at stating the constraint — it immediately investigates whether the restriction is truly enforced or if there's a workaround. The bash command searches the SGLang server_args.py file for any references linking topk to spec_v2:

grep -n "topk.*1\|topk.*v2\|spec_v2.*topk\|ENABLE_SPEC_V2\|enable_spec_v2"

The results are revealing. Lines 1412-1413 and 1433-1434 show that SGLANG_ENABLE_SPEC_V2 gets automatically set to True under certain conditions — specifically when speculative_num_steps <= 2 and speculative_eagle_topk == 1. This is the enforcement mechanism: the code explicitly checks for topk=1 before enabling spec_v2. Lines 1788-1789 show that topk > 1 also disables certain attention backend optimizations (FA3 on Hopper, trtllm_mha on Blackwell), suggesting that the tree-based speculation path (topk > 1) uses fundamentally different CUDA kernels than the chain-based path (topk=1).

The assistant is probing the boundaries of the system. It's asking: "Is this restriction a hard architectural constraint, or is it a conservative default that we can override?" The answer, implied by the grep output, is that it's a hard constraint baked into the server_args validation logic.

The Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are well-founded:

topk=2 would improve C=1 acceptance rate. This is almost certainly correct. A tree of 7 draft tokens has strictly more coverage than a chain of 3. However, the assistant implicitly assumes that the acceptance rate improvement would be enough to close the 6.4% gap. This is uncertain — the gap might be caused by other factors (e.g., the drafter model quality, the verify step overhead) that topk=2 wouldn't address.
spec_v2 truly requires topk=1. The grep output confirms this is enforced in code, but the assistant doesn't check why. The spec_v2 overlap path likely relies on the simpler scheduling of a linear chain — with a tree, the overlap scheduler would need to handle multiple branches, which complicates the CUDA graph capture and the token-by-token overlap logic. This is a reasonable architectural constraint.
The grep search is sufficient to answer the question. The assistant searches only for specific patterns in one file. There could be runtime checks elsewhere (e.g., in the eagle worker initialization) that also enforce topk=1. The grep gives a strong signal but isn't exhaustive. One subtle point worth examining: the assistant claims "7 draft tokens with num_steps=2" for topk=2. In EAGLE-3's tree construction, the exact count depends on how the tree is structured. With topk=2 and num_steps=2, step 1 produces 2 candidate tokens, and step 2 produces 2 candidates branching from each of those 2 tokens, yielding 4 more — totaling 6 unique draft tokens, or 7 if the initial hidden state is counted as a separate position. The assistant's count of 7 is therefore plausible depending on the tree construction convention. This minor detail does not affect the core reasoning: a tree of ~6-7 draft tokens is indeed a richer structure than a linear chain of 3, and should improve the acceptance rate at single-stream concurrency.

Input Knowledge Required

To fully understand this message, the reader needs:

Understanding of speculative decoding: How draft models propose tokens and target models verify them, and the difference between chain speculation (topk=1) and tree speculation (topk>1).
Knowledge of EAGLE-3's architecture: That EAGLE-3 uses a separate drafter model to predict hidden states rather than tokens directly, and that the "topk" parameter controls how many candidate tokens the drafter considers at each step.
Familiarity with the spec_v2 overlap path: That spec_v2 is an experimental SGLang feature that overlaps the draft and verify steps to hide speculation overhead, but that it has constraints on which configurations it supports.
Context of the benchmark results: That topk=1+v2 beat baseline at high concurrency but lagged at C=1, creating the motivation for exploring topk=2.
Understanding of CUDA graph capture: That SGLang uses CUDA graphs to accelerate repetitive computation patterns, and that tree-based speculation (topk>1) complicates graph capture because the execution path depends on which branches are accepted.

Output Knowledge Created

This message produces several valuable insights:

The topk=2 experiment is blocked by a hard constraint. The assistant confirms that spec_v2 enforces topk=1, meaning the user cannot simply change topk to 2 while keeping the overlap scheduler. This saves the user from wasting time on a configuration that would silently fall back to the slower v1 path.
The constraint is enforced at multiple levels. The grep shows both automatic enabling of spec_v2 (lines 1412-1434) and attention backend restrictions (lines 1788-1789) that depend on topk. This suggests the constraint is fundamental to the architecture, not just a configuration oversight.
The trade-off is explicit and quantified. The assistant articulates the exact trade-off: topk=2 would improve acceptance rate at C=1, but it would lose the overlap scheduling that makes speculation viable at high concurrency. The user now has a clear picture of the decision space.
A direction for future work is identified. If someone wanted to support topk=2 with spec_v2, they now know exactly which parts of the codebase need modification (server_args.py validation, CUDA graph capture, overlap scheduling logic).## The Thinking Process Visible in the Message One of the most instructive aspects of this message is what it reveals about the assistant's thinking process — even though the reasoning is compressed into just three sentences. The assistant is performing multi-factor optimization under constraints. It's simultaneously considering: - The geometry of speculation: chain vs. tree, acceptance rate, draft token count - The scheduler architecture: spec_v2 overlap vs. v1, and their respective constraints - The benchmark data: the C=1 gap of 6.4%, the high-concurrency victory - The codebase reality: what's enforced in server_args.py, what can be changed The assistant's thinking follows a clear pattern: hypothesis generation ("topk=2 would improve acceptance rate"), constraint identification ("but spec_v2 requires topk=1"), and empirical verification ("let me check if that restriction is still enforced"). This is the scientific method applied to systems engineering — propose, check, validate. The bash command is particularly revealing. The assistant doesn't just accept the constraint at face value; it goes to the source code to verify. The grep patterns are carefully chosen: topk.*1 catches conditions that check for topk=1, topk.*v2 catches any references to v2 that mention topk, spec_v2.*topk catches the reverse relationship, and ENABLE_SPEC_V2 catches the env var that controls the feature. This is a systematic search designed to find all possible interactions between the two parameters. The truncated output (the grep result ends with "an..." at line 2371) is also telling. The assistant only requested the first 20 lines, which was enough to confirm the constraint but not enough to see the full picture. This is a pragmatic trade-off: the assistant could have searched more exhaustively, but the evidence was already sufficient to answer the user's question. The cost of additional investigation (time, complexity) outweighed the benefit.

Conclusion

Message [msg 5655] is a masterclass in concise, actionable technical reasoning. In three sentences and one bash command, the assistant:

Validates the user's intuition about topk=2
Articulates the precise mechanism by which it would help (improved acceptance rate via tree structure)
Identifies the critical constraint (spec_v2 requires topk=1)
Empirically verifies the constraint in the source code
Provides enough context for the user to make an informed decision The message also implicitly defines the next steps in the optimization journey. The user now faces a clear fork in the road: either accept the C=1 gap and stay with topk=1+spec_v2 (which dominates at high concurrency), or abandon spec_v2 and try topk=2 with the v1 scheduler (which might improve C=1 but regress at high concurrency). Neither option is obviously better — that's what makes the decision interesting. This message also serves as a boundary marker in the conversation. After this point, the assistant will pivot away from EAGLE-3 tuning and toward deploying a new model (Qwen3.5-397B-A17B-NVFP4), as documented in segment 38's summary. The topk=2 question represents the last unexplored branch of the EAGLE-3 optimization tree — and the assistant's response shows that even this branch is blocked by architectural constraints. The optimization campaign has reached a natural conclusion, and the assistant's disciplined response to the user's question helps the team recognize that it's time to move on. In the broader narrative of the coding session, this message exemplifies the kind of thinking that separates effective system optimization from aimless experimentation: always know why a parameter change might help, always check what constraints prevent it, and always provide enough evidence for the team to make the right call.