Probing the Upper Bound: The Quest for Optimal Speculation Depth in EAGLE-3 Decoding

In the relentless pursuit of inference throughput, every token counts. This article examines a single, deceptively simple message from an opencode coding session where an AI assistant responds to a user's suggestion to test an extreme speculative decoding parameter: speculative-num-steps=10. The message, reproduced below, is a brief but pivotal moment in a systematic optimization campaign for the Qwen3.5-122B-A10B model running on four NVIDIA RTX PRO 6000 Blackwell GPUs:

Assistant (msg 6523): Let me try steps=10 to see if there's more headroom at very high speculation depth. ``bash ssh root@10.1.230.174 'systemctl stop sglang-qwen.service' 2>&1; sleep 2; ssh root@10.1.2.6 'pct exec 129 -- bash -c "ps aux | grep python3 | grep -v grep | awk \"{print \\\$2}\" | xargs -r kill -9"' 2>&1; sleep 2; ssh root@10.1.230.174 'fuser -k /dev/nvidia* 2>/dev/null' 2>&1 ``

At first glance, this appears to be a routine service restart. But beneath the surface lies a rich story of diminishing returns, resource trade-offs, and the scientific method applied to large-scale inference optimization. This message is the culmination of hours of careful benchmarking, representing a deliberate probe into the outer limits of speculative decoding performance.

The Optimization Campaign: From 123 to 282 tok/s

To understand why this message matters, one must appreciate the journey that led to it. The assistant had been engaged in a systematic, multi-hour effort to tune the speculative decoding parameters of the Qwen3.5-122B-A10B model, a 122-billion-parameter mixture-of-experts architecture with 10 billion active parameters per token. The model was deployed using SGLang with EAGLE-3 speculative decoding, a technique where a lightweight draft model generates candidate tokens in advance, which the main model then verifies in parallel. When the draft tokens are accepted, the effective throughput increases dramatically.

The assistant's benchmark series had been nothing short of methodical. Starting with speculative-num-steps=1 (the default, generating 2 draft tokens per step), the assistant measured a baseline of approximately 123 tokens per second (tok/s) for a single concurrent request. Each increment of the steps parameter increased the number of draft tokens generated per speculation round — steps=2 produced 3 draft tokens, steps=3 produced 4, and so on, following SGLang's internal adjustment rule of num_draft_tokens = steps + 1 when speculative_eagle_topk == 1.

The results were striking. At steps=2, single-request throughput jumped to 186 tok/s (+51%). At steps=3, it reached 234 tok/s (+90% over baseline). Steps=4 pushed to 277 tok/s (+125%), and steps=5 eked out 282 tok/s (+129%). Each increment brought meaningful gains, though the marginal improvement was clearly shrinking — from +51% to +36% to +18% to a mere +2% between steps 4 and 5.

More importantly, the assistant had observed a critical trade-off: higher speculation depths consumed more KV cache memory per request, reducing the maximum number of concurrent requests the system could handle. At steps=1, max_running_requests was 26; at steps=5, it had fallen to an unknown but lower value. The aggregate throughput at high concurrency (C=16+) actually decreased from steps=4 to steps=5, suggesting the system had crossed an inflection point where the overhead of longer speculation steps outweighed the per-request gains.

The User's Suggestion and the Assistant's Response

It was at this juncture that the user interjected with a provocative suggestion: "Try 10 steps to see if we unplateu" (msg 6522). The typo — "unplateu" for "unplateau" or perhaps "see if we plateau" — captured the experimental spirit perfectly. The user was asking: have we truly hit the ceiling, or is there hidden headroom waiting to be unlocked at extreme speculation depths?

The assistant's response in msg 6523 is a model of concise, focused execution. The assistant acknowledges the request with a clear statement of intent — "Let me try steps=10 to see if there's more headroom at very high speculation depth" — and then executes a three-step cleanup sequence: stopping the systemd service, killing any residual Python processes on the Proxmox container (via pct exec 129), and freeing GPU memory with fuser -k /dev/nvidia*. This cleanup ritual, repeated throughout the session, ensures a clean state for the next server launch with the modified configuration.

The Reasoning Behind the Message

The assistant's thinking is captured in the phrase "to see if there's more headroom at very high speculation depth." This reveals a nuanced understanding of the optimization landscape. The assistant recognizes that the relationship between speculation depth and throughput is unlikely to be linear — at some point, the overhead of generating and verifying an ever-larger tree of draft tokens will overwhelm the benefits. The question is where that saturation point lies.

Several factors influence this saturation point. First, the EAGLE-3 draft model itself has a fixed accuracy: if the draft tokens are frequently rejected by the main model, the speculation overhead is wasted. Second, the KV cache memory constraint means that deeper speculation reduces the batch size, which in turn reduces the main model's utilization. Third, the Triton kernel configurations for the MoE layers may not be optimal for the unusual batch shapes created by deep speculation trees.

By testing steps=10, the assistant is probing whether the system has truly saturated or whether there is a second regime of improvement at very high depths. This is a classic exploration-vs-exploitation dilemma: the assistant could continue fine-tuning around the known optimum (steps=4 or 5), or explore the uncharted territory of extreme speculation. The user's suggestion pushes toward exploration.

Assumptions Embedded in the Message

This message rests on several implicit assumptions. The assistant assumes that the benchmark methodology (temperature=0, ignore_eos=True) remains valid at extreme speculation depths — but this is questionable, since greedy decoding inflates acceptance rates. At steps=10, the draft model would generate 11 candidate tokens per speculation round; if even one of those tokens is wrong, the entire speculation tree collapses and the model must regenerate. In real-world usage with non-zero temperature, the acceptance rate would be lower, and the optimal speculation depth would likely be smaller.

The assistant also assumes that the system's memory capacity can accommodate steps=10. Each additional step consumes KV cache memory for the draft tokens. At steps=5, max_running_requests was already significantly reduced; at steps=10, it might drop to single digits, making the configuration impractical for any workload requiring concurrent requests.

Furthermore, the assistant assumes that the EAGLE-3 implementation in SGLang handles extreme speculation depths correctly. There could be software bugs, numerical instabilities, or performance pathologies that only manifest at steps=10. The assistant is implicitly trusting the SGLang codebase's robustness.

Input Knowledge Required

To fully appreciate this message, one needs substantial background knowledge. The reader must understand speculative decoding — the concept of using a draft model to generate candidate tokens that the main model verifies in parallel. They must know about EAGLE-3 specifically, which uses a feature-level draft model rather than a full transformer. They must understand the speculative-num-steps parameter and its relationship to speculative-num-draft-tokens (adjusted to steps + 1 when topk=1). They must be familiar with the KV cache memory constraint and how it limits the number of concurrent requests. And they must know the hardware context: four NVIDIA RTX PRO 6000 Blackwell GPUs with 96 GB of memory each, connected via NVLink, running CUDA 13.0.

Output Knowledge Created

This message itself does not produce benchmark results — those would arrive in subsequent messages. But it creates several forms of knowledge. First, it establishes the intent to probe the upper bound, which frames the interpretation of whatever results follow. Second, it documents the experimental methodology: the cleanup sequence, the parameter modification, and the deployment process. Third, it captures a moment of scientific curiosity — the willingness to test an extreme parameter value even when the marginal gains appear to be diminishing.

The Broader Significance

This message exemplifies a pattern that recurs throughout high-performance computing: the tension between optimization and exploration. The assistant could have declared victory at steps=4 (277 tok/s, a 125% improvement over baseline) and moved on to other optimizations. Instead, the user's suggestion prompted a deeper probe. This willingness to explore the edges of the parameter space is what distinguishes thorough engineering from mere tinkering.

The message also highlights the importance of clean experimental methodology. Every parameter change in this session follows the same pattern: stop the service, kill residual processes, free GPU memory, modify the configuration, copy the file, reload systemd, start the service, wait for loading, verify the endpoint, and run the benchmark. This rigor ensures that each measurement is independent and reproducible — a lesson that applies far beyond inference optimization.

In the end, whether steps=10 proved to be a breakthrough or a bust is almost beside the point. The value lies in the asking — in the willingness to challenge assumptions about where the ceiling lies and to test hypotheses with disciplined experimentation. This message, for all its brevity, captures that spirit perfectly.