The Speculation Tuning That Doubled Inference Speed: A Deep Dive into EAGLE-3 Optimization

In the high-stakes world of large language model inference, every token per second matters. When you're running a 122-billion-parameter model like Qwen3.5-122B-A10B on four NVIDIA RTX PRO 6000 Blackwell GPUs, squeezing out maximum throughput is a complex optimization problem with many interacting knobs. One of the most impactful knobs is the speculation depth of EAGLE-3, SGLang's multi-token prediction (MTP) system. In message [msg 6549], the assistant delivers the culmination of a systematic tuning session, presenting a comprehensive summary table that distills hours of benchmarking into a clear, data-driven deployment decision.

This message is not merely a status update. It is the conclusion of an empirical optimization journey, a decision document, and a knowledge artifact all in one. Let us examine it in depth.

The Message in Full

The assistant writes:

Server is up with steps=4. Here's the summary of the speculation tuning:

>

| Steps | Draft tokens | C=1 per-req | C=4 per-req | C=16 agg | C=32 agg | Status | |---|---|---|---|---|---|---| | 1 | 2 | 123 | 119 | 1213 | 1515 | baseline | | 2 | 3 | 186 | 162 | 1530 | 1673 | +51% C=1 | | 3 | 4 | 234 | 200 | 1733 | 1979 | +90% C=1, best agg | | 4 | 5 | 277 | 236 | 1596 | 1773 | +125% C=1, best low-C | | 5 | 6 | 282 | 222 | 1119 | 1273 | diminishing, agg drops | | 10 | 11 | — | — | — | — | OOM at init |

>

Steps=4 is deployed — 277 tok/s single-request, more than double the original 123 tok/s. The caveat is these numbers are with temperature=0, ignore_eos=True (synthetic benchmark with ~100% accept rate). Real-world accept rates will be lower, but even at 70% accept rate the improvement should be substantial.

At first glance, this is a simple report. But beneath the surface lies a rich story of systematic experimentation, hardware constraints, and careful trade-off analysis.

Why This Message Was Written: The Context and Motivation

To understand why this message exists, we must trace the conversation that precedes it. The assistant had been iterating on the speculative_num_steps parameter — the number of EAGLE-3 speculation steps — since [msg 6507]. Each step value required a full deployment cycle: editing the systemd service file, copying it to the remote machine, reloading systemd, starting the server, waiting ~100 seconds for the model to load, then running a multi-concurrency benchmark script.

The journey began with steps=1 (baseline) at 123 tok/s single-request. Steps=2 pushed to 186 tok/s. Steps=3 hit 234 tok/s. Steps=4 achieved 277 tok/s. The user then asked at [msg 6522]: "Try 10 steps to see if we unplateu." The assistant dutifully attempted steps=10, which led to a cascade of failures — OOM during initialization, a restart loop caused by zombie processes holding GPU memory, and a confusing diagnostic process where the assistant had to distinguish between OOM kills and systemctl stop timeouts ([msg 6529] through [msg 6545]).

The subject message is written immediately after the assistant successfully cleaned up the crash loop, reverted to steps=4, and confirmed the server was running ([msg 6548]). It serves multiple purposes:

  1. Reporting completion: The server is back up and stable.
  2. Summarizing the full tuning sweep: All tested configurations in one place.
  3. Justifying the decision: Why steps=4 was chosen over steps=3 (better single-request) and steps=5 (diminishing returns).
  4. Documenting the failure mode: Steps=10 OOMs at init — a valuable data point for future reference.
  5. Managing expectations: The caveat about synthetic benchmark conditions prepares the user for potentially lower real-world gains.

The Decision-Making Process: How Steps=4 Was Chosen

The assistant's decision-making is a textbook example of empirical optimization with multiple objectives. The key insight is that there is no single "best" setting — the optimal choice depends on the workload.

The assistant identifies two distinct use cases:

Assumptions Embedded in the Analysis

The message makes several important assumptions that are worth examining:

1. Greedy decoding (temperature=0) yields ~100% acceptance rate. This is the most significant assumption. EAGLE-3's speculation works by having a lightweight draft model predict multiple tokens ahead, then the full model verifies them in parallel. When temperature=0, the draft model's predictions are deterministic and highly accurate, leading to near-perfect acceptance. In real-world usage with sampling (temperature > 0), the draft model's predictions may diverge from the sampled distribution, reducing the acceptance rate. The assistant acknowledges this caveat explicitly.

2. The benchmark uses ignore_eos=True. This means the model never stops generating due to end-of-sequence tokens, which is unrealistic for most real prompts. Real conversations have variable-length responses with natural stopping points.

3. The benchmark prompt is representative. The bench_qwen.py script uses a specific prompt structure. Different prompt lengths and types may yield different speculation efficiency.

4. Hardware conditions are stable. The benchmarks assume consistent GPU performance, no thermal throttling, and no interference from other processes.

The assistant's caveat — "even at 70% accept rate the improvement should be substantial" — is a reasonable hedge. A 70% accept rate would reduce the effective throughput gain, but starting from a baseline of 123 tok/s, even a modest improvement would be significant.

Mistakes and Incorrect Assumptions Along the Way

The path to this message was not without errors. Several incorrect assumptions were made and corrected during the tuning process:

1. The initial assumption that steps=10 would fit in memory. The user's suggestion to try 10 steps seemed reasonable — perhaps there was more headroom. But the KV cache memory required for 11 draft tokens per request exceeded available GPU memory. The assistant had to diagnose whether the crash was OOM or a systemd timeout ([msg 6532] through [msg 6534]).

2. Confusion between OOM and systemd stop timeouts. At [msg 6532], the assistant initially thought the steps=10 instance was OOM-killed: "It got SIGKILLed — probably OOM killed (no swap on this machine)." But upon closer inspection at [msg 6534], the assistant realized: "Not OOM — it was the systemctl stop from earlier that SIGKILLed it after the stop timeout." The first crash was actually from the previous steps=5 teardown, not from steps=10 itself. The steps=10 instance had actually served requests successfully before being killed by the stop command.

3. The restart loop diagnosis. After the steps=10 deployment, the server entered a crash loop. The assistant initially thought it was a continuous OOM cycle, but eventually identified the real cause: zombie processes from the previous instance were holding GPU memory, causing the next instance to fail during CUDA graph capture ([msg 6537]). The fix required manually killing processes and using fuser -k /dev/nvidia* to release GPU resources.

4. The assumption that topk=2 might work. At [msg 6520], the assistant considered trying tree-based speculation (topk=2) but then discovered that SGLang's code prevents topk>1 with the triton attention backend and page_size>1 ([msg 6521]). This avenue was correctly abandoned before wasting time on a non-viable configuration.

These mistakes are not failures — they are the natural process of empirical investigation. Each incorrect assumption was corrected through careful observation and diagnostic work.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, one needs familiarity with several concepts:

EAGLE-3 / Multi-Token Prediction (MTP): A speculative decoding technique where a lightweight draft model predicts multiple future tokens simultaneously. The full model then verifies these predictions in a single forward pass, potentially generating multiple tokens per step. This is distinct from autoregressive decoding, which generates one token at a time.

Speculation steps vs. draft tokens: The number of speculation steps determines how many tokens the draft model predicts ahead. With topk=1 (linear chain), the number of draft tokens equals steps + 1. So steps=4 produces 5 draft tokens per verification step.

Per-request vs. aggregate throughput: Per-request tok/s measures the speed experienced by a single client. Aggregate tok/s measures the total throughput across all concurrent clients. These diverge at high concurrency due to batching effects.

KV cache memory pressure: Each request in the batch requires KV cache entries for all its tokens, including draft tokens. More speculation steps mean more draft tokens per request, reducing the maximum number of concurrent requests that can fit in GPU memory.

CUDA graph capture: SGLang uses CUDA graphs to accelerate inference by pre-recording GPU operations. The graph must be sized for the maximum expected batch size, and larger speculation depths require larger graphs, consuming more memory.

Output Knowledge Created by This Message

This message creates several forms of valuable knowledge:

1. A definitive performance curve for EAGLE-3 speculation depth on this specific hardware. The table provides exact throughput numbers for steps 1-5 at multiple concurrency levels. This is immediately actionable — anyone deploying Qwen3.5-122B on 4× Blackwell GPUs with SGLang can use these numbers to choose their configuration.

2. A clear decision rule for configuration selection. The message establishes that steps=3 is optimal for high-concurrency batch serving, while steps=4 is optimal for low-concurrency agentic workloads. This is a reusable heuristic.

3. A documented memory ceiling. Steps=10 with 11 draft tokens exceeds available memory, establishing an upper bound on speculation depth for this hardware configuration.

4. A caveat framework. The explicit acknowledgment of synthetic benchmark conditions (temperature=0, ignore_eos=True) provides a mental model for translating these results to real-world deployments.

5. A methodology template. The systematic approach — test each step value, measure at multiple concurrency levels, compare per-request and aggregate metrics, identify trade-offs, and select based on use case — is a reusable pattern for any speculation tuning effort.

The Thinking Process: What the Assistant's Reasoning Reveals

The assistant's reasoning throughout this tuning session reveals a sophisticated mental model of the inference system. Several patterns are noteworthy:

Trade-off awareness. The assistant consistently thinks in terms of trade-offs rather than seeking a single "best" number. When steps=4 shows lower aggregate throughput than steps=3 at high concurrency, the assistant doesn't discard it — instead, it identifies the crossover point and reasons about which workload matters more.

Diminishing returns detection. The progression from steps=4 (277 tok/s) to steps=5 (282 tok/s) shows only a 1.8% gain for 20% more draft tokens. The assistant immediately flags this as diminishing returns and notes the aggregate throughput penalty.

Memory budgeting intuition. The assistant understands that draft tokens consume KV cache memory, and that this has a direct impact on maximum concurrency. The comment "More draft tokens = less KV cache" at [msg 6508] shows this mental model clearly.

Systematic isolation of variables. When the steps=10 deployment enters a crash loop, the assistant methodically checks: systemd status, journalctl logs, GPU memory usage, process lists. Each diagnostic step narrows the possible causes until the real issue (zombie processes) is identified.

Use-case-driven decision making. The final choice of steps=4 is not based on a single metric but on an understanding of how the system will be used. The assistant explicitly connects the configuration to the expected workload pattern (agentic coding).

Conclusion

Message [msg 6549] is far more than a simple status update. It is the distilled result of a systematic empirical investigation spanning multiple deployment cycles, crash diagnoses, and benchmark runs. The assistant transformed raw benchmark data into actionable knowledge: a clear table of trade-offs, a justified deployment decision, and a caveat framework for real-world application.

The 277 tok/s achieved at steps=4 represents a 125% improvement over the baseline of 123 tok/s — more than doubling the inference speed through configuration alone, without any code changes or hardware upgrades. This is the power of systematic speculation tuning, and this message captures the complete reasoning process that made it possible.

For anyone deploying large language models with speculative decoding, the methodology demonstrated here — test systematically, measure at multiple concurrency levels, understand the trade-offs, and choose based on your workload — is a template worth following.