The Systematic Sweep: Finding the Optimal EAGLE-3 Step Count

In the middle of an intensive optimization session for EAGLE-3 speculative decoding on a Kimi-K2.5 model, the assistant issued a seemingly routine command: restart the SGLang server with --speculative-num-steps 3 and --speculative-num-draft-tokens 4. This single message ([msg 4621]) — a bash command launching an inference server — is far more than a mundane operational action. It represents a critical data point in a methodical, empirically-driven optimization sweep that would ultimately yield a 5.9% throughput improvement over the non-speculative baseline. To understand this message fully, one must trace the reasoning that led to it, the assumptions it embodies, and the knowledge it would generate.

The Message

The assistant executed the following command:

ssh root@10.1.230.174 'SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 nohup ~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --trust-remote-code \
  --tp-size 8 \
  --mem-fraction-static 0.88 \
  --host 0.0.0.0 \
  --port 8000 \
  --num-continuous-decode-steps 4 \
  --disable-custom-all-reduce \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_100k_sglang/4 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-num-steps 3 \
  > /data/eagle3/synth_100k/logs/sglang_eagle3_prod_3step.log 2>&1 &'
echo "Starting with 3 draft steps..."

The server is configured with tensor parallelism across 8 GPUs (--tp-size 8), using the EAGLE-3 speculative decoding algorithm with a draft model trained on 100K samples. The key parameters being varied are --speculative-num-steps 3 and --speculative-num-draft-tokens 4 — meaning the draft model will generate 4 candidate tokens across 3 autoregressive steps, and then the target model will verify all of them in a single forward pass.

The Reasoning: Why 3 Steps?

This message did not appear in a vacuum. It was the third point in a deliberate sweep across the step-count hyperparameter, and understanding why requires examining the two preceding data points.

The 5-step baseline (6 draft tokens): Earlier in the session, the assistant had launched the server with num_steps=5 and num_draft_tokens=6, achieving an average of 71.3 tok/s. This was a significant improvement over the broken 46.7–54.8 tok/s range from earlier debugging sessions, but still well below the 90 tok/s non-speculative baseline. The accept length was hovering around 2.0–2.3 tokens per verify cycle — meaning the draft model was only getting about 2 out of 6 proposed tokens accepted.

The 10-step experiment (11 draft tokens): Reasoning that more draft steps might increase the number of accepted tokens per verify cycle, the assistant then tried num_steps=10 with num_draft_tokens=11. The result was worse: 60.0 tok/s. The accept rate per token dropped to 0.16–0.27, indicating severe diminishing returns beyond step 5. Each additional draft step added overhead without producing proportionally more accepted tokens.

The 3-step hypothesis: With two data points showing that 10 steps was too many and 5 steps was better, the assistant naturally moved in the opposite direction: fewer steps. The reasoning was clear and quantitative. Each verify cycle has a fixed overhead: the target model must process all draft tokens in a prefill-like forward pass. If the accept length is ~2.1 tokens, then generating 6 draft tokens (5 steps) means 4 out of 6 are wasted — but the target model still pays the cost of verifying all 6. With fewer steps, the wasted tokens are reduced, and the verify cycle becomes shorter. The tradeoff is that with fewer steps, each cycle produces fewer accepted tokens, so more cycles are needed per second. The optimal point balances these competing forces.

The Decision-Making Process

What makes this message remarkable is not the command itself but the decision-making framework it represents. The assistant was conducting a systematic, empirical hyperparameter sweep — not guessing randomly, but using each result to inform the next choice.

The assistant's thinking, visible in the preceding messages, reveals a sophisticated mental model of the system's dynamics:

1. Bottleneck identification: Through profiling instrumentation added to the eagle worker, the assistant had discovered that the target model verify forward consumes 95%+ of the cycle time (21–28ms), while the draft model is negligible (<5%). This means the key optimization lever is not making the draft model faster, but maximizing the ratio of accepted tokens per verify cycle.
2. Cost-benefit analysis: Each verify cycle has a fixed cost (the target model forward pass) and a variable benefit (the number of accepted tokens). The draft steps have a small but non-zero cost. The assistant was empirically mapping the function: throughput = f(num_steps).
3. Diminishing returns awareness: The 10-step result confirmed that acceptance probability per step decays rapidly. If step 1 has a 47% acceptance rate, step 2 might have 30%, step 3 might have 20%, and so on. Beyond some point, each additional step contributes negligible expected accepted tokens while still incurring the draft model's forward cost. The 3-step configuration was the logical next point in this sweep. It tested the hypothesis that the optimal lies below 5 steps, potentially at 3 or even 2 steps.

Assumptions Embedded in the Message

Every decision carries assumptions, and this message is no exception:

• The step count is the primary lever: The assistant assumes that varying num_steps (and correspondingly num_draft_tokens) is the most impactful configuration change to test. Other parameters like --speculative-eagle-topk, --num-continuous-decode-steps, and --mem-fraction-static are held constant.
• Monotonicity of the relationship: The assistant implicitly assumes that the optimal step count is somewhere between 1 and 10, and that testing 3 steps will reveal whether the optimum is below or above 5. This is a reasonable assumption but not guaranteed — the relationship could have multiple local optima.
• The draft model quality is fixed: By keeping the draft model checkpoint constant (/data/eagle3/output_100k_sglang/4), the assistant assumes that the acceptance characteristics are determined solely by the step count, not by model quality. In reality, a better-trained draft model would shift the optimal step count higher.
• CUDA graph capture is reproducible: The server uses CUDA graphs (evident from the cuda graph: True entries in the logs). The assistant assumes that graph capture completes successfully and that the captured graphs are equally efficient across different step counts.
• Single-request throughput is the right metric: The benchmark uses a single concurrent request. The assistant assumes that optimizing for single-stream throughput will also benefit multi-request scenarios, which is not necessarily true — speculative decoding can interact differently with batching.

Input Knowledge Required

To understand this message, one needs substantial domain knowledge:

• Speculative decoding architecture: Understanding that EAGLE-3 uses a lightweight draft model to propose tokens, which the target model then verifies in parallel. The tradeoff is between draft overhead (multiple small forward passes) and verify efficiency (one large batched forward pass).
• SGLang server configuration: Knowledge of the specific flags (--speculative-num-steps, --speculative-num-draft-tokens, --tp-size, --disable-custom-all-reduce, etc.) and their meanings. The --speculative-num-steps parameter controls how many autoregressive steps the draft model takes, while --speculative-num-draft-tokens is the total number of draft tokens generated (which is num_steps + 1 for EAGLE-3, since the first token is from the target model's previous output).
• The Kimi-K2.5 model architecture: A Mixture-of-Experts model with 8 tensor-parallel GPUs, using INT4 quantization. The target model's verify forward pass processes all draft tokens in a single prefill-like operation, which is more efficient than processing them individually.
• Previous benchmark results: The 71.3 tok/s (5 steps) and 60.0 tok/s (10 steps) results that motivated this experiment. Without this context, the 3-step trial appears arbitrary.
• The training data pipeline: The draft model was trained on 100K samples of hidden state pairs extracted from the target model. The accept rate of ~2.1 tokens reflects the quality of this training data.

Output Knowledge Created

This message generated several forms of knowledge:

1. A benchmark result for 3-step speculation: The assistant would later benchmark this configuration, producing a throughput measurement that could be compared against the 5-step and 10-step results.
2. Accept rate statistics: The server logs would reveal the accept length and accept rate for 3-step speculation, showing how the acceptance probability decays across steps 1, 2, and 3.
3. A data point in the optimization curve: Together with the other step counts, this result would allow the assistant to identify the optimal configuration. The chunk summary reveals that 2 steps (3 draft tokens) ultimately proved optimal at 94 tok/s — meaning the 3-step result likely showed improvement over 5 steps but was still suboptimal.
4. Confirmation of the diminishing returns hypothesis: If 3 steps performed better than 5 steps, it would confirm that the marginal benefit of additional steps decays faster than the marginal cost. If 3 steps performed worse, it would suggest the optimal lies at 5 or higher.

The Thinking Process

The assistant's reasoning, reconstructed from the surrounding messages, reveals a methodical optimization approach:

Step 1: Measure the baseline. The non-speculative throughput was ~90 tok/s. This established the target to beat.

Step 2: Test a reasonable default. Starting with 5 steps (a common default for EAGLE-style speculation) yielded 71 tok/s — below baseline. This was disappointing but informative.

Step 3: Diagnose the gap. The assistant analyzed the cycle composition: 5 draft forward passes (~3ms each = 15ms) plus 1 target verify pass (~15-20ms for 6 tokens) = ~30-35ms per cycle, producing 2.1 tokens. This matched the observed 60-70 tok/s.

Step 4: Test the extremes. Trying 10 steps tested whether more aggressive speculation could overcome the overhead. The result (60 tok/s) was worse, confirming that diminishing returns dominate.

Step 5: Test the opposite extreme. The 3-step trial (this message) tested whether fewer steps reduce overhead enough to increase throughput despite lower accept volume per cycle.

Step 6 (later): Test 2 steps. The chunk summary reveals that 2 steps (3 draft tokens) was the optimal configuration, achieving 94 tok/s — 5.9% above baseline. This is the culmination of the sweep.

What's notable is the absence of guesswork. Each step was motivated by quantitative reasoning based on measured data. The assistant didn't just try random configurations — it built a mental model of the system's dynamics and used each result to refine that model.

Mistakes and Incorrect Assumptions

While the systematic approach was sound, several assumptions proved incomplete:

• The optimal was 2 steps, not 3: The assistant hypothesized that the optimum lay between 3 and 5 steps. In reality, 2 steps was optimal. This means the assistant underestimated the overhead of even a single extra draft step, or overestimated the marginal acceptance probability of step 3.
• The draft model's TP8 configuration: Earlier in the conversation, the user had asked whether running the draft model on all 8 GPUs was wasteful. The assistant investigated but found no built-in SGLang support for separate TP groups. The draft model was running on TP8, meaning every single-token draft forward paid the full cross-GPU communication overhead. This likely made each draft step more expensive than the assistant's mental model assumed, which would explain why fewer steps performed better than expected.
• Focusing on step count rather than training data: The chunk summary notes that AQ-MedAI's Kimi-K2-Instruct-eagle3 model, trained on 38× more data (1.4M vs 37K samples), achieved accept lengths of 3.2–3.5. This suggests that the highest-leverage improvement would be more training data, not step count tuning. The assistant's optimization sweep, while valuable, was optimizing within a local maximum constrained by draft model quality.

Broader Significance

This message exemplifies a crucial skill in ML systems engineering: the ability to systematically optimize a complex system with multiple interacting parameters. The assistant didn't just try one configuration and declare victory or defeat. It conducted a principled sweep, using each result to inform the next decision, and ultimately found a configuration that beat the non-speculative baseline.

The 3-step trial was a pivotal data point in this sweep. It helped establish that the optimal step count was below 5, narrowing the search space and leading directly to the 2-step optimum. Without this trial, the assistant might have concluded that 5 steps was optimal and accepted the 71 tok/s result as the best possible.

In the broader narrative of this coding session, this message represents the transition from "does speculation work at all?" to "what's the best way to configure speculation?" — a shift from debugging to optimization that marks the maturation of the deployment.