The Art of Systematic Optimization: Tuning EAGLE-3 Speculative Decoding Parameters

In the high-stakes world of large language model inference, every millisecond counts. When deploying a 236-billion-parameter model like Kimi-K2.5 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a good and great token throughput can determine whether speculative decoding is a net win or a costly overhead. Message 3638 in this coding session captures a pivotal moment in that optimization journey — a single bash command that represents the third iteration of a systematic parameter search for EAGLE-3 speculative decoding.

The Message

ssh root@10.1.230.174 'sleep 3 && nohup bash -c "NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS \
  NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 \
  SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 ~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 --trust-remote-code --tp-size 8 \
  --mem-fraction-static 0.85 --host 0.0.0.0 --port 8000 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_10k_sglang/4 \
  --speculative-num-steps 3 --speculative-eagle-topk 4 --speculative-num-draft-tokens 5 \
  --num-continuous-decode-steps 4 --disable-custom-all-reduce \
  --log-level info" > /data/eagle3/sglang_eagle3_cg_5tok.log 2>&1 &'
echo "Starting EAGLE3 CUDA graphs 5 draft tokens..."

At first glance, this looks like any other server launch command — a wall of environment variables, flags, and paths. But in the context of the preceding 25 messages, it represents a carefully reasoned experimental step in a debugging and optimization saga that began with a fundamental bug fix and has now moved into fine-grained performance tuning.

The Reasoning Behind the Command

To understand why this message was written, we need to trace the chain of events that led to it. The story begins with a critical bug discovered in the EAGLE-3 speculative decoding pipeline. The assistant had been battling with a zero acceptance rate — the draft model's predictions were being rejected entirely, meaning speculative decoding was providing zero benefit. The root cause turned out to be a subtle flag mismatch: the server was being started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. This seemingly minor difference had devastating consequences because the is_eagle3() check in the SGLang codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. Without this concatenation, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states, causing the fusion layer to be silently bypassed and all trained weights to be useless.

Once this bug was fixed (msg 3614-3615), the acceptance rate jumped from zero to approximately 15% (accept_len ~2.1-2.5 out of 16 draft tokens). This was a significant improvement — the draft model was finally receiving correct hidden states and making predictions that sometimes matched the target model's distribution. However, the throughput remained disappointing: approximately 56.7 tok/s with CUDA graphs disabled, compared to a 90 tok/s non-speculative baseline. The speculation was actually slowing down inference rather than speeding it up.

The assistant identified three contributing factors: the accept rate of ~15% was too low to overcome speculation overhead; CUDA graphs were disabled, imposing a massive per-step penalty; and the draft model had only been trained on 10,000 samples, whereas the EAGLE-3 paper demonstrated clear scaling with more data.

A Systematic Exploration of the Parameter Space

What followed was a methodical exploration of the speculative decoding parameter space. The assistant formulated a clear hypothesis: if the accept rate is fixed (determined by the quality of the draft model, which is constrained by limited training data), then the only way to improve throughput is to reduce the overhead per speculation step. The primary lever for this is the number of draft tokens — fewer draft tokens means fewer verification passes, lower latency per step, and a better overhead-to-gain ratio even with a modest accept length.

The first experiment (msg 3618) tested 5 draft tokens without CUDA graphs, yielding 53.2 tok/s — actually slightly worse than the 16-token configuration. The assistant then tested the AQ-MedAI drafter (a general-purpose K2 drafter trained on more data) with 16 draft tokens, which achieved 50.5 tok/s — confirming that the custom K2.5-trained drafter was better but still data-limited. The critical breakthrough came when CUDA graphs were enabled with 16 draft tokens (msg 3632): throughput jumped to 74.9 tok/s, a 40% improvement over the non-CUDA-graph configuration.

This brings us to message 3638. The assistant now has a clear picture: CUDA graphs are essential (providing a ~40% boost), but 74.9 tok/s is still below the 90 tok/s baseline. The reasoning is that with 16 draft tokens and an accept rate of ~13-16%, the verification overhead for 16 candidates is still too high. The next logical step is to combine CUDA graphs with fewer draft tokens — specifically 5 — to see if the reduced verification cost can push throughput past the baseline. The command in message 3638 is identical to the earlier 5-token test (msg 3618) except for one critical difference: --disable-cuda-graph has been removed, allowing CUDA graphs to be enabled.## Assumptions Embedded in the Command

This message makes several implicit assumptions worth examining. First, the assistant assumes that the NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, etc.) represent an optimal configuration for the 8-GPU topology. These settings were tuned in earlier segments (segment 24) to maximize single-stream throughput, and the assistant is carrying them forward without re-validation — a reasonable assumption given the time cost of re-tuning, but one that could mask interaction effects between NCCL settings and speculative decoding.

Second, the assistant assumes that the draft model checkpoint at /data/eagle3/output_10k_sglang/4 is the best available. This is the model trained on 10,000 SGLang-extracted samples in segment 25. While it demonstrably works (accept_len ~2.1), the assistant has not explored whether intermediate checkpoints or different training hyperparameters might yield better acceptance. The decision to accept the current draft model quality as fixed and optimize around it rather than invest in better training data is a pragmatic trade-off — but it's an assumption that constrains the entire optimization search.

Third, the assistant assumes that the --speculative-num-steps 3 and --speculative-eagle-topk 4 parameters are near-optimal. These were carried forward from the initial EAGLE-3 configuration and never systematically varied. The num_steps parameter controls how many draft generation steps the draft model runs before verification, while topk controls the beam width for candidate generation. Both interact with num_draft_tokens in complex ways — reducing draft tokens while keeping 3 steps and topk 4 may leave the draft model underutilized. A more thorough search would vary all three parameters simultaneously, but the assistant is proceeding one variable at a time.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, reveals a structured experimental methodology. The thought process follows a clear pattern: observe a metric (throughput), identify the bottleneck (overhead > gain), formulate a hypothesis (fewer draft tokens reduces overhead), test the hypothesis (msg 3618 with 5 tokens, no CUDA graphs), observe the result (53.2 tok/s, worse), pivot to a different lever (enable CUDA graphs, msg 3632), observe the result (74.9 tok/s, better but still below baseline), and then combine the two levers (message 3638: CUDA graphs + 5 tokens).

This is textbook scientific optimization: isolate variables, measure systematically, and build incrementally. The assistant is not randomly trying configurations but is guided by a mental model of how speculative decoding works. The key insight is that speculation overhead has two components: the draft model forward pass cost (fixed per step) and the verification cost (proportional to number of draft tokens). CUDA graphs reduce the fixed per-step cost by pre-compiling execution traces, while reducing draft tokens reduces the verification cost. The assistant correctly identifies that these are complementary optimizations and tests them in combination.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The SGLang inference engine's flag system is essential — understanding that --speculative-algorithm EAGLE3 triggers specific code paths for hidden state concatenation, and that --disable-cuda-graph toggles between eager-mode execution and graph-compiled execution. The EAGLE-3 speculative decoding algorithm itself must be understood: how the draft model generates candidate tokens, how the target model verifies them in parallel, and how the acceptance rate and accept length metrics relate to throughput. The NCCL environment variables require knowledge of NVIDIA's collective communications library and how protocols like LL (Low Latency) and algorithms like Ring affect inter-GPU communication in an 8-way tensor-parallel configuration. Finally, the model architecture — Kimi-K2.5 with Multi-head Latent Attention (MLA) and 236B parameters — constrains what performance is even theoretically possible.

Output Knowledge Created

This message creates several forms of knowledge. Most immediately, it produces a new server instance with a specific configuration that will be benchmarked in the following messages. The log file at /data/eagle3/sglang_eagle3_cg_5tok.log will contain the acceptance metrics, CUDA graph capture times, and per-step throughput that will inform the next optimization decision. But more broadly, this message contributes to a growing empirical map of the EAGLE-3 performance landscape on this specific hardware configuration. Each data point — 53.2 tok/s with 5 tokens no graphs, 74.9 tok/s with 16 tokens with graphs, and the forthcoming result with 5 tokens with graphs — helps the assistant understand the shape of the optimization surface and where the true bottlenecks lie.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption in this message is that reducing draft tokens from 16 to 5, combined with CUDA graphs, will yield a net improvement. In the earlier test without CUDA graphs (msg 3622), 5 draft tokens produced 53.2 tok/s compared to 56.7 tok/s with 16 tokens — a slight decrease. The assistant is betting that CUDA graphs change the cost structure enough to make 5 tokens the better choice. This is a reasonable hypothesis, but it's not guaranteed. In fact, looking ahead to the results (which appear in later messages), the 5-token CUDA graph configuration achieves approximately 82.3 tok/s — an improvement over 74.9 tok/s with 16 tokens, but still below the 90 tok/s baseline. The assistant's assumption was partially validated (fewer tokens did help with CUDA graphs), but the improvement was not enough to cross the threshold.

Another subtle assumption is that the --disable-custom-all-reduce flag is still needed. This flag was introduced during the CUDA graph debugging phase because custom all-reduce was causing hangs. The assistant never verified whether this restriction could be lifted now that CUDA graphs are working correctly. Custom all-reduce typically provides a 10-20% throughput improvement in tensor-parallel configurations, so leaving it disabled may be costing more performance than the assistant realizes.

The Broader Context

This message sits at a inflection point in the session. The assistant has resolved the critical bug, validated that the draft model works, and is now in the fine-tuning phase — trying to squeeze every last token per second out of the system. The results of this experiment will determine the next major decision: whether to invest more effort in tuning speculation parameters or to pivot to the only remaining lever — scaling up the training data. As the assistant noted earlier, the EAGLE-3 paper shows clear scaling with more data, and a parallel effort to generate 83,000 training samples is already underway. Message 3638 is the last systematic attempt to make the current draft model work before accepting that more data is the only path forward.