The Art of the Incremental Tweak: Optimizing EAGLE-3 Speculative Decoding One Parameter at a Time

Introduction

In the world of large language model inference, every millisecond counts. When you're running an 8-GPU server with a 236-billion-parameter model like Kimi-K2.5, the difference between 82 tokens per second and 90 tokens per second represents not just a 9% performance gap, but potentially hours of saved computation over a long-running workload. The message at index 3642 captures a pivotal moment in a systematic optimization campaign: the assistant, having just achieved its best EAGLE-3 speculative decoding result yet, pauses to analyze the data and then launches the next experiment in a carefully orchestrated search for better performance.

The Message

The subject message is deceptively brief. It reads:

Good data. Accept_len ~2.0-2.4, accept_rate ~0.38-0.47 with 5 draft tokens. Bursts up to 113 tok/s.

Let me also try num_steps=2 with topk=4 to see if fewer speculation steps but lower overhead helps:

[bash] ssh root@10.1.230.174 'pkill -f "sglang.launch_server" && sleep 2 && pkill -9 -f python3; sleep 3 && fuser -k /dev/nvidia* 2>/dev/null; sleep 3 && nohup bash -c "NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 ~/ml-env/bin/python3 -m sglang.launch_server \ --model-path /shared/kimi-k2.5-int4 --trust-remote-code --tp-size 8 \ --mem-fraction-static 0.85 --host 0.0.0.0 --port 8000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path /data/eagle3/output_10k_sglang/4 \ --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 5 \ --num-continuous-decode-steps 4 --disable-custom-all-reduce \ --log-level info" > /data/eagle3/sglang_eagle3_cg_2step.log 2>&1 &' echo "Starting 2-step EAGLE3..." Starting 2-step EAGLE3...

On the surface, this looks like a routine server restart with a parameter change. But beneath this terse exchange lies a rich story of empirical optimization, careful reasoning about trade-offs, and the relentless pursuit of marginal gains that characterizes serious ML engineering work.

The Context: A Long Road to a Working EAGLE-3 Pipeline

To understand why this message matters, we need to appreciate the journey that led to it. The assistant and user had been working for days on deploying EAGLE-3 speculative decoding for the Kimi-K2.5 model. The path was fraught with obstacles. A critical bug had been discovered and fixed just hours earlier: the server was being started with --speculative-algorithm EAGLE instead of EAGLE3. This one-character difference was devastating because SGLang's is_eagle3() check is strict — only the string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states, causing the fusion layer to be silently bypassed and all trained weights to be useless.

After fixing that bug, the assistant systematically benchmarked different configurations. The non-speculative baseline ran at 90 tok/s. Initial EAGLE-3 attempts without CUDA graphs achieved only 53.2 tok/s — far worse than the baseline. Enabling CUDA graphs jumped performance to 74.9 tok/s. Reducing draft tokens from 16 to 5 pushed throughput further to 82.3 tok/s, with bursts reaching 101.8 tok/s on code-generation prompts. Each improvement came from a single parameter change, tested methodically.

The Reasoning: Why Change `num_steps`?

The message at index 3642 represents the next logical step in this optimization sequence. The assistant had just observed that with 5 draft tokens and CUDA graphs, the average accept_len was approximately 2.0–2.4 tokens per verification step, with an accept_rate of 0.38–0.47. The key insight is that accept_rate — the fraction of generated draft tokens that are accepted — is quite high at 38–47% with 5 tokens, compared to only 13% with 16 tokens. This suggests that with fewer draft tokens, the draft model's predictions are more accurate per token, and less computation is wasted on tokens that get rejected.

However, the overall throughput of 82.3 tok/s still lagged behind the 90 tok/s baseline. The assistant's hypothesis is that the overhead of running the speculation pipeline — even with CUDA graphs — is eating into the gains from accepted tokens. Each speculation step involves running the draft model autoregressively, then running the target model for verification. With num_steps=3 and topk=4, the draft model runs three times, generating a tree of candidates. Reducing to num_steps=2 means one fewer draft model execution per verification cycle, which should reduce overhead.

The reasoning is explicit in the message: "to see if fewer speculation steps but lower overhead helps." This captures the central trade-off in speculative decoding: more steps generate more candidates and potentially higher acceptance, but each step adds computational cost. The assistant is probing whether the marginal benefit of the third step is worth its cost. If accept_len drops only slightly (say from 2.1 to 1.8) while overhead drops significantly, the net throughput could improve.

The Decision-Making Process

The decision to try num_steps=2 is not arbitrary — it follows a clear empirical methodology. The assistant has been systematically exploring the hyperparameter space:

Start with the default configuration (16 draft tokens, 3 steps, no CUDA graphs) → 53.2 tok/s
Enable CUDA graphs → 74.9 tok/s (massive improvement from reduced per-step overhead)
Reduce draft tokens to 5 → 82.3 tok/s (higher accept_rate compensates for fewer candidates)
Now: reduce steps to 2 → hypothesis: lower overhead may improve net throughput Each step isolates one variable. The assistant holds other parameters constant (CUDA graphs enabled, 5 draft tokens, topk=4, same draft model checkpoint) while changing only num_steps from 3 to 2. This is textbook experimental design — control for confounding variables, change one thing at a time, measure the result. The assistant also demonstrates good operational practice. The bash command first kills any existing server processes (pkill -f "sglang.launch_server"), waits for cleanup, forcefully terminates any remaining Python processes, frees GPU memory with fuser -k /dev/nvidia*, and only then launches the new server. The NCCL environment variables for performance tuning are preserved from previous successful runs. The log is written to a distinct file (sglang_eagle3_cg_2step.log) that encodes the configuration parameters, making it easy to correlate results with settings later.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message, some of which may prove incorrect:

That accept_len won't collapse with fewer steps. With num_steps=3 and topk=4, the draft model generates a tree of candidates: at step 1, it generates 4 candidates from the initial hidden state; at step 2, it generates 4 candidates from each of those 4 candidates (16 total); at step 3, it generates 4 from each of those 16 (64 total). The verification step then selects the longest prefix that matches the target model's distribution. With only 2 steps, the tree has at most 4 + 16 = 20 candidates instead of 4 + 16 + 64 = 84. The assistant assumes the third step contributes marginal acceptance gains that don't justify its cost — but this is an empirical question.

That the server will start successfully. Previous attempts sometimes required multiple restarts due to CUDA graph capture failures or memory issues. The assistant's command includes sleep 3 between kill and launch, but doesn't verify GPU memory is actually freed before proceeding.

That the NCCL tuning parameters remain optimal. The environment variables NCCL_PROTO=LL, NCCL_ALGO=Ring, etc., were tuned for the previous configuration. A different number of speculation steps changes the communication pattern between GPUs, potentially shifting the optimal NCCL settings.

That the benchmark is representative. The benchmark script uses 5 fixed prompts. If the 2-step configuration happens to perform particularly well or poorly on these specific prompts, the results may not generalize.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

Speculative decoding fundamentals: Understanding that EAGLE-3 uses a lightweight draft model to propose multiple candidate tokens, which the target model then verifies in a single forward pass. The acceptance rate determines how many draft tokens are kept.

EAGLE-3's tree attention mechanism: The num_steps parameter controls the depth of the candidate tree, while topk controls the branching factor at each step. Together they determine the total number of candidates and the computational cost.

CUDA graphs: A feature that captures a sequence of GPU operations into a reusable graph, eliminating kernel launch overhead. This is critical for speculative decoding where the same sequence of operations repeats many times.

SGLang server architecture: Understanding flags like --tp-size 8 (tensor parallelism across 8 GPUs), --disable-custom-all-reduce, and --num-continuous-decode-steps.

NCCL tuning: The environment variables control how NVIDIA's collective communications library handles inter-GPU communication, which is critical for tensor-parallel inference.

The project history: The draft model was trained on only 10,000 samples extracted from Kimi-K2.5's own outputs, which the assistant suspects is insufficient for good acceptance rates.

Output Knowledge Created

This message creates several forms of knowledge:

A testable hypothesis: That reducing speculation steps from 3 to 2 improves net throughput by reducing overhead more than it reduces acceptance.

An experimental record: The server log file sglang_eagle3_cg_2step.log will capture accept_len, accept_rate, and throughput metrics that can be compared against the previous runs.

A reproducible configuration: The exact command with all flags is recorded, allowing the experiment to be reproduced or the configuration to be deployed if successful.

A data point in the optimization landscape: Regardless of outcome, this experiment narrows the search space. If 2 steps performs worse, the assistant learns that 3 steps is the minimum viable depth. If it performs better, a new direction opens (perhaps even trying 1 step).

The Thinking Process

The assistant's reasoning in this message reveals a sophisticated mental model of the system's behavior. The brief phrase "Good data" is not casual — it signals that the assistant has absorbed the benchmark results and is satisfied with their quality. The accept_len range of 2.0–2.4 and accept_rate of 0.38–0.47 are interpreted as promising but insufficient.

The key insight is the recognition that "fewer speculation steps but lower overhead" might help. This shows the assistant is thinking about the cost per accepted token — not just the acceptance rate in isolation. A configuration that accepts 2.1 tokens with low overhead might beat one that accepts 2.4 tokens with high overhead.

The burst of 113 tok/s mentioned in the message is particularly telling. It suggests that under favorable conditions (certain prompt types, certain positions in the generation), the draft model achieves much higher acceptance. This hints that the average case is being dragged down by hard cases, and that more training data might help smooth out the distribution — which is exactly what the assistant pursues in parallel (scaling the training dataset by 10×).

Conclusion

The message at index 3642 is a masterclass in systematic optimization. It shows how serious ML engineering proceeds not through grand redesigns but through a disciplined sequence of small, measurable experiments. Each parameter change is motivated by a clear hypothesis derived from previous results. Each experiment is carefully controlled, logged, and reproducible. The assistant demonstrates deep understanding of the speculative decoding pipeline, from the mathematical trade-offs of tree attention to the practicalities of NCCL tuning and CUDA graph capture.

Whether the 2-step configuration proves better or worse, the message represents progress — because every experiment, regardless of outcome, narrows the search space and deepens understanding of the system. This is the essence of empirical engineering: not knowing the answer in advance, but knowing exactly how to ask the question.