The Critical Parameter Tuning Decision: Why Reducing Draft Tokens Was the Right Call After Fixing EAGLE-3

In the middle of an intense debugging session spanning dozens of messages, message [msg 3616] represents a quiet but crucial pivot. After spending hours tracing a devastating bug where EAGLE-3 speculative decoding was silently failing due to a flag mismatch (--speculative-algorithm EAGLE vs EAGLE3), the assistant had finally confirmed the fix worked: hidden states were now correctly arriving as 21504-dimensional concatenated tensors instead of the broken 7168-dim final-layer-only states. But the victory was bittersweet. The benchmark results showed the EAGLE-3 drafter achieving only 56.7 tok/s — well below the 90 tok/s non-speculative baseline. The draft model was working, but it wasn't helping.

This message captures the moment the assistant stepped back from the celebration of a successful bug fix to confront a harder question: why isn't this faster, and what can we do about it?

The Context: A Bug That Sabotaged Everything

To understand the weight of message [msg 3616], one must understand what came before. The assistant had been building an EAGLE-3 speculative decoding pipeline for the Kimi-K2.5 model — a massive 8-GPU deployment. EAGLE-3 is a sophisticated speculative decoding technique that uses a lightweight "draft" model to predict multiple future tokens, which are then verified in parallel by the full target model. When it works, it can dramatically accelerate inference. But for days, the draft model was achieving zero acceptance — every token it predicted was being rejected by the target model, making speculation purely overhead.

The root cause, traced across messages [msg 3601] through [msg 3605], was shockingly simple. The server had been started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. In SGLang's codebase, the is_eagle3() method performs a strict string match — only the literal string "EAGLE3" triggers the auxiliary hidden state capture mechanism. With "EAGLE", the target model never concatenated intermediate layer hidden states, so the draft model received only 7168-dimensional final-layer embeddings instead of the expected 21504-dimensional concatenated states. The draft model's fusion layer (fc, a 21504→7168 projection) was silently bypassed, and all trained weights were effectively dead.

After restarting with the correct flag in [msg 3607], the debug logs confirmed the fix: hidden states arriving as torch.Size([21, 21504]). The acceptance length jumped from 1.0 (zero acceptance) to ~2.2. But the throughput still lagged behind the baseline.

The Reasoning in Message 3616

Message [msg 3616] opens with a deliberate analytical step: "Let me understand the parameters better." This is the assistant taking stock of the situation, performing a back-of-the-envelope calculation to understand why the numbers don't add up.

The assistant reasons through the math:

accept_len ≈ 2.2: On average, 2.2 draft tokens are accepted per verification step.
num_draft_tokens = 16: The server generates 16 candidate tokens in each speculation round.
accept rate = 2.2 / 16 ≈ 0.14 (14%): Only 14% of drafted tokens are accepted.
num_steps = 3 and topk = 4: These parameters control the tree-structured speculation — the draft model generates 4 candidates at each of 3 steps, creating a branching tree of 4 × 4 × 4 = 64 possible sequences (though the actual number is managed differently). The key insight is that 16 draft tokens are being generated, but only ~2.2 are accepted on average. The overhead of generating and verifying 16 candidates far outweighs the benefit of accepting 2.2. In speculative decoding, the speedup formula is roughly: $$ \text{speedup} = \frac{\text{accept\_len}}{\text{overhead\_factor}} $$ When accept_len is only 2.2 and the overhead of running the draft model plus verifying 16 candidates is high (especially without CUDA graphs, which were disabled), the net result is a slowdown. The assistant's decision: reduce to 5 or 8 draft tokens. This is a classic speculative decoding tuning strategy. Fewer draft tokens means: 1. Less computation in the draft model (fewer forward passes) 2. Less verification cost (fewer parallel target model runs) 3. Lower absolute acceptance (you can't accept more tokens than you draft) 4. But potentially higher accept rate (the first few tokens are easier to predict) The tradeoff is that with fewer draft tokens, the maximum possible speedup is capped. But when acceptance rates are low, reducing draft tokens can dramatically improve efficiency by cutting overhead.

Assumptions and Knowledge

This message rests on several implicit assumptions:

The draft model is fundamentally limited by training data. The assistant had trained the drafter on only 10,000 samples. The EAGLE-3 paper shows clear scaling laws — more training data yields higher acceptance rates. The accept_len of ~2.2 might be near the ceiling for a 10K-sample drafter. The assistant doesn't assume that parameter tuning alone will solve the problem, but rather that it's the most immediately actionable lever.

CUDA graphs are a major factor. The server was started with --disable-cuda-graph, which eliminates the overhead of recompiling CUDA kernels but also removes optimization opportunities. With CUDA graphs enabled, the overhead of running 16 draft tokens might be significantly lower, changing the optimal tradeoff.

The target model's verification cost is the bottleneck. In speculative decoding, the target model must verify all drafted tokens in parallel. With 8 GPUs and tensor parallelism, this verification step is expensive. Reducing draft tokens directly reduces this cost.

The Kimi-K2.5 model's architecture matters. The model uses MLA (Multi-head Latent Attention), which may interact differently with speculative decoding than standard attention. The EAGLE-3 paper's results may not directly transfer.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A confirmed baseline: The EAGLE-3 drafter, even with correct hidden states, achieves accept_len ~2.2 on this model — far below the ~5-6 needed for significant speedup.
A tuning hypothesis: Reducing draft tokens from 16 to 5-8 may improve throughput by reducing overhead, even if absolute acceptance drops.
A decision point: The assistant implicitly decides that parameter tuning is worth trying before scaling up training data (which would take 24-55 hours for 83K samples).
A methodology: The message demonstrates how to interpret speculative decoding metrics — connecting accept_len, accept_rate, num_draft_tokens, and throughput in a coherent mental model.

The Broader Arc

Message [msg 3616] sits at a inflection point in the session. The assistant has just fixed the critical bug that made EAGLE-3 completely non-functional. Now it faces the harder problem: making it actually faster than the baseline. This message represents the first step in that optimization journey — a deliberate, analytical approach to parameter tuning before resorting to the expensive option of scaling up training data.

The decision to kill the server and restart with new parameters (executed in the bash command) is a microcosm of the entire session's rhythm: diagnose, fix, benchmark, analyze, tune, repeat. Each cycle brings the system closer to the goal of faster inference, but each cycle also reveals new layers of complexity in the interaction between speculative decoding algorithms, model architectures, and hardware constraints.