The 53.2 tok/s Benchmark: A Diagnostic Data Point in the EAGLE-3 Optimization Journey

In the course of deploying speculative decoding for the Kimi-K2.5 language model on an 8-GPU server, a single benchmark result arrived at message index 3622, reporting an average throughput of 53.2 tokens per second. This figure, produced by running a Python benchmark script against an SGLang inference server configured with the EAGLE-3 speculative decoding algorithm, represents far more than a simple performance measurement. It is a diagnostic data point that crystallizes the outcome of a multi-hour debugging saga, tests a specific hypothesis about speculation overhead, and ultimately forces a strategic pivot in the entire optimization approach.

The Context: A Bug Fixed, But Performance Still Lacking

To understand why this message was written, one must trace the events that immediately preceded it. The assistant had just resolved a critical bug that had rendered the EAGLE-3 draft model completely ineffective. The root cause, discovered through painstaking code tracing across multiple task subagents ([msg 3603], [msg 3604]), was a flag mismatch: the SGLang server had been started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in the SGLang codebase is strict — only the literal string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only hidden states instead of the expected 21504-dimensional concatenated states, causing the fusion layer (fc.weight) to receive input of the wrong dimensionality and effectively bypassing all trained weights.

After restarting the server with the correct EAGLE3 flag ([msg 3607]), the assistant confirmed that hidden states were now arriving as 21504-dimensional tensors ([msg 3609]). The initial benchmark with --speculative-num-draft-tokens 16 produced an average of 56.7 tok/s ([msg 3612]), which was still substantially slower than the 90 tok/s non-speculative baseline established earlier in the session ([msg 3624]). The server logs revealed an accept length of approximately 2.0–2.5 tokens per verification step and an accept rate of roughly 12–16% ([msg 3614]).

This created a puzzle: the draft model was now receiving correct hidden states and some predictions were being accepted, yet speculation was slower than no speculation at all. The assistant's analysis in [msg 3615] identified three contributing factors: (1) the accept rate of ~15% was too low for the overhead of verifying 16 draft candidates, (2) CUDA graphs were disabled, incurring a significant per-step overhead penalty, and (3) the draft model had been trained on only 10K samples, which the EAGLE-3 paper's scaling curves suggested was insufficient.

The Hypothesis: Reducing Overhead by Cutting Draft Tokens

The message in question ([msg 3622]) was written to test a specific hypothesis: if the speculation overhead is too high relative to the accept length, reducing the number of draft tokens should improve the overhead-to-gain ratio. The reasoning, articulated in [msg 3616], was that with num_draft_tokens=16 and an accept length of ~2.2, the accept rate is only 2.2/16 = 0.14 — meaning 86% of the draft computation is wasted. By reducing to 5 draft tokens, the accept rate would rise to 2.2/5 = 0.44, meaning a much higher fraction of draft computation is productive. Even though the absolute number of accepted tokens per step remains the same (~2.1), the verification cost drops because fewer candidates need to be scored against the target model's logits.

The assistant killed the existing server and launched a new instance with --speculative-num-draft-tokens 5 ([msg 3618]), keeping all other parameters identical: --speculative-num-steps 3, --speculative-eagle-topk 4, and --disable-cuda-graph. After waiting for the server to become healthy ([msg 3621]), the assistant ran the benchmark script that produced the subject message.

The Message Itself: A Benchmark Result

The message consists of a single bash command execution that runs a Python benchmark script on the remote server. The script sends five chat completion requests to the SGLang server's OpenAI-compatible API endpoint, each requesting 1024 tokens of generation on a different prompt. It measures elapsed time per request and computes throughput in tokens per second. The results show:

The Interpretation: Hypothesis Not Confirmed

The critical finding is that 53.2 tok/s with 5 draft tokens is actually worse than 56.7 tok/s with 16 draft tokens. This is counterintuitive — reducing overhead should improve throughput, not degrade it. The explanation lies in the accept length. As the assistant noted in [msg 3623], the server logs for the v3b configuration showed accept_len ~2.0-2.2 and accept_rate ~0.41-0.45. The accept rate increased from ~0.14 to ~0.42 (as expected), but the accept length remained essentially unchanged at ~2.1 tokens per verification step.

This means the draft model's predictive quality is the bottleneck, not the speculation overhead. The draft model can only predict ~2 tokens correctly on average before making a mistake, regardless of how many candidates it generates. Reducing the number of draft tokens from 16 to 5 didn't help because the fundamental limitation is the draft model's accuracy, not the cost of verifying extra candidates. In fact, with fewer draft tokens, the server may run the draft model less efficiently (worse GPU utilization for smaller batch sizes), explaining the slight regression from 56.7 to 53.2 tok/s.

The assistant's subsequent analysis in [msg 3624] crystallized this understanding: "The fundamental problem: accept_len of ~2.1 is too low. With an ideal EAGLE-3 drafter, accept_len should be 3-4+. Our drafter trained on only 10K samples just isn't good enough."

The Strategic Pivot

This benchmark result was the turning point. It demonstrated conclusively that tweaking speculation parameters could not overcome the draft model's data limitation. The assistant immediately pivoted to two parallel strategies: (1) testing the AQ-MedAI drafter (a general K2 drafter trained on more data) to see if a better-trained model could achieve higher acceptance ([msg 3624]), and (2) scaling up the training dataset by 10× to retrain the custom drafter with substantially more data ([msg 3630] onward).

The AQ-MedAI drafter benchmark ([msg 3628]) produced 50.5 tok/s with accept_len ~1.8-2.1, slightly worse than the custom drafter. This confirmed that the custom K2.5-trained drafter was better but still data-limited. The assistant then enabled CUDA graphs ([msg 3630]) and achieved 82.3 tok/s — still below the 90 tok/s baseline but much closer, suggesting that CUDA graph overhead was a significant factor.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, the concept of speculative decoding: a technique where a small "draft" model generates multiple candidate tokens cheaply, and a large "target" model verifies them in parallel, accepting correct predictions and discarding incorrect ones. The accept length measures how many draft tokens are accepted on average per verification step, and the accept rate is accept length divided by the number of draft tokens.

Second, the EAGLE-3 architecture specifics: it captures hidden states from intermediate layers of the target model (layers 2, 30, and 58 for the 60-layer Kimi-K2.5), concatenates them into a 21504-dimensional vector (3 × 7168), and feeds them through a fusion layer (fc.weight) that projects down to 7168 dimensions. The draft model then predicts future tokens autoregressively using these fused representations.

Third, the SGLang server configuration: the --speculative-num-draft-tokens flag controls how many tokens the draft model generates per speculation step, --speculative-num-steps controls how many draft steps to run, and --disable-cuda-graph disables CUDA graph optimization (which reduces per-step overhead but requires static shapes).

Fourth, the training data context: the draft model was trained on only 10,000 samples extracted from the Kimi-K2.5 model's own outputs using a custom hidden state extraction pipeline. The EAGLE-3 paper demonstrates that acceptance rate scales with training data quantity, suggesting that orders of magnitude more data may be needed.

Output Knowledge Created

This message created a critical piece of empirical knowledge: the EAGLE-3 draft model trained on 10K samples achieves an accept length of ~2.1 tokens regardless of speculation configuration, and this is insufficient to overcome the overhead of speculative decoding on this hardware. The 53.2 tok/s figure serves as a baseline for the "reduced overhead" configuration, demonstrating that the bottleneck is predictive quality, not computational efficiency.

More importantly, this message established that the optimization strategy needed to change from parameter tuning to data scaling. The subsequent 10× dataset expansion (from 10K to ~88K samples) and the launch of a 24-55 hour inference pipeline to regenerate responses for 83K prompts were direct consequences of the insight gained from this benchmark.

Assumptions and Limitations

The assistant operated under several assumptions. It assumed that reducing draft tokens from 16 to 5 would improve throughput by reducing verification overhead while maintaining the same accept length. This assumption was partially correct — the accept length remained stable at ~2.1 — but the throughput actually decreased slightly, suggesting that GPU utilization efficiency for smaller speculation batches may be worse. The assistant also assumed that the benchmark script's five prompts were representative of general performance, though the 40% variation between prompts suggests significant prompt-dependent behavior.

A potential mistake was not testing intermediate values (e.g., 8 or 10 draft tokens) before jumping to 5. The non-monotonic relationship between draft tokens and throughput (16 tokens → 56.7 tok/s, 5 tokens → 53.2 tok/s) suggests there may be an optimum between these values. However, given that even the best configuration (16 tokens) was far below the 90 tok/s baseline, exploring the parameter space more finely would not have changed the strategic conclusion: the draft model needs more training data.

Conclusion

The 53.2 tok/s benchmark in [msg 3622] is a deceptively simple message that encapsulates a pivotal moment in a complex optimization effort. It represents the convergence of a bug fix, a hypothesis test, and a strategic decision point. The message's true significance lies not in the number itself but in what it revealed about the system's behavior: that the EAGLE-3 draft model's accept length is fundamentally limited by training data quantity, not by speculation configuration. This insight redirected the entire effort from parameter tuning to data scaling, ultimately leading to the 10× dataset expansion that would define the next phase of the project.