The 50.5 tok/s Verdict: When Fixing the Bug Isn't Enough

In the long arc of a speculative decoding deployment, there are moments of triumph and moments of reckoning. Message 3628 in this opencode session belongs squarely to the latter category. It is a single benchmark result — a terse block of numbers showing that the AQ-MedAI EAGLE-3 drafter achieves an average of 50.5 tokens per second on the Kimi-K2.5 model. To an outsider, it looks like a routine performance measurement. To anyone who has followed the preceding 23 messages of debugging, it is the quiet closing of a door. This message is the final, unambiguous data point that forces a strategic pivot: after fixing the root-cause bug, after verifying hidden states are correctly concatenated, after trying multiple configurations of draft tokens and step counts, and now after testing an alternative drafter trained on substantially more data, the assistant must confront the fact that EAGLE-3 speculative decoding is not delivering a speedup on this hardware stack. The message is not just a benchmark — it is a verdict.

The Context: A Bug Fixed, But Performance Still Missing

To understand why this message was written, one must understand the debugging odyssey that preceded it. The assistant had been wrestling with a perplexing problem: a custom EAGLE-3 draft model trained on 10,000 samples was achieving zero acceptance rate when deployed on the SGLang server. The draft model's predictions were never being accepted by the target model's verification step, meaning speculative decoding was producing no speedup whatsoever. The root cause, identified in message 3605, was a single flag mismatch: the server had been launched with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. The is_eagle3() check in SGLang's codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states from layers 2, 30, and 58. With the EAGLE flag, the draft model received only 7168-dimensional final-layer hidden states instead of the expected 21504-dimensional concatenated states. The fusion layer (fc) that projects 21504 dimensions down to 7168 was being silently bypassed, and all trained weights were effectively useless.

After restarting the server with the correct flag (message 3607), the assistant verified that hidden states were now arriving at the correct 21504-dimensional shape (message 3609). The debug prints confirmed the fix. But when the first benchmark was run (message 3612), the results were sobering: the custom drafter achieved only 56.7 tok/s on average, compared to the 90 tok/s non-speculative baseline. The acceptance length was approximately 2.0–2.5 tokens per verification step, with an acceptance rate of roughly 15%. The speculation overhead — running the draft model forward pass, generating 16 candidate tokens, and performing the verification step — was costing more than the ~2.3× acceptance length was saving.

The assistant then tried reducing the number of draft tokens from 16 to 5 (message 3618), reasoning that fewer candidates would reduce verification overhead. The result was 53.2 tok/s — even worse. The acceptance rate improved to ~41% (because the denominator was smaller), but the acceptance length remained stubbornly at ~2.0–2.2 tokens. The fundamental problem was clear: the draft model simply wasn't predicting well enough to justify the overhead of speculative decoding.

The Experiment: Testing a Better-Trained Drafter

This brings us to the subject message. The assistant's reasoning, articulated in message 3624, was straightforward: "Let me also try the AQ-MedAI drafter with EAGLE3 to see if that one does better (it was a general K2 drafter, not K2.5-specific, but it was trained on more data)." The assumption was that the custom 10K-sample drafter might be data-limited — trained on only 10,000 examples, it might not have learned the target model's token distribution well enough. The AQ-MedAI drafter, by contrast, was a publicly available EAGLE-3 drafter for the Kimi-K2 model family, presumably trained on a much larger corpus. If the problem was simply insufficient training data, the AQ-MedAI drafter should perform better.

The assistant first verified that the AQ-MedAI drafter had the same architectural configuration: same hidden size (7168), same draft vocabulary size (32000), same auxiliary hidden state layer IDs ([2, 30, 58]), and the use_aux_hidden_state flag set to true (message 3624). This was a critical validation step — if the architectures were incompatible, the benchmark would be meaningless. Finding them compatible, the assistant killed the existing server (message 3625), launched a new instance pointing at the AQ-MedAI checkpoint (message 3626), waited for it to become healthy (message 3627), and then ran the benchmark.

What the Numbers Say

The benchmark result in message 3628 is devastating in its clarity:

=== Single-stream EAGLE3 benchmark ===
  Prompt 1: 924 tokens in 20.8s = 44.4 tok/s (prompt: 19 tok)
  Prompt 2: 1024 tokens in 17.3s = 59.2 tok/s (prompt: 20 tok)
  Prompt 3: 1024 tokens in 20.4s = 50.1 tok/s (prompt: 24 tok)
  Prompt 4: 1024 tokens in 18.7s = 54.9 tok/s (prompt: 18 tok)
  Prompt 5: 1024 tokens in 22.2s = 46.1 tok/s (prompt: 18 tok)

  Average: 50.5 tok/s (5020 tokens / 99.4s)

The AQ-MedAI drafter achieves 50.5 tok/s — worse than both configurations of the custom drafter (56.7 tok/s with 16 draft tokens, 53.2 tok/s with 5 draft tokens). The variance across prompts is substantial, ranging from 44.4 tok/s to 59.2 tok/s, but the average tells the story: this is 44% slower than the 90 tok/s non-speculative baseline. The drafter trained on more data is performing worse, not better.

This result is significant because it eliminates the "more training data" hypothesis as a sufficient explanation. If the AQ-MedAI drafter — presumably trained on orders of magnitude more data — cannot achieve a speedup, then the problem is not simply data quantity. The assistant had already identified other contributing factors: CUDA graphs were disabled (adding overhead), the EAGLE-3 paper's scaling curves suggest acceptance length improves with more data but perhaps not enough at this model scale, and the MLA (Multi-head Latent Attention) architecture of Kimi-K2.5 might interact poorly with EAGLE-3's verification mechanism. But this benchmark provides the empirical evidence that the approach itself, on this hardware and model combination, is not viable.

Assumptions and Their Collapse

Several assumptions underlay this experiment. The first was that a better-trained drafter would yield higher acceptance. This assumption was reasonable — the EAGLE-3 paper explicitly shows acceptance length scaling with training data. But it failed in practice, suggesting either that the AQ-MedAI drafter was not actually better trained (perhaps it was trained on a different data distribution or with different hyperparameters), or that the ceiling for acceptance length on this model is fundamentally lower than needed.

The second assumption was that the architectural compatibility verified in message 3624 was sufficient for the drafter to work correctly. While the config parameters matched, there could be subtle differences in how the draft model was trained — the number of MLP layers in the midlayer, the specific normalization scheme, or the initialization of the fusion layer — that affect performance. The assistant did not investigate these deeper architectural details.

The third, more implicit assumption was that EAGLE-3 would eventually yield a speedup if the implementation bugs were fixed. This benchmark, combined with the previous two, effectively disproves that assumption for this specific deployment. The overhead of speculative decoding — running the draft model, generating multiple candidates, and verifying them against the target model — simply exceeds the benefit of accepting ~2 tokens at a time.

The Knowledge Created

This message creates critical output knowledge: the AQ-MedAI EAGLE-3 drafter, despite being a publicly released, presumably well-trained checkpoint, achieves only 50.5 tok/s on the Kimi-K2.5 model with 8-GPU tensor parallelism on RTX PRO 6000 Blackwell GPUs. This is 44% slower than the non-speculative baseline. Combined with the previous benchmarks, this establishes that EAGLE-3 speculative decoding is not beneficial on this hardware stack with currently available draft models.

The input knowledge required to interpret this message includes: familiarity with speculative decoding and the EAGLE-3 algorithm, understanding of the acceptance rate/length metrics and their relationship to throughput, knowledge of the Kimi-K2.5 model architecture (including its 60-layer, 7168-dim hidden structure and MLA attention), awareness of the SGLang server's speculative decoding implementation, and understanding of the previous debugging effort that fixed the EAGLE vs EAGLE3 flag bug.

The Thinking Process

The assistant's thinking process is visible in the chain of messages leading to this benchmark. After fixing the bug, the assistant did not simply declare victory — it immediately benchmarked, found the performance inadequate, and began iterating on parameters. The progression from 16 draft tokens (56.7 tok/s) to 5 draft tokens (53.2 tok/s) to the AQ-MedAI drafter (50.5 tok/s) shows a systematic search for a configuration that works. Each step is motivated by a clear hypothesis: reduce overhead, try a better model. The assistant is thinking like an engineer running a controlled experiment, isolating one variable at a time.

The decision to test the AQ-MedAI drafter specifically reveals an important piece of reasoning: the assistant recognized that the custom drafter might be data-limited and sought to test a drafter that was presumably trained on more data. This is a classic scientific control — if the AQ-MedAI drafter performs better, the bottleneck is training data; if it performs the same or worse, the bottleneck is something more fundamental.

The message itself contains no reasoning text — it is purely a tool call and its output. But the reasoning is embedded in the experimental design: the assistant chose to run the same benchmark script (/tmp/eagle3_bench.py) that was used for the previous measurements, ensuring comparability. The script sends five diverse prompts (general relativity, merge sort, climate change, photosynthesis, ML vs traditional programming) with max_tokens=1024 and temperature=0 for deterministic output. This consistency across experiments is itself a methodological decision worth noting.

The Aftermath

This message is the last benchmark before a strategic pivot. In the messages that follow, the assistant shifts focus to scaling up the training dataset by 10× (from 10K to 88K samples) and launching a large inference pipeline to generate responses for 83,000 prompts. The EAGLE-3 approach is not abandoned, but the immediate expectation of a speedup is abandoned. The assistant instead invests in building a much larger, higher-quality training dataset, accepting that the current draft models are simply not good enough. The 50.5 tok/s benchmark is the data point that makes this pivot necessary — it proves that no amount of parameter tuning or drafter swapping will work with the current level of training data quality.

In the broader narrative of this coding session, message 3628 is the moment when the assistant stops debugging the infrastructure and starts investing in the fundamentals. It is a quiet message, containing only numbers, but those numbers speak volumes about the gap between a working implementation and a performant one.