The 60 Tok/s Benchmark: A Pivotal Negative Result in EAGLE-3 Speculative Decoding Optimization

The Message

[assistant] It's ready! Let me benchmark:
[bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /tmp/benchmark_eagle3.py --server-url http://localhost:8000 --max-tokens 500 --num-runs 5 --warmup 1'
Benchmarking http://localhost:8000
  max_tokens=500, num_runs=5, warmup=1

  Warmup 1/1 done (41 tokens)

  Run 1/5: 500 tokens in 8.60s = 58.1 tok/s (prompt: 32 toks)
  Run 2/5: 500 tokens in 8.92s = 56.1 tok/s (prompt: 31 toks)
  Run 3/5: 500 tokens in 7.99s = 62.6 tok/s (prompt: 26 toks)
  Run 4/5: 500 tokens in 8.42s = 59.4 tok/s (prompt: 28 toks)
  Run 5/5: 500 tokens in 7.74s = 64.6 tok/s (prompt: 34 toks)

  ========== Results ==========
  Overall tok/s: 60.0
  Avg tok/s:     60.1
  Min t...

At first glance, this message looks like a routine benchmark result — just another data point in a long optimization session. But in the context of the broader EAGLE-3 speculative decoding effort, this single benchmark run represents a critical turning point. The result — 60.0 tok/s with 10 draft steps — was decisively worse than the 71.3 tok/s achieved with 5 steps just minutes earlier. More importantly, it was a full 33% below the 90 tok/s baseline without speculation. This negative result forced a fundamental rethinking of the optimization strategy and ultimately led to the discovery of the optimal configuration.

Why This Message Was Written: The Hypothesis Being Tested

To understand why this benchmark was run, we need to trace the reasoning that led to it. In the preceding messages ([msg 4598] through [msg 4602]), the assistant had just benchmarked a 5-step EAGLE-3 configuration and achieved 71.3 tok/s — a significant improvement from the broken 46.7–54.8 tok/s range seen earlier, but still well below the 90 tok/s baseline without speculation. The logs showed an accept length of approximately 2.0–2.3 tokens per verify cycle.

The assistant then performed a detailed back-of-the-envelope calculation in [msg 4602]:

"With accept_len 2.1, each verify cycle produces 2.1 tokens on average. The cycle consists of: 5 draft model forward passes (sequential, each single-token), 1 target model verify pass (6 tokens). If baseline decode is ~11ms per token (90 tok/s), then 6 tokens of prefill-like verify should be maybe ~15-20ms. Plus 5 draft model passes at maybe ~3ms each = 15ms. Total: ~30-35ms per cycle, producing 2.1 tokens → ~60-70 tok/s."

This analysis revealed that the draft model overhead — 5 passes at ~3ms each — was consuming nearly as much time as the target verify itself. The assistant hypothesized two possible solutions: fewer draft steps (to reduce overhead) or more draft steps (to increase the number of accepted tokens per verify cycle, amortizing the overhead). The assistant decided to test both, starting with the more aggressive option: 10 steps (11 draft tokens).

The message we're analyzing is the result of that first test. The assistant launched the 10-step server in [msg 4602], waited through a 15-minute CUDA graph capture and model loading process (visible in the "Still loading..." messages through [msg 4616]), and finally ran the benchmark once the server was ready.

The Assumption That Was Tested — and Falsified

The core assumption behind this benchmark was that increasing the number of draft steps would improve throughput by generating more accepted tokens per verify cycle. The logic was straightforward: if each verify cycle costs a fixed overhead (draft model passes + target verify), then producing more tokens per cycle should increase the overall token rate. With 5 steps yielding accept_len ~2.1, the assistant hoped that 10 steps might yield accept_len ~3.5–4.0, more than doubling the output per cycle while only modestly increasing the draft overhead.

This assumption turned out to be wrong — and the data was unambiguous. Instead of improving, throughput dropped from 71.3 tok/s to 60.0 tok/s. The accept rate likely didn't scale linearly with steps because the marginal acceptance probability decreases with each additional token — the draft model becomes less accurate the further it extrapolates from the ground truth. By step 10, the extra draft tokens were being rejected at such a high rate that the additional overhead of 5 more draft passes couldn't be justified by the meager increase in accepted tokens.

This is a classic pattern in speculative decoding optimization: there exists a sweet spot for the number of draft steps, determined by the tradeoff between the draft model's accuracy (which degrades with distance from the verified prefix) and the overhead of running additional draft passes. The 10-step configuration overshot this sweet spot.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this benchmark, one needs to understand several layers of context:

Speculative decoding architecture: EAGLE-3 is a speculative decoding framework where a small "draft" model generates multiple candidate tokens, and the large "target" model verifies them in parallel. The key metric is the accept length — how many draft tokens are accepted on average per verify cycle. The draft model runs sequentially (one token at a time), while the target verifies all draft tokens in a single forward pass.

The TP8 constraint: Both the draft and target models were running with tensor parallelism across 8 GPUs (TP8). This means every single-token draft forward pass required communication across all 8 GPUs via NCCL allreduce operations. For a small 2.6B parameter draft model, the communication overhead dominates the computation time — each allreduce for a 7168-dimensional vector is latency-bound at perhaps 50–100μs, but the cumulative effect across multiple steps becomes significant.

The baseline: Without speculation, the target model achieved ~90 tok/s. Any speculative configuration that falls significantly below this is actively harmful — it's better to run without speculation at all.

The benchmark methodology: The benchmark script (benchmark_eagle3.py) measures single-stream token generation throughput by sending prompts to the SGLang server's /generate endpoint, timing the response, and dividing the number of generated tokens by the elapsed time. Five runs with different prompts are averaged to smooth out variance from prompt length differences and server state.

Output Knowledge Created by This Message

This benchmark produced several pieces of critical knowledge:

  1. 10 steps is worse than 5 steps: The throughput dropped from 71.3 to 60.0 tok/s — a 16% regression. This definitively ruled out the "more steps" direction.
  2. The optimal step count is not monotonic: The relationship between step count and throughput is not simply "more is better" or "fewer is better." There is a peak somewhere between 1 and 10 steps that must be found empirically.
  3. The draft model's accuracy degrades with distance: The fact that 10 steps produced worse results suggests that the marginal acceptance probability for tokens beyond the first few is very low. Each additional draft token is increasingly likely to be rejected, wasting the compute used to generate it.
  4. The overhead of additional steps is real: Even if some extra tokens are accepted, the cost of running 10 draft passes (each requiring TP8 communication) outweighs the benefit. The draft model overhead is not negligible — it's a first-order term in the throughput equation. This knowledge directly shaped the next actions. The assistant would go on to test step counts from 1 to 10 systematically, eventually finding that 2 steps (3 draft tokens) was optimal at 94 tok/s — beating the baseline by 5.9%. But that discovery depended on first ruling out the 10-step configuration.

The Thinking Process Visible in the Reasoning

The assistant's thinking process in the lead-up to this benchmark reveals a methodical, hypothesis-driven approach to performance optimization. In [msg 4602], the assistant explicitly laid out the tradeoff:

"Let me think about this more strategically before trying random configurations. The key question is: why is 71 tok/s with accept_len 2.1 slower than 90 tok/s baseline?"

This is a crucial moment of reflection. Rather than blindly tweaking parameters, the assistant stepped back to build a mental model of the bottleneck. The back-of-the-envelope calculation that followed — estimating verify time at 15–20ms and draft overhead at ~15ms — was an attempt to decompose the cycle time into its constituent parts. This kind of first-principles analysis is essential for effective optimization because it reveals which parameters are worth tuning.

The assistant then formulated two competing hypotheses:

Mistakes and Incorrect Assumptions

While the benchmark itself was correctly executed, several assumptions embedded in the reasoning deserve scrutiny:

The estimated draft model overhead (~3ms per step) was likely an underestimate. The actual overhead includes not just the forward pass compute but also CUDA graph launch overhead, NCCL synchronization across 8 GPUs, and the eagle feature concatenation (the fc layer that combines the target model's hidden states with the draft model's input). Later profiling in the same chunk would reveal that the target verify consumes 21–28ms (95%+ of cycle time), but this doesn't mean the draft overhead is negligible — it means the verify step is so expensive that even a 10–15% reduction in its frequency (from more accepted tokens) could be overshadowed by increased draft cost.

The assumption that accept_len would scale significantly with more steps was optimistic. The draft model was trained on only 37K samples (as revealed later in the chunk summary), and its accuracy likely plateaued after 2–3 steps. A more conservative estimate would have predicted accept_len of perhaps 2.5–2.8 with 10 steps, which the back-of-the-envelope math would have shown as marginal or negative improvement.

The assumption that the 5-step result (71 tok/s) was a fair baseline for comparison was reasonable but incomplete. The 5-step benchmark had been run with CUDA graphs enabled (as shown by the "cuda graph: True" flag in the logs at [msg 4599]), while the 10-step benchmark also used CUDA graphs. However, the CUDA graph capture itself can introduce variability — different graph shapes for different step counts may have different performance characteristics.

The Broader Significance

This message, standing alone, appears to be just a disappointing benchmark result. But in the arc of the optimization journey, it was a necessary negative result that eliminated a plausible but incorrect hypothesis. The path to the optimal 2-step configuration (94 tok/s) required first ruling out both extremes: 10 steps (too much overhead) and, as would be tested later, 1 step (too few accepted tokens).

The systematic, empirical approach on display here — formulate hypothesis, test, measure, learn, iterate — is the essence of performance engineering. The assistant didn't guess the optimal step count; it swept the parameter space and let the data decide. This benchmark was the data point that said "not this direction," and that knowledge was just as valuable as a positive result would have been.

In the end, the 60.0 tok/s result served as a crucial calibration. It told the assistant that the draft model's accuracy was the binding constraint — not the step count, not the TP configuration, but the fundamental quality of the draft model itself. This realization would later lead to the conclusion that more training data (scaling from 37K to 1.4M samples, matching AQ-MedAI's approach) was the highest-leverage improvement available. But that insight was only possible because the assistant had thoroughly explored and exhausted the configuration space, one benchmark at a time.