The 82.3 tok/s Plateau: A Pivotal Benchmark in the EAGLE-3 Speculative Decoding Saga

Introduction

In the high-stakes world of large language model deployment, every token per second counts. When you're running an 8-GPU cluster with a 671B-parameter model like Kimi-K2.5, the difference between 50 tok/s and 90 tok/s can mean hours saved on every inference run. This article examines a single, deceptively simple message from an opencode coding session — message index 3641 — in which the assistant reports a benchmark result of 82.3 tok/s for an EAGLE-3 speculative decoding configuration, still 9% short of the 90 tok/s non-speculative baseline. The message captures a moment of both progress and frustration: the critical hidden-state concatenation bug has been fixed, CUDA graphs are finally working, and yet the speculative drafter still cannot beat the baseline.

The Message

The subject message reads as follows:

Getting closer! 82.3 tok/s average with CUDA graphs + 5 draft tokens. Prompt 2 (code) hit 101.8 tok/s!

>

`` $ ssh root@[REDACTED] 'grep "accept len" /data/eagle3/sglang_eagle3_cg_5tok.log | tail -15' [2026-02-23 18:04:46 TP0] Decode batch, #running-req: 1, #token: 871, token usage: 0.01, accept len: 1.90, accept rate: 0.38, cuda graph: True, gen throughput (token/s): 64.43, #queue-req: 0 [2026-02-23 18:04:47 TP0] Decode batch, #running-req: 1, #token: 958, token usage: 0.01, accept len: 2.17, accept rate: 0.43, cuda graph: True, gen throughput (token/s): 71.71, #queue-req: 0 [2026-02-23 18:04:49 TP0] Decode batch, #running-req: 1, #token: 1027, token usage: 0.01, accept len: 1.73, accept rat... ``

The message consists of two parts: an enthusiastic but measured status update ("Getting closer!"), and a shell command that queries the server logs for acceptance statistics. The output is truncated in the conversation view, but the visible lines tell a clear story.

Context: The Long Road to This Benchmark

To understand why this message matters, we must trace the events that led to it. The session had been wrestling with EAGLE-3 speculative decoding for Kimi-K2.5 across multiple segments. The core idea of EAGLE-3 is to train a small "draft" model that predicts the target model's next tokens. If the draft model's predictions are good enough, the target model can verify them in parallel, yielding a net speedup. But the entire pipeline had been broken by a subtle bug: the server was started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in the SGLang codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states. The fusion layer was silently bypassed, and all trained weights were effectively useless. The draft model's acceptance length was exactly 1.0 — meaning no predictions were ever accepted.

The fix was simple but transformative. Once the flag was corrected, the accept length jumped to ~2.1, and the draft model's predictions began actually being accepted. But the throughput remained below the non-speculative baseline. The assistant then embarked on a systematic hyperparameter search: varying the number of draft tokens (16 vs 5), toggling CUDA graphs, adjusting speculation steps (3 vs 2), and even testing an alternative drafter (AQ-MedAI). Each configuration was benchmarked with a five-prompt test suite.

Why This Message Was Written

The assistant wrote this message for several interconnected reasons. First and foremost, it needed to document the benchmark result of the latest configuration — CUDA graphs enabled with 5 draft tokens — which had just completed its run. The previous configuration (CUDA graphs with 16 draft tokens) had achieved 74.9 tok/s, and the assistant was testing whether reducing draft tokens would lower the verification overhead enough to close the gap with the 90 tok/s baseline.

Second, the assistant was communicating progress to the user in a conversational format. The exclamation "Getting closer!" and the bolded "101.8 tok/s" for the code-generation prompt serve as encouragement — a signal that the approach is on the right track, even if the average still lags.

Third, the assistant was gathering diagnostic data by running the grep command on the server logs. The accept length and accept rate metrics are the key indicators of draft model quality. With 5 draft tokens, the accept rate is ~0.38-0.47 (much higher than the ~0.13-0.16 seen with 16 draft tokens), but the accept length remains stubbornly around 2.0-2.2. This tells the assistant that the draft model's predictions are decent but not good enough: on average, only about 2 out of 5 draft tokens get accepted per verification step.

How Decisions Were Made

The decision to test this particular configuration — CUDA graphs + 5 draft tokens — was the result of a deliberate reasoning chain visible in the preceding messages. The assistant had already tested:

  1. No CUDA graphs, 16 draft tokens: 53.2 tok/s, accept_len ~2.1, accept_rate ~0.14
  2. No CUDA graphs, 5 draft tokens: 53.2 tok/s, accept_rate ~0.41 (better rate but same throughput due to overhead)
  3. CUDA graphs, 16 draft tokens: 74.9 tok/s, accept_len ~2.1-2.5, accept_rate ~0.13-0.16
  4. AQ-MedAI drafter (alternative model): 50.5 tok/s, accept_len ~1.8-2.1 Each test isolated one variable. The assistant's reasoning was: CUDA graphs dramatically reduce per-step overhead (74.9 vs 53.2 tok/s), but the accept length of ~2.1 is still too low to overcome the remaining overhead. By reducing draft tokens from 16 to 5, the verification cost per step drops proportionally, which should improve the overhead-to-gain ratio even with the same accept length. The benchmark confirmed this: 82.3 tok/s vs 74.9 tok/s, a ~10% improvement from the draft token reduction alone.

Assumptions Made

The assistant operated under several assumptions in this message. It assumed that the benchmark script (/tmp/eagle3_bench.py) provides a representative measure of real-world performance. The five prompts used are relatively short (18-24 tokens of prompt generating ~1000 tokens of output), which may not reflect the behavior on longer contexts or more varied input distributions. The assistant also assumed that the accept length of ~2.1 is a fundamental property of the current draft model rather than an artifact of the specific test prompts. Additionally, it assumed that CUDA graph capture succeeded correctly — the logs confirmed capture completion, but there could be edge cases where graph capture degrades performance for certain input patterns.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption visible in this message's context is the belief that reducing draft tokens alone would close the gap with the baseline. The assistant achieved 82.3 tok/s, still 8.5% below 90 tok/s. The accept length of ~2.1 is simply insufficient — the EAGLE-3 paper suggests that accept lengths of 3-4+ are needed for meaningful speedup, and those require substantially more training data. The assistant had trained the draft model on only 10K samples, while the EAGLE-3 paper used hundreds of thousands.

Another subtle mistake was the assumption that the benchmark average (82.3 tok/s) tells the full story. The individual prompts show high variance: Prompt 2 (code generation) hit 101.8 tok/s, while Prompt 1 achieved only 75.6 tok/s. This variance suggests that the draft model's performance is highly dependent on the input domain — it works well for code but poorly for other types of text. The assistant does not explicitly address this variance in the message.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. Speculative decoding is the core concept: using a smaller draft model to predict tokens that the larger target model then verifies in parallel. EAGLE-3 is a specific speculative decoding algorithm that uses intermediate hidden states from the target model to condition the draft model's predictions. CUDA graphs are a CUDA optimization that pre-compiles a sequence of GPU operations into a single fused kernel, reducing launch overhead. The accept length is the number of consecutive draft tokens accepted by the target model in a single verification step, and the accept rate is the accept length divided by the number of draft tokens proposed. The SGLang framework is the inference engine hosting the model. The Kimi-K2.5 model is a 671B-parameter Mixture-of-Experts language model deployed across 8 GPUs with tensor parallelism (tp-size 8).

Output Knowledge Created

This message produced several concrete pieces of knowledge. It established that 82.3 tok/s is the best achievable throughput with the current 10K-sample draft model on this hardware configuration. It demonstrated that reducing draft tokens from 16 to 5 improves throughput by ~10% (74.9 → 82.3 tok/s) when CUDA graphs are enabled. It showed that code generation benefits disproportionately from speculative decoding (101.8 tok/s vs 75-82 tok/s for other prompt types). It confirmed that the accept length of ~2.1 is a hard ceiling for the current draft model regardless of configuration changes — neither CUDA graphs nor draft token count affects the fundamental quality of the draft model's predictions. And it implicitly established that more training data is the next required step, which the assistant would act on in subsequent messages by launching a 10× data scaling pipeline.

The Thinking Process

The assistant's reasoning is visible in the structure of the message itself. The opening line — "Getting closer!" — reveals the assistant's mental model of the problem as a gradual optimization process rather than a binary success/failure. The assistant is tracking progress along a continuum, and 82.3 tok/s represents genuine progress from the 53.2 tok/s of earlier configurations, even if the ultimate goal of exceeding 90 tok/s remains out of reach.

The decision to run the grep command alongside the status update reveals a data-driven mindset. Rather than simply reporting the benchmark number, the assistant immediately seeks the underlying metrics (accept length, accept rate) to understand why the throughput is what it is. The assistant is not just collecting data but actively interpreting it: the accept rate of ~0.38-0.47 with 5 draft tokens is much higher than the ~0.13-0.16 with 16 draft tokens, but the accept length remains ~2.1 in both cases. This tells the assistant that the draft model's absolute prediction quality is the bottleneck, not the verification overhead.

The truncation of the grep output is itself informative. The conversation only shows three lines of log output before cutting off with "accept rat..." — but those three lines already tell the story: accept lengths of 1.90, 2.17, and 1.73, with throughputs of 64.43, 71.71, and (presumably) ~60 tok/s. The assistant could have requested more lines or parsed the data more thoroughly, but the pattern was already clear.

Aftermath and Significance

This message sits at a critical inflection point in the session. Immediately after this benchmark, the assistant tries one more configuration (2 speculation steps instead of 3) which achieves 81.7 tok/s — essentially the same. The user then reports a server crash, and after cleanup, the assistant pivots entirely. The realization sets in: no amount of hyperparameter tuning will overcome a draft model trained on only 10K samples. The assistant launches a massive data collection pipeline, dispatching ten parallel agents to find and prepare datasets totaling 88,088 samples, and begins an inference run expected to take 24-55 hours to regenerate responses through the target model.

In retrospect, message 3641 marks the moment when the assistant accepted the fundamental limitation of the current approach and committed to the data scaling path. The 82.3 tok/s plateau was not a failure — it was the necessary diagnostic that revealed the true bottleneck. The assistant's systematic, data-driven approach to debugging speculative decoding — isolating variables, measuring precisely, and reasoning about root causes — is a model of how to approach complex ML engineering problems. And the pivot to data scaling, while not yet complete at this point in the conversation, represents the correct strategic response to the evidence gathered in this message.