The Moment of Truth: Benchmarking EAGLE-3 Acceptance with CUDA Graphs

The Message

ssh root@10.1.230.174 'grep "accept len" /data/eagle3/sglang_eagle3_cudagraph.log | tail -15'
[2026-02-23 17:53:14 TP0] Decode batch, #running-req: 1, #token: 671, token usage: 0.01, accept len: 2.15, accept rate: 0.13, cuda graph: True, gen throughput (token/s): 66.89, #queue-req: 0
[2026-02-23 17:53:16 TP0] Decode batch, #running-req: 1, #token: 756, token usage: 0.01, accept len: 2.12, accept rate: 0.13, cuda graph: True, gen throughput (token/s): 64.25, #queue-req: 0
[2026-02-23 17:53:17 TP0] Decode batch, #running-req: 1, #token: 838, token usage: 0.01, accept len: 2.05, accept rate...

At first glance, this appears to be a routine diagnostic command — a simple grep into a log file to check a few metrics. But in the arc of this coding session, message 3636 represents a pivotal moment: the first time the EAGLE-3 speculative decoding pipeline runs with both the correct algorithm flag and CUDA graphs enabled. After hours of debugging a silent hidden-state concatenation bug, after training a draft model on only 10,000 samples, after testing multiple configurations and two different draft models, this message is where the team finally sees whether all the pieces work together. The answer, captured in three log lines, is simultaneously encouraging and sobering.

The Long Road to This Message

To understand why this particular grep command was issued, one must trace the tortured path that led to it. The team had been working for days to deploy EAGLE-3 speculative decoding on top of the Kimi-K2.5 model running on an 8-GPU server with SGLang. The core idea of speculative decoding is elegant: use a small, fast "draft" model to predict multiple future tokens, then have the large "target" model verify them in parallel. If the draft model's predictions are good enough, the system generates tokens faster than running the target model alone.

The EAGLE-3 architecture is particularly sophisticated. Instead of using a standalone draft model, it extracts intermediate hidden states from the target model itself — specifically from layers 2, 30, and 58 of the 60-layer Kimi-K2.5 model — concatenates these 7168-dimensional vectors into a single 21504-dimensional representation, and feeds them through a lightweight fusion layer to predict future tokens. This design means the draft model is deeply coupled to the target model's internal representations.

For days, this coupling was broken. The team had trained a draft model on 10,000 samples of hidden states extracted from Kimi-K2.5, but when they deployed it with SGLang, the acceptance rate was essentially zero — every draft token was rejected, meaning the system was running the full target model for every token with no speedup at all. The root cause, discovered in the messages immediately preceding this one, was devastatingly simple: the server had been started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. The is_eagle3() check in SGLang's codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate those intermediate layer hidden states. With "EAGLE", the draft model received only the final-layer 7168-dimensional hidden states instead of the expected 21504-dimensional concatenated states. The fusion layer was silently bypassed, and all trained weights were useless.

Why This Message Was Written

Message 3636 was written to answer a single, critical question: does the fix actually work? The team had just completed a multi-step diagnostic and remediation process:

  1. They identified the --speculative-algorithm EAGLE vs EAGLE3 flag mismatch ([msg 3615]).
  2. They restarted the server with the correct flag and benchmarked, finding accept_len had risen from ~1.0 to ~2.0-2.5, but throughput was only 56.7 tok/s — well below the 90 tok/s non-speculative baseline ([msg 3622]).
  3. They experimented with fewer draft tokens (5 instead of 16) to reduce verification overhead, but accept_len remained ~2.0-2.2 and throughput was 53.2 tok/s ([msg 3623]).
  4. They tested a third-party draft model (AQ-MedAI's K2 drafter) to see if a model trained on more data would perform better, but it achieved only 50.5 tok/s with similar accept_len ([msg 3628]).
  5. They removed debug prints from the EAGLE-3 implementation to eliminate print overhead ([msg 3619]).
  6. Finally, they restarted the server with CUDA graphs enabled — a critical optimization that pre-compiles GPU kernel launches into efficient execution graphs, dramatically reducing per-step overhead ([msg 3632]). The benchmark with CUDA graphs (message 3635) showed a promising improvement: 74.9 tok/s average, up from 56.7 tok/s without graphs. But the benchmark script only reports throughput — it doesn't show the underlying acceptance metrics. Message 3636 reaches into the server's own log to extract those metrics directly, providing the granular diagnostic data needed to understand why the throughput is what it is.

What the Output Reveals

The three log lines tell a consistent story:

The Thinking Process Visible in This Message

The reasoning behind message 3636 reflects a methodical, data-driven debugging approach. The assistant does not simply declare victory after the CUDA graph benchmark shows 74.9 tok/s. Instead, it digs into the server's internal metrics to understand the root cause of the remaining performance gap. This is characteristic of the entire session: every hypothesis is tested with concrete measurements, and every configuration change is benchmarked before moving on.

The assistant is operating with a clear mental model of how speculative decoding works. It knows that throughput is a function of three factors: the baseline target model speed, the draft model's prediction quality (measured by accept_len), and the overhead of running the draft model and verification steps. By isolating each factor — first fixing the hidden state bug, then removing debug prints, then enabling CUDA graphs — the assistant systematically eliminates sources of overhead to reveal the fundamental bottleneck: accept_len is too low because the draft model is undertrained.

Assumptions and Their Implications

Several assumptions underpin this message and the surrounding work:

Assumption 1: The draft model architecture is correct. The team assumes that with the EAGLE3 flag, the hidden state concatenation works as designed and the draft model receives the correct 21504-dimensional input. The benchmark results (accept_len ~2.1 vs ~1.0) validate this assumption.

Assumption 2: CUDA graphs would significantly reduce overhead. This assumption is confirmed by the data — throughput jumped from 56.7 tok/s to 74.9 tok/s when CUDA graphs were enabled, a 32% improvement.

Assumption 3: More training data would improve accept_len. This is the key untested assumption. The EAGLE-3 paper shows scaling laws where accept_len increases with training data, but the team's 10,000-sample dataset may be insufficient. The assistant implicitly accepts this assumption and begins planning a 10× scale-up in the following messages.

Assumption 4: The server metrics are accurate. The assistant trusts that accept_len and accept_rate as reported by SGLang's logging are correct measurements. Given the careful instrumentation of the SGLang codebase, this is reasonable.

Input Knowledge Required

To fully understand this message, one needs:

  1. Speculative decoding fundamentals: The concept of using a draft model to predict tokens and a target model to verify them, and the relationship between accept_len, accept_rate, and overall throughput.
  2. EAGLE-3 architecture specifics: That EAGLE-3 uses intermediate hidden states from layers [2, 30, 58] concatenated into 21504 dimensions, and that the is_eagle3() flag controls whether these states are captured.
  3. CUDA graphs: The optimization technique of pre-recording GPU kernel launches to reduce launch overhead, and its particular importance for small, frequent operations like draft model inference.
  4. SGLang's logging format: Understanding that accept_len is the average number of tokens accepted per verification step, accept_rate is accept_len divided by num_draft_tokens, and gen throughput is the instantaneous token generation rate.
  5. The preceding diagnostic work: Knowledge that the hidden state concatenation bug was just fixed, that two draft models were tested, and that CUDA graphs were just enabled for the first time with the correct EAGLE3 flag.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. Confirmation that the EAGLE3 flag fix works end-to-end with CUDA graphs. The draft model is receiving correct hidden states and producing predictions that are accepted at a non-trivial rate.
  2. Quantification of the remaining performance gap. With accept_len ~2.1 and CUDA graphs, the system achieves 74.9 tok/s — still 17% below the 90 tok/s baseline. The gap is attributable to insufficient draft model quality, not infrastructure issues.
  3. Evidence that the custom K2.5-trained drafter is slightly better than the AQ-MedAI drafter. The custom model achieved 53.2 tok/s (without CUDA graphs) vs 50.5 tok/s for AQ-MedAI, suggesting that training on K2.5-specific data provides a marginal advantage.
  4. A clear direction for improvement. The accept_len of ~2.1, combined with the EAGLE-3 paper's scaling curves, strongly suggests that more training data is the primary lever. This directly motivates the 10× data scaling effort launched in the subsequent messages.

Mistakes and Incorrect Assumptions

The most significant mistake visible in the broader context is the original --speculative-algorithm EAGLE vs EAGLE3 flag error. This was a silent, devastating bug — the server started successfully, the draft model loaded, inference appeared to work, but the hidden states were fundamentally wrong. The error was only caught by carefully examining the hidden state dimensions and comparing them to the expected 21504-dim concatenated representation.

A subtler issue is the assumption that the draft model trained on 10,000 samples would be sufficient. The EAGLE-3 paper's training datasets are orders of magnitude larger (millions of tokens), and the team's 10,000-sample dataset (each sample containing perhaps a few hundred tokens) is almost certainly undersized. The accept_len of ~2.1, while dramatically better than 1.0, is still in the regime where speculation is a net loss.

There's also an implicit assumption that the AQ-MedAI drafter, being trained on more data, would outperform the custom drafter. In fact, it performed slightly worse (50.5 vs 53.2 tok/s), likely because it was trained on DeepSeek-K2 rather than K2.5, and the two models may have subtle differences in their hidden state distributions despite sharing the same architecture.

The Broader Significance

Message 3636 is a checkpoint — a moment of assessment before a major strategic pivot. The data it reveals tells the assistant that the infrastructure is now correct (CUDA graphs work, hidden states flow properly, the draft model loads and runs) but the draft model itself is not good enough. This diagnosis directly drives the next phase of work: scaling the training dataset by 10×, launching an inference pipeline to regenerate 83,000 prompts through Kimi-K2.5, and retraining the draft model from scratch.

In the larger narrative of the coding session, this message represents the transition from debugging to scaling. The hidden state bug is fixed, the debug prints are removed, CUDA graphs are working, and the remaining bottleneck is purely a data quantity problem. The team shifts from "does it work?" to "how do we make it work better?" — a fundamentally different and more productive mode of operation.

The three log lines in message 3636, mundane as they appear, contain the answer to a question that had consumed hours of debugging: is our draft model actually predicting tokens that the target model agrees with? The answer is yes — barely. And that "barely" is enough to justify the massive data collection effort that follows.