The Moment of Truth: Diagnosing EAGLE-3 Speculative Decoding Performance After the Hidden State Fix

Introduction

In the high-stakes world of large language model deployment, few moments are as tense as the first benchmark after fixing a critical bug. Message [msg 3615] captures exactly such a moment in an opencode coding session: the assistant has just confirmed that a hidden state concatenation bug has been resolved, the EAGLE-3 draft model is finally receiving the correct 21504-dimensional hidden states, and the acceptance rate has climbed from a useless 1.0 to a promising ~2.0-2.5. Yet the throughput benchmark tells a sobering story: 56.7 tokens per second, well below the 90 tok/s non-speculative baseline. This message is the assistant's real-time analysis of why the fix hasn't yet translated into a speedup, and its decision about what to try next.

The Message in Full

The assistant writes:

OK, so the accept length is now ~2.0-2.5 (up from 1.0 before!), and the accept rate is ~0.12-0.16 with num_draft_tokens=16. That's a real improvement — the draft model IS now getting correct hidden states and some predictions are being accepted. But 56.7 tok/s average is still slower than the 90 tok/s baseline without speculation.

>

The issue is: 1. Accept rate ~15% is too low for speedup — speculation overhead (running the draft model + verifying 16 candidates) costs more than the ~2.3× accept length saves 2. CUDA graphs are disabled — huge overhead penalty 3. Only 10K training samples — the EAGLE-3 paper shows clear scaling with more data

>

Let me try with fewer draft tokens to reduce overhead (fewer draft tokens = less verification cost, better overhead/speedup ratio even with low accept rate):

The message then executes a bash command to grep the latest server logs for acceptance metrics.

Context and Background

To understand the significance of this message, we must trace the arc of the preceding debugging saga. The session had been working on deploying an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model on an 8-GPU server. EAGLE-3 is a sophisticated speculative decoding technique where a lightweight "draft" model predicts multiple future tokens, which the target model then verifies in parallel. The key architectural requirement is that the draft model needs access to intermediate hidden states from multiple layers of the target model — specifically layers [2, 30, 58] — concatenated into a 21504-dimensional vector (3 × 7168). This allows the draft model to predict tokens based on richer contextual information than just the final layer's output.

For days, the team had been training draft models and observing a perplexing failure: the acceptance rate was exactly 1.0, meaning the draft model's predictions were never accepted — the system was effectively running without speculation. The root cause, traced across messages [msg 3601] through [msg 3604], was a single flag mismatch: the server had been started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. The is_eagle3() check in the SGLang codebase is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states. With "EAGLE", the target model returned only final-layer hidden states (7168-dim), which silently bypassed the draft model's fusion layer and rendered all trained weights useless.

After restarting with the correct flag in [msg 3607], the debug prints confirmed the fix: hidden states were arriving as torch.Size([21, 21504]) instead of the previous 7168-dim. The draft model's weights were confirmed to be non-zero and properly loaded ([msg 3610]-[msg 3611]). The first benchmark in [msg 3612] showed 56.7 tok/s average — and this is where message [msg 3615] picks up.

The Reasoning Process: Why the Fix Didn't Fix Performance

The assistant's analysis in this message reveals a sophisticated understanding of speculative decoding dynamics. The key insight is that acceptance rate and accept length are two different metrics, and both matter for throughput.

The accept length (~2.0-2.5) tells us how many draft tokens on average are accepted per verification step. An accept length of 2.3 means the system generates ~2.3 tokens per forward pass of the target model, compared to 1 token without speculation. This is a genuine improvement — the draft model is now producing useful predictions.

However, the accept rate (~0.12-0.16) tells us the fraction of draft tokens that are accepted. With 16 draft tokens generated per step, a 15% accept rate means only ~2.4 tokens are accepted on average — which matches the accept length. The remaining ~13.6 draft tokens were rejected, meaning the verification step had to recompute them.

The critical calculation the assistant performs is a cost-benefit analysis of speculative decoding overhead:

  1. Draft model execution cost: Generating 16 draft tokens requires running the lightweight draft model 16 times (or in parallel where possible).
  2. Verification cost: The target model must verify all 16 candidates in a single forward pass (this is the key efficiency of speculative decoding — parallel verification).
  3. Net gain: If the accept length is 2.3, the system produces 2.3 tokens per verification pass. Without speculation, it produces 1 token per pass. So the gain factor is 2.3× — but only if the overhead of running the draft model and the verification pass is less than 2.3× the cost of a single target model forward pass. In this case, the overhead of generating and verifying 16 draft tokens (most of which are rejected) exceeds the benefit of the ~2.3× accept length. The assistant correctly identifies this as the core problem.

Assumptions and Their Validity

The assistant makes several assumptions in this analysis:

Assumption 1: The baseline of 90 tok/s is accurate. This is a reasonable assumption — the non-speculative baseline was established in earlier benchmarks (segment 25). However, the baseline was measured with CUDA graphs enabled, while the current EAGLE-3 configuration has CUDA graphs disabled (as noted in point 2). This introduces a confound: part of the performance gap may be due to the missing CUDA graph optimization rather than speculation overhead itself.

Assumption 2: More training data will improve acceptance. The assistant references "the EAGLE-3 paper shows clear scaling with more data." This is based on published research results showing that EAGLE-3 draft model quality improves with training data volume. The assumption is reasonable but not guaranteed — the specific distribution of the 10K training samples may have limitations that more data alone cannot fix.

Assumption 3: Fewer draft tokens will improve the overhead/speedup ratio. This is a well-known trade-off in speculative decoding: fewer draft tokens means less verification cost but also lower maximum possible accept length. The assistant's intuition is that with accept length ~2.3, generating 16 draft tokens is wasteful — most are never accepted. Reducing to, say, 5 draft tokens would reduce overhead while still covering the typical accept length. This is a sound engineering judgment.

Assumption 4: The CUDA graphs penalty is "huge." This is correct — CUDA graphs eliminate kernel launch overhead by pre-recording sequences of GPU operations. For speculative decoding, which involves multiple small draft model invocations, this overhead can be substantial. The assistant later confirms this in subsequent messages when enabling CUDA graphs boosts performance to 82.3 tok/s.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of speculative decoding: The concept of using a draft model to predict tokens that the target model verifies in parallel, and the metrics (accept length, accept rate) used to evaluate it.
  2. Knowledge of the EAGLE-3 architecture: Specifically that it requires concatenated hidden states from multiple layers (21504-dim = 3 × 7168), and that the fc fusion layer projects this down to the draft model's hidden dimension.
  3. Context about the bug fix: That the server was restarted with --speculative-algorithm EAGLE3 instead of EAGLE, enabling the hidden state capture mechanism.
  4. Familiarity with CUDA graphs: The optimization technique that pre-records GPU kernel sequences to reduce launch overhead, and why it's particularly important for speculative decoding with many small operations.
  5. Knowledge of the training pipeline: That the draft model was trained on only 10K samples extracted via SGLang, and that the EAGLE-3 paper demonstrated scaling benefits with larger datasets.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. Empirical confirmation: The hidden state fix works — the draft model is now producing accepted predictions (accept length ~2.0-2.5 vs 1.0 before). This validates the entire debugging effort.
  2. Performance baseline: EAGLE-3 with 16 draft tokens, CUDA graphs disabled, and 10K training samples achieves 56.7 tok/s — significantly below the 90 tok/s non-speculative baseline.
  3. Diagnosis of three independent bottlenecks: The assistant identifies three separate issues that each need addressing: low accept rate (data-limited draft model), disabled CUDA graphs (configuration issue), and excessive draft tokens (parameter tuning).
  4. Decision to tune draft token count: The assistant decides to experiment with fewer draft tokens as the immediate next step, which is a testable hypothesis that can be evaluated quickly without retraining or server restarts.

The Thinking Process Revealed

What makes this message particularly interesting is the structure of the assistant's reasoning. It follows a clear pattern:

Step 1 — Celebrate the win: "That's a real improvement — the draft model IS now getting correct hidden states and some predictions are being accepted." This acknowledges the successful fix before diving into the remaining problems.

Step 2 — State the disappointing metric: "But 56.7 tok/s average is still slower than the 90 tok/s baseline." This frames the problem honestly.

Step 3 — Diagnose root causes: The assistant enumerates three issues in a numbered list, each with a brief explanation of why it matters. This shows systematic thinking rather than jumping to conclusions.

Step 4 — Formulate a testable hypothesis: "Let me try with fewer draft tokens to reduce overhead." This is the key decision — the assistant chooses the most immediately actionable fix (changing a runtime parameter) over the more impactful but slower fixes (enabling CUDA graphs requires a server restart, more training data requires hours of inference).

Step 5 — Execute: The assistant runs a bash command to check the latest logs, gathering data to inform the next decision.

This pattern — celebrate, diagnose, prioritize, hypothesize, execute — is a hallmark of effective debugging. The assistant doesn't just identify problems; it ranks them by how quickly they can be tested and resolved.

Mistakes and Incorrect Assumptions

One subtle issue in the assistant's analysis is the comparison to the 90 tok/s baseline. The baseline was established with CUDA graphs enabled, while the current EAGLE-3 configuration has them disabled. This means the 56.7 tok/s figure includes the CUDA graph penalty, making the comparison somewhat unfair. A fairer comparison would be EAGLE-3 with CUDA graphs enabled vs. non-speculative with CUDA graphs enabled. The assistant acknowledges this indirectly in point 2 ("CUDA graphs are disabled — huge overhead penalty") but doesn't adjust the comparison accordingly.

Another potential blind spot: the assistant assumes that the accept rate of ~15% is purely a function of training data quantity. However, it could also be affected by the quality of the hidden state extraction pipeline used to generate the training data. If the extraction pipeline (set up in segment 25) had its own issues — such as not properly capturing the multi-layer hidden states during training data generation — then the draft model was trained on incorrect data even before the deployment bug was fixed. The assistant doesn't explore this possibility in this message.

The Broader Significance

Message [msg 3615] represents a pivotal moment in the session. It's the point where the team confirms that the fundamental architectural issue (hidden state concatenation) is resolved, and the remaining challenges shift to optimization and scaling. The assistant's decision to try fewer draft tokens leads, in subsequent messages, to finding that 5 draft tokens with CUDA graphs enabled achieves 82.3 tok/s — much closer to the baseline. And the decision to scale training data by 10× (launched as a parallel effort later in the same chunk) addresses the fundamental data limitation.

This message also illustrates a universal truth about speculative decoding: fixing the correctness bug is necessary but not sufficient for performance. Even with correct hidden states, the draft model must be good enough to justify its overhead, and the system must be configured optimally. The assistant's systematic approach to diagnosing and prioritizing these issues — rather than declaring victory or defeat based on the first benchmark — is what makes this message a model of effective engineering reasoning.

Conclusion

Message [msg 3615] is a masterclass in post-fix analysis. It shows an AI assistant that doesn't just celebrate a successful bug fix but immediately pivots to understanding why the fix hasn't yet delivered the expected performance improvement. The assistant correctly identifies the three independent bottlenecks, formulates a testable hypothesis, and executes the next experiment — all within a single message. For anyone working on speculative decoding or large model deployment, this message offers a valuable case study in how to think about performance optimization: celebrate the wins, diagnose honestly, prioritize by actionability, and iterate quickly.