The Moment of Truth: Benchmarking the EAGLE-3 Fix

Introduction

In any complex debugging journey, there comes a pivotal moment when the fix has been applied and the system must be tested. All the hours of tracing code paths, reading source files, and reasoning about hidden state dimensions culminate in a single question: did it work? Message [msg 3612] in this opencode session captures exactly that moment. After tracing a devastating EAGLE-3 hidden state concatenation bug to a single flag mismatch — --speculative-algorithm EAGLE instead of EAGLE3 — the assistant restarts the SGLang server and runs a benchmark to see if the draft model's predictions are finally being accepted. The results are simultaneously validating and sobering: the fix works, the hidden states are now 21504-dim as expected, and the acceptance length has jumped from 1.0 to ~2.1, but the overall throughput of 56.7 tok/s remains well below the non-speculative baseline of 90 tok/s. This message is a microcosm of the entire speculative decoding endeavor — a story of incremental progress, persistent bottlenecks, and the relentless pursuit of more training data.

The Context: A Bug That Broke Everything

To understand the weight of this benchmark, one must appreciate the debugging odyssey that preceded it. The session had been working toward deploying an EAGLE-3 draft model for the Kimi-K2.5 large language model, a sophisticated speculative decoding technique where a smaller "draft" model predicts multiple future tokens in parallel, and the larger "target" model verifies them. When predictions are correct, the system effectively generates multiple tokens per forward pass, yielding speedups.

But the draft model kept achieving zero acceptance — every prediction was rejected, meaning the speculation overhead was pure waste. The assistant traced the issue through layer after layer of the SGLang codebase: the eagle_worker.py that orchestrates draft generation, the logits_processor.py that concatenates hidden states, the kimi_k25.py wrapper that delegates to the underlying language model, and the model_runner.py that initializes the auxiliary hidden state capture mechanism. The root cause, discovered in [msg 3604], was astonishingly simple: the server had been started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in the SGLang code is strict — only the string "EAGLE3" triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With EAGLE, the target model only returned final-layer hidden states (7168 dimensions), while the draft model expected concatenated states from three layers (21504 dimensions). The draft model's fc fusion layer — a learned projection from 21504 → 7168 — was silently bypassed, and all trained weights were effectively dead.

The fix was a single command-line parameter change. After killing the server and restarting with --speculative-algorithm EAGLE3, the logs confirmed the hidden states were now arriving as torch.Size([21, 21504]) — three layers concatenated as expected. But a fix that works in theory must be validated in practice.

The Benchmark: Methodology and Execution

Message [msg 3612] begins with the assistant's characteristic methodical approach: "Good, inference is working. Now let me run the benchmark and check the server metrics for acceptance rate." This sentence reveals the dual purpose of the message — not just to measure raw throughput, but to understand why the system performs as it does by examining the acceptance metrics that the SGLang server exposes.

The assistant crafts a Python benchmark script (/tmp/eagle3_bench.py) that tests five diverse prompts:

  1. "Explain the theory of general relativity in simple terms."
  2. "Write a Python function that implements merge sort with detailed comments."
  3. "What are the main causes of climate change and what can we do about it?"
  4. "Describe the process of photosynthesis step by step."
  5. "Compare and contrast machine learning and traditional programming approaches." Each prompt requests 1024 tokens of output at temperature 0 (deterministic generation). The choice of five prompts is a practical compromise — enough to smooth out variance from prompt-specific effects (short vs. long prompts, topic-specific token distributions) without running an exhaustive evaluation. The 1024-token length ensures each test runs long enough for the speculative decoding system to reach steady-state behavior, where the draft model's acceptance statistics stabilize. Temperature 0 eliminates sampling randomness, making the benchmark reproducible and isolating the speculative decoding performance from stochastic effects. The script measures elapsed time per request, computes tokens-per-second, and then queries the server's /metrics endpoint for acceptance statistics. This dual measurement — external throughput plus internal acceptance metrics — is crucial for diagnosing why the system is fast or slow.

The Results: Validation and Disappointment

The benchmark produces these results:

Prompt 1: 1024 tokens in 20.8s = 49.2 tok/s (prompt: 19 tok)
Prompt 2: 1024 tokens in 13.9s = 73.7 tok/s (prompt: 20 tok)
Prompt 3: 1024 tokens in 19.0s = 53.9 tok/s (prompt: 24 tok)
Prompt 4: 1024 tokens in 18.3s = 56.0 tok/s (prompt: 18 tok)
Prompt 5: 1024 tokens in 18.3s = 56.0 tok/s (prompt: 18 tok)

Average: 56.7 tok/s (5120 tokens / 90.3s)

The variation is striking. Prompt 2 achieves 73.7 tok/s while Prompt 1 barely reaches 49.2 tok/s. This spread hints at the sensitivity of speculative decoding to the specific token distribution of the prompt and generated text. The merge sort prompt (Prompt 2) likely produces more predictable, structured output that the draft model can predict accurately, while the general relativity explanation (Prompt 1) may involve more varied vocabulary and less predictable token sequences.

The average of 56.7 tok/s is a clear improvement over the pre-fix state where acceptance was zero — but it is still substantially below the 90 tok/s non-speculative baseline established earlier in the session. This is the central tension of the message: the fix worked in the sense that the draft model is now receiving correct hidden states and producing accepted predictions, but it did not succeed in the sense of achieving the speedup that speculative decoding promises.

Interpreting the Acceptance Metrics

The subsequent messages ([msg 3614], [msg 3615]) reveal the server's internal metrics: acceptance length of ~2.0-2.5 (up from 1.0 before the fix) and acceptance rate of ~12-16% with 16 draft tokens. The acceptance length of ~2.1 means that on average, the draft model correctly predicts about 2 tokens before the target model rejects a prediction. This is a genuine improvement — before the fix, the acceptance length was exactly 1.0 (meaning no draft tokens were ever accepted, and the system fell back to generating one token at a time).

But the acceptance rate of ~15% reveals the problem: with 16 draft tokens generated per step, only about 2-3 are accepted. The overhead of running the draft model forward pass and then verifying all 16 candidates against the target model consumes more time than the ~2.3× effective speedup from accepted tokens saves. This is a classic speculative decoding failure mode: the draft model is not accurate enough to overcome the computational overhead of the speculation mechanism.

The assistant's reasoning in [msg 3615] shows clear understanding of this dynamic: "Accept rate ~15% is too low for speedup — speculation overhead (running the draft model + verifying 16 candidates) costs more than the ~2.3× accept length saves." This analysis correctly identifies that with low acceptance rates, reducing the number of draft tokens can actually improve throughput by lowering verification cost, even though it also reduces the maximum possible speedup.

Assumptions and Their Validity

The benchmark makes several implicit assumptions that deserve examination. First, it assumes that five prompts of 1024 tokens each provide a representative sample of the model's behavior. For a system as complex as speculative decoding, where performance depends on the alignment between the draft model's predictions and the target model's actual token distribution, this is a reasonable but limited assumption. The variation between prompts (49.2 to 73.7 tok/s) confirms that prompt-specific effects are significant.

Second, the benchmark assumes that temperature 0 (greedy decoding) is the right setting for evaluating speculative decoding. This is standard practice — deterministic generation removes sampling noise and makes results reproducible — but it may not reflect the performance the system would achieve with sampling, where the draft model's distribution might align differently with the target's.

Third, the assistant assumes that the server metrics (acceptance length, acceptance rate) are the right diagnostic tools for understanding performance. This assumption is well-founded — these metrics directly measure the effectiveness of the draft model and are exposed precisely for this purpose by the SGLang framework.

The Thinking Process: From Data to Decision

What makes this message particularly rich is the thinking process visible in the assistant's actions. The assistant does not simply run a benchmark and report results — it structures the test to produce actionable diagnostic information. The benchmark script measures both throughput and acceptance metrics, anticipating that the raw throughput numbers will need explanation. The choice to query /metrics alongside the benchmark shows a sophisticated understanding of how to debug performance issues: measure the outcome, then measure the internal state that produced it.

The assistant's reasoning in the subsequent message ([msg 3615]) demonstrates the logical chain: acceptance rate is too low → overhead dominates → try fewer draft tokens → also consider CUDA graphs and more training data. This chain connects the benchmark results to concrete next steps, transforming a disappointing throughput number into a research direction.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains. First, speculative decoding — the concept that a small draft model can predict multiple tokens that a large target model then verifies, with accepted tokens providing free speedup. Second, the EAGLE-3 architecture specifically, which uses hidden states from intermediate layers of the target model as input to the draft model, requiring careful concatenation of those states. Third, the SGLang serving framework and its configuration flags (--speculative-algorithm, --speculative-num-steps, --speculative-eagle-topk, etc.). Fourth, the Kimi-K2.5 model architecture, which wraps a DeepseekV3 language model in a vision-language model, adding complexity to the hidden state propagation path.

The reader also needs to understand the debugging history: that the hidden states were previously 7168-dim (single layer) instead of 21504-dim (three layers concatenated), that this was traced to the is_eagle3() check in spec_info.py, and that the fix was a single flag change. Without this context, the benchmark results would seem like random numbers rather than a validation of a specific hypothesis.

Output Knowledge Created

This message produces several important pieces of knowledge. First, it confirms that the hidden state concatenation fix is working — the draft model is now receiving correct multi-layer hidden states and producing accepted predictions. Second, it establishes a quantitative baseline for EAGLE-3 performance with the current 10K-sample draft model: 56.7 tok/s average, ~2.1 acceptance length, ~15% acceptance rate. Third, it reveals the gap between current performance and the target: 90 tok/s non-speculative baseline, meaning the draft model is actually slowing things down. Fourth, it identifies the primary bottleneck as insufficient training data, pointing toward the next major effort: scaling up the training dataset.

The Broader Significance

In the arc of the session, this message marks the transition from debugging to optimization. The hidden state concatenation bug was the last major blocker preventing EAGLE-3 from functioning at all. With the fix validated, the assistant can now focus on the harder problem of making it fast. The benchmark results directly motivate the next major effort: scaling the training data from 10K samples to 88K samples, launching an inference pipeline that will run for 24-55 hours to regenerate responses through the Kimi-K2.5 model.

The message also reveals a fundamental truth about speculative decoding: the draft model must be good enough to justify the overhead of running it alongside the target model. A draft model that predicts only 2 out of 16 tokens correctly is worse than no draft model at all. The EAGLE-3 paper's scaling curves suggest that more training data is the primary lever for improving acceptance rates, and the assistant's subsequent actions — searching for datasets, preparing prompts, launching a massive inference pipeline — are a direct response to this insight.

Conclusion

Message [msg 3612] is a classic "moment of truth" in a debugging session: the fix is applied, the test is run, and reality delivers its verdict. The verdict is mixed — the fix works, but not well enough. The assistant's response to this mixed result is exemplary: rather than celebrating a partial victory or despairing at the gap to the baseline, it methodically analyzes the metrics, identifies the root cause (insufficient training data), and pivots to the next phase of work. The benchmark script itself, with its careful design and dual measurement strategy, reflects a sophisticated understanding of how to evaluate and debug complex ML systems. This message is not just about numbers — it is about the reasoning process that turns those numbers into actionable insights, driving the session forward toward its ultimate goal of making EAGLE-3 speculation actually faster than the baseline.