The Moment of Truth: Benchmarking EAGLE-3 Speculative Decoding at 56.8 tok/s

The Message

[assistant] [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /tmp/benchmark_eagle3.py --server-url http://localhost:8000 --max-tokens 2048 --num-runs 5 --warmup 2'

>

Benchmarking http://localhost:8000 max_tokens=2048, num_runs=5, warmup=2

>

Warmup 1/2 done (41 tokens) Warmup 2/2 done (41 tokens)

>

Run 1/5: 2048 tokens in 36.32s = 56.4 tok/s (prompt: 32 toks) Run 2/5: 2048 tokens in 33.90s = 60.4 tok/s (prompt: 31 toks) Run 3/5: 2048 tokens in 38.80s = 52.8 tok/s (prompt: 26 toks) Run 4/5: 2038 tokens in 35.29s = 57.8 tok/s (prompt: 28 toks) Run 5/5: 2048 tokens in 35.89s = 57.1 tok/s (prompt: 34 toks)

>

========== Results ========== Overall ...

Introduction

In any complex engineering debugging session, there comes a single moment when all the accumulated fixes, patches, and adjustments are finally put to the test. Message 4356 in this opencode session is precisely that moment. After rounds of fixing weight key mismatches, correcting hidden state concatenation bugs, patching the SGLang inference engine to capture embedding outputs, and restarting the server with a corrected draft model configuration, the assistant runs a benchmark to measure whether the EAGLE-3 speculative decoding system actually delivers on its promise. The answer, delivered across five runs averaging roughly 56.8 tokens per second, is a sobering disappointment — and the catalyst for the next phase of deep debugging.

This message is deceptively simple: a single bash command executing a Python benchmark script on a remote server. But understanding why this particular command was issued at this exact moment, what assumptions it carries, what knowledge it presupposes, and what it reveals requires reconstructing the entire trajectory of the EAGLE-3 deployment effort. This article examines that single message as a pivotal artifact in a much larger debugging narrative.

Context and Motivation: Why This Message Was Written

To understand why message 4356 exists, we must trace the chain of events that led to it. The assistant and user had been working for hours — across multiple segments and chunks — to deploy a custom-trained EAGLE-3 draft model for the Kimi-K2.5 large language model. The draft model had been trained on 100,000 synthetic samples and achieved 74.7% validation accuracy, which should have been sufficient to provide meaningful speculative decoding speedups.

However, earlier benchmarks had been disappointing. In segment 30, after deploying the draft model with SGLang speculation, performance was poor. In segment 31 (the current chunk), the assistant had already identified and fixed two significant bugs:

  1. The --speculative-num-steps override: The server was started with --speculative-num-steps 1, which silently overrode the --speculative-num-draft-tokens 16 setting, reducing effective draft tokens to just 2 due to a SGLang constraint when topk=1. After fixing this to --speculative-num-steps 15, performance actually worsened to 46.7 tok/s, revealing that the draft model itself wasn't predicting well.
  2. The hidden state input format mismatch: A standalone test revealed that the training pipeline used cat([embed_output, layer3, layer31]) as input to the draft model's fully-connected layer, but SGLang was passing cat([layer3, layer31, layer59]) — the three auxiliary hidden states captured by the target model, completely missing the embedding output. The assistant modified deepseek_v2.py in SGLang to capture the embedding output when layer_id=-1 is specified, updated the draft model config from [2, 30, 58] to [-1, 2, 30], and restarted the server. Message 4356 is the first benchmark after applying the hidden state fix. It represents the assistant's attempt to answer a critical question: "Did fixing the input format mismatch resolve the performance problem?" The benchmark had been carefully prepared: the assistant modified the original script to use max_tokens=2048 instead of 512 (to accommodate reasoning models that generate long <think> chains), increased the number of runs to 5 for statistical significance, and added a warmup phase. The server was confirmed running with EAGLE3 active and 16 draft tokens. All the pieces were in place for a definitive measurement.

The Benchmark Results: What the Numbers Reveal

The five runs show a consistent pattern:

| Run | Tokens | Time (s) | tok/s | Prompt Tokens | |-----|--------|----------|-------|---------------| | 1 | 2048 | 36.32 | 56.4 | 32 | | 2 | 2048 | 33.90 | 60.4 | 31 | | 3 | 2048 | 38.80 | 52.8 | 26 | | 4 | 2038 | 35.29 | 57.8 | 28 | | 5 | 2048 | 35.89 | 57.1 | 34 |

The throughput ranges from 52.8 to 60.4 tok/s, with a rough average around 56.8 tok/s. The variation across runs is relatively modest (about ±4 tok/s), suggesting the measurement is stable and the results are reproducible. The prompt lengths vary between 26 and 34 tokens — short prompts typical of the benchmark's factual/general knowledge questions.

But the critical context, which the assistant already knows from prior benchmarks, is that the baseline without speculation is approximately 90 tok/s. At 56.8 tok/s, the EAGLE-3 speculative decoding system is delivering only about 63% of the baseline throughput. This is worse than running without speculation at all — the overhead of generating and evaluating draft tokens is actively harming performance.

The message itself truncates the output at "Overall ...", but the subsequent message (msg 4357) reveals the assistant's immediate reaction: "56.8 tok/s with EAGLE3 + 16 draft tokens — that's significantly worse than the 90 tok/s baseline. The speculation overhead is killing performance."

Assumptions Embedded in This Message

This benchmark message carries several implicit assumptions, some of which turn out to be incorrect or incomplete:

Assumption 1: The hidden state fix would substantially improve performance. The assistant had just corrected a fundamental wiring error — the draft model was receiving the wrong input features during inference. The standalone test confirmed that with the correct input format, the draft model achieved 76.9% accuracy, matching training metrics. It was reasonable to expect that fixing this mismatch would translate into better acceptance rates and higher throughput. The benchmark results disprove this assumption: despite the fix, performance remains poor.

Assumption 2: 16 draft tokens is an appropriate configuration. The server was launched with --speculative-num-draft-tokens 16, which means the draft model generates 16 candidate tokens per step. However, with an accept length of only ~1.6 (as revealed in msg 4357), the system is generating 16 tokens and then immediately discarding almost all of them. This suggests either that 16 is far too many (the draft model's predictions degrade rapidly beyond the first few tokens) or that there is a more fundamental issue with how the draft model interacts with the target model.

Assumption 3: The benchmark methodology is sound. The assistant increased max_tokens to 2048 and added warmup runs, which are reasonable choices. However, the prompts used are short factual questions (e.g., "Explain the theory of general relativity in detail..."), which may not be representative of the kinds of queries where speculative decoding would shine. The assistant also didn't capture server-side metrics like accept length in this benchmark run, requiring a separate check in the next message.

Assumption 4: The server configuration is optimal. The assistant assumed that the server was correctly configured after the restart. However, the next message reveals that the accept length is only ~1.6 out of 16 draft tokens, and the accept rate is about 0.79-0.84. This means the target model is accepting fewer than 2 tokens on average per speculative step, which is catastrophically low for effective speculation.

Input Knowledge Required to Understand This Message

A reader needs substantial domain knowledge to interpret this message correctly:

  1. EAGLE-3 speculative decoding architecture: Understanding that EAGLE-3 uses a lightweight draft model that predicts future hidden states, which are then fed to the target model's output layers for acceptance verification. The draft model operates on concatenated hidden states from multiple layers of the target model.
  2. SGLang server infrastructure: The benchmark targets http://localhost:8000, which is the SGLang inference server running on a remote machine (10.1.230.174). The Python environment is at ~/ml-env/bin/python3, indicating a custom virtual environment. The benchmark script was SCP'd to /tmp/.
  3. The hardware context: The server has 8 NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96GB of memory (approximately 89GB used). The model being served is Kimi-K2.5 in 4-bit quantized format (/shared/kimi-k2.5-int4).
  4. The training pipeline: The draft model was trained on 100,000 synthetic samples generated via OpenRouter API and through an inference pipeline. It achieved 74.7% validation accuracy, which should theoretically provide good speculative decoding performance.
  5. Prior debugging history: The reader must know about the weight key name mismatch fix, the hidden state concatenation bug, and the --speculative-num-steps override issue that were all addressed before this benchmark.

Output Knowledge Created by This Message

This single benchmark run produces several critical pieces of knowledge:

  1. Quantified performance gap: The EAGLE-3 system delivers ~56.8 tok/s versus a ~90 tok/s baseline — a 37% deficit. This is a concrete, measurable result that reframes the debugging effort from "is it working?" to "why is it so much worse than the baseline?"
  2. Reproducibility evidence: Five runs with consistent results demonstrate that the poor performance is not a fluke or transient condition. The narrow variance (52.8-60.4 tok/s) indicates a stable, systemic issue.
  3. Confirmation that the hidden state fix alone is insufficient: The most important finding is negative evidence — the input format mismatch was not the root cause of poor performance. Something deeper is wrong.
  4. Baseline for future comparisons: These numbers become the reference point for subsequent debugging. Any future fix can be evaluated against this 56.8 tok/s benchmark.
  5. Trigger for deeper investigation: The poor results immediately prompt the assistant to check server-side metrics (accept length, accept rate) in the next message, leading to the discovery that the accept length is only ~1.6 — meaning the draft model's predictions are being rejected almost immediately.

The Thinking Process Visible in the Reasoning

While message 4356 itself contains no explicit reasoning (it is purely a tool call with its output), the reasoning is visible in the surrounding messages and in the structure of the benchmark itself.

In msg 4354, the assistant explicitly states: "The benchmark script uses max_tokens=512 which is a bit short for a reasoning model that generates <think> chains. Let me increase it and also add more runs for statistical significance." This reveals deliberate experimental design: the assistant anticipates that a reasoning model may produce long thinking chains, so a 512-token limit might truncate the measurement. Increasing to 2048 tokens and adding 5 runs with 2 warmup runs shows a methodical approach to benchmarking.

The choice of 5 runs with warmup reflects an understanding of GPU inference dynamics: the first few iterations may be slower due to CUDA graph compilation, cache warmup, and other initialization overhead. The warmup runs (producing only 41 tokens each) suggest that the server was already past the initial CUDA graph capture phase (visible in the server logs from msg 4352 showing "Capture draft extend cuda graph begin").

The fact that the assistant runs this benchmark immediately after fixing the hidden state issue and restarting the server (without any intermediate validation) indicates a "measure first, then analyze" methodology. Rather than trying to reason about whether the fix would work, the assistant lets the data speak first. This is characteristic of effective debugging: form a hypothesis, implement a fix, measure the result, and let the outcome guide the next step.

Mistakes and Incorrect Assumptions Revealed

The most significant incorrect assumption exposed by this message is that fixing the hidden state input format would be sufficient to achieve good speculative decoding performance. The standalone test showed 76.9% accuracy with the correct input format, which should have translated into meaningful speedups. The fact that it didn't suggests one of several possibilities:

  1. The standalone test is not representative: The test may have used different data distributions, different sampling strategies, or different evaluation criteria than the actual inference pipeline. Accuracy on a test set does not guarantee good speculative decoding behavior.
  2. There are additional bugs beyond the input format: The draft model may be correctly receiving hidden states now, but there could be issues with how the speculative decoding loop integrates with the target model's KV cache, attention mechanisms, or sampling logic.
  3. The draft model itself may be fundamentally flawed: Despite 74.7% training accuracy, the model may not generalize well to the diverse prompts used in the benchmark. The training data (synthetic responses generated via OpenRouter API) may differ significantly from the benchmark prompts.
  4. Configuration issues remain: The --speculative-num-draft-tokens 16 setting may be inappropriate for this model. With an accept length of ~1.6, the system is wasting compute on 14+ tokens that are almost always rejected. A smaller number of draft tokens might actually improve throughput by reducing overhead. Another subtle mistake is that the benchmark doesn't capture server-side metrics (accept length, accept rate, draft token efficiency) in the same run. The assistant has to make a separate request in msg 4357 to get this information. Integrating metrics collection into the benchmark script would have provided a more complete picture in a single step.

Conclusion

Message 4356 is a pivotal moment in a complex debugging session — the point where a carefully prepared hypothesis meets empirical reality. The assistant had invested significant effort in diagnosing and fixing a hidden state input format mismatch, only to discover that the fix moved performance from 46.7 tok/s to 56.8 tok/s — still far below the 90 tok/s baseline. This negative result is itself valuable knowledge: it rules out one hypothesis and forces the investigation to go deeper.

The message exemplifies the scientific method in engineering debugging: form a hypothesis based on available evidence, implement a targeted fix, measure the outcome precisely, and let the data guide the next iteration. The benchmark's design — multiple runs, warmup phases, increased token limits — reflects a methodical approach to measurement. And the disappointing results serve as the catalyst for the next phase of investigation, which will uncover the accept length problem and drive further refinements.

In the broader narrative of the EAGLE-3 deployment, this message marks the transition from "fixing obvious bugs" to "understanding deeper systemic issues." The input format mismatch was a bug that could be identified and corrected through code inspection. The remaining performance gap requires understanding the complex dynamics of speculative decoding — why a draft model with 74.7% accuracy fails to achieve meaningful speedups in practice. That investigation continues in the messages that follow.