The Verdict on Speculative Decoding: When EAGLE-3 Fails on Blackwell

Introduction

In the high-stakes world of large language model serving, every millisecond counts. When you have eight NVIDIA RTX PRO 6000 Blackwell GPUs powering a 547GB Kimi-K2.5 INT4 model, the difference between a good serving setup and a great one can be hundreds of tokens per second. This article examines a single, decisive message in an opencode coding session where the assistant—after an extensive multi-day campaign of building, patching, training, and benchmarking—finally renders a definitive verdict on EAGLE-3 speculative decoding for the Kimi-K2.5 model on Blackwell hardware. The message, <msg id=3218>, is a moment of reckoning: the assistant has just finished benchmarking a custom-trained EAGLE-3 drafter and must now confront the uncomfortable truth that speculative decoding, despite weeks of effort, is not delivering any benefit.

The Message in Full

The subject message begins with a stark diagnosis:

Our custom drafter has accept rate 0.25 and accept len 1.00 — meaning it accepts exactly 0 draft tokens per step (accept len 1.00 = only the original token). The accept rate of 25% = 1/4, which is the rate of accepting 0 out of 4 draft tokens per step (i.e., the drafter is proposing all wrong tokens). This is essentially random — our drafter is completely broken.

The assistant then presents a comparison table across four configurations, summarizes four key conclusions, offers a root-cause hypothesis, and executes a cleanup command to kill the failed server and prepare for the next pivot.

Context: The Road to This Message

To understand why this message carries such weight, we must trace the path that led here. The session began in Segment 19 with a comprehensive profiling campaign of Kimi-K2.5 INT4 on 8x Blackwell GPUs. Profiling revealed that AllReduce dominated decode time at 51.5%, and the team pivoted to investigating speculative decoding as a software-only optimization path. Over the next several segments, the assistant:

The Reasoning and Motivation Behind the Message

The assistant's primary motivation in <msg id=3218> is to deliver a clear, data-driven verdict on whether EAGLE-3 speculative decoding is viable for this deployment. This is not a casual observation—it is the culmination of days of engineering effort spanning model patching, library compatibility fixes, training pipeline construction, synthetic data generation, and multiple benchmarking rounds across two different inference engines (vLLM and SGLang).

The reasoning structure of the message reveals a methodical analytical process:

  1. Direct observation: The assistant first states the raw numbers—accept rate 0.25, accept len 1.00—and translates them into plain English: the drafter is accepting exactly 0 out of 4 proposed draft tokens per step.
  2. Comparative analysis: The assistant immediately contextualizes this result by comparing it to the AQ-MedAI drafter's ~42% accept rate, noting that even that mediocre result is better than the custom drafter's performance.
  3. Comprehensive table: The assistant constructs a four-row comparison table spanning all configurations tested: vLLM baseline, SGLang base, SGLang + AQ-MedAI EAGLE-3, and SGLang + Custom K2.5 EAGLE-3. This table serves as the definitive summary of the entire speculative decoding investigation.
  4. Structured conclusions: The four numbered conclusions distill the table into actionable insights, ranked from most general to most specific.
  5. Root cause hypothesis: The assistant proposes a theory for why the custom drafter failed—the hidden state extraction pipeline used BF16/FP16 hidden states during training but the serving model is INT4 quantized, creating an alignment gap.
  6. Decisive action: The message concludes with a cleanup command to kill the failed server, signaling a clear pivot away from the current approach.

How Decisions Were Made

This message is itself a decision document. The key decisions embedded within it are:

Decision 1: Abandon the custom EAGLE-3 drafter. The accept rate of 25% (effectively random) and single-stream throughput of 40.8 tok/s (worse than the base SGLang's 63.6 tok/s) leaves no room for ambiguity. The custom drafter is not merely underperforming—it is actively harmful to throughput.

Decision 2: Abandon EAGLE-3 speculative decoding entirely for this deployment. Even the better-performing AQ-MedAI drafter, with its ~42% accept rate, yields lower peak throughput (849 tok/s) than the base SGLang server (2,370 tok/s). The speculative overhead—running 4 draft tokens per step across 3 steps—exceeds the benefit at high concurrency, and the max_running_requests=48 limit that SGLang imposes for speculative mode cripples throughput.

Decision 3: Restore the SGLang base server as the best throughput option. The assistant explicitly states "Let me kill this and restore the SGLang base server (best throughput option)" before executing the cleanup command. This is a clear recommitment to the non-speculative configuration.

Decision 4: Accept the root cause hypothesis as a working theory. The assistant identifies the INT4 quantization mismatch as the likely cause of the drafter's failure, but does not commit to fixing it immediately. This hypothesis will inform the next phase of work—the planned retraining with SGLang-based extraction—but the immediate priority shifts to tuning SGLang's single-stream performance.

Assumptions Made

Several assumptions underpin the analysis in this message:

Assumption 1: The expected acceptance rate for a well-trained EAGLE-3 drafter is 70%+. The assistant states that even the AQ-MedAI drafter's 42% "should be 70%+." This assumption is based on published results for EAGLE-3 on other model architectures and may not account for the unique challenges of the Kimi-K2.5 architecture with MLA (Multi-head Latent Attention).

Assumption 2: The INT4 quantization is the root cause of the alignment failure. This is a plausible hypothesis but remains unproven. The hidden states from an INT4 quantized model may indeed differ significantly from BF16 hidden states, but there could be other factors at play—differences in the training data distribution, architectural mismatches in the drafter, or issues in the training pipeline itself.

Assumption 3: The AQ-MedAI drafter's mediocre performance is due to being trained on Kimi-K2 rather than K2.5. This was stated in the previous message (<msg id=3210>) and is carried forward as an implicit assumption in <msg id=3218>. While plausible, the gap between K2 and K2.5 may not fully explain the 42% vs 70%+ discrepancy.

Assumption 4: The max_running_requests=48 limit is a hard constraint imposed by SGLang for speculative mode. The assistant treats this as an immutable limitation, though in principle it might be configurable. The assumption is that SGLang's internal resource accounting for speculative decoding requires this limit, and raising it would cause out-of-memory errors.

Mistakes and Incorrect Assumptions

Mistake 1: The custom drafter was expected to outperform the AQ-MedAI drafter. The assistant launched the custom K2.5-trained drafter with the implicit expectation that training on K2.5 data would yield better acceptance rates than the K2-trained AQ-MedAI model. The opposite occurred—the custom drafter was dramatically worse. This suggests a fundamental flaw in the training pipeline that was not caught during validation.

Mistake 2: The training pipeline may have had an undiagnosed issue. The assistant had previously validated the training pipeline on 10 samples and scaled to 1000 samples, but the resulting drafter performs at chance level. This indicates either a bug in the training script, a mismatch between the hidden state extraction format and the drafter's expected input, or a problem with the synthetic data quality.

Mistake 3: The assumption that SGLang would handle EAGLE-3 better than vLLM. The pivot from vLLM to SGLang in Segment 23 was motivated by vLLM's broken 15% acceptance rate. While SGLang did improve to 42% with the AQ-MedAI drafter, the overall conclusion remains that EAGLE-3 is not beneficial on this hardware. The pivot was necessary to get a fair test, but it did not change the fundamental outcome.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. Speculative decoding: The technique of using a smaller "draft" model to propose multiple candidate tokens, which the larger "target" model then verifies in parallel. Acceptance rate measures what fraction of proposed tokens are accepted.
  2. EAGLE-3: A specific speculative decoding framework that uses hidden state alignment between the draft and target models. The "accept len" metric indicates how many draft tokens are accepted per step on average.
  3. SGLang and vLLM: Two popular open-source LLM serving frameworks. The assistant has been testing both and comparing their performance on the same hardware.
  4. Kimi-K2.5 and its architecture: A large language model from Moonshot AI, using MLA (Multi-head Latent Attention) and a Mixture-of-Experts (MoE) architecture. The model is deployed in INT4 quantization to fit across 8 GPUs.
  5. CUDA Graphs: A technique for capturing and replaying GPU operations to reduce kernel launch overhead, used by both SGLang and vLLM to accelerate inference.
  6. Tensor Parallelism (TP): The model is split across 8 GPUs using TP=8, meaning each GPU holds a shard of each layer.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A definitive benchmark comparison across four configurations on the same hardware, providing a clear ranking: SGLang base (2,370 tok/s) > vLLM base (1,536 tok/s) > SGLang + AQ-MedAI EAGLE-3 (849 tok/s) > SGLang + Custom EAGLE-3 (597 tok/s).
  2. The empirical finding that EAGLE-3 speculative decoding provides no benefit on 8x Blackwell GPUs for Kimi-K2.5 INT4. This is a non-obvious result—speculative decoding is widely reported to improve throughput, but the combination of INT4 quantization, MLA attention, and Blackwell's raw compute power makes the overhead of running the drafter exceed the benefit.
  3. The hypothesis that INT4 quantization creates a hidden state alignment gap that breaks EAGLE-3's core mechanism. This is a valuable insight for anyone attempting speculative decoding with quantized models.
  4. The observation that SGLang's max_running_requests=48 limit for speculative mode is a throughput bottleneck. This is a practical deployment constraint that future work must address.
  5. A clear decision to pivot away from speculative decoding toward tuning SGLang's single-stream performance, establishing a new direction for the remainder of the session.

The Thinking Process Visible in the Message

The assistant's reasoning unfolds in a clear, logical progression:

Step 1 — Data interpretation: The assistant receives the raw acceptance logs from <msg id=3217> and immediately interprets them. The accept rate of 0.25 is not 25% in the sense of "one in four draft tokens is accepted"—rather, it's the rate at which the drafter's entire proposal is accepted. With 4 draft tokens per step, an accept rate of 0.25 means the drafter is essentially guessing randomly.

Step 2 — Translation to practical impact: The assistant translates the accept rate into single-stream throughput (40.8 tok/s) and compares it to the baseline (63.6 tok/s). This makes the abstract metric concrete: the drafter is making the model slower, not faster.

Step 3 — Comparative analysis: The assistant places the custom drafter's results alongside the AQ-MedAI drafter and both baselines in a single table. This reveals the full landscape of what has been tested.

Step 4 — Synthesis into conclusions: The four numbered conclusions extract the key insights from the comparison. Notably, the assistant ranks them from most general (EAGLE-3 fundamentally not working) to most specific (the max_running_requests limit).

Step 5 — Root cause theorizing: The assistant proposes the INT4 quantization mismatch as the likely explanation. This is a sophisticated inference that connects the training pipeline (which used BF16 hidden states) with the inference pipeline (which uses INT4). The assistant does not present this as proven fact but as a hypothesis to guide future work.

Step 6 — Decisive action: The message ends with a cleanup command, demonstrating the assistant's commitment to acting on its analysis rather than continuing to investigate a dead end.

The Broader Significance

This message represents a critical juncture in the coding session. The assistant has invested enormous effort in building an EAGLE-3 speculative decoding pipeline from scratch—patching model files, fixing library incompatibilities, writing training scripts, generating synthetic data, and running multiple training iterations. The conclusion that all of this work has not improved throughput is a significant setback.

However, the message also demonstrates intellectual honesty and scientific rigor. The assistant does not try to spin the results or find excuses—it presents the data clearly, acknowledges the failure, and pivots to a more promising direction. The root cause hypothesis about INT4 quantization provides a clear path forward: retrain the drafter using SGLang-based hidden state extraction that matches the actual inference quantization.

The message also contains an implicit lesson about the nature of optimization work: not every promising technique delivers results in practice. Speculative decoding, for all its theoretical elegance, can be defeated by the combination of quantization, architectural complexity, and raw GPU compute power. The Blackwell GPUs are so fast at running the base model that the overhead of running a drafter—even a well-trained one—can exceed the benefit.

Conclusion

Message <msg id=3218> is a masterclass in data-driven decision making under uncertainty. The assistant takes a set of disappointing benchmark numbers, interprets them correctly, contextualizes them against prior results, synthesizes them into actionable conclusions, proposes a plausible root cause, and executes a decisive pivot. The message is honest about failure without being defeatist—it acknowledges that the custom drafter is "completely broken" while simultaneously identifying the likely path to a fix.

For anyone working on LLM serving optimization, this message offers several valuable lessons: benchmark everything, compare against multiple baselines, be willing to abandon approaches that don't work, and always look for the deeper structural cause behind surface-level metrics. The EAGLE-3 experiment on Kimi-K2.5 may have failed, but the knowledge gained—about the limits of speculative decoding with quantized models, about the performance characteristics of SGLang versus vLLM on Blackwell, and about the importance of alignment between training and inference pipelines—will inform every subsequent optimization decision.