The 0.66x Reality: When Speculative Decoding Backfires

Introduction

In the high-stakes world of large language model inference, speculative decoding is supposed to be a silver bullet—a technique that uses a smaller, faster "draft" model to predict tokens ahead of time, which the larger "target" model then validates in parallel, achieving throughput gains with no loss in quality. The theory is elegant, the math is sound, and for many architectures it delivers 1.5–2.5x speedups in practice. But theory and practice do not always align, especially when exotic hardware, quantized models, and bleeding-edge attention mechanisms collide.

Message 3082 in this opencode session captures the precise moment when that collision became undeniable. After days of work—building an EAGLE-3 training pipeline from scratch, generating 10,000 synthetic training samples, extracting hidden states at 3,165 tokens per second, finetuning a drafter model over five epochs, and patching vLLM's internals with three separate monkey-patches to support the DeepSeek V3 / Kimi-K2.5 architecture—the assistant finally launched the speculative decoding server, verified it was producing correct output with reasoning, and ran a benchmark. The result: 54.6 tokens per second, compared to a baseline of 82.5 tokens per second without speculation. A speedup of 0.66x. EAGLE-3 was not just failing to accelerate inference; it was actively making it slower.

This article examines that message in depth: what led to it, what the assistant discovered, the reasoning process that unfolded, the assumptions that proved incorrect, and the strategic pivot it triggered.

The Long Road to a Disappointing Number

To understand why message 3082 carries such weight, one must appreciate the effort that preceded it. The session began in a completely different place—installing NVIDIA drivers and CUDA Toolkit on Ubuntu 24.04, resolving flash-attention build issues by reducing parallel compilation jobs from 128 to 20 to avoid memory exhaustion, and eventually stabilizing a working environment with PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1 on eight RTX PRO 6000 Blackwell GPUs.

The deployment of Kimi-K2.5 (a 1-trillion-parameter Mixture-of-Experts model, quantized to INT4) went through multiple iterations: benchmarking NVFP4 variants, switching to MiniMax-M2.5 FP8, and finally settling on the native INT4 version of Kimi-K2.5. A comprehensive profiling campaign identified AllReduce as the dominant bottleneck at 51.5% of decode time, leading the assistant to investigate speculative decoding as a software-only optimization path.

The EAGLE-3 pipeline itself was a massive undertaking. The assistant built a complete training pipeline, tested n-gram speculation (ruled it out as slower), then constructed an EAGLE-3 training infrastructure from scratch. This required patching the speculators library for vLLM 0.16 API compatibility, rewriting custom workers for the DeepSeekV2 forward pass, fixing KV cache config API mismatches, patching Scheduler and Request constructors, and fixing a collective_rpc unique_reply_rank bug. Hidden state extraction ran on 10 test samples, then scaled to 1,000, then to 10,000. The synthetic data generation script had to be rewritten to properly wrap reasoning with thinking/ response tokens. The 10K inference run completed in ~5.3 hours with zero errors. Hidden state extraction produced 828 GB of training data, and the 5-epoch finetune from the AQ-MedAI checkpoint completed in 2.6 hours.

When it came time to test the trained drafter with vLLM's EAGLE-3 integration, three separate patches were required just to make the model load: adding the model to a whitelist, handling image token edge cases, and implementing the SupportsEagle3 interface on both the DeepseekV3 and KimiK25 model classes. The first three attempts crashed with syntax errors and import bugs. Attempt 4 finally succeeded—vLLM started, loaded the 547 GB model, and began serving requests.

The Message: Benchmark Results

Message 3082 opens with the assistant's reaction to the benchmark output:

That's slower than baseline — 0.66x. EAGLE-3 is actively hurting performance. This could mean: 1. Poor draft model quality (low acceptance rate) 2. Overhead of running the draft model on TP=8 isn't worth it 3. The drafter's predictions don't match K2.5's distribution well enough

The benchmark was thorough: five diverse prompts (general relativity, N-queens, TCP vs UDP, photosynthesis, infinite primes), each requesting 512 tokens. The results were consistent across all five prompts, ranging from 49.0 to 65.1 tokens per second—all well below the 82.5 tok/s baseline. The assistant's immediate reaction is measured and analytical. Rather than expressing frustration or jumping to conclusions, it lists three hypotheses and immediately moves to gather diagnostic data.

The message then executes a curl command to fetch vLLM's Prometheus metrics, filtering for speculative decoding statistics:

curl -s http://localhost:8000/metrics 2>/dev/null | grep -i -E "accept|specul|draft|eagle" | head -20

The output reveals the raw metrics: vllm:spec_decode_num_drafts_total at 1506, vllm:spec_decode_num_drafts_created at 1.77 billion (a gauge value, likely cumulative token count), and the beginning of draft token statistics. The message cuts off at "Number of d..." but the intent is clear—the assistant is hunting for the acceptance rate, which would be computed in the subsequent message (3083) as 1,127 accepted out of 7,530 drafted = 15.0%.

The Reasoning Process

The thinking visible in this message reveals a methodical diagnostic approach. The assistant does three things simultaneously:

  1. Interprets the result: It immediately recognizes that 0.66x is worse than no speculation, and frames this as "actively hurting performance"—a precise characterization. The three hypotheses cover the main failure modes: drafter quality (the model itself is bad), overhead (the cost of running the drafter exceeds its benefit), and distribution mismatch (the drafter was trained on data that doesn't match the target model's output distribution).
  2. Formulates a diagnostic plan: Rather than speculating further, the assistant goes straight to the source of truth—vLLM's internal metrics. The Prometheus metrics endpoint exposes counters for drafts created, draft tokens generated, and tokens accepted. These numbers would allow computing the exact acceptance rate, which is the single most informative diagnostic for speculative decoding.
  3. Executes the diagnostic: The curl command is precise, filtering for the relevant metric names and limiting output to 20 lines. This is a production-quality debugging technique—query the running system's instrumentation rather than adding logging or restarting. The message also implicitly communicates a fourth activity: the assistant is processing the implications. The benchmark was run with num_speculative_tokens: 5, meaning the drafter proposes 5 tokens ahead, and the target model verifies them in a single forward pass. With a 15% acceptance rate (as we learn in the next message), the expected speedup is roughly 1 / (1 - 0.15 + 0.15*5) ≈ 0.57x before overhead, and the actual 0.66x is in the right ballpark. The assistant likely recognizes this pattern instantly.

Assumptions and Their Failure

Several assumptions underpinned the EAGLE-3 effort, and message 3082 is where they collide with reality:

Assumption 1: The AQ-MedAI checkpoint provides a good starting point for K2.5. The finetune started from AQ-MedAI/Kimi-K2-Instruct-eagle3, a pre-trained EAGLE-3 drafter for Kimi-K2 (not K2.5). The assumption was that finetuning on 10,000 samples of K2.5 output would adapt the drafter sufficiently. The 15% acceptance rate suggests this was incorrect—the distributional gap between K2 and K2.5, combined with the INT4 quantization, may be too large for 10K samples to bridge.

Assumption 2: Hidden states extracted during inference match those needed during decode. The training pipeline extracted hidden states from the target model during a separate inference pass. But during actual speculative decoding, the drafter receives hidden states from the decode phase, which may differ due to KV cache state, batch size effects, or subtle differences in the model's internal pathways when running with the speculative decoding wrapper. This is a known challenge in EAGLE-style training.

Assumption 3: The overhead of running the drafter on 8 GPUs is negligible. With tensor parallelism of 8, every operation involves communication across all GPUs. The drafter, while smaller than the target model, still requires GPU resources and allreduce synchronization. The 0.66x result suggests the overhead of running the drafter (and the verification forward pass) outweighs the benefit of the few tokens that are accepted.

Assumption 4: vLLM's EAGLE-3 implementation is efficient for MLA-based models. vLLM's EAGLE-3 support was clearly experimental for DeepSeek V3 / Kimi-K2.5—the assistant had to add the model to a whitelist and implement interfaces that didn't exist. The implementation may have suboptimal performance for Multi-head Latent Attention (MLA), which is the core attention mechanism in these models.

Knowledge Created

Message 3082 produces several critical pieces of knowledge:

  1. Empirical throughput measurement: 54.6 tok/s average across 5 diverse prompts, with individual results ranging from 49.0 to 65.1 tok/s. This is the ground truth for EAGLE-3 on Kimi-K2.5 INT4 with TP=8.
  2. Speedup factor: 0.66x relative to the 82.5 tok/s baseline. This definitively answers the question "does EAGLE-3 help on this setup?" with a clear "no."
  3. Diagnostic pathway: The message demonstrates how to query vLLM's Prometheus metrics for speculative decoding statistics, establishing a reusable debugging technique.
  4. Hypothesis space: The three enumerated hypotheses provide a framework for understanding why speculative decoding might fail, which is valuable for future debugging.
  5. Confirmation of a pattern: The consistency across all five prompts (none exceeded 65.1 tok/s) rules out prompt-specific effects and confirms the issue is systemic.

The Strategic Implications

The 0.66x result in message 3082 is a turning point. The assistant had already researched SGLang as an alternative (in a parallel task spawned at message 3074), discovering that SGLang has first-class EAGLE-3 support, lists it as the recommended speculative decoding method, and explicitly tests with Kimi-K2 drafters achieving ~1.8x throughput. The user had suggested SGLang at message 3063: "Also consider sglang if eagle3 support there is significantly better."

With the vLLM result now in hand—not just "not great" but actively worse than baseline—the case for pivoting to SGLang becomes compelling. The assistant's next actions (in messages 3083 and beyond) confirm the 15% acceptance rate, test the untrained AQ-MedAI baseline (which also achieves ~15%), and then pivot to installing and debugging SGLang on the SM120 Blackwell GPUs.

Conclusion

Message 3082 is a masterclass in how to handle negative results in machine learning engineering. The assistant invested enormous effort in building a pipeline, training a model, patching a framework, and deploying a service—only to discover that the result was worse than doing nothing. Rather than rationalizing, hand-waving, or diving into premature optimization, the assistant calmly measured, diagnosed, and documented the failure mode. The three hypotheses are precise and testable. The diagnostic query is immediate and targeted. The tone is professional and data-driven.

This message also illustrates a deeper truth about speculative decoding: it is not a universal accelerator. The technique works best when the draft model is well-matched to the target model's distribution, when the overhead of running the drafter is small relative to the target model's cost, and when the framework implementation is efficient for the specific architecture. All three conditions failed simultaneously here, producing a result that is both disappointing and instructive.

In the broader narrative of the session, message 3082 is the moment of reckoning that forces a strategic pivot from vLLM to SGLang—a pivot that would itself encounter new challenges (SM120 deadlocks, kernel compilation issues) but ultimately represent a more promising path. The 0.66x number is not the end of the story; it is the beginning of a new chapter.