The 0.66x Verdict: How a Single Benchmark Message Exposed a Fundamental Integration Flaw

The Message

In the middle of an intensive speculative decoding campaign for the 1-trillion-parameter Kimi-K2.5 INT4 model running on 8x Blackwell GPUs, the assistant issued this succinct but devastating benchmark result:

AQ-MedAI drafter loaded. Let me benchmark it:

``` === EAGLE-3 Single-Stream Benchmark ===

Prompt 1: 512 tokens in 11.0s = 46.7 tok/s (21.4 ms/tok) Prompt 2: 512 tokens in 7.7s = 66.1 tok/s (15.1 ms/tok) Prompt 3: 512 tokens in 9.6s = 53.4 tok/s (18.7 ms/tok) Prompt 4: 512 tokens in 9.4s = 54.2 tok/s (18.4 ms/tok) Prompt 5: 512 tokens in 9.3s = 55.2 tok/s (18.1 ms/tok)

TOTAL: 2560 tokens in 47.0s AVG: 54.4 tok/s (18.4 ms/tok) Baseline (no spec decode): 82.5 tok/s (12.1 ms/tok) Speedup: 0.66x ```

At first glance, this looks like a routine performance measurement. But this single benchmark run — the second of two nearly identical tests — was the moment the entire speculative decoding strategy collapsed. The numbers tell a story that goes far deeper than a failed optimization attempt.

Why This Message Was Written

The message was written as a controlled experiment to isolate the root cause of a catastrophic failure. The assistant had just spent hours building a complete EAGLE-3 training pipeline: generating 10,000 synthetic reasoning samples from Kimi-K2.5's own outputs, extracting hidden states at 3,165 tok/s producing 828 GB of training data, and fine-tuning an EAGLE-3 drafter from the AQ-MedAI/Kimi-K2-Instruct-eagle3 checkpoint over 5 epochs in 2.6 hours.

When the custom-trained drafter was first tested with vLLM's EAGLE-3 speculative decoding ([msg 3081]), the result was deeply disappointing: only 54.6 tok/s, a 0.66x slowdown compared to the baseline of 82.5 tok/s without any speculation. The acceptance rate — the fraction of draft tokens accepted by the target model — was measured at a mere 15.0% ([msg 3083]), far below the 60-80% that EAGLE-3 typically achieves.

The assistant faced a critical ambiguity: was the poor performance caused by insufficient or flawed training data, or by a fundamental integration problem between vLLM's EAGLE-3 implementation and the DeepSeek V3 / Kimi-K2.5 architecture with its Multi-Head Latent Attention (MLA)? To resolve this, the assistant designed a clean experiment: load the pre-trained AQ-MedAI drafter — the exact checkpoint used as the starting point for fine-tuning — and benchmark it under identical conditions. If the pre-trained drafter performed well, the training pipeline was at fault. If it performed equally poorly, the problem lay in vLLM's integration layer.

This message is the result of that experiment. It is the second data point in a controlled pair, and its meaning derives entirely from its relationship to the first measurement.

How Decisions Were Made

The decision to benchmark the AQ-MedAI baseline was a deliberate diagnostic pivot. After the custom-trained drafter showed 15% acceptance, the assistant could have pursued several paths: retrain with more data, adjust hyperparameters, investigate the hidden state extraction pipeline, or modify the training loss function. Instead, the assistant chose the most information-efficient experiment: test the pre-existing checkpoint that was known to work with SGLang (the inference engine it was designed for) to see if it also failed on vLLM.

This decision reflects a mature debugging methodology. Rather than chasing training-quality hypotheses (which would have required hours of additional computation), the assistant isolated the variable that would most quickly distinguish between two competing root-cause theories. The experiment was designed to be maximally informative with minimal additional work: load a different model path, run the same benchmark script (/tmp/bench_eagle3.py), compare results.

The operational decisions were also notable. The assistant had to kill the previous vLLM server instance (running with the custom-trained drafter), clear GPU memory, and launch a new server with the AQ-MedAI drafter path. This required careful process management: killing all Python processes, waiting for GPU memory to release, clearing /dev/shm/, and launching with the same configuration flags (--tensor-parallel-size 8, --max-model-len 32768, --speculative-config with num_speculative_tokens: 5). The 30-minute wait for model loading (visible in [msg 3086]) reflects the sheer scale of the 547 GB model being loaded across 8 GPUs.

Assumptions Made

Several assumptions underpin this message, and one of them proved to be incorrect in a way that reshaped the entire project direction.

Assumption 1: The AQ-MedAI drafter was a valid baseline. The assistant assumed that the AQ-MedAI/Kimi-K2-Instruct-eagle3 checkpoint, being the foundation from which fine-tuning began, would serve as a meaningful "before" measurement. This was correct in principle but carried a subtle complication: the AQ-MedAI drafter was trained for Kimi-K2, not Kimi-K2.5. While the two models share the same architecture, there could be distributional differences that affect acceptance rates. The assistant acknowledged this earlier ([msg 3077]) but proceeded anyway, treating the baseline as informative even if not perfectly matched.

Assumption 2: The benchmark script was reliable. The assistant reused /tmp/bench_eagle3.py, the same script written for the first benchmark ([msg 3081]). This script sends 5 prompts, each requesting 512 tokens, and measures throughput. The assumption was that this provides a representative measurement. In practice, single-stream latency benchmarks can be noisy, but the consistency across both runs (46.9s total for the custom drafter, 47.0s for AQ-MedAI) suggests the measurement was stable.

Assumption 3: vLLM's EAGLE-3 implementation was functionally correct even if suboptimal. This assumption was the one that broke. The assistant had already applied three patches to vLLM to make EAGLE-3 work with DeepSeek V3/Kimi-K2.5: a model whitelist patch, an image token handling fix, and a SupportsEagle3 interface implementation. The assumption was that these patches, while hacky, would produce correct behavior. The results proved otherwise.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was that vLLM's EAGLE-3 integration, once patched to load, would function correctly for MLA-based models. The 15% acceptance rate — barely above random guessing for a 5-token speculation window — indicates that the auxiliary hidden states being extracted from the target model's attention layers during decode are fundamentally misaligned with what the draft model expects.

This is not a training problem. The AQ-MedAI drafter, which achieves ~1.8x speedup on SGLang with Kimi-K2, achieved only 0.66x on vLLM with Kimi-K2.5. The identical failure mode across both drafters (15.0% vs 15.6% acceptance) points to a systematic issue in how vLLM extracts and passes hidden states from DeepSeek V3's MLA attention mechanism to the EAGLE-3 draft model. The MLA architecture uses a compressed latent representation for keys and values, which differs from standard multi-head attention. If vLLM's EAGLE-3 integration extracts hidden states from the wrong layer or fails to decompress the MLA latent representation correctly before passing it to the draft model, the drafter receives garbage input and produces useless predictions.

A secondary mistake was the assumption that patching vLLM's model whitelist and interface system would be sufficient. The three patches addressed loading-time errors (SyntaxError from a broken import, missing interface methods, image token handling), but they did not address the runtime semantics of hidden state propagation. The assistant was operating with incomplete knowledge of vLLM's internal architecture — a necessary compromise when working with bleeding-edge hardware (SM120 Blackwell GPUs) and recently released model architectures.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

Speculative decoding fundamentals: The concept of using a smaller "draft" model to predict multiple tokens that the target model then verifies in parallel. EAGLE-3 is a specific architecture that uses auxiliary hidden states from the target model's intermediate layers to condition the draft model's predictions, achieving higher acceptance rates than independent draft models.

The EAGLE-3 training pipeline: The assistant had just completed a full training run: generating synthetic data from Kimi-K2.5's own outputs (10K samples, 17.3M tokens), extracting hidden states from intermediate layers (specifically layers 2, 30, and 58 as defined in the AQ-MedAI config), and fine-tuning the drafter. The 828 GB of training data reflects the scale of storing hidden state vectors for every token position.

vLLM's architecture: Understanding that vLLM uses a multi-process serving architecture with separate worker processes for each tensor-parallel rank, and that EAGLE-3 integration requires coordination between the draft model forward pass and the target model's hidden state extraction. The metrics endpoint (/metrics) provides counters for draft tokens and accepted tokens, enabling acceptance rate calculation.

The DeepSeek V3 / Kimi-K2.5 architecture: Specifically the use of Multi-Head Latent Attention (MLA), which compresses key-value cache into a latent space. This is the architectural feature that likely causes the integration failure, as standard EAGLE-3 implementations may assume standard attention hidden state shapes.

The hardware context: 8x NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 compute capability), which required custom kernel compilation and limited software compatibility. The assistant had previously built sgl-kernel for SM120 (a 48-minute compilation) and was running vLLM specifically because it had been verified to work on this hardware.

Output Knowledge Created

This message produced several critical pieces of knowledge:

1. The vLLM EAGLE-3 integration for MLA models is broken. The identical 15% acceptance rate across two different drafters (one custom-trained, one pre-trained and known to work on SGLang) conclusively demonstrates that the problem is not training quality but integration correctness. This is a valuable finding for the broader ML community — anyone attempting to use EAGLE-3 with DeepSeek V3 or similar MLA-based models on vLLM should expect failure.

2. The training pipeline was validated. While the end-to-end speedup was negative, the training pipeline itself was successful: the custom-trained drafter performed identically to the pre-trained baseline, confirming that the training data, hidden state extraction, and fine-tuning procedure were correct. The pipeline can be reused once the integration layer is fixed.

3. A clear pivot direction was established. The identical failure of both drafters on vLLM, combined with the research showing SGLang's tested EAGLE-3 support for Kimi-K2 (see [msg 3074]), provided a clear next step: abandon vLLM for speculative decoding and pursue SGLang. The user explicitly endorsed this direction in the following message ([msg 3093]): "Get the models up on SGLang."

4. A diagnostic methodology was demonstrated. The controlled experiment — testing the pre-trained baseline under identical conditions to isolate the root cause — serves as a model for debugging complex ML system failures. The assistant could have spent hours retraining with different parameters; instead, a single 30-minute benchmark run provided definitive evidence.

The Thinking Process

The reasoning visible in this message and its surrounding context reveals a systematic, hypothesis-driven approach. After the custom drafter showed 0.66x throughput ([msg 3081]), the assistant immediately checked the metrics endpoint to compute acceptance rate ([msg 3083]). Finding 15%, the assistant generated three hypotheses:

Training data quality (too few samples, distribution mismatch)
Finetune inadequacy (10K samples insufficient to adapt from K2 to K2.5)
Vocab mapping differences But rather than pursuing any of these, the assistant designed the most efficient falsification experiment: test the pre-trained baseline. The reasoning was: if the pre-trained drafter (which was explicitly tested with SGLang and achieves 1.8x speedup there) also fails on vLLM, then the problem is vLLM, not training. The assistant then killed the server, cleared GPU memory, launched a new instance with the AQ-MedAI drafter path, and waited 30 minutes for the 547 GB model to load ([msg 3084]-[msg 3086]). The benchmark in this message is the result of that wait. After this message, the assistant immediately checked the metrics again ([msg 3088]), finding 1119 accepted out of 7190 draft tokens — 15.6% acceptance, essentially identical to the custom drafter's 15.0%. The conclusion was then stated explicitly ([msg 3089]): "The problem is NOT our training — the pre-existing AQ-MedAI drafter performs identically poorly. The problem is the EAGLE-3 + vLLM + DeepSeekV3/MLA integration itself." This message, then, is the fulcrum on which the entire speculative decoding effort pivoted. Before it, the team had invested days in building a training pipeline and patching vLLM. After it, the direction shifted decisively toward SGLang. A single 47-second benchmark, comprising 2560 tokens across 5 prompts, was enough to invalidate an entire approach and redirect a multi-day engineering effort.