The 15% Wall: How a Single Todo Update Marked the Death of vLLM EAGLE-3 on Kimi-K2.5

Introduction

In the sprawling narrative of a multi-day effort to deploy speculative decoding for a 1-trillion-parameter Kimi-K2.5 model on 8x Blackwell GPUs, few messages carry as much weight as the one that quietly closes a chapter. Message [msg 3091] is, on its surface, nothing more than a todowrite tool call — a routine update to a structured todo list. The assistant marks three items as completed and one as in-progress, all in a few lines of JSON. But beneath this mundane format lies a pivotal moment: the formal acknowledgement that weeks of work building an EAGLE-3 training pipeline, patching vLLM source code, and running multi-hour inference jobs had culminated in a definitive failure. The acceptance rate was 15%. The throughput was 0.66x of baseline — worse than running without speculative decoding at all. This article examines that message in depth: why it was written, what decisions it crystallized, the assumptions that were tested and broken, and the knowledge it created for the next phase of the project.

The Message in Full

The subject message is a todowrite tool invocation that updates the assistant's persistent todo list. It reads:

[todowrite] {"todos":[{"content":"EAGLE-3 training pipeline: COMPLETE (10K samples, 5 epochs)","priority":"high","status":"completed"},{"content":"vLLM EAGLE-3 testing: DONE - 15% acceptance, 0.66x (WORSE than baseline)","priority":"high","status":"completed"},{"content":"AQ-MedAI baseline: also 15% acceptance ...

The message is truncated in the conversation record, but the visible portion contains three completed items and the beginning of a fourth (restoring the normal service). The full todo list rendered in the assistant's response shows:

Why This Message Was Written: The Reasoning and Motivation

To understand why this particular message exists, one must trace the arc of the preceding several hours of work. The assistant had invested enormous effort into building an EAGLE-3 speculative decoding pipeline from scratch. This involved:

  1. Generating 10,000 synthetic training samples by running the Kimi-K2.5 model through its own vLLM server, capturing both reasoning traces and final responses. This job ran for approximately 5.3 hours with zero errors.
  2. Extracting hidden states from the target model at 3,165 tokens per second, producing 828 GB of training data — the raw material needed to train an EAGLE-3 drafter that could predict the target model's next-token distribution.
  3. Fine-tuning the drafter from the AQ-MedAI/Kimi-K2-Instruct-eagle3 checkpoint for 5 epochs, completing in 2.6 hours.
  4. Patching vLLM with three separate monkey-patches to enable EAGLE-3 support for DeepSeek V3 architecture models: a model whitelist patch, an image token handling fix, and a SupportsEagle3 interface implementation on both the DeepseekV3 and KimiK25 model classes.
  5. Loading and testing the model, which took approximately 25 minutes per server start due to the 547 GB model size. When the first benchmark came back at 54.6 tok/s — compared to a baseline of 82.5 tok/s without speculation — the result was shocking. The speedup was 0.66x, meaning EAGLE-3 was actively hurting performance. The acceptance rate was a mere 15%, far below the 60-80% that EAGLE-3 typically achieves. The assistant then performed a critical diagnostic step: testing the pre-trained AQ-MedAI drafter (the one they had finetuned from) without any of their training modifications. This drafter achieved the same 15.6% acceptance rate and identical 0.66x throughput. This was the smoking gun. It proved that the problem was not the quality of the training data or the finetuning process — it was a fundamental incompatibility between vLLM's EAGLE-3 implementation and the DeepSeek V3 / Multi-head Latent Attention (MLA) architecture. The todo update message was written to formalize this diagnosis. It marks the completion of the vLLM EAGLE-3 investigation and signals that the approach has been exhausted. The message serves as a boundary: everything before this point was the vLLM EAGLE-3 effort; everything after will be the pivot to SGLang.

How Decisions Were Made

Several critical decisions are embedded in this message, though they appear only as status changes:

Decision 1: Accept the vLLM EAGLE-3 experiment as concluded. The assistant could have continued tweaking — trying different numbers of speculative tokens, adjusting batch sizes, or attempting additional patches. Instead, the controlled experiment (testing both trained and untrained drafters yielding identical results) provided sufficient evidence that the problem was architectural, not parametric. The decision to mark "vLLM EAGLE-3 testing: DONE" reflects this conclusion.

Decision 2: Attribute the failure to a vLLM integration bug, not training quality. This was the key insight from comparing the two drafters. If the trained drafter had performed poorly but the AQ-MedAI baseline had performed well, the diagnosis would have been a training data or methodology problem. But both performed identically, isolating the root cause to the vLLM EAGLE-3 → DeepSeek V3 interface. The todo item explicitly states "confirms vLLM integration bug" — a decision about causality.

Decision 3: Restore the production service. By killing the EAGLE-3 server and restarting the normal vLLM service (the INT4 model without speculation), the assistant implicitly decided that the 0.66x throughput degradation was unacceptable for production use. Better to run without speculation than to run with broken speculation.

Decision 4: Prepare for the SGLang pivot. Although not explicit in this message, the todo update clears the mental and project-management space for the next phase. The subsequent message ([msg 3092]) immediately lays out the SGLang alternative, showing that the assistant had already researched SGLang's EAGLE-3 support (in a parallel task at [msg 3074]) and was ready to proceed.

Assumptions Made and Broken

The EAGLE-3 effort rested on several assumptions, most of which were proven wrong:

Assumption 1: EAGLE-3 + vLLM would work for DeepSeek V3 because vLLM supports EAGLE-3. This was partially true — vLLM does support EAGLE-3, but only for specific model architectures (primarily Llama-family models). DeepSeek V3's MLA attention mechanism was never properly integrated, requiring the three monkey patches just to load. Even after loading, the hidden state extraction during decode was apparently misaligned.

Assumption 2: Finetuning from the AQ-MedAI checkpoint would adapt the drafter to Kimi-K2.5. This was correct in terms of the training pipeline working — the loss converged, the checkpoint saved correctly, and vLLM loaded it without errors. But the underlying assumption that the drafter's predictions would match the target model's distribution was broken by the vLLM integration issue.

Assumption 3: 10,000 synthetic samples would be sufficient for adaptation. This may still be true — we simply cannot evaluate it because the vLLM integration masked any training improvements. Both the trained and untrained drafters performed identically, suggesting the hidden states being fed to the drafter during inference were wrong regardless of the drafter's weights.

Assumption 4: The hidden state extraction during training matched the hidden state extraction during inference. This was the most subtle and dangerous assumption. During training, hidden states were extracted from the target model via a custom vLLM worker that intercepted the forward pass. During inference with EAGLE-3, vLLM's speculative decoding engine extracts hidden states differently — through the get_hidden_states() method on the model class. If these two extraction paths produce different tensors (different layers, different formats, different normalization), the drafter would receive inputs it was never trained on, explaining the ~15% acceptance rate (barely above random guessing for a vocabulary of 160K+ tokens).

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of speculative decoding and EAGLE-3 architecture. EAGLE-3 is a draft-model-based speculative decoding technique where a small transformer (the "drafter") predicts the target model's next tokens using intermediate hidden states from the target model's layers. The acceptance rate measures how often the target model accepts the drafter's predictions — 60-80% is typical for well-tuned systems.
  2. Knowledge of vLLM's model architecture support matrix. vLLM supports EAGLE-3 primarily through a whitelist of model classes that implement the SupportsEagle3 interface. DeepSeek V3 and Kimi-K2.5 were not on this whitelist, requiring source patches.
  3. Knowledge of the DeepSeek V3 / MLA attention mechanism. Multi-head Latent Attention uses a compressed latent representation for the KV cache, which may interact differently with hidden state extraction compared to standard multi-head attention.
  4. Knowledge of the project's infrastructure. The model is deployed on 8x NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 compute capability), the model is a 1T-parameter INT4 quantized Kimi-K2.5, and the baseline throughput without speculation is 82.5 tok/s.
  5. Knowledge of the AQ-MedAI checkpoint. This is a pre-trained EAGLE-3 drafter for Kimi-K2 (not K2.5) that served as the starting point for finetuning. Its identical poor performance was the key diagnostic signal.

Output Knowledge Created

This message, combined with its context, creates several pieces of actionable knowledge:

  1. vLLM EAGLE-3 is incompatible with DeepSeek V3 / MLA architectures. This is a documented finding that future practitioners can reference. The 15% acceptance rate is not a training quality issue but a framework integration issue.
  2. The EAGLE-3 training pipeline works correctly. The assistant built a complete pipeline — synthetic data generation, hidden state extraction, and finetuning — that produces valid drafters. The pipeline itself is reusable; only the inference-time integration needs to change.
  3. AQ-MedAI/Kimi-K2-Instruct-eagle3 is not a viable drafter for vLLM-based inference on Kimi-K2.5. Even the pre-trained checkpoint, which reportedly achieves 1.8x speedup on SGLang, yields 0.66x on vLLM. This strongly suggests the problem is vLLM-specific.
  4. The correct path forward is SGLang. The subsequent message explicitly states this, but the todo update sets the stage by clearing the vLLM effort from the active work queue.

The Thinking Process Visible in the Message

Though the message itself is just a structured data update, the thinking process is visible in the content of the todo items. Note the precise language used:

Conclusion

Message [msg 3091] is a masterclass in how a seemingly trivial operational update can carry enormous narrative weight. In three todo items and a truncated fourth, the assistant formally closes a chapter that spanned days of work, terabytes of data processing, and dozens of source code patches. The 15% acceptance rate is a wall — a hard boundary where the vLLM approach meets its architectural limits. But the message is not an ending; it is a transition. By clearing the vLLM EAGLE-3 effort from the todo list, the assistant creates space for what comes next: the pivot to SGLang, with its first-class EAGLE-3 support and tested compatibility with Kimi-K2 drafters. The 15% wall would not stand forever — but first, the assistant had to acknowledge that it existed.