The 15% Verdict: A Single Curl Command That Ended EAGLE-3 on vLLM

In the course of a sprawling machine learning engineering session spanning dozens of hours and hundreds of commands, the most consequential message can be the shortest. Message [msg 3088] is exactly that: a single curl command piped through grep, returning two floating-point numbers. Yet those two numbers — 7,190 and 1,119 — delivered a definitive verdict that ended a multi-day effort to integrate EAGLE-3 speculative decoding with the Kimi-K2.5 model on vLLM, forcing a complete pivot to SGLang.

The Message

The assistant executed the following command on the remote server:

ssh root@10.1.230.174 'curl -s http://localhost:8000/metrics 2>/dev/null | grep -E "spec_decode_num_(draft_tokens|accepted_tokens)_total{" | head -4'

And received the response:

vllm:spec_decode_num_draft_tokens_total{engine="0",model_name="/shared/kimi-k2.5-int4"} 7190.0
vllm:spec_decode_num_accepted_tokens_total{engine="0",model_name="/shared/kimi-k2.5-int4"} 1119.0

Why This Message Was Written: The Reasoning and Context

To understand why this seemingly trivial metrics query was written, one must appreciate the immense effort that preceded it. The assistant and user had spent the better part of a week building a complete EAGLE-3 training pipeline for Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model running on 8x NVIDIA Blackwell GPUs. The pipeline included:

• Generating 10,000 synthetic reasoning traces from the target model itself (~5.3 hours of inference)
• Extracting hidden states from the model's internal layers at 3,165 tok/s, producing 828 GB of training data
• Fine-tuning a drafter model from the AQ-MedAI K2 checkpoint over 5 epochs (~2.6 hours)
• Applying three separate monkey-patches to vLLM to enable EAGLE-3 support for DeepSeek V3 / Kimi-K2.5 architectures (model whitelist, image token handling, and the SupportsEagle3 interface) When the assistant finally got vLLM with EAGLE-3 to load and serve requests ([msg 3079]), the initial throughput benchmark was devastating: 54.6 tok/s vs a baseline of 82.5 tok/s — a 0.66x speedup, meaning speculative decoding was making performance worse, not better ([msg 3081]). This immediately raised the question: was the problem the trained drafter specifically, or was there a deeper integration issue? The assistant then tested the pre-trained AQ-MedAI K2 drafter — the baseline checkpoint from which the fine-tuning had started — and got essentially identical results: 54.4 tok/s, also 0.66x speedup ([msg 3087]). This was the critical clue: the problem wasn't the training quality or the 10K sample size. The pre-trained drafter, which had been explicitly tested by its authors and known to achieve ~1.8x speedup on K2, was also failing on K2.5. But throughput benchmarks alone don't reveal why. The assistant needed to isolate the root cause. vLLM exposes Prometheus-style metrics at the /metrics endpoint, including counters for draft tokens proposed and accepted. Message [msg 3088] is the assistant reaching for those metrics to compute the acceptance rate — the single most important diagnostic for speculative decoding.

The Input Knowledge Required

To interpret this message, one needs to understand several layers of context:

1. Speculative decoding mechanics: EAGLE-3 works by having a small "draft" model predict multiple future tokens cheaply, which the large target model then verifies in a single forward pass. The acceptance rate is the fraction of draft tokens that the target model accepts. Typical EAGLE-3 acceptance rates are 60–80%; below ~40%, the overhead of running the draft model outweighs the benefit.
2. vLLM's metrics architecture: The vLLM inference server exposes Prometheus-format metrics at the /metrics HTTP endpoint. The spec_decode_num_draft_tokens_total and spec_decode_num_accepted_tokens_total counters track cumulative totals across all requests since server startup.
3. The hardware and model constraints: The target model is Kimi-K2.5, a 1T-parameter MoE model using Multi-Head Latent Attention (MLA) — a memory-efficient attention mechanism. The draft model runs on the same 8 GPUs (tensor-parallel size 8), meaning it consumes GPU memory and compute that could otherwise be used for the target model. On PCIe-connected GPUs, the communication overhead is significant.
4. The experimental history: The assistant had already established that both the trained drafter and the pre-trained AQ-MedAI baseline produced identical throughput, ruling out training quality as the cause. This message was the final diagnostic step before concluding that the integration itself was fundamentally broken.

What the Numbers Revealed

The metrics returned two numbers:

• Draft tokens proposed: 7,190
• Draft tokens accepted: 1,119 The acceptance rate is 1,119 / 7,190 = 15.6%. This is catastrophically low — roughly one-quarter of the 60–80% that EAGLE-3 typically achieves. At this acceptance rate, the draft model is essentially generating random tokens that the target model rejects almost all the time, yet the system still pays the overhead of running the draft model's forward pass. The assistant had previously benchmarked the target model alone at 82.5 tok/s with 12.1 ms/tok latency ([msg 3081]). With EAGLE-3, each speculative step proposes 5 draft tokens (configured via num_speculative_tokens: 5), but only ~0.78 are accepted on average (15.6% × 5). The overhead of running the draft model on TP=8 GPUs — including the communication cost of gathering hidden states and scattering draft predictions — adds enough latency that the net throughput drops to 54.6 tok/s.

The Thinking Process Visible in This Message

This message reveals a methodical diagnostic approach. The assistant had already:

1. Benchmarked throughput and found 0.66x speedup ([msg 3081])
2. Ruled out training quality by testing the pre-trained baseline and getting identical results ([msg 3087])
3. Now reached for the acceptance rate to quantify exactly how poorly the draft model was performing The choice of the /metrics endpoint rather than computing acceptance from the benchmark logs is telling. The assistant could have instrumented the benchmark script to report acceptance rates, but instead reached for the production metrics endpoint — the same one used for monitoring deployed services. This shows an engineering mindset: use the observability infrastructure that already exists rather than adding custom instrumentation. The grep pattern is also precise: spec_decode_num_(draft_tokens|accepted_tokens)_total{ captures both counters while excluding related but irrelevant metrics like spec_decode_num_drafts_total (number of draft sequences, not tokens) and spec_decode_num_drafts_created (a gauge, not a counter). The head -4 ensures only the two desired lines are returned.

The Output Knowledge Created

This message produced a single, unambiguous data point: 15.6% acceptance rate. Combined with the earlier finding that both drafters produced identical throughput, this confirmed that the problem was architectural, not training-related.

The assistant's reasoning, visible in the subsequent messages, converged on the root cause: MLA attention hidden state extraction during decode. The EAGLE-3 draft model needs access to the target model's internal hidden states from specific layers to generate its predictions. With MLA attention, the hidden state representation differs from standard multi-head attention, and vLLM's integration apparently extracts states that don't align with what the drafter was trained on — or extracts them at the wrong point in the computation graph. This explains why even the pre-trained AQ-MedAI drafter (which achieves ~1.8x on K2) fails on K2.5: the hidden state extraction path is fundamentally broken for MLA-based models in vLLM's EAGLE-3 implementation.

The Consequences: A Pivot to SGLang

The 15.6% acceptance rate was the final piece of evidence. Within minutes of this message, the assistant killed the vLLM server and began investigating SGLang as an alternative ([msg 3089]). The research had already been done in parallel ([msg 3074]), revealing that SGLang has first-class EAGLE-3 support with explicit testing on Kimi-K2 drafters. The pivot was swift: build sgl-kernel for SM120 (48 minutes), install SGLang, and attempt to load the model.

The SGLang attempt would encounter its own challenges — the server deadlocked after weight loading on SM120 GPUs — but the vLLM EAGLE-3 path was definitively closed. Message [msg 3088] was the moment the assistant stopped debugging vLLM's EAGLE-3 integration and accepted that a different approach was needed.

Assumptions and Potential Mistakes

The assistant made several assumptions in interpreting these metrics:

1. The metrics counters are accurate and cumulative: This is a reasonable assumption — Prometheus counters are designed for this purpose — but there's always a risk of counter resets or race conditions in multiprocess servers.
2. The acceptance rate is stable across request types: The benchmark used 5 diverse prompts with 512 tokens each. The 15.6% rate might vary with prompt length, temperature, or other sampling parameters, but the consistency across 5 prompts in the throughput test suggests it's representative.
3. The drafter's failure is due to vLLM integration, not model architecture: This was the correct inference, but it's worth noting that the AQ-MedAI drafter was trained for K2, not K2.5. While the architectures are similar, K2.5 may have subtle differences in its internal representations. The fact that the trained drafter (fine-tuned on K2.5 data) performed identically to the baseline strongly supports the integration hypothesis. One potential mistake is that the assistant didn't test with num_speculative_tokens=1 to isolate the acceptance rate from the overhead of multi-token speculation. With 1 speculative token, even a 15% acceptance rate would add at most 15% overhead (one rejected draft token per step), which might have been tolerable. However, the assistant correctly judged that a 15% acceptance rate is fundamentally broken — no amount of tuning num_speculative_tokens can fix a drafter that predicts tokens the target model almost never accepts.

Conclusion

Message [msg 3088] is a masterclass in diagnostic efficiency. In a single curl command, the assistant extracted the one number that explained a multi-day debugging effort: the acceptance rate. Two floating-point values — 7,190 and 1,119 — told the story of a broken integration, not a broken training pipeline. The message itself is barely a line of bash, but it represents the culmination of hours of patching, loading, benchmarking, and reasoning. It is the moment when the assistant stopped trying to fix vLLM's EAGLE-3 support and accepted that the path forward lay elsewhere.