When Speculative Decoding Backfires: The N-gram Experiment That Changed the Plan for Kimi-K2.5

Introduction

In the high-stakes world of deploying trillion-parameter Mixture-of-Experts (MoE) language models, every millisecond of inference latency is scrutinized, every bottleneck identified, and every optimization path explored. Message 2504 in this opencode session represents a critical inflection point: the moment when a promising optimization hypothesis—n-gram speculative decoding—was empirically tested, found to be actively harmful, and decisively abandoned in favor of a more sophisticated approach. This single message encapsulates the scientific method applied to ML engineering: form a hypothesis, test it rigorously, accept the results even when they contradict expectations, and pivot to a better-informed strategy.

The message is a summary delivered by the AI assistant to the user after an intensive multi-hour investigation. It distills two parallel threads of work into actionable conclusions: (1) an empirical benchmark campaign that definitively ruled out n-gram speculation for the Kimi-K2.5 INT4 model, and (2) a comprehensive research effort that produced a complete EAGLE-3 training pipeline documented in next-steps-eagle.md. Between these two findings, the message charts a clear path forward—one that involves renting specialized hardware rather than persisting with suboptimal approaches on the existing 8×RTX PRO 6000 setup.

The Context: A Bottleneck Hunt That Led to Speculative Decoding

To understand why this message was written, we must trace back through the preceding conversation. The team had spent days deploying and optimizing the Kimi-K2.5 INT4 model—a ~1 trillion parameter MoE model—on 8×NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. The previous profiling session (Segment 19) had delivered a sobering finding: AllReduce communication accounted for 51.5% of decode time, making it the dominant bottleneck. With NVLink unavailable on the PCIe-only topology, the team was stuck at approximately 82.5 tok/s single-stream throughput.

The natural next question was: can we improve throughput without touching the hardware bottleneck? Speculative decoding—a technique where a small "draft" model proposes tokens that a large "target" model verifies in parallel—offered a tantalizing software-only path to 2-4x speedups. The assistant launched parallel research agents to investigate speculative decoding fundamentals, framework support in vLLM and SGLang, candidate draft models, and training feasibility. The research surfaced three options: n-gram speculation (no training required, uses the target model's own history as a draft), using an existing EAGLE-3 draft model trained for K2 (not K2.5), or training a custom EAGLE-3 head specifically for K2.5.

The user directed the assistant to begin implementing the EAGLE-3 training pipeline while simultaneously testing n-gram speculation as a quick baseline. Message 2504 is the report back on both fronts.

The N-gram Experiment: A Hypothesis Tested and Refuted

The most striking content in message 2504 is the empirical demolition of n-gram speculative decoding for reasoning models. The assistant didn't just theorize about why it might fail—they ran the experiment, collected the data, and presented the results with surgical precision.

The test was methodical. First, the assistant established a baseline benchmark using a custom script (bench_spec_decode.py) that measured tokens-per-second across five diverse prompts: a coding task (binary search tree implementation), a math problem (fencing optimization), a structured output task (JSON generation), and an explanatory prompt (TCP/UDP comparison). The baseline measurements showed approximately 74 tok/s average, with individual prompts ranging from 62.6 tok/s (coding, with its long reasoning chain) to 79.5 tok/s (explanatory).

Then came the intervention. The assistant created a new systemd service (vllm-kimi-k25-int4-ngram.service) with the --speculative-config flag configured for n-gram speculation with 5 speculative tokens and a prompt lookup window of 4. This required stopping the running service, killing any lingering GPU processes, waiting for memory to free, and restarting—a process that took approximately 90 minutes due to the model's 30-minute load time plus additional CUDA graph compilation overhead from the speculative decoding configuration.

The results were unambiguous and damning:

| Prompt | Baseline (tok/s) | N-gram (tok/s) | Change | |--------|------------------|----------------|--------| | Coding (BST) | 62.6 | 46.4 | −26% | | Math (fencing) | 78.0 | 64.8 | −17% | | Structured (JSON) | 76.1 | 65.5 | −14% | | Explanatory (TCP/UDP) | 79.5 | 72.5 | −9% |

Every single prompt was slower with n-gram speculation enabled. The degradation ranged from 9% to 26%, with the most reasoning-heavy prompts suffering the most. This is the opposite of what speculative decoding is supposed to do.

The assistant didn't stop at throughput numbers. They dug into vLLM's internal speculative decoding metrics, which told an even more revealing story. The mean acceptance length was only 1.86–2.55 tokens out of 5 speculated. The per-position acceptance rates collapsed rapidly: Position 1 had a 40–56% chance of being accepted, but by Position 5, the acceptance rate had fallen to a dismal 4–13%. The overall draft acceptance rate was just 17–31%.

Why N-gram Fails for Reasoning Models: The MoE Verification Tax

The message's explanation for this failure is worth quoting directly: "This is exactly what the MoE papers predicted—the overhead of verifying speculative tokens (loading more experts) outweighs the benefit of occasional n-gram matches. For a reasoning model with novel/unpredictable thinking token sequences, n-gram matches are rare."

This insight is crucial and deserves deeper examination. N-gram speculation works by looking for repeated token sequences in the model's own generation history. It's effective for tasks with high repetition—code generation with boilerplate patterns, conversational responses with common phrases, or structured outputs with predictable formatting. But reasoning models like Kimi-K2.5 are designed to generate novel "thinking chains"—internal monologues where the model explores different approaches, catches its own errors, and refines its reasoning. These thinking tokens are, by design, unpredictable and non-repetitive. There are no n-gram patterns to exploit.

The verification step compounds the problem. When the target model verifies draft tokens, it must compute the full forward pass for each speculated position. For a dense transformer, this verification is relatively cheap. But for an MoE model like Kimi-K2.5, each verification step activates a subset of experts—and loading those experts from GPU memory, computing their outputs, and routing tokens through the gating network adds significant overhead. When most draft tokens are rejected (as they are here), this verification overhead is pure waste. The model is doing extra work for no benefit.

The assistant's conclusion is blunt and correct: "This is dead on arrival." It's a rare example in ML engineering where a hypothesis is tested, the data is clear, and the decision to abandon the approach is unambiguous. No equivocation, no "more research needed," no "maybe with different hyperparameters." Just a clean rejection based on evidence.

The EAGLE-3 Training Plan: A Documented Path Forward

The second major finding in message 2504 is the EAGLE-3 training plan, documented in next-steps-eagle.md. While the n-gram experiment was a closed case (tested and failed), the EAGLE-3 plan is an open proposal—a roadmap for the next phase of work.

The plan identifies three approaches:

Option A (Recommended): Train a custom EAGLE-3 head for K2.5. This is the "Baseten approach"—the same technique used by the company that popularized EAGLE-style speculative decoding for large models. It involves extracting hidden states from specific layers of the target model (layers 2, 30, and 58 for K2.5), using them as training data for a small transformer head (~1B parameters) that learns to predict the target model's next-token distribution.

Option B: Use the existing K2 EAGLE-3 drafter. The model AQ-MedAI/Kimi-K2-Instruct-eagle3 is a pre-trained EAGLE-3 draft model for the K2 architecture. However, K2.5 has a different architecture (multimodal wrapper, different layer structure), so the drafter would need adaptation and acceptance rates would be lower.

Option C: N-gram speculation (already tested and eliminated).

The plan includes detailed cost and time estimates for two hardware scenarios:

| Phase | On 8×RTX PRO 6000 | On rented B200 NVL8 | |-------|-------------------|---------------------| | Data generation (500K samples) | 3–7 days | 12–24 hours | | Training (10 epochs) | 6–12 hours | 8–16 hours | | Total | 4–8 days | ~1–2 days ($600–2000) |

The bottleneck is clearly data generation. Extracting hidden states from a 1T-parameter model requires running prefill on hundreds of thousands of prompts. On the PCIe-bound RTX PRO 6000 setup, this is painfully slow. On a B200 NVL8 with NVLink interconnects, the same work completes in a fraction of the time.

The plan also evaluates two training frameworks: Speculators v0.3.0 (from the vLLM project) and SpecForge v0.2 (from the SGLang project). Both support MoE verifiers, but SpecForge has an intriguing advantage: it supports online training, where hidden states are generated on-the-fly during training rather than pre-computed and stored. This eliminates the disk I/O bottleneck entirely and could make the training feasible on the existing hardware.

The Recommendation: Rent, Don't Build

The message closes with a pragmatic recommendation: "If you want to pursue this, renting a B200/B300 NVL8 for 1-2 days is the most time-efficient path." This is a mature engineering judgment. The assistant has calculated that the cost of renting ($600–2000 for 1-2 days) is justified by the time savings (3–6 days saved on data generation alone). It's a classic build-vs-buy decision, applied to cloud compute.

The expected speedup from a successful EAGLE-3 deployment is 1.6–2.2x single-stream (130–180 tok/s from the 82.5 baseline), discounted from the usual 3–4x for dense models due to the same MoE expert activation overhead that killed n-gram speculation. Even with this discount, the improvement would be substantial.

What This Message Reveals About the Engineering Process

Message 2504 is a masterclass in structured decision-making under uncertainty. It demonstrates several principles that distinguish effective ML engineering from mere tinkering:

1. Test hypotheses before committing. Rather than assuming n-gram speculation would work (or spending weeks debating it theoretically), the assistant ran a controlled experiment with measurable outcomes. The 90-minute reload time was an investment that paid for itself by preventing wasted effort on a dead-end approach.

2. Measure what matters. The assistant didn't just check whether the model loaded with speculative decoding enabled. They measured throughput, token acceptance rates, per-position acceptance probabilities, and compared everything against a carefully established baseline.

3. Understand why things fail. The explanation for n-gram's failure goes beyond "it's slower" to "here's the mechanism: MoE verification overhead exceeds the benefit of rare n-gram matches in reasoning chains." This understanding prevents the team from revisiting the same dead end later with different parameters.

4. Document alternatives. The EAGLE-3 training plan isn't just a single recommendation—it's a decision tree with multiple options, each with clear trade-offs in time, cost, and expected performance.

5. Make concrete recommendations. The message ends with a clear ask: "Want me to try Option C next, or do you want to go straight to renting hardware?" This gives the user agency while providing a structured choice.

Conclusion

Message 2504 represents the culmination of a significant engineering investigation into speculative decoding for the Kimi-K2.5 INT4 model. It delivers a clear negative result (n-gram speculation is harmful for reasoning MoE models), a comprehensive positive artifact (the EAGLE-3 training plan and pipeline), and a pragmatic path forward (rent specialized hardware for the training). The message is notable for its empirical rigor, its clear reasoning, and its willingness to accept inconvenient results. In a field often driven by hype and wishful thinking, this message stands as a model of evidence-based engineering decision-making.

The original service is restarting as the message concludes—a symbolic return to the status quo, but one informed by hard-won knowledge. The team now knows that n-gram speculation is not the answer, that EAGLE-3 training is feasible but hardware-bound, and that the most efficient path involves a temporary cloud rental. These are not the answers anyone hoped for at the start of the investigation, but they are the answers supported by data—and that is what makes them valuable.