Closing the Loop: How Empirical Evidence Killed N-Gram Speculation for a 1T MoE Reasoning Model

"Now let me update the training doc with these empirical findings and mark tasks complete."

At first glance, message 2502 appears trivial — a single edit operation updating a markdown file. The assistant's words are unremarkable: a brief statement of intent followed by a file edit that succeeded. But this message is the culmination of a multi-hour experimental odyssey spanning over twenty messages, a 90-minute model reload, and the definitive falsification of a hypothesis about speculative decoding. It represents the moment when empirical evidence was formally codified into the project's knowledge base, transforming a hunch into documented fact.

The Long Road to This Moment

To understand why this message was written, one must trace the thread back to its origin. The conversation had reached an impasse: deep profiling of the Kimi-K2.5 INT4 model — a 1-trillion-parameter Mixture-of-Experts (MoE) model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs — had revealed that AllReduce communication was the dominant bottleneck, consuming 51.5% of decode time. With PCIe-based inter-GPU communication as the hard ceiling, the user and assistant began exploring speculative decoding as a software-only path to higher throughput.

The user's instruction in [msg 2479] was precise: try Option A (n-gram speculation), but first document what it would take to train a custom EAGLE-3 draft model (Option D). The assistant launched parallel efforts — researching the EAGLE-3 training pipeline via the speculators library while simultaneously preparing to test n-gram speculation on the live inference server.

The Experiment That Cost an Hour and a Half

The n-gram speculation test was not trivial. The Kimi-K2.5 INT4 model takes approximately 30 minutes to load into GPU memory, and enabling speculative decoding requires a full restart with the --speculative-config flag. The assistant wrote a benchmark script ([msg 2484]), collected baseline measurements (~74 tok/s average), created a new systemd service with n-gram configuration ([msg 2490]), stopped the production service, killed residual GPU processes, and started the n-gram variant.

Then came the wait. The assistant polled the server every 30 seconds ([msg 2497]) as minute after minute passed. What normally took 30 minutes stretched to 90 — the speculative decoding configuration added CUDA graph compilation overhead. Finally, the model came online, and the benchmark ran.

The results were unequivocal: n-gram speculation was slower than baseline across every prompt type. The coding prompt dropped from 62.6 tok/s to 46.4 tok/s (a 26% regression). The math prompt fell from 78.0 to 64.8 tok/s (17% slower). Even the best-case explanatory prompt managed only 72.5 tok/s versus 79.5 baseline — a 9% loss. The vLLM internal metrics told the same story: mean acceptance length of just 1.86–2.55 tokens out of 5 speculated, with per-position acceptance rates collapsing from 40–56% at position 1 to a dismal 4–13% at position 5.

The assistant immediately restored the original service ([msg 2500], [msg 2501]), but the experiment was far from wasted.

Why This Message Matters

Message 2502 is the act of knowledge capture. The assistant had just spent over an hour of wall-clock time — and significant cognitive effort in setup, benchmarking, and analysis — to learn something important: n-gram speculation is counterproductive for this specific model and hardware combination. The next-steps-eagle.md document, originally written in [msg 2482] as a plan for training a custom EAGLE-3 draft model, needed to be updated with this empirical finding.

The edit served multiple purposes:

Closing the experimental loop: The todo items created in [msg 2480] — "Write next-steps-eagle.md with detailed EAGLE-3 training plan" and "Test n-gram speculative decoding on current vLLM setup" — could now be marked complete. The hypothesis had been tested and falsified.
Preventing future wasted effort: Without this update, someone reading the training plan later might wonder, "Why not just use n-gram speculation? It's simpler." The document now contains the empirical evidence explaining why that path was tried and rejected.
Strengthening the case for EAGLE-3 training: The original research in [msg 2481] had predicted that n-gram speculation would perform poorly on reasoning models, citing MoE-Spec research. The experiment confirmed this prediction with real data. The document now transitions from "EAGLE-3 training is theoretically better" to "EAGLE-3 training is the only viable path, as n-gram speculation has been empirically ruled out."

Assumptions Tested and Corrected

The experiment exposed several assumptions that proved incorrect:

Assumption 1: N-gram speculation provides a net benefit even for reasoning models. This was the central hypothesis, and it was wrong. The reasoning model generates novel thinking chains — internal monologue tokens that are inherently unpredictable. N-gram matching, which relies on token sequences repeating from earlier in the prompt or generation, finds almost nothing to match against.

Assumption 2: The verification overhead is acceptable for MoE models. This proved to be the killer. Each speculated token requires a verification forward pass through the full model, including loading the top-K MoE experts. When most speculated tokens are rejected (70–83% rejection rate), the cost of loading experts for those tokens is pure overhead with no benefit. The MoE architecture, designed to be sparse during normal inference, becomes effectively denser during speculative verification.

Assumption 3: The 30-minute reload was worth the test. This assumption was actually correct, though it might not have seemed so during the 90-minute wait. The experiment provided definitive evidence that saved the team from pursuing a dead end. Without this test, the team might have spent days or weeks trying to tune n-gram parameters, or worse, deployed it to production and discovered the regression under load.

Input and Output Knowledge

Input knowledge required to understand this message includes: the architecture of MoE transformer models and why expert activation matters; the mechanics of speculative decoding (draft model generates candidates, target model verifies in parallel); the distinction between n-gram speculation (pattern-matching against existing text) and learned draft models (EAGLE-3, Medusa); the vLLM inference framework and its speculative decoding configuration; the characteristics of reasoning models (long chains of novel thinking tokens); and the PCIe-bound communication topology of the 8-GPU setup.

Output knowledge created by this message is the documented empirical finding that n-gram speculation degrades throughput by 9–26% on this specific model-hardware combination, with the root cause identified as low acceptance rates (17–31%) combined with high verification overhead from MoE expert loading. This knowledge directly shapes the project's future direction: custom EAGLE-3 training is now the only serious path forward.

The Thinking Process

The assistant's reasoning in this message is compact but reveals a clear cognitive structure. First, there is the recognition that experimental results must be captured — not just observed and acted upon, but written down where they can inform future decisions. Second, there is the understanding that the training document is the appropriate repository for this knowledge, since it directly addresses the question of which speculative decoding approach to pursue. Third, there is the operational awareness that tasks should be marked complete when their objectives have been met — the n-gram experiment was not successful in improving throughput, but it was successful as an investigation, and that success deserves documentation.

The edit itself is a single line — [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/next-steps-eagle.md — but it carries the weight of hours of work, the disappointment of a failed hypothesis, and the clarity of empirical truth. In the lifecycle of an engineering project, these moments of knowledge capture are often invisible but essential. They are what separate an ad-hoc experiment from institutional learning.

Conclusion

Message 2502 is a quiet but pivotal moment in the conversation. It represents the transition from exploration to decision, from hypothesis to evidence, from ephemeral observation to documented knowledge. The n-gram speculation experiment failed — but the project succeeded in learning something important, and that learning was preserved. The EAGLE-3 training pipeline would go on to be built and tested in subsequent messages, but it would proceed with the confidence that the simpler alternative had been tried, measured, and found wanting. In engineering, knowing what doesn't work is often as valuable as knowing what does.