The Moment of Truth: Testing N-Gram Speculative Decoding on a 1T-Parameter MoE Reasoning Model

"Model is loaded with n-gram speculation enabled. That took about 90 minutes (longer than usual — the spec decode adds CUDA graph compilation overhead). Let me now run the benchmark"

This single message — message index 2498 in a sprawling, multi-session coding conversation — represents the culmination of hours of preparation, research, and a 90-minute wait. It is the moment of truth for an experiment that would determine the entire trajectory of the project. The assistant, having spent the preceding messages researching speculative decoding options, writing an EAGLE-3 training plan, creating benchmark infrastructure, and restarting the vLLM inference server with a new configuration, finally gets to see whether n-gram speculative decoding can improve throughput for the Kimi-K2.5 INT4 model running on 8× RTX PRO 6000 Blackwell GPUs.

The Message Itself

The full message reads:

[assistant] Model is loaded with n-gram speculation enabled. That took about 90 minutes (longer than usual — the spec decode adds CUDA graph compilation overhead). Let me now run the benchmark:
[bash] python3 bench_spec_decode.py --label ngram --max-tokens 4096 --concurrency 1 8
Benchmark label: ngram
API: http://[REDACTED]:8000/v1
Model: /shared/kimi-k2.5-int4

============================================================
SINGLE-STREAM BENCHMARK (max_tokens=4096)
============================================================

--- Prompt 1/5: Write a Python function that implements a binary search tree...
  Tokens: 1111, TTFT: 30.148s, TPOT: 21.6ms, TPS: 46.4

--- Prompt 2/5: A farmer has 120 meters of fencing. What dimensions should h...
  Tokens: 435, TTFT: 8.715s, TP...

The message is truncated — we see only the first two prompts of the five-prompt benchmark before the output cuts off. But even these first two data points tell a damning story. The baseline measurements from earlier in the conversation (msg 2488-2489) showed Prompt 1 achieving 62.6 tok/s with 16.0ms TPOT. The n-gram version shows 46.4 tok/s with 21.6ms TPOT — a 26% degradation. Prompt 2 went from 78.0 tok/s (12.8ms TPOT) to 64.8 tok/s — a 17% loss. The assistant does not yet have the full picture within this message, but the trend is already visible.

Why This Message Was Written

This message exists because of a chain of reasoning that began in the previous segment (Segment 19), where a comprehensive profiling campaign had identified AllReduce as the dominant bottleneck in the Kimi-K2.5 INT4 inference pipeline, consuming 51.5% of decode time. The user and assistant had explored various optimization paths — GEMM kernel tuning, Expert Parallelism, NCCL algorithm tuning — but all ran into fundamental hardware limitations of the PCIe-only 8-GPU topology. Speculative decoding emerged as a "software-only optimization path" that could potentially improve throughput without any hardware changes.

The user had given explicit instructions in msg 2479: "Try option A, before tho, write down what it would take to train Option D in next-steps-eagle.md." Option A was n-gram speculation — the simplest form of speculative decoding, requiring no draft model, no training, and minimal configuration changes. The assistant dutifully wrote the EAGLE-3 training plan document and then proceeded to test n-gram speculation, recognizing it as "a low-risk experiment that'll quickly tell us whether spec decode is viable on our SM120 hardware before investing in the heavier training path" (msg 2478).

The Decision-Making Process

The assistant's decision to test n-gram speculation reveals a careful, methodical approach to experimentation. Several key decisions are visible:

1. Restarting vLLM with a new service file. Rather than attempting to hot-add speculative decoding to a running instance (which vLLM does not support), the assistant created a separate systemd service file (vllm-kimi-k25-int4-ngram.service) with the --speculative-config flag. This was the correct approach, but it came with a cost: the model takes approximately 30 minutes to load, and the assistant noted that spec decode added CUDA graph compilation overhead, extending the wait to 90 minutes.

2. Choosing n-gram parameters. The assistant configured n-gram speculation with num_speculative_tokens: 5 and prompt_lookup_max: 4. These are relatively conservative parameters — the model would attempt to predict up to 5 tokens ahead by looking for matching n-grams in the prompt context, with a maximum lookup window of 4 tokens. The choice of 5 speculative tokens is standard for n-gram speculation in vLLM.

3. Creating a benchmark script. Before the experiment, the assistant wrote bench_spec_decode.py to measure tokens-per-second, time-to-first-token (TTFT), and tokens-per-output-token (TPOT) across multiple prompts with varying characteristics (coding, math, structured output, explanatory text). This ensured apples-to-apples comparison with the baseline.

4. Running the benchmark at concurrency levels 1 and 8. The assistant tested both single-stream (concurrency=1) and multi-stream (concurrency=8) scenarios, recognizing that speculative decoding might behave differently under different load patterns.

Assumptions Made

The assistant operated under several assumptions, some explicit and some implicit:

Assumption 1: N-gram speculation might help. The assistant acknowledged in msg 2478 that this was an experiment, not a sure bet. However, the very act of investing 90 minutes in reloading the model implies a belief that there was a non-trivial chance of improvement.

Assumption 2: The CUDA graph compilation overhead was a one-time cost. The assistant noted that the 90-minute load time was "longer than usual" and attributed it to spec decode adding CUDA graph compilation. This was correct — the overhead occurs at load time, not during inference.

Assumption 3: The benchmark script would work correctly with reasoning models. The assistant had to fix the script earlier (msg 2486-2488) because reasoning models return tokens in reasoning_content rather than content, and the initial script was counting zero tokens. The fix counted both reasoning and content tokens, giving accurate throughput measurements.

Assumption 4: The n-gram acceptance rate would be non-trivial. This was the critical assumption being tested. The assistant's research (from the parallel task agent in msg 2481) had suggested that n-gram speculation is "poorly suited for reasoning models (which generate novel thinking chains with little repetition)." Despite this warning, the assistant proceeded with the experiment because it was quick to set up and would provide empirical evidence.

Input Knowledge Required

To fully understand this message, one needs:

  1. The profiling context. The message builds directly on Segment 19's finding that AllReduce dominates decode time at 51.5%. Without this context, the pivot to speculative decoding seems arbitrary.
  2. Understanding of speculative decoding. The reader needs to know that speculative decoding works by having a cheap "draft" model predict several tokens, which are then verified by the target model in a single forward pass. N-gram speculation uses the input prompt itself as the draft model — it looks for matching token sequences in the context.
  3. Knowledge of MoE architecture overhead. The key insight — which the assistant's research had uncovered — is that for Mixture-of-Experts models, the verification step in speculative decoding activates more experts than a normal forward pass, creating additional overhead that can outweigh the benefits of speculation.
  4. The hardware topology. The 8 GPUs are connected only via PCIe, with no NVLink or NVSwitch. This makes AllReduce particularly expensive and limits the benefits of any technique that increases per-token computation.
  5. The model characteristics. Kimi-K2.5 is a 1T-parameter MoE reasoning model. Reasoning models generate long, novel "thinking" token sequences that are unlikely to repeat n-grams from the prompt.

Output Knowledge Created

This message produces several forms of knowledge:

1. Empirical benchmark data. The message captures the first two data points of the n-gram benchmark, showing 46.4 tok/s and 64.8 tok/s for the first two prompts. These numbers, when compared to the baseline of 62.6 tok/s and 78.0 tok/s, immediately suggest that n-gram speculation is degrading performance.

2. Confirmation of the 90-minute load time. The message documents that adding speculative decoding to vLLM increases model load time from ~30 minutes to ~90 minutes due to CUDA graph compilation. This is valuable operational knowledge for anyone deploying speculative decoding with large models.

3. A methodological template. The assistant's approach — baseline measurement, service modification, controlled restart, benchmark comparison — provides a reproducible methodology for testing inference optimizations.

4. The seed of a pivot. Although the full results are not visible within this message, the trend it reveals would lead the assistant to conclude (in msg 2499) that "n-gram is making things worse" and to pivot to the EAGLE-3 training path. This message is the turning point.

The Thinking Process Visible in the Message

The assistant's reasoning is compact but revealing. The opening sentence — "Model is loaded with n-gram speculation enabled. That took about 90 minutes (longer than usual — the spec decode adds CUDA graph compilation overhead)" — shows the assistant accounting for the unexpected delay and correctly attributing it to CUDA graph compilation. This demonstrates systems-level thinking: the assistant understands that speculative decoding requires vLLM to compile additional CUDA graphs for the verification forward pass, and that this compilation happens at load time.

The phrase "Let me now run the benchmark" is deceptively simple. It signals that the assistant is following the scientific method: first establish a baseline (done in msg 2488-2489), then introduce the experimental variable (n-gram speculation), then measure the effect. The assistant does not jump to conclusions based on the slow load time or the theoretical predictions from the research — it waits for empirical data.

The choice to run the benchmark with --concurrency 1 8 (testing both single-stream and 8-stream concurrency) shows the assistant thinking about different operational regimes. Single-stream tests measure raw decode throughput, while multi-stream tests measure how well the system handles concurrent requests. Speculative decoding could theoretically help more in the multi-stream case by reducing per-request latency, even if single-stream throughput degrades.

The Broader Significance

This message, though brief, sits at a critical juncture in the conversation. The assistant had spent the previous segment (Segment 19) exhaustively profiling and optimizing the Kimi-K2.5 INT4 deployment, hitting a wall with the PCIe AllReduce bottleneck. The pivot to speculative decoding represented a creative attempt to bypass that bottleneck through algorithmic means rather than hardware upgrades. The n-gram experiment was the first test of that hypothesis, and the results — visible even in the truncated output — were unequivocally negative.

The failure of n-gram speculation was not a dead end but a redirection. It confirmed the theoretical prediction from the MoE-Spec research that the assistant had cited earlier: for reasoning models generating novel token sequences, the overhead of verification outweighs the benefits of speculation. This empirical confirmation would lead the assistant to abandon n-gram speculation and invest effort in the more promising EAGLE-3 training path — a custom-trained draft model that could achieve much higher acceptance rates.

In the subsequent message (msg 2499), the assistant would examine the vLLM spec decode metrics and find a mean acceptance length of only 2.33 out of 5 speculated tokens, with acceptance rates dropping from 56% at position 1 to just 13% at position 5. The smoking gun data confirmed that n-gram speculation was fundamentally mismatched to the reasoning model's token distribution. The assistant then restored the original service and pivoted to building the EAGLE-3 training pipeline — the work that would occupy the remainder of Segment 20.

This message, then, is the hinge point. It is the moment when a hypothesis meets reality, when a 90-minute investment yields a clear answer, and when the conversation pivots from a dead-end optimization to a more ambitious and ultimately more rewarding path. It is a testament to the value of empirical testing in systems optimization: no amount of theoretical analysis could substitute for the concrete benchmark numbers that this message captures.