The N-Gram Gamble: When a Training-Free Shortcut Falls Short in Speculative Decoding

Introduction

In the high-stakes world of large language model inference optimization, every millisecond counts. When a team spends days training custom draft models for speculative decoding, only to watch them plateau at 38% accuracy, the temptation to reach for a simpler, training-free alternative is overwhelming. This is precisely the situation captured in message 5023 of this coding session, where the assistant launches an n-gram speculation server for the Kimi-K2.5 model and benchmarks its performance. The message represents a critical inflection point in a broader optimization journey—a moment when a promising shortcut is tested against reality and found wanting.

The Context: A Cascade of Failed Approaches

To understand message 5023, one must appreciate the long chain of failures that preceded it. The session began with a bold goal: improve Kimi-K2.5 inference throughput using speculative decoding. The team had already trained a from-scratch EAGLE-3 draft model on 37K samples, achieving a respectable 74.7% validation accuracy and ~2.0 accept length. But they wanted more—the AQ-MedAI team had achieved 3.2-3.5 accept length by training on 1.4M samples, and the gap seemed bridgeable.

The first attempt was a direct probe of the AQ-MedAI K2 EAGLE-3 drafter (Phase 0). This confirmed architectural compatibility but revealed misaligned hidden state representations, yielding only ~1.5 accept length and 52 tok/s. Phase 1 attempted to fine-tune the K2 drafter on K2.5 data. Initially, the loss was random (~18-20), which the assistant diagnosed as a critical vocab mapping mismatch—only 252 out of 32,000 draft-to-target token positions matched between the AQ-MedAI and K2.5 mappings. Fixing this dropped the loss to ~9 and improved accuracy to ~24%. But the fine-tuning plateaued at ~38% accuracy, converging slower than the from-scratch model (which reached 75% by epoch 5). The K2 weights were a poor initialization for K2.5, and the approach was abandoned.

At this point, the user asked a pivotal question in message 5010: "Do we have a simpler multi token predictors that could be used for now?" This query redirected the entire optimization strategy away from data-centric improvements (more training data, fine-tuning) toward simpler, training-free alternatives.

The Discovery: N-Gram Speculation in SGLang

The assistant investigated what SGLang supported out of the box and discovered the NGRAM speculative algorithm. Unlike EAGLE-3, which requires a trained draft model and hidden state extraction pipeline, n-gram speculation is training-free. It builds an n-gram cache from tokens generated so far (including prompt tokens) and uses n-gram matches to speculate future tokens. The key parameters were configurable: speculative_ngram_min_match_window_size, speculative_ngram_max_match_window_size, speculative_ngram_min_bfs_breadth, speculative_ngram_max_bfs_breadth, and speculative_ngram_match_type supporting both BFS and PROB strategies.

The assistant reasoned that for a reasoning model like K2.5, which generates thinking blocks with lots of repeated reasoning patterns, n-gram speculation could work well. The thinking blocks in reasoning models often contain repetitive patterns—self-correction phrases, step enumeration, and structural markers—that n-gram matching could exploit. And critically, it required no draft model, no hidden state capture, and no training.

However, the assistant also noted a crucial concern: the n-gram verify path uses ForwardMode.TARGET_VERIFY—the same extend/prefill path as EAGLE-3 verification. This meant the verify cost would be similar (~30ms), dominated by NCCL all-reduce operations across 8 GPUs. The question was whether n-gram matching could deliver a better accept length than the EAGLE-3 drafter's ~2.0, making the fixed verify cost worthwhile.

Message 5023: The Benchmark

Message 5023 captures the moment of truth. The assistant writes:

Server is ready. Let me ensure the benchmark script is there and run it. [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/eagle3-train/benchmark_eagle3.py root@10.1.230.174:/tmp/benchmark_eagle3.py 2>&1 && ssh root@10.1.230.174 '/root/ml-env/bin/python3 /tmp/benchmark_eagle3.py' 2>&1 Benchmarking http://localhost:8000 max_tokens=2048, num_runs=5, warmup=2

>

Warmup 1/2 done (41 tokens) Warmup 2/2 done (41 tokens)

>

Run 1/5: 2048 tokens in 51.74s = 39.6 tok/s (prompt: 32 toks) Run 2/5: 2048 tokens in 48.15s = 42.5 tok/s (prompt: 31 toks) Run 3/5: 2046 tokens in 57.19s = 35.8 tok/s (prompt: 26 toks) Run 4/5: 1708 tokens in 38.57s = 44.3 tok/s (prompt: 28 toks) Run 5/5: 2048 tokens in 48.34s = 42.4 tok/s (prompt: 34 toks)

>

========== Results ========== Overall ...

The benchmark script, originally written for EAGLE-3 evaluation, was repurposed to test the n-gram speculation server. It performs 5 runs of 2048-token generation with short prompts (26-34 tokens), measuring throughput in tokens per second. The results range from 35.8 to 44.3 tok/s, with the output cut off before the final aggregate.

The message itself is deceptively simple—a bash command and its output. But it represents the culmination of a complex decision tree: the abandonment of fine-tuning, the exploration of SGLang's speculative algorithms, the launch of the n-gram server, and the long wait for it to become ready (23+ polling attempts in message 5022, indicating roughly 11.5 minutes of server startup time).

The Results: A Disappointing Reality

The follow-up message (5024) reveals the full picture: 41 tok/s—even worse than the EAGLE-3 drafter (60 tok/s) and far below the baseline (82 tok/s). The accept length was only ~1.3-1.5 with an accept rate of ~16-18%, worse than EAGLE-3's ~2.0. And the verify was actually more expensive because it was verifying 8 tree-structured tokens instead of 3 chain tokens.

This was a significant failure. The n-gram speculation, which required no training and seemed like an elegant shortcut, performed worse than both the existing EAGLE-3 drafter and the baseline without any speculation. The training-free alternative was not just unhelpful—it was actively detrimental to throughput.

Assumptions and Their Consequences

Several assumptions underpinned this experiment, and examining them reveals why the results were disappointing.

Assumption 1: N-gram matching would work well for reasoning models. The assistant assumed that K2.5's thinking blocks would contain repetitive patterns amenable to n-gram matching. While this is true for long generations within a single session, the benchmark used short, independent requests (2K tokens each, fresh context every time). The n-gram cache barely had anything to match against. As the user astutely observed in message 5027: "Doesn't ngram need a decent amount of data to start being good?"

Assumption 2: The verify cost would be comparable to EAGLE-3. The assistant noted that n-gram uses the same TARGET_VERIFY forward mode, implying similar cost. But the n-gram configuration used 8 draft tokens with tree-structured speculation, which is inherently more expensive to verify than EAGLE-3's 3-token chain. More tokens to verify means more compute and communication overhead.

Assumption 3: The benchmark script was adequate for testing n-gram. The assistant reused the EAGLE-3 benchmark script without modification. This script was designed to measure single-request throughput with short prompts—the worst-case scenario for n-gram speculation, which needs a warm cache to perform well.

Assumption 4: Training-free meant risk-free. The appeal of n-gram speculation was that it required no training investment. But the cost of "free" was hidden in the verify overhead—the same NCCL all-reduce bottleneck that plagued EAGLE-3 was still present, and with more tokens to verify, it was actually worse.

The Broader Significance

Message 5023 is a turning point in the session. The failure of n-gram speculation closed the second major optimization path (after K2 fine-tuning), leaving the team with a stark realization: the verify step itself was the bottleneck, not the quality of the draft model. This insight drove the pivot to system-level optimization—analyzing the NCCL all-reduce overhead, creating the eagle-fast-verify.md optimization plan, and implementing changes like FlashInfer allreduce fusion for SM120 Blackwell architecture and NCCL tuning parameter adjustments.

The message also reveals something about the assistant's decision-making process. The assistant correctly identified n-gram speculation as a viable alternative, verified SGLang's support for it, and launched it efficiently. But the assistant also made a subtle error in benchmarking methodology—testing n-gram with short, independent requests rather than long, sustained generations where the cache could build up. The user's follow-up question about n-gram needing data to start being good shows that the user had a deeper intuitive understanding of the technique's limitations.

Input and Output Knowledge

To understand message 5023, the reader needs knowledge of: speculative decoding fundamentals (draft models, verify steps, accept length), SGLang's server architecture and speculative algorithm options, n-gram cache mechanics, NCCL all-reduce communication patterns in multi-GPU inference, and the specific hardware configuration (8× RTX PRO 6000 Blackwell GPUs with PCIe interconnect).

The message creates several important outputs: a benchmarked throughput of ~41 tok/s for n-gram speculation on K2.5, confirmation that n-gram accept length (~1.3-1.5) is worse than EAGLE-3 (~2.0), and the critical insight that the verify overhead dominates regardless of the speculation method. This knowledge directly informs the subsequent pivot to system-level optimization.

Conclusion

Message 5023 captures a moment of pragmatic experimentation in a complex optimization journey. The n-gram speculation experiment was a rational response to the failure of K2 fine-tuning—a search for a simpler path forward. Its failure was not due to poor execution but to a mismatch between the benchmarking methodology and the technique's requirements, and to the fundamental reality that on PCIe-bound multi-GPU systems, the verify communication overhead dominates regardless of how the draft tokens are generated. The message stands as a testament to the iterative nature of systems optimization: each failure narrows the search space and reveals deeper truths about the bottleneck.