The Pivot from Research to Execution: Testing N-gram Speculation on a 1T-Parameter MoE Model

In the long arc of an engineering session, there is often a single message that marks the transition from analysis to action — the moment when all the research, planning, and deliberation crystallize into a concrete operational decision. Message 2483 in this opencode session is precisely such a pivot point. After an exhaustive research phase that produced a deep speculative decoding report ([msg 2477]) and a comprehensive EAGLE-3 training document ([msg 2482]), the assistant now turns to executing the simplest of the proposed options: testing n-gram speculation on the running vLLM instance serving the Kimi-K2.5 INT4 model.

The message itself is deceptively brief:

Good, doc is written. Now let me test n-gram speculation. First, I need to write a quick test script and figure out how to add the speculative config to our running vLLM. Since the model takes 30 minutes to load, let me check if vLLM supports adding spec decode to a running instance, or if we need to restart with the flag.

This is followed by a single bash command that inspects the running vLLM process's command-line arguments, revealing the current launch configuration:

/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --model /shared/kimi-k2.5-int4 --tensor-parallel-size 8 --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --trust-remote-code --max-model-len 131072 --gpu-memory-utilization 0.95 --port 8000 --disable-log-requests --enable-auto-tool-choice

The Context: Why This Message Exists

To understand why this message was written, we must trace back through the preceding conversation. The session had been engaged in a deep profiling campaign of the Kimi-K2.5 INT4 model running on 8× RTX PRO 6000 Blackwell GPUs. The profiling revealed that AllReduce communication was the dominant bottleneck, consuming 51.5% of decode time — a direct consequence of the PCIe-only interconnect topology. The user, recognizing that hardware-level improvements (like NVLink) were not available, pivoted to investigate speculative decoding as a software-only optimization path.

The assistant had responded by launching parallel research agents that investigated speculative decoding fundamentals, vLLM and SGLang framework support, candidate draft models, and training feasibility. The resulting research report ([msg 2477]) was thorough and sobering. It identified five possible approaches, from the trivial (n-gram speculation, "Option A") to the cutting-edge (custom MoE-Spec expert budgeting, "Option E"). The report's honest assessment was that speculative decoding is "primarily a latency optimization (single-stream / low-QPS), not a throughput optimization" — meaning it could improve single-request latency from ~82 tok/s to perhaps 150-200 tok/s, but would not meaningfully change the peak throughput plateau of ~1,536 tok/s at high concurrency.

The user's response ([msg 2479]) was pragmatic: "Try option A, before tho, write down what it would take to train Option D in next-steps-eagle.md." This directive set the stage for message 2483. The assistant was to execute two parallel workstreams: document the full EAGLE-3 training pipeline (the "hero" approach that Baseten used to achieve 340+ tok/s on B200 GPUs), and empirically test the simplest possible speculation method.

The Decision Point: Hot-Reload vs. Full Restart

Message 2483 represents the precise moment when the assistant transitions from the documentation workstream (writing next-steps-eagle.md) to the experimental workstream (testing n-gram speculation). The reasoning visible in the message reveals a critical operational concern: the model takes 30 minutes to load.

This 30-minute load time is not an abstract concern. The Kimi-K2.5 INT4 model is approximately 540 GB distributed across 8 GPUs, and loading it involves reading the sharded weights from disk, quantizing them, distributing them across the tensor-parallel ranks, and initializing the CUDA graphs and KV cache. Every restart costs half an hour of wall-clock time. The assistant's first instinct is therefore to ask: can we avoid this cost? Can the speculative decoding configuration be added to a running vLLM instance without restarting?

The question itself reveals an assumption: that vLLM might support hot-reconfiguration of its decoding strategy. This is not an unreasonable assumption — some inference servers do support live configuration changes. But for vLLM's speculative decoding, the answer is no. The speculative configuration (--speculative_config) is parsed at engine initialization time and baked into the model runner's forward pass structure. The SchedulerConfig, the ModelConfig, and the draft model worker are all constructed during engine startup. There is no API to dynamically attach a drafter to a running engine.

The bash command that follows is the assistant's systematic verification of this question. By inspecting /proc/<pid>/cmdline, it retrieves the exact command-line arguments of the running vLLM process. The output confirms the absence of any --speculative_config flag. The model is running in vanilla autoregressive mode.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of contextual knowledge:

First, the architecture of speculative decoding itself. N-gram speculation (also called prompt lookup decoding or n-gram matching) is the simplest form of speculative decoding: it scans the existing prompt and previously generated text for matching n-gram sequences, and uses those matches as draft tokens. It requires no training, no draft model, and no additional GPU memory. The trade-off is that it only works well for text that contains repetitive patterns — code with repeated boilerplate, structured outputs like JSON, or text that reuses phrases from the prompt. For open-ended reasoning or creative generation, n-gram speculation offers minimal benefit because the model is generating novel content that doesn't appear in the prompt.

Second, the specific hardware constraints of this deployment. The 8 RTX PRO 6000 Blackwell GPUs are connected via PCIe only, with no NVLink. This means AllReduce communication — which is required for every transformer layer's attention and MoE computations — must traverse the PCIe bus, which has approximately 10-20× less bandwidth than NVLink. The profiling had shown that AllReduce alone consumed 51.5% of decode time. Any speculative decoding scheme must account for the fact that verifying multiple draft tokens in parallel will activate more MoE experts, increasing the AllReduce burden.

Third, the research findings from the preceding messages. The MoE-Spec paper (published just four days before this message) had identified a critical problem for speculative decoding on MoE models: during verification of k draft tokens, the target model runs a forward pass on k+1 tokens simultaneously, and each token routes to 8 out of 384 experts. Collectively, k+1 tokens may activate 30-60 unique experts per layer, dramatically increasing memory pressure and communication overhead. The paper warned that this can cause speculative decoding to be slower than baseline — exactly the risk the assistant is about to test empirically.

The Output Knowledge Created

This message produces two concrete outputs. First, it confirms the current operational state of the inference server: the model is running with tensor parallelism 8, using the Kimi K2 reasoning parser, with auto-tool-call enabled, and without any speculative decoding configuration. This is a baseline snapshot that will be used to compare against the n-gram experiment.

Second, and more importantly, it establishes the decision that a full restart is required. The assistant now knows it must stop the vLLM service, modify the systemd unit file to add --speculative_config, and endure the 30-minute reload. This is not a trivial decision — it means the production service will be unavailable for the duration of the reload. The assistant's next actions ([msg 2484] onward) will involve writing a benchmark script to capture baseline performance before the restart, so that the before-and-after comparison is meaningful.

The Thinking Process

The assistant's reasoning in this message follows a clear pattern: verify before acting. Rather than assuming the speculative config can or cannot be hot-added, it checks the running process. Rather than blindly restarting, it considers the cost (30 minutes) and investigates alternatives first. This is characteristic of an operator who has been burned by long reload times before — the careful, deliberate approach of someone managing a production-adjacent system.

The message also reveals a subtle tension between the research findings and the operational reality. The research had predicted that n-gram speculation would be poorly suited for reasoning models like Kimi-K2.5, which generate novel thinking chains with little repetition. Yet the assistant proceeds with the test anyway, because the cost of trying is low (a config change and a restart) and the empirical data will be more informative than theoretical predictions. This is the engineering mindset: test everything, trust nothing until you have measurements.

What Comes Next

The messages immediately following this one ([msg 2484] through [msg 2489]) show the assistant executing the plan: writing a benchmark script, running baseline measurements, discovering that the reasoning model's output format (with reasoning_content consuming most tokens) requires adjustments to the benchmark, and finally preparing to restart vLLM with n-gram speculation enabled. The results of that experiment — which would ultimately confirm that n-gram speculation is 9-26% slower than baseline due to MoE expert activation overhead — are still in the future at the moment of this message.

In retrospect, this message captures a moment of optimism before the empirical reality sets in. The assistant has done the research, documented the training pipeline, and is about to test the simplest option. The 30-minute reload that follows will produce disappointing numbers, but those numbers will validate the MoE-Spec paper's predictions and steer the project toward the more ambitious EAGLE-3 training path. Message 2483 is the hinge point — the moment when analysis ends and experiment begins.