The 40 tok/s Problem: Diagnosing Speculative Decoding Performance in Production

In the middle of a sprawling coding session spanning model deployment, training pipeline optimization, and speculative decoding research, a single user message cuts through the noise with a precise performance observation and a fork-in-the-road decision:

We're at only 40tok/s, were around 70 before. We may want more MTP or vllm?

This message (index 8187) is deceptively short. It contains a measurement, a historical baseline, a diagnosis, and two proposed remedies. To understand its full weight, we must trace the thread of reasoning that led to it and the investigation it triggered.

The Context: Deploying Qwen3.6-27B with Stock MTP

The assistant had just finished deploying Qwen3.6-27B on the CT129 server — a machine with two NVIDIA RTX A6000 GPUs (48 GB each). The deployment used SGLang's built-in speculative decoding algorithm called NEXTN, which leverages the model's native MTP (Multi-Token Prediction) heads. The model itself has mtp_num_hidden_layers: 1 in its configuration, meaning it was pre-trained with a single MTP layer that can predict one additional token ahead of the main model.

The launch command was:

--speculative-algorithm NEXTN
--speculative-num-steps 1
--speculative-eagle-topk 1
--tensor-parallel-size 2
--mamba-scheduler-strategy extra_buffer

This is a conservative configuration: one speculative step, top-1 sampling for the draft, and the mamba scheduler set to handle the model's hybrid GDN (Gated Dilated Network) layers — a mamba-style architecture that SGLang's standard scheduler doesn't support natively.

The assistant reported the server was live with a 19K token context window and max_running_requests=1, noting that the MTP weights consumed 2.79 GB per GPU. It then asked: "Want me to adjust anything?"

The User's Observation: 40 vs 70 tok/s

The user's response reveals they had been testing the deployed server and found it underwhelming. At 40 tok/s, the throughput was roughly 57% of the 70 tok/s they recalled from a previous deployment. This is a significant gap — the difference between a responsive interactive experience and a sluggish one, especially for agentic coding workloads where the model generates hundreds or thousands of tokens per request.

The "70 before" reference is crucial. It tells us that the user has a clear performance baseline in mind, likely from an earlier deployment of the same model on the same hardware. This implies that the current 40 tok/s is not a hardware limitation but a configuration issue — something about the current deployment is suboptimal compared to what was done previously.

The Proposed Remedies: More MTP or vLLM?

The user proposes two distinct paths forward, each representing a different hypothesis about the root cause:

"More MTP" — This means increasing the number of speculative draft steps. The current deployment uses --speculative-num-steps 1, meaning the MTP head predicts one token ahead. If the acceptance rate is low (the model rejects the draft token and falls back to the main model's prediction), the speculative overhead might not be worth it. Increasing to 2 or 3 MTP steps could improve the acceptance length — but it also adds compute and memory overhead for the additional MTP heads. The user is hypothesizing that the current 1-step MTP is insufficient to realize the full benefit of speculative decoding.

"vLLM" — This means switching inference engines entirely, from SGLang to vLLM. The user is implicitly asking: is SGLang's speculative decoding implementation slower than vLLM's for this particular model and hardware combination? This is a non-trivial question. Both frameworks support MTP-style speculative decoding, but their implementations differ in scheduler design, CUDA graph capture, memory management, and GDN/mamba layer handling. The user suspects that vLLM might have a more optimized path for Qwen3.6-27B's hybrid architecture.

The Hidden Assumptions

The message rests on several assumptions worth examining:

1. The 70 tok/s baseline is recoverable. The user assumes the previous configuration that achieved 70 tok/s is still applicable. In reality, the assistant had deployed with a stripped-down configuration — no --mem-fraction-static, no --context-length, no --max-running-requests tuning. The previous deployment (found later in a systemd service file) used --context-length 131072, --mem-fraction-static 0.88, --max-running-requests 16, and --mamba-full-memory-ratio 0.5. These settings dramatically change memory allocation and throughput characteristics.
2. The bottleneck is speculative decoding. The user attributes the throughput gap to the speculative decoding algorithm (too few MTP steps, or SGLang's implementation). But the real bottleneck might be something more mundane: the tiny context window (19K vs 131K), the single running request slot, or the missing memory fraction optimization.
3. More MTP steps will help. Increasing speculative steps is not guaranteed to improve throughput. Each additional MTP step adds compute cost (running the MTP head) and memory pressure (storing additional KV cache entries for draft tokens). If the acceptance rate per step is already low (~75% as the assistant later measured), adding more steps could actually reduce throughput by wasting compute on frequently-rejected drafts.
4. vLLM is a drop-in replacement. Switching inference engines is a major undertaking. The model loading, quantization, tensor parallelism, and speculative decoding configurations would all need to be re-tuned. The user's suggestion of "vLLM" as an alternative reflects a pragmatic willingness to swap tools, but it glosses over the engineering effort involved.

The Assistant's Investigation

The assistant's response (msg 8188-8190) is a masterclass in data-driven debugging. Rather than immediately acting on either proposal, it gathers evidence:

First, it runs a realistic coding prompt — "Write a Python function that implements a red-black tree" — and measures throughput directly from the server logs. The logs confirm the user's observation: gen throughput (token/s): 35.81, accept len: 1.75, accept rate: 0.75. The MTP head predicts one token and is accepted 75% of the time, yielding an effective 1.75× speedup over single-token decoding. But 35-40 tok/s is still far from 70.

Second, it searches for the previous deployment configuration. It finds a systemd service file (/etc/systemd/system/sglang-qwen.service) with dramatically different settings:

--context-length 131072
--max-running-requests 16
--mem-fraction-static 0.88
--mamba-full-memory-ratio 0.5
--reasoning-parser qwen3
--tool-call-parser qwen3_coder

This is the smoking gun. The previous deployment allocated 88% of GPU memory to the model (vs whatever default the current deployment used), allowed 16 concurrent requests, and supported 131K token context. The current deployment, launched without these flags, defaulted to conservative settings that crippled throughput.

The assistant also checks the vLLM version installed (0.1.dev16016+g3cfc8f8b7 — a development build) and finds it's present but not currently used.

What the Message Reveals About the User

This message is a window into the user's mental model and working style:

• They measure before they act. The user didn't just accept the assistant's report that the server was "up and responding." They tested it, measured throughput, and compared it to a known baseline.
• They think in terms of architectural levers. "More MTP or vllm" shows the user understands the speculative decoding landscape: you can either increase the number of draft steps within the current framework, or switch to a different framework entirely.
• They communicate efficiently. The message is 13 words, but it conveys a measurement, a comparison, a diagnosis, and two action items. There's no hand-holding, no explanation of why 40 is bad or why 70 is expected — the assistant is trusted to understand the implications.
• They have production experience. The 40 vs 70 tok/s comparison implies the user has run this exact model on this exact hardware before, which means they're not just experimenting — they're deploying and iterating on a real system.

The Broader Significance

This message sits at a critical juncture in the conversation. The assistant had just finished deploying the model and was asking for direction. The user's response reframes the task: it's not "is it running?" but "is it running well enough?" This shifts the session from deployment to performance tuning — a much more open-ended and challenging phase.

The two proposed remedies also reveal the user's strategic thinking. "More MTP" is a within-framework optimization (tweak a parameter in SGLang). "vLLM" is a cross-framework migration (swap the entire inference engine). By offering both, the user signals that they're open to either incremental improvement or a radical change, depending on what the investigation reveals.

In the end, the assistant's investigation shows that neither "more MTP" nor "vLLM" was the right answer — the real fix was restoring the previous configuration's memory and context settings. But the user couldn't have known that without the data the assistant gathered. The message was the catalyst that turned a "deployment done" handoff into a proper performance investigation, ultimately leading to a much better understanding of what makes this model run fast on these GPUs.