The Configuration Detective: How One SGLang Parameter Halved Inference Throughput

Introduction

In the course of deploying a large language model for production use, few moments are as disorienting as expecting 70 tokens per second and measuring 35. This was precisely the situation facing the assistant in message 8191 of an opencode coding session, where a freshly deployed Qwen3.6-27B model on a dual-RTX A6000 server was delivering half the performance of a previous deployment. What followed was a masterclass in configuration archaeology—a process of discovering, comparing, and understanding the subtle flags that separate a mediocre deployment from a well-tuned one. This message captures the exact moment of insight: the construction of a comparison table that revealed the root cause, and the decisive action to restore performance.

The Context: A Deployment Gone Wrong

The story begins with a simple user request: "Can you start Qwen3.6-27B on CT129 with stock MTP that we had deployed?" (msg 8168). The assistant dutifully checked the server, found SGLang 0.5.11 installed alongside the model files, and launched the server with what seemed like reasonable defaults. The model came up successfully, responding to API requests, but at a disappointing 35-40 tokens per second. The user's response was direct: "We're at only 40tok/s, were around 70 before. We may want more MTP or vllm?" (msg 8187).

This complaint triggered an investigation. The assistant first ran a benchmark request—a 512-token code generation prompt—and examined the server logs. The logs revealed troubling numbers: an acceptance length of only 1.75 (meaning the MTP speculative decoder was accepting roughly 75% of its single draft token), a context window limited to 19K tokens, and max_running_requests=1. The assistant hypothesized that the GDN mamba state cache was consuming too much memory, leaving insufficient room for the speculative decoding pipeline to operate efficiently.

But the real breakthrough came when the assistant checked the server's bash history and discovered an old systemd service file (/etc/systemd/system/sglang-qwen.service). This file contained the previous deployment configuration—the one that had achieved 70 tok/s. The assistant retrieved it in msg 8190, and in msg 8191—the subject of this article—laid out the comparison that explained everything.

The Message: A Table That Tells a Story

The message (msg 8191) opens with a carefully constructed comparison table:

| Setting | Old (70 tok/s) | Current (35 tok/s) | |---|---|---| | --speculative-num-steps | 3 | 1 | | --speculative-num-draft-tokens | 4 | 2 (auto) | | --mem-fraction-static | 0.88 | default (~0.8) | | --context-length | 131072 | 262144 (auto) | | --max-running-requests | 16 | 1 (auto) | | --mamba-full-memory-ratio | 0.5 | default | | --reasoning-parser | qwen3 | none | | --tool-call-parser | qwen3_coder | none |

The table is the centerpiece of the message—a concise, visual explanation of why performance had collapsed. Every row represents a choice the assistant had made implicitly (by accepting defaults) versus the explicit choices encoded in the previous systemd service file. The assistant then identifies the most impactful difference: "The big one is --speculative-num-steps 3 — the old config was running 3 MTP steps with 4 draft tokens, not just 1."

This single sentence captures the root cause. The assistant had launched the server with --speculative-num-steps 1 (the default), which meant the MTP head could only predict one token ahead per decoding step. The old configuration used --speculative-num-steps 3, allowing three tokens of lookahead, which dramatically improved the speculative decoding acceptance rate and overall throughput. The difference between 1-step and 3-step speculation is not linear—it compounds because each accepted draft token saves an entire forward pass through the base model.

The Reasoning Process: From Symptom to Cause

The assistant's thinking process in this message is worth examining in detail. It proceeds through several logical stages:

Stage 1: Performance measurement. The assistant had already benchmarked the server and observed 35-40 tok/s with an acceptance length of ~1.75. This established the symptom quantitatively.

Stage 2: Hypothesis formation. The initial hypothesis was that memory pressure from the GDN mamba state cache was constraining the speculative decoding pipeline. The logs showed "mamba usage: 0.43" (43% of mamba slots occupied) and a context window of only 19K tokens, which seemed tight.

Stage 3: Historical discovery. Rather than continuing to debug from first principles, the assistant checked for historical configuration files. The systemd service file was a goldmine—it contained the exact command-line arguments used in the previous, successful deployment.

Stage 4: Comparative analysis. The assistant constructed a side-by-side comparison of seven parameters. This is the critical reasoning step: instead of guessing which parameter mattered, the assistant enumerated all differences and then identified the most impactful one.

Stage 5: Action. Having identified the root cause, the assistant immediately acted: "Let me kill the current server and restart with the old config." The bash command kills the running SGLang processes (PID 47345 and its zombie children 47571/47572) to prepare for a fresh launch with the correct parameters.

Assumptions and Their Consequences

This message reveals several assumptions that shaped the assistant's behavior:

Assumption 1: Defaults are reasonable. The assistant assumed that SGLang's default values for speculative decoding parameters would provide acceptable performance. This was incorrect—the defaults are conservative, designed for compatibility rather than throughput. The old configuration had been explicitly tuned for this specific model and hardware combination.

Assumption 2: The previous configuration was lost. The assistant did not initially check for a systemd service file or other persistent configuration. It launched the server from scratch, effectively discarding the tuning work that had been done previously. This was a process failure—the assistant should have checked for existing configuration before deploying.

Assumption 3: MTP step count is the primary lever. While the assistant correctly identified --speculative-num-steps as the most impactful parameter, the comparison table shows seven differences. Some of these (like --mem-fraction-static 0.88 and --mamba-full-memory-ratio 0.5) affect memory allocation, which in turn affects how many tokens can be cached and how many concurrent requests can be served. The --max-running-requests 16 setting allows batch parallelism, which can significantly improve throughput under load. The assistant's focus on speculative-num-steps was correct for the immediate throughput problem, but the other parameters would matter for production robustness.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

Speculative decoding architecture. The MTP (Multi-Token Prediction) technique uses a lightweight "draft" model—in this case, a single additional transformer layer attached to the base Qwen3.6-27B model—to predict multiple future tokens in parallel. The base model then verifies these predictions in a single forward pass. The speculative-num-steps parameter controls how many tokens the draft model predicts ahead. With 3 steps, the model can accept up to 3 tokens per decoding iteration (though the actual acceptance length depends on the quality of predictions).

GDN (Gated Differential Network) layers. Qwen3.6-27B uses a hybrid architecture with both attention layers and GDN (mamba-style) layers. These require special memory management in SGLang, controlled by --mamba-scheduler-strategy and --mamba-full-memory-ratio. The GDN state cache consumes significant GPU memory, which competes with the KV cache for attention layers and the MTP model weights.

SGLang server architecture. Understanding the relationship between --mem-fraction-static, --context-length, and --max-running-requests is essential. These parameters jointly determine how much GPU memory is allocated to the KV cache, which directly limits the maximum context length and number of concurrent requests. The old config's --mem-fraction-static 0.88 allocates 88% of available GPU memory to the model and cache, while the default (~0.8) leaves more headroom but reduces capacity.

GPU memory constraints on 2× A6000. Each RTX A6000 has 48 GB of VRAM. With tensor parallelism across two GPUs, the 27B-parameter model in BF16 consumes approximately 27 GB for weights, leaving ~21 GB per GPU for KV cache, MTP weights, GDN state, and activations. The old configuration's choices reflect careful tuning to fit within these constraints.

Output Knowledge Created

This message produces several valuable outputs:

1. A verified performance baseline. The assistant now knows that the previous configuration achieved 70 tok/s, establishing a target to restore. This is not a theoretical maximum but a demonstrated capability on the same hardware and model.

2. A configuration diff. The comparison table serves as a permanent record of the differences between two deployment configurations. This is valuable documentation for future deployments and debugging.

3. A causal explanation. The message establishes that speculative-num-steps is the primary driver of throughput for this model on this hardware. This insight would guide future tuning efforts—if 3 steps give 70 tok/s, would 4 or 5 steps give even more? (The answer depends on the MTP model's predictive accuracy and the memory cost of additional steps.)

4. An action plan. The message concludes with a concrete next step: kill the current server and restart with the old configuration. This decisiveness is important—the assistant does not propose further analysis or debate, but acts immediately.

The Broader Significance

This message exemplifies a pattern that recurs throughout engineering work: the tension between starting fresh and recovering prior art. The assistant's initial impulse—to launch the server with defaults—is understandable. Defaults are supposed to be sensible, and the assistant had no reason to suspect that the previous deployment had been specially tuned. But the systemd service file represented accumulated knowledge: the hours of trial and error, the benchmarking runs, the edge-case discoveries that led to that specific set of flags.

The lesson is that configuration files are a form of institutional memory. The assistant's mistake was not in choosing wrong defaults, but in failing to look for existing configuration before acting. The recovery—finding the systemd file, comparing parameters, identifying the root cause—demonstrates the value of systematic investigation over guesswork.

Conclusion

Message 8191 is a compact but rich artifact of the engineering process. In a few paragraphs and a table, it captures the journey from symptom (35 tok/s) to cause (speculative-num-steps=1) to action (kill and restart). It demonstrates the importance of configuration archaeology, the power of side-by-side comparison, and the danger of assuming that defaults are optimal. For anyone deploying large language models in production, this message is a reminder that the difference between 35 tok/s and 70 tok/s is often not a hardware upgrade or a model change—it is a single command-line flag that encodes the lessons of previous tuning efforts.