The Smoke Test That Exposed Everything: Deploying Qwen3.6-27B with MTP Speculative Decoding

A Moment of Triumph and Disappointment

Message <msg id=6853> arrives at a pivotal juncture in a long and grueling deployment session. After more than thirty messages of wrestling with NVIDIA driver installations, LXC container quirks, memory allocation puzzles, and process persistence issues, the assistant finally reports: "Yes, it's up! 'The server is fired up and ready to roll!'" The exclamation is telling—it carries the palpable relief of an engineer who has just conquered a stubborn infrastructure problem. But the message is not merely a victory lap; it is a smoke test, and the results it reveals are deeply troubling.

This single message encapsulates the gap between "the server is running" and "the model is working." The assistant had spent the preceding hour migrating the Qwen3.6-27B deployment from a decommissioned host (kpro6) to a new one (kpro5), battling through NVIDIA driver 580.126.09 installation, GPU unbinding from vfio-pci, LXC container configuration, and the delicate art of getting SGLang's MTP (Multi-Token Prediction) speculative decoding to initialize without crashing. Now, with the server accepting HTTP requests, the real test begins—and it fails spectacularly.

The Long Road to "It's Up"

To understand the weight of this moment, one must trace the path that led to it. The assistant had been migrating the Qwen3.6-27B model—a 27-billion-parameter BF16 model weighing 52GB—to a new host. The migration involved installing matching NVIDIA userspace libraries inside an LXC container, downloading the model weights, and configuring SGLang 0.5.9, which initially produced degenerate output due to incompatible GDN hybrid attention handling. Upgrading to SGLang 0.5.11 (as recommended by the model card) resolved the immediate quality issues, and the assistant achieved 73.5 tok/s single-request throughput with MTP speculation.

But then the deployment hit a wall. The server repeatedly crashed with RuntimeError: Not enough memory, even with --mem-fraction-static 0.88. The culprit was the model's GDN hybrid architecture—a mix of full attention layers and Gated DeltaNet (Mamba-like) linear attention layers. The linear attention layers require recurrent state memory on top of the KV cache for the full attention layers, and the default --mamba-full-memory-ratio 0.9 was consuming too much GPU memory. The assistant reduced --max-running-requests from 48 to 16 and --mamba-full-memory-ratio from 0.9 to 0.5, which finally allowed the model to load.

Then came the LXC container process persistence problem. When the assistant used nohup to launch the server over SSH, the process died as soon as the SSH session closed. The assistant diagnosed this as an LXC container issue—SSH sessions were being killed when the parent connection terminated. The solution was setsid, which detaches the process from the session's controlling terminal. With setsid, the server finally stayed alive.

The assistant then waited through the model loading sequence: checkpoint shards loading at 80%, then 87%, then CUDA graph capture beginning with only 3.18 GB of available memory. The CUDA graph capture is a critical optimization step where SGLang pre-records GPU operations for common batch sizes, and it can take several minutes. When the log showed "INFO: 127...." (Application startup complete), the assistant knew the server was accepting requests.

The user's simple query—"ready?"—prompted the assistant's triumphant response.

The Smoke Test: What the Message Actually Contains

The message consists of two parts: the assistant's verbal confirmation and a bash command that executes a curl request to the SGLang server's chat completions endpoint, followed by a Python script that parses and displays the response.

The curl request sends a straightforward prompt: "What is 2+2? Answer briefly." The sampling parameters are carefully chosen: temperature=1.0, top_p=0.95, top_k=20. These are the recommended parameters from the model card, intended to produce diverse, non-repetitive output. The assistant is deliberately avoiding the earlier mistake of using temperature=0.7 which had produced degenerate repetitive output in a previous test (see <msg id=6837>).

The Python parsing script extracts three key fields: reasoning_content, content, and usage statistics. The reasoning content is the model's internal chain-of-thought before producing its final answer, while content is the actual response to the user.

The results are devastating:

=== Reasoning ===
We are tasked with a simple arithmetic problem: 2 +  2  = ? We are tasked with a simple arithmetic problem: 2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 + 2 = 2.2 
=== Content ===
None
=== Tokens ===
{'prompt_tokens': 20, 'total_tokens': 520, 'completion_tokens': 500, 'prompt_tokens_details': None, 'reasoning_tokens': 0}
finish: length

The model generates 500 tokens of pure repetition—the same phrase "2 + 2 = 2." repeated endlessly—and produces no content at all. The finish_reason is "length," meaning the model hit the max_tokens=500 limit without generating a stop token. It would have continued repeating indefinitely.

What This Reveals: Diagnosing the Failure

This smoke test exposes a fundamental problem with the model deployment. The MTP speculative decoding was working perfectly—earlier logs showed an acceptance rate of 97-100% and throughput of 60 tok/s (see <msg id=6845>). The server infrastructure was solid. But the model itself was generating degenerate output.

The assistant's reasoning in the message is visible in the choice of sampling parameters. Having seen repetitive output with temperature=0.7 earlier, the assistant tried the model card's recommended parameters (temperature=1.0, top_p=0.95, top_k=20), which should maximize diversity and minimize repetition. The fact that the model still produced pure repetition rules out a simple sampling parameter issue.

Several possible root causes emerge:

  1. MTP speculative decoding corruption: The MTP draft model might be producing corrupted tokens that steer the target model into a repetition loop. Even though the acceptance rate was high, the draft tokens could be subtly wrong in a way that compounds over 500 tokens.
  2. Chat template mismatch: Earlier tests (see <msg id=6836>) showed the model generating double thinking tags, suggesting the chat template was adding a thinking tag and the model was also generating one. This template mismatch could confuse the model's generation.
  3. GDN hybrid attention bug: The Qwen3.6-27B model uses a hybrid architecture with both full attention and Gated DeltaNet (Mamba-style) linear attention layers. SGLang's handling of this hybrid architecture might have a bug that causes the model to lose track of its generation state after a few tokens.
  4. Mamba state management: The model's 48 linear attention layers maintain recurrent states that must be correctly managed during generation. If the Mamba state is not being properly reset or propagated, the model could fall into a repetition loop.
  5. CUDA graph capture issues: The CUDA graph was captured with only 3.18 GB of available memory, which is extremely tight. If the graph capture didn't cover all execution paths, the model might fall back to unoptimized code paths that produce incorrect results.

Assumptions and Their Consequences

The assistant made several assumptions in this message:

Assumption 1: The model card's recommended sampling parameters would fix the repetition issue. This was a reasonable assumption—model cards typically provide well-tested parameters. But the assumption proved incorrect, suggesting the problem is deeper than sampling configuration.

Assumption 2: The MTP speculative decoding was working correctly because the acceptance rate was high. The assistant had earlier noted "MTP was working at 100% acceptance rate and 62 tok/s throughput" (see <msg id=6846>). But a high acceptance rate only means the draft tokens were accepted by the target model's rejection sampler—it doesn't mean the generated text is coherent.

Assumption 3: The server was "up" in a meaningful sense. The assistant's enthusiastic "Yes, it's up!" conflates server availability with model functionality. The server was accepting HTTP requests and returning JSON responses, but the model was producing garbage.

Knowledge Required and Created

To fully understand this message, the reader needs knowledge of: speculative decoding and MTP (Multi-Token Prediction), the GDN hybrid architecture combining full attention with Gated DeltaNet linear attention layers, SGLang's server architecture and CUDA graph capture mechanism, LXC container networking and process management, and the Qwen3.6-27B model's chat template and reasoning format.

The message creates important diagnostic knowledge: the Qwen3.6-27B model, when deployed with SGLang 0.5.11 and MTP speculative decoding on 2× RTX A6000 GPUs, produces repetitive output that never reaches a coherent response. The model generates 500 tokens of pure repetition in the reasoning block and produces no content. This is a critical finding that will drive the next phase of debugging—the assistant's very next message (see <msg id=6854>) begins investigating whether mamba_ssm_dtype needs to be set explicitly or if there's an attention backend issue.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the smoke test itself. The curl request uses the chat completions API (not the raw generate endpoint), which means the server applies the chat template and reasoning parser. The Python parser extracts reasoning_content separately from content, showing the assistant is specifically checking whether the model's reasoning mechanism is working correctly.

The choice of max_tokens=500 is deliberate—earlier tests used smaller values (200 tokens) which might have masked the repetition pattern. By requesting 500 tokens, the assistant forces the model to reveal its long-range behavior.

The fact that reasoning_tokens is reported as 0 in the usage statistics, even though the reasoning content field contains text, is another clue. This suggests the server's reasoning parser is not correctly identifying the thinking tags, which could be related to the earlier double-tag issue.

Conclusion

Message <msg id=6853> is a study in contrasts—the joy of a successful deployment immediately undercut by the reality of a broken model. The assistant's "Yes, it's up!" is technically true but practically meaningless. The server is running, the GPUs are loaded, the CUDA graphs are captured, but the model cannot produce a coherent sentence. This message marks the transition from infrastructure debugging to model debugging, from "can we get the server to start" to "can we get the model to generate correct text." The smoke test has done its job: it has revealed a critical problem that would have remained hidden if the assistant had simply declared victory and moved on.