The Second Attempt: Diagnosing and Fixing an MTP Deployment Crash for Qwen3.5-122B on SGLang

[assistant] [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service root@10.1.230.174:/etc/systemd/system/sglang-qwen.service && ssh root@10.1.230.174 'systemctl daemon-reload && systemctl start sglang-qwen && echo "starting..."'

starting...

At first glance, this message looks unremarkable — a routine file copy followed by a systemd service restart. But in the context of the conversation, it represents a pivotal moment: the second attempt to deploy a 234 GB Qwen3.5-122B-A10B model on four Blackwell GPUs, after the first attempt crashed with a cryptic error. This message is the culmination of a rapid diagnosis-and-fix cycle that spanned barely two minutes of conversation time, yet required deep knowledge of SGLang's speculative decoding internals, the hybrid GDN architecture of Qwen3.5, and the interaction between MTP (Multi-Token Prediction) and attention scheduling.

The Crash That Preceded This Message

The story of this message begins with the first deployment attempt at [msg 6117]. The assistant had just downloaded the Qwen3.5-122B-A10B model (234 GB across 39 safetensor shards), prepared a systemd service file based on the earlier Qwen3.5-397B NVFP4 configuration, and started the server. The service file included MTP flags (--speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4) but omitted two critical parameters needed for the hybrid GDN architecture.

When the user asked "crashed?" at [msg 6120], the assistant investigated. The systemd status at [msg 6121] showed the service was in an auto-restart loop: Active: activating (auto-restart) (Result: exit-code). The journal logs at [msg 6122] revealed the root cause: "MTP with the hybrid GDN model needs --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1."

This error message is itself a piece of output knowledge — it tells us that SGLang's codebase contains explicit validation logic that detects when a hybrid GDN model (one combining linear attention layers with full attention layers, as Qwen3.5 does with 36 linear + 12 full attention layers) is used with MTP speculative decoding, and refuses to proceed without the correct scheduler configuration.

The Fix and Its Deployment

The assistant acted immediately. It stopped the restart loop at [msg 6123], read the service file at [msg 6124], and edited it at [msg 6125] to add the two missing parameters. Then came the target message: copying the fixed service file to the remote server and starting the service again.

The command itself is a compound operation: scp copies the local service file to /etc/systemd/system/sglang-qwen.service on the remote machine, then systemctl daemon-reload informs systemd that the service definition has changed, and systemctl start sglang-qwen launches the server. The && chaining ensures each step only proceeds if the previous one succeeded. The echo "starting..." at the end confirms the start command was issued — though it does not confirm the server actually initialized successfully.

Why the First Attempt Failed: Hybrid GDN and MTP

To understand why this fix was necessary, we need to examine the architecture of Qwen3.5-122B-A10B. The model uses a "hybrid GDN" (Gated Dense Network) architecture with 48 layers: 36 use linear (or "mamba-style") attention, and every 4th layer (12 total) uses full attention. This hybrid design is a key innovation in the Qwen3.5 family — it reduces the computational cost of full attention while maintaining long-context capabilities through periodic full-attention layers.

When MTP (Multi-Token Prediction, also called NEXTN in SGLang's terminology) is combined with this hybrid architecture, the speculative decoding process must handle the state transitions between linear and full attention layers correctly. The --mamba-scheduler-strategy extra_buffer flag tells SGLang to allocate additional buffer space for the Mamba-like states that linear attention layers maintain, ensuring that the speculative draft tokens can be processed without corrupting the KV cache or hidden states. The SGLANG_ENABLE_SPEC_V2=1 environment variable activates an alternative speculative decoding pathway (spec_v2) that was specifically designed to handle the complexities of hybrid architectures.

The first deployment attempt at [msg 6117] omitted these flags because the assistant was adapting the service file from the earlier Qwen3.5-397B NVFP4 configuration, which used FP4 quantization and a different attention backend. The assumption was that the MTP flags alone would suffice — an assumption that proved incorrect when SGLang's validation logic rejected the configuration.

Assumptions and Their Consequences

This message reveals several assumptions made by the assistant. First, the assistant assumed that the service file template from the 397B NVFP4 deployment would work for the 122B BF16 model with only minor modifications (changing model path, removing quantization flags, adding MTP flags). This was reasonable — both models share the same hybrid GDN architecture — but it overlooked the scheduler requirements that the 397B NVFP4 deployment didn't need because it wasn't using MTP.

Second, the assistant assumed that systemctl daemon-reload followed by systemctl start would be sufficient to deploy the fix. This is correct for systemd, but it doesn't address potential issues with GPU state — if the previous crash left the GPUs in an inconsistent state, a manual fuser -k /dev/nvidia* or GPU reset might be needed. The assistant had already run such cleanup at [msg 6116] before the first attempt, so this was likely not an issue.

Third, there is an implicit assumption that the fix is complete — that adding these two parameters resolves all issues and the server will start successfully. This is a high-confidence assumption given that the error message explicitly stated what was needed, but it doesn't account for other potential problems like CUDA version mismatches, NCCL configuration issues, or insufficient VRAM.

Input and Output Knowledge

The input knowledge required to understand this message includes: familiarity with systemd service management (the daemon-reload and start commands), understanding of SGLang's deployment workflow (service files, environment variables, command-line flags), knowledge of the Qwen3.5 hybrid GDN architecture (linear vs. full attention layers), and awareness of the MTP speculative decoding mechanism.

The output knowledge created by this message is the deployed fix — the corrected service file is now on the remote server, and the SGLang server is attempting to start with the proper scheduler configuration. Whether it succeeds or fails will be determined in subsequent messages. This message also implicitly documents the fix: anyone reading the conversation can see that --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1 are required for MTP with hybrid GDN models on this version of SGLang.

The Thinking Process

The assistant's thinking process across messages [msg 6120] through [msg 6126] is a textbook example of structured debugging. When the user reported a crash, the assistant didn't guess at the cause — it checked systemd status to confirm the crash mode, then examined the journal logs for the specific error. The error message was unusually helpful, telling the assistant exactly which flags were missing. The assistant then stopped the restart loop (preventing resource waste), read the current service file, edited it to add the missing flags, and redeployed.

This sequence demonstrates a key principle of ML infrastructure debugging: always check the logs first. The journalctl output at [msg 6122] contained the diagnostic information needed to fix the problem. Without it, the assistant might have spent time investigating GPU errors, CUDA compatibility, or model loading issues — all of which would have been red herrings.

The speed of the fix is also notable. From the user's "crashed?" query to the redeployment, only four messages elapsed (approximately 30 seconds of wall time). This rapid iteration is characteristic of experienced ML engineers who have internalized the common failure modes of model serving frameworks.

Conclusion

This message, while syntactically simple, captures a critical moment in the model deployment workflow: the transition from failure to attempted recovery. It demonstrates that deploying large language models on production infrastructure is rarely a one-shot process — it requires iterative debugging, log analysis, and domain-specific knowledge of the serving framework's internals. The assistant's methodical approach — diagnose from logs, identify the exact fix, apply it, redeploy — is a pattern that repeats throughout successful ML infrastructure work. Whether this second attempt succeeds or reveals further issues, the process itself is the valuable artifact: a record of how to diagnose and fix MTP deployment failures on hybrid GDN architectures with SGLang.