The Production Handoff: Deploying Qwen3.5-397B-A17B-NVFP4 with Systemd

Introduction

In the life cycle of any machine learning deployment, there comes a critical moment when experimental benchmarking gives way to hardened production serving. Message 5828 in this opencode session captures exactly that transition — the assistant creates a systemd service file for the newly downloaded nvidia/Qwen3.5-397B-A17B-NVFP4 model, codifying dozens of prior decisions about CUDA stack versions, NCCL tuning, SM120 compatibility patches, and model quantization into a single, repeatable unit file. This message is not merely a configuration file; it is the crystallization of an entire session's worth of debugging, optimization, and infrastructure work.

The message is deceptively simple on its surface: a cat heredoc piped over SSH to write a service file, followed by an echo "Service created". But the decisions embedded within it — the choice of tensor parallelism degree, the selection of attention backend, the omission of certain flags that were present in the previous Kimi-K2.5 service — reveal a rich tapestry of reasoning about model architecture, hardware constraints, and operational reliability.

Context: The Journey to This Point

To understand why message 5828 was written, one must appreciate the path that led to it. The session had spent considerable effort deploying the Kimi-K2.5 INT4 model with EAGLE-3 speculative decoding, culminating in a systemd service (sglang-kimi.service) that served as the production baseline. That deployment had required solving numerous challenges: upgrading to CUDA 13, patching SGLang's all-reduce utilities for SM120 (Blackwell) compute capability, enabling FlashInfer allreduce fusion and Torch symmetric memory, and fixing a crash in the dynamic speculation disable patch.

The pivot to Qwen3.5-397B-A17B-NVFP4 was motivated by the arrival of a newer, more efficient model. The model card described a 397-billion-parameter Mixture-of-Experts architecture with 512 experts (10 active per token), 60 layers, a hidden size of 4096, and — critically — only 2 KV heads (a 16:1 GQA ratio). The checkpoint was quantized to NVFP4 using NVIDIA's ModelOpt framework, a 4-bit floating-point format that promised significant memory savings. At 223 GB, it was less than half the size of the Kimi-K2.5 INT4 model (547 GB), which meant faster loading and more room for KV cache.

The assistant had already completed the model download (verified at 223 GB, all 19 files and 6 safetensor shards), built the latest SGLang main branch from source (installed as an editable package from /root/sglang-main/python), and applied the SM120 patches to the all-reduce utilities and Torch symmetric memory configuration. The stage was set for production deployment.

The Message Itself: Anatomy of a Systemd Service

Message 5828 is a single tool call — a bash command that writes a systemd unit file to the remote server. The assistant leads with a brief rationale: "Since this is a different model, I'll use tp=8 (we have 8 GPUs, model card says 4 is minimum but 8 will give more KV cache and throughput)." This one sentence encapsulates the key design decision of the entire service.

The tp=8 Decision

The choice of tensor parallelism degree (tp) is one of the most consequential decisions in deploying a large language model. It determines how the model's weights and activations are sharded across GPUs, directly impacting memory usage, communication overhead, and throughput. The model card recommended tp=4 as the minimum, meaning the model could fit across 4 GPUs. But the assistant chose tp=8, using all available GPUs.

The reasoning is explicit: "8 will give more KV cache and throughput." This is correct in principle — spreading the model across more GPUs reduces the memory footprint per GPU, leaving more room for KV cache (which scales with batch size and sequence length). However, it also increases all-reduce communication overhead, since each tensor parallelism operation must synchronize across 8 GPUs instead of 4. The assistant was implicitly betting that the Blackwell NVLink and NCCL optimizations (which had been extensively tuned earlier in the session) would keep communication costs manageable.

This decision also reflected a key assumption: that the model's architecture (60 layers, 512 experts, 10 experts per token) would benefit from the additional parallelism. With only 2 KV heads, the attention computation was already extremely memory-efficient, so the bottleneck was likely to be the MoE feed-forward computation. Distributing the 512 experts across 8 GPUs meant each GPU hosted 64 experts, reducing the expert computation per token.

Service Configuration Choices

The service file itself reveals several deliberate choices, both present and absent:

Present flags:

Assumptions Embedded in the Message

Message 5828 makes several assumptions, some justified and some that would soon be challenged:

  1. That --attention-backend flashinfer works on Blackwell for this model. This was the most consequential assumption. The assistant had used flashinfer successfully for the Kimi-K2.5 model, which used standard full attention. But the Qwen3.5 model is a "hybrid GDN" architecture — it interleaves full attention layers with linear attention (mamba-style) layers. On Blackwell GPUs, the flashinfer backend does not support this hybrid pattern. The server would crash with: AssertionError: triton or trtllm_mha or fa4 backend are the only supported backends on Blackwell GPUs for hybrid GDN models. This was a genuine surprise — the assistant had no way to know this restriction without attempting to start the server.
  2. That the default FP4 GEMM runner backend (flashinfer_cutlass) works on SM120. This assumption would also prove wrong. The server would start but produce NaN outputs, with finish_reason: "NaN happened". The FP4 kernels in flashinfer_cutlass were not compatible with Blackwell's SM120 architecture, requiring a switch to flashinfer_cudnn for the FP4 GEMM runner and flashinfer_cutlass for the MoE runner.
  3. That tp=8 is strictly better than tp=4. While more GPUs provide more KV cache memory, the communication overhead of all-reduce across 8 GPUs can sometimes negate the throughput benefits, especially for small batch sizes. The assistant implicitly assumed the NCCL optimizations (FlashInfer allreduce fusion, Torch symmetric memory) would keep overhead low.
  4. That the service structure from Kimi-K2.5 is a good template. This was a reasonable assumption — the systemd service pattern (environment variables, restart policy, timeouts, logging) was proven to work. The assistant reused the same structure with model-specific modifications.

The Thinking Process Visible in the Message

The assistant's reasoning is compact but revealing. The opening line — "Now let me create the service. Since this is a different model, I'll use tp=8" — shows the assistant actively comparing against the previous deployment. The phrase "we have 8 GPUs" grounds the decision in hardware reality. The mention of "model card says 4 is minimum but 8 will give more KV cache and throughput" demonstrates a cost-benefit analysis: the assistant understood the trade-off and made an explicit choice.

The choice of --reasoning-parser qwen3 and --tool-call-parser qwen3_coder reflects research done in the preceding messages. The assistant had checked the SGLang source code for Qwen3.5-specific parsers (finding none) and then fallen back to the general Qwen3 parsers. This was a pragmatic decision — use the closest available parser and verify later.

The inclusion of --enable-flashinfer-allreduce-fusion alongside --disable-custom-all-reduce shows the assistant carrying forward the optimization stack from the Kimi-K2.5 deployment. These two flags work together: FlashInfer's allreduce fusion replaces the custom all-reduce implementation, so the custom one must be disabled. This was a lesson learned from earlier debugging.

Input Knowledge Required

To fully understand message 5828, one needs knowledge of:

Output Knowledge Created

Message 5828 produces:

  1. A systemd service file at /etc/systemd/system/sglang-qwen.service that can be started, stopped, enabled, and monitored like any standard Linux service
  2. A production configuration that encodes all the model-specific and hardware-specific choices into a repeatable format
  3. A baseline for iteration — the service would be stopped, modified, and restarted multiple times in subsequent messages as the attention backend and FP4 GEMM runner issues were discovered and fixed The service file itself is a form of documentation. It captures the exact command-line arguments, environment variables, and operational parameters needed to serve this model. Anyone with access to the server can inspect the service file and understand exactly how the model is being served.

What Came Next: The Service in Action

The immediate aftermath of message 5828 revealed the assumptions that needed correction. When the assistant started the service (message 5829), the server began loading but quickly hit the hybrid GDN assertion error. The assistant then iterated: switching to --attention-backend triton, which allowed the server to start, but then discovering the NaN output issue caused by incompatible FP4 GEMM and MoE backends. The user contributed a critical gist (message 5852) documenting the exact flags needed for SM120: --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn.

This pattern — create a service, test it, discover issues, fix them — is exactly the production hardening workflow that message 5828 inaugurated. The service file was not a final artifact but a living document that would be edited as new information emerged.

Conclusion

Message 5828 represents the moment when a complex, multi-day debugging and optimization session crystallizes into production infrastructure. The systemd service file it creates is the tangible output of dozens of earlier decisions about CUDA versions, kernel patches, NCCL tuning, and model quantization. But it is also a hypothesis — a set of assumptions about what configuration will work for this model on this hardware. Some of those assumptions (tp=8, the environment variables, the restart policy) proved correct. Others (the attention backend, the FP4 GEMM runner defaults) required immediate correction.

This is the nature of deploying cutting-edge ML models: you cannot know the correct configuration until you try. The service file is both the culmination of past learning and the starting point for future iteration. Message 5828 captures that duality perfectly — a confident deployment that simultaneously acknowledges, through its very existence, that the real test is yet to come.