From Build Hell to Production: Deploying Qwen3.5-397B-A17B-NVFP4 on Blackwell SM120 GPUs

Introduction

Deploying a 397-billion-parameter Mixture-of-Experts (MoE) model on cutting-edge GPU hardware is never a straightforward task. When the hardware in question is NVIDIA's latest Blackwell architecture — specifically the RTX PRO 6000 Server Edition with compute capability SM120 — and the model uses the novel NVFP4 quantization format, the challenge multiplies. This article synthesizes a remarkable opencode coding session that took a machine from bare-metal Ubuntu 24.04 installation through a gauntlet of build failures, backend incompatibilities, and accuracy pitfalls, ultimately achieving a production deployment serving over 2100 tokens per second at high concurrency.

The session unfolded across two major phases: first, establishing a working ML environment with the correct driver stack, CUDA toolkit, and compiled dependencies; second, deploying the Qwen3.5-397B-A17B-NVFP4 model with SGLang and systematically testing backend configurations to find a combination that was both performant and correct. The narrative that emerges is one of methodical troubleshooting, deep codebase investigation, and a steadfast commitment to output accuracy over raw performance.

Phase One: Building the Foundation

The journey began with a clean Ubuntu 24.04 machine equipped with two NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant's first task was to install the latest NVIDIA drivers (590.48.01) and CUDA Toolkit 13.1, establishing the hardware-software interface that everything else would depend on. This foundational step succeeded cleanly, with nvidia-smi confirming both GPUs were operational and the CUDA runtime was correctly configured.

With the driver stack in place, the assistant created a Python virtual environment using uv — a modern, fast package manager — and installed PyTorch alongside the core dependencies. This established the base layer for what would become a complex dependency stack.

The Flash-Attention Ordeal

The installation of flash-attn became the session's first major bottleneck, illustrating the fragility of building CUDA extensions at scale. The system's CUDA 13.1 conflicted with PyTorch's CUDA 12.8 base, requiring the installation of a secondary CUDA 12.8 toolkit. Even then, the build process repeatedly exhausted system memory, forcing a collaborative trial-and-error reduction of parallel compilation jobs (MAX_JOBS) from 128 down to 20. This issue was only fully resolved after the machine was rebooted with an expanded 432GB of RAM.

Further dependency conflicts arose when vLLM downgraded PyTorch, breaking the compiled flash-attn binary. This necessitated a targeted rebuild against the correct PyTorch version (2.9.1). The final stabilized stack included PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1 — a carefully balanced set of versions that would serve as the foundation for the deployment phase.

Phase Two: The Nightly Upgrade and SM120 Kernel Build

The user then issued a directive that would reshape the entire stack: "update all to nightly." This triggered an aggressive upgrade to PyTorch 2.12.0.dev20260307+cu130, flashinfer 0.6.5, and the latest SGLang main branch. The machine was also upgraded from 2 to 8 GPUs, setting the stage for tensor-parallel deployment of the massive Qwen3.5 model.

The critical technical achievement of this phase was building sgl-kernel from source with SM120 (Blackwell) support. This required applying patches from the SGLang developer catid for CMake policy guards, CUDA 13 cccl include paths, and FA3 (FlashAttention-3) fallback. The kernel was compiled with TORCH_CUDA_ARCH_LIST=12.0a, enabling the FP4 kernels essential for NVFP4-quantized model inference. This build step was the linchpin of the entire deployment — without SM120-compatible kernels, the model would either fail to load or produce incorrect results.

Phase Three: The Great Backend Hunt

With the software stack in place, the assistant embarked on an exhaustive investigation of SGLang's GEMM backend options for SM120. This research, documented across dozens of messages in the session, involved systematically probing the SGLang codebase to understand which backends were available, which supported SM120 versus only SM100 (the datacenter Blackwell variant), and which SM120-specific optimizations existed.

Mapping the Backend Landscape

The assistant discovered that SGLang maintains a three-tier architecture detection system. The is_blackwell_supported() function checks for compute capability majors 10, 11, or 12 with CUDA 12.8+, providing a broad umbrella for all Blackwell variants. The is_sm100_supported() function narrows this to major version 10 only (B200/B100 datacenter GPUs), while is_sm120_supported() checks for major version 12 specifically (RTX PRO 6000 and similar desktop Blackwell GPUs). This distinction proved critical: many auto-selection paths in SGLang gate on is_sm100_supported(), which returns False on SM120 hardware, silently excluding SM120 from optimized backend selection.

The MoeRunnerBackend enum revealed ten backend options: auto, deep_gemm, triton, triton_kernel, flashinfer_trtllm, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, and marlin. The Fp4GemmRunnerBackend enum (for FP4-specific GEMM operations) was more restricted, offering only auto, flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm. Critically, the default for --fp4-gemm-backend was "flashinfer_cutlass" — not "auto" — suggesting this was the preferred or most mature FP4 GEMM implementation.

The SM100/SM120 Divide

A critical discovery emerged from tracing TensorRT-LLM (TRTLLM) references across the codebase. Comments in server_args.py explicitly stated: # trtllm-gen only supports SM100 and Note: trtllm_mha does not support SM120, which will fall back to flashinfer. Every reference to flashinfer_trtllm as an MoE runner backend was conditioned on SM100 — never SM120. This was not an oversight but a deliberate architectural decision: the TRTLLM-based backends were designed and optimized for datacenter Blackwell (SM100), not the desktop-class SM120.

Similarly, the FP8 GEMM backend selection in fp8_utils.py contained hardcoded rules: # SM120/121: CUTLASS only. # SM100/103: TRTLLM only. The MXFP4 quantization path also diverged between architectures, with SM120 using a non-persistent, non-TMA kernel path (StridedLayout) while SM100 used a persistent/TMA path.

The Working Configuration

After extensive testing, the assistant identified that flashinfer_cutlass for the MoE runner backend and flashinfer_cudnn for the FP4 GEMM backend produced correct output on SM120. In contrast, flashinfer_trtllm and flashinfer_cutedsl crashed or produced NaN/garbage. This narrowed the viable backend combinations considerably.

Phase Four: The Accuracy Crisis

Just as the deployment was coming together, the assistant identified a critical accuracy issue that could have silently degraded model quality. The checkpoint's default FP8 KV cache was being applied without proper scaling factors — a configuration that would work for short contexts but would produce increasingly inaccurate results as the context length grew, particularly for the agentic coding tasks the deployment was intended to support.

The assistant traced this issue to the KV cache configuration. The model checkpoint specified FP8 KV cache by default, but the scaling factors required for correct FP8 quantization were not being properly loaded. This meant that while the model would produce seemingly reasonable outputs for short prompts, the KV cache would accumulate errors over longer sequences, degrading the quality of multi-turn conversations and code generation tasks.

The fix was elegantly simple: force the KV cache to BF16 precision using the --kv-cache-dtype bf16 flag. This provided approximately 1.57 million tokens of high-precision cache — more than sufficient for the intended workload. The BF16 KV cache consumed more memory than FP8, but the 8× RTX PRO 6000 setup had ample VRAM to accommodate this trade-off. This decision exemplified the session's guiding philosophy: correctness over raw performance.

Phase Five: Production Deployment

The final production configuration was codified into a systemd service, ensuring the model would restart automatically after any system interruption. The performance numbers were impressive: approximately 172 tokens per second at single-request concurrency, scaling to over 2100 tokens per second aggregate at high concurrency (C=32). The built-in MTP (Multi-Token Prediction) speculative decoding, known as NEXTN, was successfully loaded but showed no throughput gain on synthetic benchmarks — confirming that the baseline performance was already optimal for the hardware configuration.

Lessons and Themes

Several overarching themes emerge from this session. First, the importance of systematic codebase investigation: the assistant's methodical approach to tracing backend selection logic — starting from server argument definitions, drilling into MoE runner implementations, examining quantization dispatch, and verifying hardware detection functions — provides a template for anyone deploying on novel hardware. Second, the fragility of CUDA extension builds: the flash-attn ordeal illustrates how memory constraints, version mismatches, and parallel compilation settings can derail even straightforward installations. Third, the gap between "it runs" and "it's correct": the FP8 KV cache issue would have gone unnoticed in a less thorough investigation, silently degrading model quality over time.

The session also highlights the fragmentation within NVIDIA's Blackwell architecture. SM100 (datacenter) and SM120 (desktop) are both "Blackwell" but have different kernel support matrices, different optimal backends, and different hardware capabilities. Inference frameworks like SGLang must navigate this complexity with careful architecture detection and conditional code paths — and users must verify which paths are actually exercised on their specific hardware.

Conclusion

This session transformed a bare-metal Ubuntu installation into a production-grade LLM serving infrastructure, navigating build failures, backend incompatibilities, and accuracy pitfalls along the way. The final deployment — Qwen3.5-397B-A17B-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs, serving over 2100 tokens per second with BF16 KV cache for guaranteed accuracy — represents a significant achievement in production ML engineering. The systematic approach to backend research, the commitment to correctness over performance hacks, and the meticulous attention to the hardware-software boundary all contributed to a deployment that is not just fast, but trustworthy.## The Subagent Research Mission: A Deep Codebase Archaeology

A significant portion of the session was devoted to a subagent research mission explicitly tasked with "Research SM120 GEMM backends." This subagent systematically explored the SGLang codebase across dozens of messages, each building on the discoveries of the previous ones. The investigation followed a clear methodological pattern: start with broad reconnaissance (listing directories, finding relevant files), then narrow to targeted searches (grepping for specific function names and architecture checks), and finally read the actual implementation code at precise line ranges.

The subagent discovered critical architectural details that would inform the deployment. The is_sm100_supported() function was found to check for device capability majors [10] only — meaning it would return False on SM120 hardware. The is_sm120_supported() function checked for major [12] with CUDA 12.8+. And is_blackwell_supported() covered majors [10, 11, 12], providing a broader umbrella. This three-tier system meant that code paths gated on is_sm100_supported() — such as the TRTLLM MoE runner selection — would silently exclude SM120 GPUs.

The subagent also traced the FP4 GEMM dispatch in modelopt_quant.py, discovering that the get_fp4_gemm_runner_backend() function returns a backend from the Fp4GemmRunnerBackend enum (auto, flashinfer_cudnn, flashinfer_cutlass, flashinfer_trtllm). The weight preparation pipeline was examined, revealing that the TRTLLM FP4 path requires a different weight layout handled by nvfp4_block_scale_interleave from FlashInfer. Critically, no SM120-specific gating was found in the FP4 dense GEMM dispatch — unlike the MoE auto-selection logic, the FP4 path was architecture-agnostic.

The Verification Loop: Closing the Gaps

A hallmark of this session was the assistant's insistence on verification. After forming a hypothesis about SM120 backend support, the assistant would pause and verify against ground truth. When a Python command failed because PyTorch wasn't in the system Python path, the assistant pivoted to nvidia-smi to confirm the GPU compute capability was indeed 12.0. When the auto-selection logic was suspected of excluding SM120, the assistant traced every call site of is_sm100_supported() in server_args.py to confirm the gap.

This verification loop — hypothesize, test, verify, refine — was executed repeatedly throughout the session. The assistant would declare "Now I have a complete picture" and then immediately issue two more verification commands to check for hidden assumptions. This intellectual rigor prevented the kind of premature conclusions that can lead to deployment failures.

The Weight Preparation Pipeline: A Final Frontier

One of the last investigations before the deployment was tracing the prepare_static_weights_for_trtllm_fp4_moe function in sglang/srt/layers/quantization/utils.py. This function, found at line 620, prepares static weights specifically for the TRTLLM FP4 MoE path. The assistant needed to understand whether this function contained SM120-specific logic or whether it was only called from SM100-specific code paths.

The investigation revealed that while the function exists and is well-defined, it is only invoked when the TRTLLM FP4 backend is selected — which, as established earlier, is not viable on SM120. This confirmed that the weight preparation pipeline was correctly scoped and would not interfere with the chosen SM120 configuration.

Final Configuration and Results

The production deployment settled on a configuration that balanced performance with correctness. The key flags were: