Choosing the Right MoE Backend: The NVFP4 Marlin Decision on SM120 Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, the difference between a successful deployment and a silent failure often comes down to a single architectural decision. Message 12505 in this opencode session captures one such pivotal moment: the assistant must decide how to route the MoE (Mixture-of-Experts) computation for DeepSeek-V4-Flash when using NVIDIA's NVFP4 quantization on RTX PRO 6000 Blackwell GPUs. The upstream code, fresh from a freshly-patched pull request, wants to take one path. The hardware constraints dictate another. The assistant's reasoning and the resulting decision form a microcosm of the entire deployment challenge—where upstream abstractions meet hardware reality, and where theoretical compatibility gives way to practical necessity.

The Scene: Deploying NVFP4 on SM120

The message occurs at a critical juncture in a larger optimization campaign. The assistant has been working to deploy DeepSeek-V4-Flash—a massive Mixture-of-Experts model—on a machine with 8× RTX PRO 6000 Blackwell GPUs. These GPUs feature the SM120 architecture (compute capability sm_120), a consumer/professional variant of the Blackwell generation. The model has been quantized by NVIDIA using their NVFP4 format, a 4-bit floating-point quantization scheme applied to the expert weights in the MoE layers.

The path to this moment has been arduous. The assistant had already deployed the stock MXFP4 version of the model and benchmarked it thoroughly, only to discover that the MoE computation was running on CUDA cores (SIMT) rather than the tensor cores, yielding abysmal throughput of ~25 tok/s at concurrency 16—far below the user's target of ~1000 tok/s. The NVFP4 checkpoint promised a fix: by routing the MoE computation through tensor-core paths (Marlin W4A16 or native cutlass FP4 grouped GEMM), the assistant could potentially close the gap.

But NVFP4 detection required a specific upstream change: Pull Request #25820, which adds logic to read the moe_quant_algo field from the checkpoint's hf_quant_config.json and route the experts through the NVFP4 handler. The assistant had fetched and applied this PR as a patch, overcoming a shallow-clone merge conflict by using git apply --3way. The patch applied cleanly across 10 files, and the NVFP4 checkpoint (149 GB, 76 files) had finished downloading. Everything was in place for the launch.

The Core Decision: Rejecting the Auto-Detected Path

The subject message opens with the assistant's reasoning, and immediately we see the crux of the decision:

"The PR's hook forces flashinfer_trtllm_routed for NVFP4 — but that's the SM100 path (tested on B200/GB300), likely broken on sm_120."

This is the key insight. PR #25820, authored by NVIDIA engineer trevor-m, was designed and tested on SM100 hardware—the datacenter-grade Blackwell GPUs like B200 and GB300. The flashinfer_trtllm_routed backend leverages NVIDIA's TensorRT-LLM (trtllm) code generation, which produces highly optimized CUDA kernels for the specific GPU architecture. However, trtllm-gen is arch-gated: it generates code for SM100 and does not support SM120. On an RTX PRO 6000, using this backend would likely result in a runtime error, a silent fallback to slow paths, or undefined behavior.

The assistant recognizes this immediately and makes a conscious, explicit decision to override the auto-detected path:

"Per our routing analysis, on sm_120 I should explicitly use marlin (NVFP4→W4A16 on BF16 tensor cores — the Kimi-class kernel)."

This decision reflects deep architectural knowledge. Marlin is a well-established kernel for 4-bit weight quantization (W4A16) that runs on tensor cores. When NVFP4 weights are loaded, they can be dequantized on-the-fly to BF16 and fed through the Marlin kernel, achieving tensor-core execution on SM120. The assistant had previously validated this path with the Kimi K2.6 model, which used INT4 quantization with Marlin on the same hardware. The NVFP4→Marlin path is essentially the same mechanism: 4-bit weights, BF16 computation, tensor-core matmul.

The assistant also considers alternatives briefly in the reasoning:

"For SM120 with NVFP4, the viable tensor-core options are marlin (proven safe with FP16/BF16 cores), triton with native FP4 support, or flashinfer_trtllm_routed (SM100-only, risky). I'll test marlin first since it's the safest, established path."

This is a risk-calibrated decision. The assistant ranks options by confidence: marlin is proven (it worked for K2.6), Triton with native FP4 support is untested but potentially viable, and flashinfer_trtllm_routed is explicitly risky. The choice of marlin is conservative—it may not be the fastest possible path, but it is the most likely to work correctly on the first try.

The Launch Configuration: Engineering for Stability

The bash command in the message writes a launch script and starts the server. Every parameter in this script reflects prior learning from the optimization campaign:

exec /root/venv_sglang211/bin/python -m sglang.launch_server \
  --model-path /root/models/DeepSeek-V4-Flash-NVFP4 --trust-remote-code \
  --tp 4 --moe-runner-backend marlin \
  --mem-fraction-static 0.70 \
  --cuda-graph-max-bs 32 \
  --host 127.0.0.1 --port 30000
[2026-06-17 20:48:02] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'attention_factor'}

The model architecture is correctly identified, and the warning about attention_factor is cosmetic—a known quirk in the DeepSeek-V4 configuration that doesn't affect functionality.

The Thinking Process: What the Assistant Weighs

The assistant's reasoning in this message is notable for its concision and confidence. Unlike earlier messages where the assistant explored multiple hypotheses and tested them, here the reasoning is direct and decisive. This reflects accumulated knowledge from the preceding optimization campaign.

The assistant considers the interaction between the patched hook and the existing SM120 marlin forcing logic:

"I need to check whether the full patched hook logic still preserves the SM120 marlin forcing that was originally in server_args.py, or if the NVFP4 routing now takes precedence and breaks SM120 compatibility."

This is a critical architectural concern. The original SGLang codebase had logic in server_args.py's _handle_model_specific_adjustments to force marlin on SM120. But PR #25820 adds new routing logic in deepseek_v4_hook.py that detects NVFP4 and routes to flashinfer_trtllm_routed. The assistant must determine which logic takes precedence. Rather than tracing through the code to verify, the assistant makes a pragmatic decision: explicitly specify --moe-runner-backend marlin, which overrides both the original SM120 forcing and the new NVFP4 routing. This is the safest approach—explicit configuration trumps implicit detection.

The assistant also performs a rough memory calculation:

"The NVFP4 model is around 149GB total, so roughly 37GB per GPU with TP4."

This quick division (149 / 4 ≈ 37.25 GB) informs the memory fraction choice. With 48 GB per GPU, 70% leaves ~33.6 GB for weights, which is close to the 37 GB estimate. The assistant is aware that this is tight—hence the note about "workspace for dequantization plus CUDA graph overhead." The 0.70 fraction is a starting point; if the server OOMs, the assistant would need to reduce it further or increase TP to spread weights across more GPUs.

Assumptions and Their Risks

Every deployment decision rests on assumptions, and this message contains several that deserve scrutiny.

Assumption 1: Marlin works correctly with NVFP4 weights on SM120. The assistant assumes that NVFP4 quantization is compatible with the Marlin kernel path. NVFP4 uses a 4-bit floating-point format (E2M1), while Marlin was originally designed for INT4 weights. The dequantization step must convert NVFP4 to BF16 correctly. If the NVFP4 weights have a different layout or scaling factor than Marlin expects, the kernel could produce incorrect results or NaNs. The assistant's confidence comes from prior experience with Kimi K2.6 (INT4 + Marlin), but NVFP4 is a different quantization format. This assumption is reasonable but not guaranteed.

Assumption 2: flashinfer_trtllm_routed will fail on SM120. The assistant states this as "likely broken on sm_120" without having tested it. This is a well-informed assumption—trtllm-gen's SM100-only support is documented—but it remains an assumption. In principle, the backend could fall back gracefully or use a different kernel path. However, the risk of a runtime crash or silent performance degradation justifies the conservative choice.

Assumption 3: TP4 is the right parallelism strategy. The assistant uses TP4 without reconsideration. Earlier in the session, TP4 was found to be optimal for MXFP4 on PCIe. But NVFP4 with Marlin has different compute characteristics—Marlin kernels are more compute-efficient than the SIMT fallback, which could shift the balance between compute and communication. TP4 might no longer be optimal, but the assistant prioritizes getting a correct baseline first before optimizing further.

Assumption 4: The memory fraction of 0.70 is sufficient. This is an educated guess. The assistant knows the model size (~37 GB per GPU with TP4) and the GPU capacity (48 GB), but doesn't account precisely for KV cache, CUDA graph workspace, activation memory, and framework overhead. The 0.70 fraction leaves ~14.4 GB for these, which should be ample, but the exact requirement depends on batch size and sequence length.

Knowledge Flow: Input and Output

This message is a nexus point where multiple streams of knowledge converge and produce a new artifact.

Input knowledge required to understand this message:

  1. The SM100 vs SM120 architecture distinction and its implications for kernel compatibility
  2. The NVFP4 quantization format and its relationship to Marlin (W4A16) kernels
  3. The role of PR #25820 in enabling NVFP4 detection in SGLang
  4. The DeepSeek-V4-Flash model architecture and its MoE structure
  5. Tensor parallelism (TP) and its memory/compute implications
  6. CUDA graph capture and its batch-size limitations
  7. The prior optimization campaign results (MXFP4 throughput, sm_120 fallback bottleneck) Output knowledge created by this message:
  8. A validated launch configuration for NVFP4 + Marlin on SM120
  9. The insight that the PR's default routing is inappropriate for SM120 hardware
  10. A baseline for further optimization (the server starts and generates, providing a correctness baseline)
  11. Documentation of the decision-making process for future reference

Broader Significance

This message illustrates a fundamental tension in ML infrastructure: upstream code is written and tested on specific hardware configurations, and deploying on different hardware requires careful adaptation. PR #25820 was a necessary step for NVFP4 support, but its default routing was optimized for SM100 datacenter GPUs. The assistant's intervention—explicitly choosing marlin over the auto-detected path—represents the kind of hardware-aware engineering that separates a working deployment from a broken one.

The decision also highlights the importance of understanding the full stack: from the quantization format (NVFP4), through the kernel backend (Marlin), to the GPU architecture (SM120), and the serving framework (SGLang). Each layer makes assumptions about the layers below it, and mismatches can cascade into failures.

The message also demonstrates a pragmatic engineering philosophy: when in doubt, choose the proven path. The assistant could have attempted to make flashinfer_trtllm_routed work on SM120, or could have tested multiple backends in parallel. Instead, it chose the path with the highest probability of immediate success, deferring optimization to later iterations. This is the right call in a deployment scenario where correctness is the primary goal.

Conclusion

Message 12505 captures a moment of architectural decision-making in the deployment of DeepSeek-V4-Flash with NVFP4 quantization on SM120 Blackwell GPUs. The assistant recognizes that the upstream PR's default routing targets a different GPU architecture, explicitly overrides it with the proven Marlin backend, and configures the launch with parameters informed by prior experience. The server starts successfully, laying the groundwork for the next round of benchmarking and optimization.

The deeper lesson is that ML infrastructure deployment is never a matter of simply running upstream code. Every hardware platform, every quantization format, and every model architecture introduces unique constraints that require human judgment to navigate. The assistant's reasoning in this message—concise, confident, and grounded in architectural knowledge—is a model of how to make those judgments effectively.