The Sanity Check: Validating Native FP4 Tensor-Core Inference on Blackwell

In the middle of a relentless optimization campaign to deploy DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant sends a message that appears, at first glance, to be trivial: a single curl command asking the model "What is 17*23?" and receiving the answer "391". But this brief exchange — message [msg 12511] in the conversation — is anything but trivial. It is a pivotal moment of validation, a sanity check that separates a working deployment from a silently broken one, and it encapsulates the disciplined, measurement-driven methodology that defines the entire session.

To understand why this message matters, one must appreciate the journey that led to it. The assistant had spent the preceding hours deploying DeepSeek-V4-Flash across multiple quantization formats and MoE backends, systematically measuring performance at every turn. The baseline FP4 deployment had achieved a disappointing ~23 tok/s at concurrency 16 — far below the user's target of ~1000 tok/s. A GPU profile revealed the culprit: the _tiled_sparse_decode_kernel, a Triton fallback for sparse MLA attention that launched only 64 blocks on ~170 SMs, serially iterating over 512 top-k tokens. This was the same low-occupancy pathology that had plagued the earlier Kimi K2.6 verify kernel.

The Marlin Interlude

The assistant's first major optimization was switching from the stock MXFP4 checkpoint to NVIDIA's official NVFP4 quantization, which routes MoE execution through tensor-core paths. This required applying PR #25820 — a patch that had to be merged via git apply --3way because the shallow clone had unrelated histories. The NVFP4 checkpoint, weighing 149 GB across 76 files, was downloaded over a saturated network link.

With NVFP4 loaded and the Marlin backend (--moe-runner-backend marlin), the assistant achieved a 24% improvement: C=16 throughput rose from 23.2 to 28.7 tok/s, and time-to-first-token roughly halved. But the assistant's reasoning in message [msg 12509] reveals a sophisticated understanding of why the gains were modest: "Marlin uses FP4 weights dequantized to BF16 for tensor-core operations, which means it still has to stream and dequantize weights from memory — it's not a native FP4×FP4 computation." Furthermore, the attention mechanism — accounting for 38% of decode time at C=16 — remained completely unchanged. Even if Marlin halved the MoE cost from 39% to ~20%, attention became the new dominant bottleneck, capping the overall improvement.

The Hypothesis: Native FP4 via CUTLASS

The assistant's reasoning in [msg 12509] then pivots to a new hypothesis: the Triton backend with cutlass_fp4_group_mm — a native W4A4 tensor-core path that avoids dequantization entirely. The key insight is that Marlin's W4A16 approach streams FP4 weights and dequantizes them to BF16 before computation, incurring both memory bandwidth and dequantization overhead. Native FP4×FP4, by contrast, keeps weights in FP4 throughout the tensor-core pipeline, potentially doubling effective memory bandwidth for weights.

The assistant recognized that the PR's intended backend — flashinfer_trtllm_routed — was SM100-only (designed for B200/GB300 GPUs) and would not work on sm_120 (RTX PRO 6000). The Triton backend, however, routes to cutlass_fp4_group_mm regardless of architecture, making it the safer bet. The assistant also noted that the deepseek_v4_hook.py logic only overrides the backend when set to auto; by explicitly specifying --moe-runner-backend triton, the hook would not interfere.

Message [msg 12509] executes the switch: it kills the Marlin server, uses sed to replace --moe-runner-backend marlin with --moe-runner-backend triton in the launch script, and starts the new server. Message [msg 12510] monitors the startup, observing that the Triton backend initializes faster — approximately 100 seconds versus the Marlin server's slower startup — and confirms the server is "fired up and ready to roll!"

The Subject Message: Verification Before Benchmarking

This brings us to message [msg 12511], the subject of this article. The assistant's reasoning block states:

NVFP4 with Triton and native CUTLASS FP4 support is now running with much faster startup times around 100 seconds and no need for Marlin repacking. I need to verify the output is correct and then benchmark performance across different C values.

This reasoning reveals several things. First, the assistant has successfully deployed the Triton/CUTLASS native FP4 path — a non-trivial achievement given that the PR #25820 was designed for SM100 hardware and the assistant is running it on sm_120. The fact that the server started without errors, loaded the 149 GB checkpoint, and initialized the CUDA graphs is itself a validation of the patch's compatibility with Blackwell consumer-grade hardware.

Second, the assistant notes "much faster startup times around 100 seconds and no need for Marlin repacking." This is a significant practical advantage: Marlin requires repacking weights into its specific format during initialization, a process that can take minutes and consumes additional GPU memory. The Triton/CUTLASS path can use the checkpoint weights as-is, reducing both startup time and memory overhead.

Third, the assistant explicitly states the workflow: "I need to verify the output is correct and then benchmark performance across different C values." This reveals a disciplined methodology: before investing time in running comprehensive benchmarks (C=1, C=8, C=16 with multiple samples each), the assistant first confirms the model produces correct outputs. A model that silently produces NaN, infinite values, or garbage text would waste the entire benchmarking run.

The Verification Itself

The verification is a single curl request to the SGLang server's /v1/chat/completions endpoint:

curl -s --max-time 35 http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"/root/models/DeepSeek-V4-Flash-NVFP4",
       "messages":[{"role":"user","content":"What is 17*23? Number only."}],
       "max_tokens":24,"temperature":0}' | python -c "import sys,json;print(repr(json.load(sys.stdin)['choices'][0]['message']['content']))"

The response is '391' — the correct answer to 17×23.

This choice of test is deliberate. 17×23 is a multiplication that requires the model to perform actual computation rather than regurgitate a memorized fact. The instruction "Number only." forces a specific output format, making the correctness check unambiguous. The temperature is set to 0 for deterministic output. The max_tokens of 24 is generous enough for the answer but short enough for a quick response.

Assumptions and Their Risks

The message rests on several assumptions. The most significant is that a single arithmetic test is sufficient to validate the correctness of the Triton/CUTLASS native FP4 backend. This is a pragmatic assumption — the assistant needs a quick go/no-go decision before proceeding to benchmarks — but it carries risk. A model could produce correct answers for simple arithmetic while exhibiting subtle numerical drift in more complex scenarios: long-context generation, multi-turn conversations, or tasks requiring precise attention patterns. The NVFP4 quantization, with its group size of 16 and mixed-precision scheme, could introduce quantization artifacts that only manifest under specific conditions.

The assistant also assumes that the Triton/CUTLASS backend's faster startup time is purely beneficial. While avoiding Marlin repacking is clearly advantageous, the faster startup could theoretically indicate that some initialization steps were skipped or deferred. The subsequent successful inference suggests this is not the case, but the assumption is worth noting.

Another assumption is that the Triton backend's cutlass_fp4_group_mm kernel actually executes on tensor cores on sm_120 hardware. The assistant's reasoning in [msg 12509] states "the triton backend with cutlass native FP4 should work on sm120 — the MoeRunner gets constructed fine and routes to cutlass_fp4_group_mm regardless." This is a reasonable inference from the code structure, but the actual kernel dispatch depends on runtime checks, CUDA capability flags, and the specific CUTLASS version compiled into SGLang. The successful inference confirms the assumption, but the performance characteristics of that kernel on sm_120 remain unknown until benchmarked.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains. First, the quantization landscape: MXFP4 (the stock format) versus NVFP4 (NVIDIA's optimized format), and how each interacts with different MoE backends (Marlin's W4A16 dequantized path versus Triton/CUTLASS's native W4A4 path). Second, the hardware architecture: sm_120 (RTX PRO 6000 Blackwell) versus SM100 (B200/GB300), and the critical difference that flashinfer_trtllm_routed is SM100-only while cutlass_fp4_group_mm is architecture-agnostic. Third, the SGLang codebase: how the deepseek_v4_hook.py routes backend selection, how the modelopt_quant.py handles NVFP4 detection, and how the auto versus explicit backend flags interact. Fourth, the benchmark methodology: what C values represent (concurrent requests), how throughput is measured in tok/s, and why the assistant benchmarks at C=1, C=8, and C=16.

Output Knowledge Created

This message creates several important outputs. First and foremost, it confirms that the Triton/CUTLASS native FP4 path produces correct inference on sm_120 hardware — a non-trivial validation given the PR's SM100-centric design. Second, it establishes that startup time is significantly faster with Triton (~100 seconds) than with Marlin, which required weight repacking. Third, it produces a validated server instance ready for comprehensive benchmarking. Fourth, it sets the stage for the critical comparison: does native FP4×FP4 outperform Marlin's W4A16 dequantized path?

The answer to that question, which the assistant will discover in subsequent messages, is revealing: the Triton/CUTLASS native FP4 path achieves approximately the same throughput as Marlin (~28 tok/s at C=16). This confirms the assistant's earlier diagnosis that attention — not MoE — is the dominant bottleneck. Even with optimal MoE throughput, the sparse-attention kernel caps performance at roughly the same level, and the attention kernel remains a Triton fallback running on CUDA cores rather than tensor cores.

The Thinking Process

The reasoning section of message [msg 12511] is concise but reveals a clear mental model. The assistant is thinking in terms of a hypothesis-testing loop: "I deployed the new backend. It started faster, which is a good sign. But I need to verify it actually works before I spend time benchmarking it." This reflects engineering discipline — measure twice, cut once.

The observation "no need for Marlin repacking" shows that the assistant is thinking about the full lifecycle cost of each backend, not just inference throughput. Marlin's repacking step adds minutes to startup and consumes GPU memory for the repacked weights, which could reduce the available KV cache capacity. The Triton/CUTLASS path's ability to use weights in-place is a genuine operational advantage, even if raw throughput is similar.

The assistant is also thinking ahead: "I need to verify the output is correct and then benchmark performance across different C values." The workflow is already planned — verify first, then measure at C=1, C=8, and C=16, the same methodology used for the Marlin benchmarks. This ensures apples-to-apples comparison.

Conclusion

Message [msg 12511] is a masterclass in disciplined optimization engineering. In a single curl command, it validates a complex deployment spanning a custom-patched SGLang build, a 149 GB NVFP4 checkpoint, and a MoE backend that was never officially tested on the target hardware. It demonstrates that the assistant understands the difference between a deployment that runs and a deployment that works correctly — and refuses to conflate the two. The message's brevity belies its significance: it is the gate that separates hypothesis from evidence, the moment where speculation meets reality. And the answer — "391" — is the quiet confirmation that the assistant's multi-hour optimization campaign remains on solid ground.