The Pivot Point: Narrowing the Backend Search Space on Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, the difference between a working inference server and a broken one often comes down to a single command-line flag. Message [msg 5971] captures one such critical decision point in an extended optimization session: the moment when an AI assistant, after systematically eliminating three incompatible backend options for the Qwen3.5-397B-A17B-NVFP4 model on 8× RTX PRO 6000 Blackwell (SM120) GPUs, pivots to compare the two remaining candidates.

The Message in Full

The assistant writes:

Confirmed: trtllm backend for dense FP4 also doesn't support SM120 (capability 120). Only cutlass and cudnn work for dense FP4 on SM120.

>

So let me now test flashinfer_cutlass for dense FP4 (instead of cudnn) to see if that's faster: [bash] ssh root@10.1.230.174 'fuser -k /dev/nvidia* 2>/dev/null; sleep 2'

At first glance, this seems like a simple status update followed by a cleanup command. But in the broader context of the session, it represents the culmination of an exhaustive search across a combinatorial space of backend configurations, and a deliberate narrowing to the two viable paths forward.

The Reasoning and Motivation

This message was written because the assistant had been systematically probing the landscape of FP4 inference backends on Blackwell GPUs—a hardware architecture so new that much of the software stack had not yet been validated for it. The session's broader goal was to deploy Qwen3.5-397B-A17B-NVFP4, a 397-billion-parameter Mixture-of-Experts model quantized to NVFP4 (NVIDIA's 4-bit floating point format), with maximum throughput and correctness on an 8-GPU Blackwell machine.

The assistant's motivation at this specific moment was twofold. First, it needed to definitively determine which backend combinations would even load and produce correct output on SM120. Second, among the working combinations, it needed to find the fastest one. The message explicitly states the next action: test flashinfer_cutlass for the dense FP4 GEMM backend instead of the currently-working flashinfer_cudnn, to see if it offers better performance.

This is classic optimization methodology: first establish the feasible set, then optimize within it. The assistant had already spent many messages (from [msg 5942] through [msg 5970]) probing each backend option, and this message marks the transition from the "feasibility" phase to the "optimization" phase.

The Decision-Making Process

The decision to test flashinfer_cutlass for dense FP4 was the result of a multi-step elimination process visible in the preceding messages:

  1. flashinfer_trtllm for MoE (<msg id=5960-5962>): Crashed immediately with an OOM-like error. The log analysis revealed that trtllm_fp4_block_scale_moe attempted to compile for SM100 (compute capability 10), while the Blackwell GPUs are SM120 (capability 12). The fused TRT-LLM MoE kernels were confirmed SM100-only.
  2. flashinfer_cutedsl for MoE (<msg id=5963-5966>): Loaded successfully and began CUDA graph capture, but produced garbage output—the infamous !!!! pattern that signals numerical corruption in FP4 inference. The smoke test caught this with an assertion: assert &#34;!&#34; * 10 not in resp.
  3. flashinfer_trtllm for dense FP4 GEMM (<msg id=5969-5970>): Also crashed on SM120, confirming that the TRT-LLM backend is uniformly incompatible with Blackwell across both MoE and dense GEMM paths.
  4. flashinfer_cutlass for MoE + flashinfer_cudnn for dense FP4 (implicitly the working baseline): This combination had been validated earlier in the chunk as producing correct output. At [msg 5971], the assistant confirms the third elimination and states the remaining viable options: cutlass and cudnn for dense FP4. The decision to test cutlass next is a natural optimization step—if cutlass is faster than cudnn, it should be used instead.

Assumptions Made

Several assumptions underpin this message. The assistant assumes that the cutlass backend for dense FP4 GEMM will actually work on SM120, given that the cutlass MoE backend worked correctly. This is a reasonable inference—both use the CUTLASS template library from NVIDIA, which is designed to be architecture-agnostic and JIT-compiles for the target GPU. However, the assumption is not yet validated; the test hasn't been run.

The assistant also assumes that performance differences between cutlass and cudnn are worth measuring. This is grounded in the understanding that different kernel implementations have different optimization strategies: CUTLASS uses template-based code generation, while cuDNN uses hand-tuned kernels. On a new architecture like Blackwell, either could be faster.

A deeper assumption is that the flashinfer_cutlass backend for dense FP4 GEMM is actually a separate code path from flashinfer_cutlass for MoE. The assistant's earlier research (<msg id=5945-5946>) showed that flashinfer.mm_fp4 accepts a backend parameter with options cudnn, trtllm, cutlass, cute-dsl, and auto. The MoE runner backend and the dense GEMM backend are independently configurable via --moe-runner-backend and --fp4-gemm-backend respectively, so this assumption is correct.

Mistakes and Incorrect Assumptions

While the message itself doesn't contain an explicit mistake, the broader context reveals that the assistant had previously assumed flashinfer_cutedsl might work on SM120. This assumption was based on the fact that CUTE DSL (CUDA Template Expressions Domain-Specific Language) JIT-compiles for the exact target architecture, which should theoretically support any SM version. In practice, however, the compiled kernels produced numerical garbage on Blackwell, suggesting either a bug in the CUTE DSL FP4 implementation for SM120 or an unhandled hardware-specific code path.

The assistant also initially assumed that flashinfer_trtllm might work for MoE on SM120, given that the code explicitly checked is_sm120_supported() ([msg 5948]). However, the actual kernel call at runtime failed because the underlying trtllm_fp4_block_scale_moe function was compiled only for SM100. This discrepancy between the guard check and the actual kernel availability is a software maturity issue typical of early hardware support.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. trtllm is definitively ruled out for dense FP4 on SM120. This is a concrete finding that future deployments on Blackwell hardware can reference. Any attempt to use --fp4-gemm-backend flashinfer_trtllm on SM120 GPUs will fail.
  2. The viable set for dense FP4 on SM120 is {cutlass, cudnn}. This narrows the search space from five options (including auto, trtllm, and cute-dsl) to two.
  3. A test plan is initiated. The fuser -k command clears GPU state for the next experiment, demonstrating the systematic testing methodology.
  4. The optimization phase begins. The session transitions from "what works" to "what's fastest," a natural progression in production deployment work.

The Thinking Process

The assistant's reasoning in this message is concise but reveals a clear mental model. The phrase "Confirmed: trtllm backend for dense FP4 also doesn't support SM120" uses the word "also" to connect this finding to the earlier MoE TRT-LLM failure, showing that the assistant is synthesizing results across different components of the system.

The statement "Only cutlass and cudnn work for dense FP4 on SM120" is a crisp summary of the feasible set, derived from the cumulative testing across messages <msg id=5960-5970>. The assistant doesn't re-list all the failures—it trusts that the reader (or the user) has been following along.

The next sentence—"So let me now test flashinfer_cutlass for dense FP4 (instead of cudnn) to see if that's faster"—reveals the assistant's optimization mindset. It has identified two working options and immediately plans to compare them. The choice to test cutlass first (rather than cudnn) is pragmatic: cutlass was already validated for MoE, so it's the natural candidate to try for dense GEMM.

The bash command fuser -k /dev/nvidia* 2&gt;/dev/null; sleep 2 is a standard cleanup step that kills any processes holding NVIDIA device files. This is necessary because previous test launches may have left GPU processes in an inconsistent state. The sleep 2 ensures the kernel has time to release resources before the next launch.

Broader Significance

In the larger narrative of this optimization session ([chunk 39.0]), message [msg 5971] sits at a crucial inflection point. The assistant had spent the earlier part of the chunk upgrading the entire software stack to nightly builds—PyTorch 2.12.0+cu130, FlashInfer 0.6.5, and the latest SGLang main branch—and building sgl-kernel from source with SM120 support. The backend testing that follows is the validation phase for that stack.

The message also demonstrates a key engineering virtue: systematic elimination. Rather than guessing at the correct configuration, the assistant tests each backend combination, records the result (crash, garbage, or correct output), and narrows the search space. This approach is especially valuable on new hardware where documentation is sparse and community knowledge is limited.

The final production configuration, as documented in the chunk summary, would use flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM—achieving ~172 tok/s at single-request concurrency and over 2100 tok/s aggregate at high concurrency. But at the moment of [msg 5971], that outcome is not yet determined. The assistant is still exploring, still testing, still narrowing down. It is the uncertainty and the methodical process of resolving it that makes this message a compelling snapshot of real-world ML engineering.