The Last Backend Standing: Systematic Elimination of FP4 Kernel Options on Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, few tasks are as painstaking as finding a working GPU kernel backend for a new architecture. Message [msg 5969] captures a pivotal moment in this process: the assistant, having already eliminated two major backend candidates for the Qwen3.5-397B-A17B-NVFP4 model on NVIDIA's Blackwell (SM120) architecture, attempts a third combination—only to see it fail before it can even begin.

The Message: A Single Bash Command

The message itself is deceptively simple—a single bash command launching the SGLang inference server:

export CUDA_HOME=/usr/local/cuda-13.0 && export PATH="/usr/local/cuda-13.0/bin:/root/ml-env/bin:$PATH" && export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64 && export OMP_NUM_THREADS=16 && nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /data/models/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name qwen3.5-397b \
  --tp 8 \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --host 0.0.0.0 \
  --attention-backend triton \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_trtllm \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --disable-custom-all-reduce \
  > /tmp/sglang_fp4trtllm_test.log 2>&1 &
echo "PID: $!"

The process is immediately killed: bash: line 14: 48326 Killed. This outcome, while negative, represents a critical data point in the assistant's systematic exploration of the backend landscape for Blackwell GPUs.

The Reasoning and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the chain of reasoning that led to it. The assistant was in the middle of a systematic search for a working and performant backend configuration for the Qwen3.5-397B-A17B-NVFP4 model, a massive 397-billion-parameter Mixture-of-Experts (MoE) model quantized with NVIDIA's NVFP4 (4-bit floating point) format. The hardware was an 8× RTX PRO 6000 Blackwell setup, using the SM120 compute capability—a brand-new architecture that was still receiving software support.

The assistant had already tested and eliminated two major backend options before this message:

  1. flashinfer_trtllm for MoE ([msg 5960]): The fused TRT-LLM MoE kernel was attempted but immediately killed. Investigation of the logs ([msg 5961]) revealed the root cause: the trtllm_fp4_block_scale_moe kernel tried to compile for SM100 (compute capability 10.x), but the Blackwell GPUs are SM120 (compute capability 12.0). The assistant correctly diagnosed this as an architecture mismatch: "The fused TRT-LLM MoE kernels are SM100-only" ([msg 5962]).
  2. flashinfer_cutedsl for MoE ([msg 5963]): This backend used CUTE DSL, which JIT-compiles for the target architecture and therefore should have worked on SM120. The server started successfully and reached CUDA graph capture ([msg 5964]), and the health check passed ([msg 5965]). However, the smoke test revealed catastrophic failure: the model produced garbage output consisting entirely of repeated exclamation marks ([msg 5966]). The assistant identified this as the same SM120 kernel issue that plagued the default auto backend ([msg 5967]). After these two failures, the assistant had established that only flashinfer_cutlass produced correct output for the MoE layers on SM120. But there was still an unexplored dimension: the dense FP4 GEMM backend (--fp4-gemm-backend), which handles the non-MoE linear layers in the model. The assistant's reasoning, stated in [msg 5967], was: "Let me also check flashinfer_trtllm for the dense FP4 GEMM (not MoE) — the --fp4-gemm-backend flag." This distinction is crucial. In the Qwen3.5 architecture, the MoE layers (expert feed-forward networks) and the dense layers (attention projections, embeddings, etc.) use different kernel paths. The --moe-runner-backend flag controls MoE routing and expert computation, while --fp4-gemm-backend controls the dense FP4 matrix multiplications. The assistant hypothesized that even though flashinfer_trtllm didn't work for the fused MoE path, it might still work for the simpler dense GEMM operations.

Assumptions and Incorrect Hypotheses

The assistant made several assumptions in crafting this test:

Assumption 1: The dense FP4 GEMM backend is independent of the MoE backend. This is architecturally correct—the two backends serve different layers and are dispatched through different code paths. However, the assumption that independence implies different SM120 support was not validated.

Assumption 2: The flashinfer_trtllm dense GEMM kernel might have different SM120 support than the flashinfer_trtllm MoE kernel. This was a reasonable hypothesis. The TRT-LLM fused MoE kernel (trtllm_fp4_block_scale_moe) is a complex, specialized kernel that fuses multiple operations (routing, GEMM1, activation, GEMM2). The dense GEMM kernel is simpler and might have been compiled with broader architecture support. The assistant was testing whether the dense kernel had been built with SM120 support while the MoE kernel had not.

Assumption 3: The process was killed due to OOM (out of memory). When the process was killed immediately ([msg 5960]), the assistant initially hypothesized OOM due to weight shuffling. However, the log investigation revealed the true cause was an SM100 architecture assertion. The same pattern repeated in this message—immediate kill—but the assistant would later confirm ([msg 5970]) that the trtllm backend for dense FP4 also "doesn't support SM120 (capability 120)."

The incorrect assumption was that the dense GEMM backend might have broader architecture support than the MoE backend. In reality, both the flashinfer_trtllm MoE and dense GEMM backends are compiled for SM100 only, as they both rely on the same TRT-LLM kernel library which hasn't yet been updated for SM120.

Input Knowledge Required

To fully understand this message, one needs:

  1. SGLang architecture knowledge: Understanding the distinction between --moe-runner-backend (controls MoE expert computation) and --fp4-gemm-backend (controls dense FP4 linear layers). These are separate flags because the Qwen3.5 model has both MoE and dense layers.
  2. NVFP4 quantization awareness: The --quantization modelopt_fp4 flag tells SGLang to use NVIDIA's ModelOpt FP4 format, which requires specific kernel support. Not all backends support this quantization format.
  3. SM120 vs SM100 architecture: Blackwell GPUs (RTX PRO 6000) use compute capability 12.0 (SM120), while the previous Hopper generation uses SM90 and the upcoming generation uses SM100. TRT-LLM kernels were compiled for SM100, not SM120, creating a compatibility gap.
  4. The CUDA 13 toolchain: The environment variables (CUDA_HOME=/usr/local/cuda-13.0, PATH, LD_LIBRARY_PATH) point to CUDA 13.0, which was a recent upgrade performed in the previous segment to enable Blackwell-native optimizations.
  5. The model's scale: The 397B-parameter model requires --tp 8 (tensor parallelism across all 8 GPUs) and substantial memory management.

Output Knowledge Created

This message, combined with its follow-up ([msg 5970]), created several important pieces of knowledge:

  1. flashinfer_trtllm is SM100-only for both MoE and dense FP4 GEMM. The assistant's hypothesis that the dense backend might work independently was disproven. Both backends share the same TRT-LLM kernel library and both fail on SM120.
  2. The only working configurations on SM120 are: flashinfer_cutlass for MoE and either flashinfer_cudnn or flashinfer_cutlass for dense FP4 GEMM. This was the result of exhaustive testing across all available backends.
  3. A systematic methodology for backend discovery: The assistant demonstrated a pattern of testing one variable at a time, isolating the MoE backend from the dense GEMM backend, and using log analysis and smoke tests to validate correctness—not just startup success.

The Thinking Process Visible in Reasoning

The assistant's reasoning chain is clearly visible across the preceding messages. After the flashinfer_cutedsl failure produced garbage output ([msg 5966]), the assistant immediately pivoted to testing the dense GEMM backend separately. This shows a structured experimental mindset: rather than assuming all backends are broken, the assistant systematically decomposed the problem into independent variables.

The assistant also demonstrated awareness of the cost of each experiment. Each server launch requires cleaning up GPU state (fuser -k /dev/nvidia*), waiting for memory to clear, and then waiting for the server to initialize. The assistant minimized this cost by reusing the same environment setup (CUDA paths, model path, TP size) and only varying the two backend flags.

The Broader Context

This message sits within a much larger optimization campaign spanning multiple segments. The assistant had already:

Conclusion

Message [msg 5969] is a textbook example of systematic experimental elimination in ML engineering. A single bash command, killed before it could produce any output, nonetheless represents a critical data point in a larger optimization landscape. The assistant's willingness to test a hypothesis that seemed unlikely to succeed—and the disciplined approach of isolating variables—demonstrates the methodical mindset required to deploy cutting-edge models on bleeding-edge hardware. In the end, only two backends survived the gauntlet: flashinfer_cutlass for MoE and flashinfer_cudnn for dense FP4 GEMM, forming the foundation of a production deployment that would ultimately achieve over 2100 tokens per second at high concurrency.