The Pivot to CUTE DSL: Testing FlashInfer's cutedsl MoE Backend on Blackwell SM120

In the high-stakes world of deploying large language models on cutting-edge hardware, few things are as frustrating as discovering that your most promising optimization path is blocked by a hardcoded architecture check. This is exactly the situation the assistant found itself in during message [msg 5963] of an extended coding session focused on deploying the Qwen3.5-397B-A17B-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message captures a single, decisive bash command — a pivot from one failed backend to a hopeful alternative — that embodies the systematic, hypothesis-driven approach to performance optimization that defines the entire session.

The Context: A Systematic Backend Search

To understand the significance of this message, we must first appreciate the journey that led to it. The assistant had been engaged in an exhaustive search for the optimal configuration of SGLang's inference engine on Blackwell (SM120) hardware. The Qwen3.5-397B-A17B-NVFP4 model is a massive Mixture-of-Experts (MoE) architecture quantized to 4-bit floating point (NVFP4), and its performance depends critically on the choice of kernel backends for two distinct operations: the MoE computation itself (routing tokens to experts, computing expert outputs, and combining them) and the dense FP4 GEMM operations (general matrix multiplications with FP4 weights).

The assistant had already established a working baseline using flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM. But the quest for maximum throughput demanded testing every available option. The most promising candidate was flashinfer_trtllm — a backend that uses NVIDIA's TensorRT-LLM fused kernels, which combine routing, GEMM1, activation, and GEMM2 into a single, highly optimized kernel. In message [msg 5960], the assistant attempted to launch the server with --moe-runner-backend flashinfer_trtllm. The result was immediate failure: the process was killed before it could even start, and the logs revealed the root cause — trtllm_fp4_block_scale_moe was compiled only for SM100 (compute capability 10.x), while the RTX PRO 6000 Blackwell GPUs are SM120 (compute capability 12.x).

The Message: A Calculated Pivot

Message [msg 5963] is the assistant's response to that failure. It contains a single bash command, executed over SSH on the remote machine hosting the GPUs:

ssh root@10.1.230.174 'export CUDA_HOME=/usr/local/cuda-13.0 && export PATH="/usr/local/cuda-13.0/bin:/root/ml-env/bin:$PATH" && export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64 && export OMP_NUM_THREADS=16 && nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /data/models/Qwen3.5-397B-A17B-NVFP4 \
  --served-model-name qwen3.5-397b \
  --tp 8 \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --host 0.0.0.0 \
  --attention-backend triton \
  --moe-runner-backend flashinfer_cutedsl \
  --fp4-gemm-backend flashinfer_cudnn \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --disable-custom-all-reduce \
  > /tmp/sglang_cutedsl_test.log 2>&1 &
echo "PID: $!"'
PID: 45597

The critical change is in the MoE runner backend: flashinfer_trtllm has been replaced with flashinfer_cutedsl. Everything else — the CUDA 13 environment, the FP4 GEMM backend (flashinfer_cudnn), the tensor parallelism of 8, the model path, the reasoning and tool-call parsers — remains identical. This is a controlled experiment: change exactly one variable and observe the result.

The Reasoning: Why cutedsl Should Work

The assistant's reasoning, though not explicitly stated in this message, is clearly legible from the surrounding context. The flashinfer_trtllm backend failed because TensorRT-LLM kernels are compiled ahead-of-time for specific GPU architectures. The TRT-LLM team has targeted SM100 (the Hopper-generation H100/H200), and the Blackwell SM120 architecture requires new kernel variants that haven't been compiled into the binary distribution. The error manifested as a CUDA kernel launch failure — the GPU simply didn't have the required machine code.

The flashinfer_cutedsl backend, by contrast, uses NVIDIA's CUTE (CUDA Template Extensions) DSL. CUTE is a header-only library of CUDA templates that generate kernel code at Just-In-Time (JIT) compilation time. Instead of relying on pre-compiled binaries for specific architectures, CUTE DSL kernels are compiled on-the-fly for the exact GPU they're running on. This means that if the CUTE templates are well-written and the CUDA compiler toolchain supports SM120, the kernels should compile and run correctly regardless of whether the developers explicitly tested on Blackwell hardware.

This is a fundamentally different approach to GPU kernel deployment. TRT-LLM represents the "bake it once, ship it everywhere" philosophy — kernels are hand-tuned and compiled for specific architectures, offering maximum performance on those targets but zero portability to newer hardware. CUTE DSL represents the "compile for the machine you're on" philosophy — sacrificing some potential optimization opportunities in exchange for broad compatibility. On a brand-new architecture like Blackwell SM120, the latter approach is often the only viable path.

Assumptions Embedded in the Command

The message makes several assumptions worth examining. First, it assumes that the CUDA 13.0 toolchain installed on the machine can compile CUTE DSL templates for SM120. This is not guaranteed — CUDA 13.0 is a very recent release, and while it should support Blackwell, the CUTE library itself may have bugs or missing features for the new architecture. Second, it assumes that the combination of flashinfer_cutedsl for MoE and flashinfer_cudnn for FP4 GEMM is compatible. These are separate kernel selection paths in SGLang's codebase, but they share GPU resources and memory pools; an incompatibility could manifest as a crash or silent corruption. Third, the command assumes that the server will launch successfully as a background process (nohup) and survive the SSH session termination. The 120-second timeout on the bash tool means the assistant won't see the server's output directly — it will need to check the log file in a subsequent message.

The Outcome: Partial Success

The subsequent messages reveal the result. In [msg 5964], the assistant checks the log and finds that the server is alive and progressing through CUDA graph capture — the process of pre-compiling execution graphs for common batch sizes. This is a strong signal that flashinfer_cutedsl works on SM120 at the compilation level. The server reaches a healthy state in [msg 5965], confirming that the backend loads without crashing.

However, the chunk summary for this segment reveals a more nuanced outcome: while flashinfer_cutedsl loaded successfully, it was later found to produce NaN or garbage output during inference. This is a classic failure mode for GPU kernels on new hardware — the code compiles and runs, but produces incorrect numerical results due to subtle differences in the architecture's floating-point behavior, memory subsystem, or instruction scheduling. The assistant would ultimately fall back to flashinfer_cutlass for MoE, which produced correct results, and flashinfer_cudnn for FP4 GEMM, which also worked correctly.

What This Teaches Us About GPU Kernel Deployment

This message is a microcosm of the challenges facing ML engineers deploying on brand-new hardware. The traditional GPU software stack — CUDA, cuDNN, TensorRT — is built on the assumption that hardware generations are discrete and well-tested before release. But the gap between hardware availability and software maturity is widening. Blackwell GPUs were shipping before the full software stack was ready, leaving early adopters to navigate a landscape where some backends crash, some produce silent corruption, and only a few work correctly.

The assistant's methodology is instructive: test each backend independently, isolate the failure mode (crash vs. NaN vs. slowdown), and maintain a working baseline to fall back on. The pivot from flashinfer_trtllm to flashinfer_cutedsl was not random — it was based on a clear understanding of the compilation model difference between the two backends. And when flashinfer_cutedsl also failed (though more gracefully), the assistant had flashinfer_cutlass ready as a known-good fallback.

Conclusion

Message [msg 5963] captures a moment of technical decision-making that is easy to overlook but deeply revealing. It is not a dramatic breakthrough or a catastrophic failure — it is a single experiment in a long chain of experiments, each one eliminating a hypothesis and narrowing the search space. The assistant's ability to reason about GPU kernel compilation models, to design controlled experiments that change exactly one variable, and to interpret the results in the context of the broader optimization goal, is what transforms a frustrating debugging session into a systematic exploration of the Blackwell inference frontier. The message stands as a testament to the value of deep technical knowledge — understanding not just which flag to flip, but why that flag might matter, and what the failure modes tell you about the underlying hardware.