Probing the GEMM Frontier: Investigating FP4 Backend Compatibility on Blackwell SM120

Introduction

In the high-stakes world of large language model inference on novel GPU architectures, every kernel backend choice can mean the difference between disappointing throughput and production-ready performance. This article examines a single message (index 5946) from an extended opencode coding session where an AI assistant was tasked with deploying and optimizing the Qwen3.5-397B-A17B-NVFP4 model—a massive 397-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 precision—on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The message captures a pivotal moment of investigation: the assistant is probing the available GEMM (General Matrix Multiply) backends for FP4 computation on the SM120 architecture, specifically checking whether a feature called Programmatic Dependent Launch (PDL) is compatible with their GPU variant.

The subject message itself is deceptively brief. It reads:

So flashinfer's mm_fp4 supports cudnn, trtllm, cutlass, cute-dsl, and auto. Let me also check enable_pdl — this might be SM100-only: [bash command inspecting flashinfer.mm_fp4 help output] Output showing PDL documentation for the cute_dsl backend

Despite its brevity, this message sits at a critical juncture in the optimization workflow. To understand its significance, we must trace the chain of reasoning that led to it and the decisions it would subsequently enable.

The Context: A User Dissatisfied with Throughput

The immediate predecessor to this investigation was a user complaint. After an extensive multi-session effort to upgrade the entire software stack—PyTorch to nightly 2.12.0+cu130, flashinfer to 0.6.5, sgl-kernel rebuilt from source with SM120 FP4 support, and SGLang updated to the latest main branch—the assistant had successfully deployed Qwen3.5-397B-A17B-NVFP4 and achieved a single-request throughput of approximately 72 tok/s ([msg 5927]). The user's reaction was pointed: "Seems really low throughput" ([msg 5928]).

The user raised two specific questions:

  1. Does the NVFP4 quant have built-in Multi-Token Prediction (MTP) heads that could be leveraged for speculative decoding?
  2. Can the GEMM backends be tuned for SM120, since the RTX PRO 6000 Blackwell has a different SM count and architecture compared to the B200 GPUs that the existing configurations were optimized for? These questions launched a parallel investigation. The assistant began probing the model checkpoint for MTP weights while simultaneously researching the available FP4 GEMM backends in flashinfer and SGLang.

The Investigation Unfolds

The assistant's investigation into MTP was remarkably successful. By inspecting the model's config.json and model.safetensors.index.json, it discovered that the NVFP4 quant does contain MTP weights—1,553 weight tensors including a full set of 512 experts in the MTP layer ([msg 5931]). Furthermore, SGLang's codebase contained a dedicated qwen3_5_mtp.py model file ([msg 5932]), and the --speculative-algorithm NEXTN flag would activate it, internally mapping to the EAGLE speculative decoding path ([msg 5935]).

On the GEMM backend front, the assistant had already discovered that flashinfer's mm_fp4 function supports five backends: cudnn, trtllm, cutlass, cute-dsl, and auto ([msg 5945]). The current production configuration used flashinfer_cudnn for FP4 GEMM and flashinfer_cutlass for the MoE runner—a combination that worked but might not be optimal.

The Subject Message: Probing PDL Compatibility

This is where the subject message (index 5946) enters the narrative. Having enumerated the five available backends, the assistant now turns its attention to a specific parameter of the mm_fp4 function: enable_pdl. The assistant's reasoning, visible in the message text, is explicit: "Let me also check enable_pdl — this might be SM100-only."

This hypothesis is grounded in the architectural distinction between two Blackwell variants:

Whether to enable Programmatic Dependent Launch (PDL) for the `cute_dsl backend, defaults to True. PDL allows overlapping the tail of one kernel with the start of the next for reduced launch latency. This parameter is only used by the cute_dsl` backend and is ignored by other backends.

Analysis of the Thinking Process

The subject message reveals several layers of the assistant's reasoning:

First, the assistant is methodically enumerating options. It has already discovered the five backend names. Now it is checking each parameter's compatibility and applicability. This systematic approach—list options, check constraints, test candidates—is characteristic of the assistant's methodology throughout the session.

Second, the assistant is proactively identifying potential incompatibilities. The hypothesis that PDL "might be SM100-only" reflects an awareness that Blackwell is not monolithic. The RTX PRO 6000 (SM120) may lack features that B200 (SM100) has, and vice versa. Rather than assuming all Blackwell features work on their hardware, the assistant actively checks.

Third, the assistant is gathering information before making decisions. At this point, no backend changes have been made. The assistant is still in the reconnaissance phase, building a mental model of which backends and parameters are viable before committing to a test plan.

Input Knowledge Required

To fully understand this message, one needs:

  1. The flashinfer library architecture: Understanding that mm_fp4 is the core FP4 matrix multiplication function, and that it dispatches to different backend implementations (cudnn, trtllm, cutlass, cute-dsl).
  2. The Blackwell GPU family: Knowing that there are multiple Blackwell variants (SM100/B200 vs SM120/RTX PRO 6000) with different compute capabilities and potentially different feature sets.
  3. Programmatic Dependent Launch (PDL): Understanding that PDL is a CUDA feature that allows overlapping kernel launches to reduce launch latency—particularly important for the many small kernel launches typical in LLM inference.
  4. The SGLang/SM120 saga: The broader context of patching SGLang and flashinfer for SM120 support, which has been a recurring theme across multiple sessions.
  5. The NVFP4 quantization format: Understanding that FP4 computation requires specialized GEMM kernels that handle the unusual 4-bit floating point format and its associated scaling factors.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. PDL is cute_dsl-only: The enable_pdl parameter is only used by the cute-dsl backend and is ignored by all others. This means if the assistant chooses cudnn, trtllm, or cutlass backends, PDL is irrelevant.
  2. PDL's purpose: It allows overlapping the tail of one kernel with the start of the next, reducing launch latency. This is a performance optimization, not a correctness requirement.
  3. The documentation does not mention SM120 vs SM100 restrictions: The help text doesn't say "SM100 only," which is notable. The assistant's hypothesis that PDL might be SM100-only is neither confirmed nor refuted by this documentation—it simply states that PDL is only used by the cute_dsl backend.
  4. The cute_dsl backend is a viable candidate: Since PDL defaults to True and is a performance optimization, the cute-dsl backend becomes an interesting option to benchmark, provided it works on SM120.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message:

That enable_pdl might be SM100-only: This is a reasonable hypothesis given the pattern of SM100-specific features throughout the codebase, but it's not confirmed. The documentation doesn't mention any SM120 restriction. The assistant is being cautious, which is appropriate for production deployment.

That the help text is complete and accurate: The assistant relies on the help() output as authoritative. In practice, documentation can lag behind implementation, especially in rapidly evolving nightly builds.

That backend selection is the primary lever for throughput improvement: While GEMM backends do affect performance, the subsequent investigation would reveal that MTP/NEXTN speculative decoding and other architectural choices may have a larger impact.

The Broader Significance

This message exemplifies a pattern that recurs throughout the session: the assistant acts as a systems detective, methodically tracing through code, inspecting APIs, and testing hypotheses. The question "is this SM100-only?" is a recurring theme—the assistant has to constantly check whether code written for B200 GPUs will work on the RTX PRO 6000 Blackwell.

The message also illustrates the importance of understanding the GPU architecture hierarchy. SM120 is not just "a different GPU"—it's a different compute capability major version (12 vs 10), which means different instruction sets, different kernel requirements, and potentially different optimal configurations. The assistant's careful attention to this distinction is what enables it to eventually find working configurations.

Conclusion

Message 5946 is a small but revealing window into the process of deploying cutting-edge AI models on novel hardware. It shows the assistant in information-gathering mode, probing the boundaries of what's possible on SM120 Blackwell GPUs. The discovery that PDL is cute_dsl-only and that the documentation doesn't restrict it to SM100 opens up the possibility of testing the cute-dsl backend—a path the assistant would indeed pursue in subsequent messages.

In the broader narrative of the session, this message represents the calm before the storm of benchmarking. The assistant is still gathering intelligence, building the knowledge base it needs to make informed decisions about which backends to test and which parameters to tune. The methodical, hypothesis-driven approach visible here—checking a specific parameter against a specific architectural concern—is what ultimately enables the session to achieve its goals of deploying a production-quality inference service on this challenging hardware configuration.