The SM120 Allreduce Fusion Dead End: A Case Study in Architecture-Specific Kernel Limitations

Introduction

In the course of optimizing inference throughput for the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, an AI assistant encountered a critical setback that illuminates a fundamental challenge in adapting high-performance computing stacks across GPU architectures. Message [msg 756] captures the moment of discovery, reversion, and strategic pivot when the assistant realized that a promising optimization path—enabling FlashInfer allreduce fusion for the SM120 architecture—was blocked not by a configuration oversight, but by the literal absence of compiled CUDA kernels for that architecture. This article examines that single message in depth, exploring the reasoning, assumptions, and technical knowledge that shaped this pivotal decision point.

The Message

The assistant wrote:

The flashinfer TRT-LLM comm module (trtllm_ar.py) uses JIT compilation but its compilation_context only supports SM90/SM100 architectures. The allreduce fusion kernels literally don't have SM120 CUDA code paths.

>

Let me revert the communicator.py change — the SM90/SM100 gate was there because the kernels don't exist for SM120. But we can still investigate what else is bottlenecking.

This was followed by a bash command to revert the change:

ssh root@10.1.230.174 "sed -i 's/(_is_sm90_supported or _is_sm100_supported or _is_sm120_supported)/(_is_sm90_supported or _is_sm100_supported)/' /root/sglang/python/sglang/srt/layers/communicator.py"

Context: The Road to Allreduce Fusion

To understand why this message was written, we must trace the chain of events that led to it. The session had been engaged in a multi-day effort to deploy and optimize GLM-5-NVFP4—a large Mixture-of-Experts (MoE) language model with FP4 quantization—on a system of eight RTX PRO 6000 Blackwell GPUs, each with a 600W TDP and 96GB of HBM3 memory. The GPUs were installed in a Proxmox virtualized environment, which introduced significant PCIe communication overhead because the GPUs were spread across multiple PCIe root complexes, preventing direct peer-to-peer (P2P) DMA transfers between them.

Earlier in the session (segment 6, chunk 0), the assistant had achieved impressive throughput gains—from ~880 tok/s to ~3,740 tok/s at 1024 concurrency—by enabling FlashInfer CUTLASS MoE autotune for SM120 and increasing --max-running-requests. However, GPU power draw remained stubbornly around 250W out of 600W TDP, indicating severe underutilization. The assistant launched a systematic investigation to identify what was starving the GPUs.

The Investigation That Led to This Message

In messages [msg 727] through [msg 730], the assistant verified the 600W TDP and began profiling. A subagent task (message [msg 729]) performed a deep analysis of the forward pass compute path and returned a critical finding: FlashInfer allreduce fusion was disabled on SM120. The communicator.py file contained a gate that checked for SM90 or SM100 architectures only:

(_is_sm90_supported or _is_sm100_supported)

This meant that every allreduce operation—156 per forward pass (2 per layer × ~78 layers)—was serialized with compute. The GPU SMs sat idle while PCIe transfers completed. The assistant hypothesized that enabling allreduce fusion for SM120 would overlap communication with computation, keeping the GPU busy and raising power draw closer to the 600W TDP.

The Fix Attempt

In messages [msg 740] through [msg 748], the assistant applied a series of surgical patches:

  1. communicator.py: Added is_sm120_supported to the import list, cached _is_sm120_supported, and extended the fusion condition to include SM120.
  2. server_args.py: Added SM120 to the allreduce fusion auto-enable logic (line 1686) and to the MoE runner backend auto-selection for GLM-5 (line 1603). The assistant then restarted the server (message [msg 751]) and waited for it to come up.

The Crash and Discovery

The server crashed. Message [msg 754] revealed the error:

RuntimeError: No supported CUDA architectures found for major versions [9, 10].

The traceback traced through flashinfer/jit/comm.py, specifically the gen_trtllm_comm_module() function, which uses JIT compilation to build CUDA kernels for allreduce. The compilation_context only had SM90 (compute capability 9.0, corresponding to Hopper) and SM100 (compute capability 10.0, corresponding to datacenter Blackwell) architecture flags. SM120 (compute capability 12.0, corresponding to consumer Blackwell like RTX PRO 6000) was not supported.

Message 756: The Reversion

This is where message [msg 756] becomes the critical inflection point. The assistant's reasoning is concise but reveals deep understanding:

The flashinfer TRT-LLM comm module (trtllm_ar.py) uses JIT compilation but its compilation_context only supports SM90/SM100 architectures. The allreduce fusion kernels literally don't have SM120 CUDA code paths.

The word "literally" is significant—it signals that this is not a configuration issue, not a missing import, not a version mismatch. The CUDA code itself does not exist for SM120. The TRT-LLM communication kernels that FlashInfer relies on for fused allreduce were written for Hopper (SM90) and datacenter Blackwell (SM100) architectures, and the consumer Blackwell (SM120) was either not yet released when the kernels were written, or the architectural differences (such as smaller shared memory and different warp-level primitives) prevented a straightforward port.

The assistant then draws the correct conclusion:

Let me revert the communicator.py change — the SM90/SM100 gate was there because the kernels don't exist for SM120.

This is a mature engineering response. Rather than attempting to hack around the limitation (which would require writing new CUDA kernels for SM120, a non-trivial undertaking), the assistant accepts the constraint and pivots to investigating other bottlenecks.

Assumptions and Mistakes

Several assumptions were made and subsequently corrected:

  1. The assumption that the gate was merely a configuration oversight: The assistant initially treated the SM90/SM100 gate as an oversight that simply needed SM120 added. This was a reasonable hypothesis given that is_sm120_supported() existed and returned True. However, the gate existed for a deeper reason: the compiled kernels themselves didn't support SM120.
  2. The assumption that adding SM120 to the condition would work: The assistant assumed that the allreduce fusion code path would gracefully handle SM120 once the gate was opened. In reality, opening the gate caused a JIT compilation failure because the underlying compilation_context didn't know about SM120.
  3. The assumption that is_sm120_supported() being True meant full support: The function checks whether the CUDA device is an SM120 device, not whether all kernel paths support SM120. This is a subtle but important distinction—device detection and kernel support are separate concerns. The mistake was not in trying the fix—that was a legitimate optimization hypothesis—but in not first verifying that the underlying TRT-LLM communication kernels supported SM120 before opening the gate. A more cautious approach would have been to inspect gen_trtllm_comm_module() and its compilation_context before patching the condition.

Input Knowledge Required

To understand this message, one needs:

  1. GPU architecture numbering: SM90 = Hopper (H100), SM100 = datacenter Blackwell (B200), SM120 = consumer Blackwell (RTX PRO 6000). These are NVIDIA's compute capability versions.
  2. FlashInfer's architecture: FlashInfer is a library of high-performance CUDA kernels for transformer inference. It uses JIT compilation (via nvcc) to build kernels at runtime, and the compilation_context manages the compiler flags for different architectures.
  3. TRT-LLM communication module: trtllm_ar.py (allreduce) implements fused allreduce operations that overlap communication with computation. It relies on architecture-specific warp-level primitives and shared memory configurations.
  4. SGLang's architecture: The communicator.py file manages inter-GPU communication, and server_args.py handles server configuration. The assistant had been modifying both files.
  5. The concept of allreduce fusion: A technique that overlaps the allreduce communication (summing gradients or activations across GPUs) with ongoing computation, hiding PCIe latency by keeping GPU SMs busy.

Output Knowledge Created

This message created several important pieces of knowledge:

  1. Documented limitation: FlashInfer allreduce fusion is definitively unsupported on SM120. Future attempts to enable it will fail at JIT compilation.
  2. Root cause identification: The failure is in gen_trtllm_comm_module() in flashinfer/jit/comm.py, specifically in compilation_context.get_nvcc_flags_list() which only supports SM90 and SM100.
  3. A clean reversion: The communicator.py change was reverted cleanly, leaving the codebase in a known state.
  4. A strategic pivot point: The assistant explicitly states "we can still investigate what else is bottlenecking," signaling that the optimization effort continues but along different axes.

The Thinking Process

The assistant's reasoning in this message follows a clear pattern:

  1. Error analysis: The crash traceback pointed to gen_trtllm_comm_module() and compilation_context. The assistant correctly traced the error to its source.
  2. Root cause inference: The error message "No supported CUDA architectures found for major versions [9, 10]" combined with knowledge that SM90=9.0 and SM100=10.0 led to the conclusion that SM120 (12.0) is missing.
  3. Constraint acceptance: Rather than attempting to patch the JIT compilation context (which would require modifying FlashInfer's CUDA code generation), the assistant accepted the fundamental limitation.
  4. Clean reversion: The assistant reverted only the specific condition change, leaving the import and cached variable changes intact (they don't cause harm and may be useful for future SM120-specific code paths).
  5. Forward-looking pivot: The message ends with "we can still investigate what else is bottlenecking," demonstrating resilience and a systematic approach to optimization.

Broader Implications

This episode illustrates a recurring challenge in the ML infrastructure space: the gap between datacenter and consumer GPU architectures. NVIDIA's datacenter GPUs (H100/SM90, B200/SM100) often receive kernel support first, while consumer variants (RTX PRO 6000/SM120) lag behind. The SM120 architecture, despite being based on the same Blackwell generation as SM100, has different characteristics—smaller shared memory, different warp sizes, and potentially different tensor core configurations—that require separate kernel implementations.

For practitioners deploying on consumer Blackwell hardware, this means: