The Moment of Truth: Benchmarking a Patched Allreduce Fusion on Blackwell SM120

The Message

ssh root@10.1.230.174 "source /root/ml-env/bin/activate && python3 -m sglang.bench_serving \
  --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 \
  --base-url http://localhost:8000 --dataset-name random --num-prompts 256 \
  --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | tail -30"

Total input text tokens:                 33093     
Total generated tokens:                  16548     
Total generated tokens (retokenized):    16536     
Request throughput (req/s):              1.22      
Input token throughput (tok/s):          157.19    
Output token throughput (tok/s):         78.60     
Peak output token throughput (tok/s):    254.00    
Peak concurrent requests:                256       
Total token throughput (tok/s):          235.79    

sleep 30 && ssh root@10.1.230.174 "nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv,noheader"
0, 100 %, 126.70 W
1, 100 %, 126.19 W
2, 100 %, 124.87 W
3, 100 %, 123.62 W
4, 100 %, 126.29 W
5, 100 %, 126.01 W
6, 100 %, 128.46 W
7, 99 %, 121.20 W

Context: The Allreduce Fusion Gamble

This message, sent by the AI assistant in an opencode coding session, represents a pivotal moment in an intense optimization campaign. The broader context is the deployment of the GLM-5-NVFP4 model—a large Mixture-of-Experts (MoE) language model quantized to NVFP4—across eight NVIDIA RTX PRO 6000 Blackwell GPUs. Over the preceding segments, the assistant had battled through driver installation, CUDA toolkit configuration, flash-attn compilation issues, NaN crashes during decode, and a fundamental PCIe topology limitation caused by running the GPUs inside a Proxmox virtual machine. Through iterative debugging and code patching, the team had achieved a respectable ~3,740 tokens per second at 1024 concurrent requests using the FlashInfer CUTLASS MoE backend.

But a puzzle remained: GPU power draw hovered around 250W per card, barely 40% of the 600W thermal design power (TDP). The hardware was starving for work. The assistant traced this to FlashInfer's allreduce fusion being disabled on SM120—the compute architecture identifier for consumer Blackwell GPUs. The fusion kernels, which combine allreduce communication with GEMM computation to reduce synchronization overhead, only supported datacenter Blackwell (SM100, corresponding to the B100/B200) and Hopper (SM90). The consumer RTX PRO 6000, despite being a Blackwell-generation GPU, uses SM120, which fell through the cracks of the architecture gates.

Why This Message Was Written

The assistant had just spent over thirty messages (indices 764–799) on a surgical patching campaign. The hypothesis was bold: the TRT-LLM allreduce fusion kernels, written in CUDA, contained no SM-specific inline PTX assembly or architecture-dependent instructions. The only barriers were software gates—#if preprocessor checks and Python version filters that explicitly excluded SM120. By patching four files across two codebases (flashinfer and sglang), the assistant hoped to unlock the fusion on consumer Blackwell hardware and close the GPU utilization gap.

The patches were:

  1. flashinfer/jit/comm.py: Changed supported_major_versions=[9, 10] to [9, 10, 12] to allow JIT compilation for SM120.
  2. flashinfer/data/include/flashinfer/comm/trtllm_allreduce.cuh: Changed __CUDA_ARCH__ >= 900 && __CUDA_ARCH__ < 1200 to __CUDA_ARCH__ >= 900, removing the exclusion.
  3. sglang/python/sglang/srt/layers/communicator.py: Extended the SM support check to include SM120.
  4. sglang/python/sglang/srt/server_args.py: Similarly extended the server argument validation. The JIT cache was cleared, the server restarted, and—against the odds—it booted successfully. The allreduce IPC handles were allocated across all eight ranks, the workspace initialized, and the server began serving requests. This was already a victory: the fusion kernels compiled and ran on SM120 without crashing, proving the code was not fundamentally incompatible. This message was written to answer the critical question: does it actually perform? The assistant ran a benchmark at 256 concurrent requests with 256-token inputs and 128-token outputs, then checked GPU power draw after 30 seconds of steady-state operation.

What the Results Revealed

The benchmark results were devastating. Total token throughput was 235.79 tok/s—a 94% collapse from the ~3,740 tok/s baseline achieved without allreduce fusion. Output token throughput was a mere 78.60 tok/s, with a peak of only 254 tok/s. The GPU utilization numbers told the real story: all eight GPUs reported 99–100% utilization, yet each was drawing only ~125W, barely 20% of their 600W TDP.

This is the signature of a synchronization bottleneck. The GPUs appear "busy" to the driver (hence 100% utilization) but are actually stalled—spinning on memory fences, waiting for inter-GPU communication over the PCIe bus, or trapped in inefficient synchronization primitives. The allreduce fusion, rather than accelerating communication, had become the bottleneck itself.

The root cause is subtle but critical. The patched CUDA header (trtllm_allreduce.cuh) contains architecture-gated code paths:

#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) && (__CUDA_ARCH__ < 1200))
  cudaGridDependencySynchronize();
#endif

By removing the &lt; 1200 restriction, the assistant inadvertently caused SM120 to fall through to the #else branch, which lacks the cudaGridDependencySynchronize() call. This CUDA cooperative grid synchronization primitive is essential for coordinating allreduce operations across multiple GPU tiles. On SM90 (Hopper) and SM100 (datacenter Blackwell), this call ensures that all participating thread blocks have reached the synchronization point before proceeding with the fused computation. On SM120, without this primitive, the kernel falls back to a less efficient synchronization mechanism—likely global memory polling or volatile loads—that serializes the GPUs and destroys throughput.

The assistant's earlier analysis (message 779) had identified these #if blocks but dismissed them as optional: "the cooperative grid sync just won't be used, which means slightly less efficient synchronization but the kernel should still work." This turned out to be a catastrophic underestimation. The synchronization primitive is not a minor optimization; it is structurally necessary for the allreduce fusion to achieve its purpose of overlapping communication with computation.

Assumptions and Their Consequences

The message exposes several assumptions that proved incorrect:

Assumption 1: Architecture gates are conservative, not functional. The assistant assumed that the __CUDA_ARCH__ &lt; 1200 restriction was a conservative choice by the flashinfer developers who hadn't tested on SM120, rather than a functional requirement. In reality, the exclusion likely exists because cudaGridDependencySynchronize() is either unavailable or behaves differently on SM120. NVIDIA's documentation for this function notes that it requires cooperative grid launch capabilities, which may have different characteristics on consumer architectures.

Assumption 2: JIT compilation success implies correctness. The kernels compiled without errors and the server initialized successfully. The assistant celebrated this as validation: "The JIT compilation worked and the TRT-LLM allreduce fusion initialized with SM120 support!" (message 795). But compilation success only proves syntactic correctness, not semantic or performance correctness. The kernels ran, but ran pathologically.

Assumption 3: The PCIe bottleneck would be mitigated by fusion. The entire motivation for enabling allreduce fusion was to reduce the impact of PCIe-based inter-GPU communication, which was identified as the primary bottleneck in earlier segments. The assistant assumed that fusing allreduce with GEMM would hide the PCIe latency. Instead, the fusion introduced its own synchronization overhead that dwarfed the original problem.

Assumption 4: "Slightly less efficient" synchronization is acceptable. The assistant's own words reveal the miscalculation: "the cooperative grid sync just won't be used, which means slightly less efficient synchronization." The 94% throughput collapse proves the inefficiency is not slight—it is catastrophic.

Input Knowledge Required

To fully understand this message, one needs:

  1. The optimization history: That the baseline was ~3,740 tok/s without fusion, and that GPU power was stuck at 250W. Without this context, 235 tok/s might seem like a reasonable starting point rather than a disaster.
  2. The architecture distinction: That SM120 (consumer Blackwell, RTX PRO 6000) and SM100 (datacenter Blackwell, B100/B200) are different compute architectures despite both being "Blackwell." Consumer GPUs have different synchronization primitives, smaller shared memory, and different performance characteristics.
  3. The allreduce fusion mechanism: That FlashInfer's TRT-LLM allreduce fusion combines the allreduce communication step with the subsequent GEMM computation, using cooperative grid synchronization to ensure all GPUs have the reduced data before computation begins. This is fundamentally different from a naive allreduce followed by separate GEMM.
  4. The PCIe topology constraint: That the eight GPUs are in a Proxmox VM with each GPU on its own PCIe root complex, preventing P2P DMA. All inter-GPU communication goes through the host memory, adding latency.
  5. The benchmark methodology: The bench_serving tool measures end-to-end serving throughput including prefill and decode phases. The 256-input/128-output configuration is a standard short-context benchmark.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. Empirical proof that SM120 allreduce fusion is non-functional: The patched kernels compile and run but deliver 235 tok/s versus 3,740 tok/s baseline—a 94% degradation. This is a definitive negative result that saves future engineers from repeating this experiment.
  2. GPU power as a diagnostic signal: The ~125W power draw at 100% utilization is a clear indicator of synchronization stall. Normal operation at this concurrency should show 400-600W per GPU. The combination of high utilization and low power is diagnostic of a synchronization-bound workload.
  3. Validation of the architecture gate's purpose: The __CUDA_ARCH__ &lt; 1200 exclusion in the flashinfer codebase was not arbitrary. It reflects a real functional limitation of SM120's cooperative grid synchronization capabilities. This validates the original developers' caution.
  4. A benchmark methodology for fusion evaluation: The paired benchmark + power measurement provides a template for evaluating communication fusion strategies. Throughput alone can be misleading; power efficiency reveals whether the GPUs are actually doing useful work.

The Thinking Process Visible in This Message

The message itself is terse—two bash commands and their output—but it speaks volumes about the assistant's thinking process. The assistant chose to run the benchmark at 256 concurrency rather than the 1024 concurrency that produced the peak 3,740 tok/s baseline. This suggests a conservative approach: validate at moderate load before scaling up. The 30-second sleep before the power check indicates an understanding that GPU power needs steady-state measurement, not transient spikes.

The assistant did not include any commentary or analysis in the message itself—just raw data. This is characteristic of a scientific mindset: let the numbers speak first. The subsequent messages (not shown in this article) would have contained the analysis, but the decision to present unadorned results shows confidence that the data would be self-explanatory to the human collaborator.

The choice of benchmark parameters is also revealing. Random input/output lengths of 256/128 with 256 prompts creates a workload that stresses both prefill (compute-bound) and decode (memory-bound) phases. The infinite request rate (--request-rate inf) ensures the server is maximally loaded, eliminating client-side bottlenecks. This is a well-designed benchmark for isolating server performance.

Conclusion

Message 800 is a moment of intellectual honesty in the optimization journey. After dozens of messages spent patching code, clearing caches, and celebrating a successful server startup, the assistant ran the benchmark and let the numbers tell the truth: the allreduce fusion experiment had failed. The 235 tok/s result and 125W power draw are not just data points—they are a diagnosis. The GPUs are stalled, not computing.

This message teaches a broader lesson about systems optimization: compilation success is not performance success. A kernel can compile, initialize, and run without crashing while delivering abysmal throughput. The architecture gates that the assistant so confidently removed were not bureaucratic restrictions; they were functional guardrails protecting against exactly this scenario.

The message also demonstrates the value of comprehensive benchmarking. Had the assistant only checked that the server started (message 795) and that it could serve a single request, the performance disaster might have gone unnoticed. The systematic benchmark with power measurement revealed the truth.

For the reader following this optimization saga, message 800 is the turning point where the allreduce fusion path is definitively closed, forcing a return to the drawing board. The next steps—exploring NCCL tuning, decode step optimization, and alternative MoE backends—would be shaped by the hard data produced in this message.