The Pivot: When Allreduce Fusion Fails on SM120

A Single Message That Captures the Art of Strategic Retreat in ML Systems Engineering

On February 19, 2026, an AI assistant working on optimizing GLM-5-NVFP4 inference across eight NVIDIA RTX PRO 6000 Blackwell GPUs sent a message that, on its surface, appears mundane: a bash command to restart an SGLang server. But this message, <msg id=803>, represents a critical inflection point in a multi-hour debugging session — a moment of strategic retreat after an ambitious optimization attempt failed catastrophically.

The Message

[assistant] Now restart the server without allreduce fusion, and focus on other optimization paths:
[bash] ssh root@10.1.230.174 "nohup bash -c 'source /root/ml-env/bin/activate && PYTHONUNBUFFERED=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 NCCL_MIN_NCHANNELS=8 OMP_NUM_THREADS=8 SAFETENSORS_FAST_GPU=1 CUDA_HOME=/usr/local/cuda-12.8 python3 -u -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp 8 --mem-fraction-static 0.92 --max-running-requests 1024 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass --disable-cuda-graph --disable-radix-cache --host 0.0.0.0 --port 8000' > /root/sglang-server.log 2>&1 &"

This is not merely a restart command. It is the culmination of a failed experiment, a documented lesson in hardware compatibility, and a deliberate reorientation of strategy. To understand its significance, we must trace the events that led to this moment.

The Context: A Desperate Bid for GPU Utilization

The assistant had been wrestling with a persistent problem: despite achieving impressive throughput of approximately 1,867 tok/s on the GLM-5-NVFP4 model, the eight RTX PRO 6000 GPUs were severely underutilized. Each GPU has a 600W TDP, yet power draw hovered around 250W — less than half of what the hardware could deliver. The bottleneck was clear: the GPUs were spending most of their time waiting for data over PCIe rather than computing.

The root cause was architectural. The machine was a Proxmox virtual machine, and each GPU was isolated on its own PCIe root complex, preventing direct peer-to-peer (P2P) memory access between GPUs. Without P2P, all inter-GPU communication — including the allreduce operations needed to synchronize gradients and activations across the eight tensor-parallel ranks — had to traverse the host CPU's memory bus and the virtualized PCIe fabric. This introduced latency that starved the compute units.

The assistant had previously identified that FlashInfer's allreduce fusion — a technique that fuses the allreduce communication with MoE (Mixture-of-Experts) computation to overlap communication with computation — was disabled on SM120 (the architecture identifier for the RTX PRO 6000's Blackwell GPU). The code in flashinfer/comm/trtllm_allreduce.cuh contained an explicit exclusion: __CUDA_ARCH__ >= 900 && __CUDA_ARCH__ < 1200. This meant that SM120 (arch 1200) fell through to fallback paths that lacked the cooperative grid synchronization primitives (cudaGridDependencySynchronize()) that make the fusion efficient.

The Failed Experiment: Patching SM120 Support

What followed was a textbook example of the "fork and modify" approach that the user had explicitly encouraged in <msg id=768>: "Please think big and don't be afraid to fork/modify code." The assistant took this directive seriously and embarked on an aggressive patching campaign across multiple codebases.

The patches touched four distinct layers:

  1. communicator.py in SGLang: The Python-level gate that checked is_sm90_supported() or is_sm100_supported() was modified to include is_sm120_supported().
  2. server_args.py in SGLang: A similar gate that controlled whether --enable-flashinfer-allreduce-fusion could be used was patched.
  3. trtllm_allreduce.cuh in FlashInfer's CUDA headers: The architecture guard __CUDA_ARCH__ >= 900 && __CUDA_ARCH__ < 1200 was changed to __CUDA_ARCH__ >= 900, removing the SM120 exclusion.
  4. JIT cache clearing: The FlashInfer JIT compilation cache was purged to force recompilation with the new architecture settings. The assistant's reasoning, visible in the investigative steps leading up to this message, was meticulous. It checked whether the CUDA source files contained SM-specific inline assembly that might break on SM120 (<msg id=774>). It verified that the fusion headers used >= 800, >= 900, and >= 1000 guards that would naturally include SM120 (<msg id=784>). It confirmed that the allreduce fusion workspace initialization succeeded across all eight ranks (<msg id=795>). The server started successfully and began serving requests. But the results were devastating.

The Catastrophic Outcome

When the assistant ran benchmarks with the allreduce fusion enabled, throughput plummeted from approximately 1,867 tok/s to just 236 tok/s — a 87% reduction. GPU power draw dropped from ~250W to ~125W, suggesting the GPUs were doing even less work than before. The assistant's own analysis in <msg id=801> identified the likely culprit: "The fusion is probably deadlocking or using an incompatible synchronization path for SM120."

The critical insight was that cudaGridDependencySynchronize() — the Hopper/Blackwell cooperative grid feature that the original < 1200 guard was protecting — does not work the same way on SM120 as it does on SM90 (H100) or SM100 (datacenter Blackwell like B200). The RTX PRO 6000, while based on the Blackwell architecture, is a consumer/professional variant with different constraints. Its SM120 compute capability may lack certain synchronization primitives or implement them differently, causing the fused allreduce kernels to stall or serialize in ways that destroy performance.

This is a crucial lesson in GPU programming: architecture version numbers are not linear indicators of capability. SM120 is not "SM100 plus 20." It is a distinct implementation with its own feature set, and the assumption that removing a < 1200 guard would simply enable a working feature was incorrect.

The Strategic Pivot

Message <msg id=803> is the response to this failure. It contains several important signals:

"Now restart the server without allreduce fusion" — This is an explicit admission that the experiment failed. The assistant is reverting the most critical change (removing --enable-flashinfer-allreduce-fusion from the launch command) while keeping all the other optimizations that were working. This is not a full rollback; it's a surgical removal of the one change that broke things.

"and focus on other optimization paths" — This phrase is the key to understanding the message's strategic importance. The assistant is not giving up on improving performance. It is acknowledging that allreduce fusion on SM120 is a dead end (at least for now) and redirecting effort toward alternative approaches. The message implicitly recognizes that there are multiple paths to better GPU utilization, and the assistant is choosing to explore them rather than continue banging on this particular wall.

The launch command itself — Every parameter in the bash command tells a story. --max-running-requests 1024 preserves the high concurrency that was working well. --moe-runner-backend flashinfer_cutlass keeps the CUTLASS-based MoE kernel that had been autotuned for SM120. --disable-cuda-graph and --disable-radix-cache remain, as they had been beneficial. The NCCL environment variables (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5, NCCL_MIN_NCHANNELS=8) are tuned for the PCIe-bound topology. The command is a carefully curated set of working optimizations, minus the one that broke.

Assumptions Made and Lessons Learned

Several assumptions underpinned the failed allreduce fusion experiment:

  1. That SM120 is a superset of SM100 features. The assistant assumed that removing the < 1200 upper bound would expose working functionality. In reality, SM120 appears to have different synchronization semantics that make the fused kernels perform poorly.
  2. That cudaGridDependencySynchronize() would work or be safely ignorable. The assistant reasoned in <msg id=781> that "the cooperative grid sync just won't be used, which means slightly less efficient synchronization but the kernel should still work." This assumption proved incorrect — the kernel did "work" in the sense of not crashing, but it performed abysmally.
  3. That the allreduce fusion code path was the primary bottleneck. The assistant had identified that GPU power was at ~250W out of 600W TDP and assumed that enabling allreduce fusion would improve utilization. The experiment disproved this — the fusion made utilization worse, suggesting that the PCIe bottleneck was not the only factor limiting performance.
  4. That patching Python-level gates and CUDA-level guards was sufficient. The assistant correctly identified all the places where SM120 was excluded, but the actual incompatibility may have been deeper — in the CUDA runtime behavior, the driver, or the hardware itself.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several forms of knowledge:

  1. A working server configuration: The bash command documents a known-good set of parameters that produce stable inference at approximately 1,867 tok/s on this specific hardware configuration.
  2. A negative result: The message implicitly documents that FlashInfer allreduce fusion is not viable on SM120 with the current codebase. This is valuable knowledge that prevents future wasted effort.
  3. A strategic direction: The phrase "focus on other optimization paths" creates a mandate for future work — explore alternative approaches rather than continuing to debug the allreduce fusion.
  4. A baseline for comparison: The server launched by this message becomes the baseline against which future optimization attempts will be measured.

The Thinking Process

The reasoning visible in the messages leading up to <msg id=803> reveals a systematic debugging methodology:

  1. Hypothesis formation: Allreduce fusion is disabled on SM120 → enabling it will improve GPU utilization.
  2. Code archaeology: Trace all the gates and guards that exclude SM120 across multiple codebases (communicator.py, server_args.py, trtllm_allreduce.cuh, trtllm_allreduce_fusion.cuh).
  3. Risk assessment: Check for inline assembly, architecture-specific intrinsics, and other potential breakage points.
  4. Controlled experiment: Apply patches, clear caches, restart, and benchmark.
  5. Outcome evaluation: The 87% throughput drop and power reduction clearly indicate failure.
  6. Revert and redirect: Remove the failing change, preserve working changes, and pivot to alternative approaches. This is a mature engineering workflow. The assistant did not over-invest in the failed approach — it recognized the failure quickly, reverted cleanly, and preserved the knowledge gained.

Conclusion

Message <msg id=803> is a masterclass in strategic engineering decision-making. It demonstrates that progress is not always about pushing forward — sometimes the most valuable action is recognizing when a path is blocked and redirecting effort elsewhere. The assistant's willingness to attempt the aggressive patch (encouraged by the user's "think big" directive), combined with its discipline to revert when the experiment failed, represents the ideal balance between ambition and pragmatism.

The message also serves as a cautionary tale about GPU architecture compatibility. The assumption that removing a version guard would unlock working functionality proved dangerously optimistic. In the world of GPU programming, architecture versions encode deep assumptions about hardware capabilities, and overriding those assumptions requires thorough validation — or acceptance that the experiment may fail.

As the assistant turns to "other optimization paths," the lessons from this failure will inform future decisions. The allreduce fusion experiment on SM120 may be revisited if the FlashInfer or TRT-LLM teams add proper SM120 support, but for now, the path forward lies elsewhere — perhaps in expert parallelism, deeper MoE kernel tuning, or accepting the PCIe bottleneck and optimizing around it.