The Blackwell Divide: From 880 to 3,740 tok/s and the Allreduce Fusion Wall

Introduction

In the high-stakes world of large language model inference optimization, few experiences are as instructive as the collision between a dramatic performance breakthrough and an immovable hardware wall. This article chronicles a single, intense segment of an opencode coding session — spanning over a hundred messages — where an AI assistant and a human user pushed a 744-billion-parameter Mixture-of-Experts (MoE) model, GLM-5-NVFP4, from a modest ~880 tokens per second to a blistering ~3,740 tok/s on eight NVIDIA RTX PRO 6000 Blackwell GPUs. But the story does not end with that triumph. The very act of achieving that throughput exposed a deeper, more troubling problem: the GPUs were barely breaking a sweat, drawing only ~250W out of their 600W thermal design power. The hardware was loafing while the software stack, designed for datacenter Blackwell (SM100), silently failed to engage the consumer Blackwell (SM120) architecture's full potential.

What followed was a deep investigative spiral through multiple layers of the software stack — from Python runtime checks, through JIT compilation orchestration, down to CUDA preprocessor directives and inline PTX assembly. The assistant systematically dismantled each architecture gate, only to discover that some hardware boundaries cannot be crossed with software patches alone. This is a story about the hidden fault lines in modern AI infrastructure — the assumptions baked into open-source frameworks about which GPU architectures matter, and the engineering detective work required to uncover and confront those assumptions when deploying on non-datacenter hardware.

The Starting Point: A Plateau at 880 tok/s

The session opened with a familiar frustration. The assistant had successfully deployed GLM-5-NVFP4 — a massive MoE model with 256 experts, NVFP4 quantization, and DeepSeek Sparse Attention — on eight RTX PRO 6000 GPUs in a Proxmox LXC container. The infrastructure migration from KVM to LXC had been arduous, involving NVIDIA driver reinstallation, CUDA initialization fixes, and P2P bandwidth validation. But the payoff was underwhelming: throughput plateaued at roughly 880 total tokens per second at 256 concurrency, and single-stream performance remained stuck at ~11 tok/s — identical to the virtualized setup they had left behind.

The user's observation was pointed: the GPUs were drawing far less power than their rated capacity, and PCIe bandwidth utilization was low. Something was starving the compute units. The assistant's todo list at this point identified three critical differences between the current GLM-5 deployment and a prior successful Kimi K2-Thinking run on the same hardware that had achieved 5,816 tok/s peak: a severely limited concurrency cap of 64 max-running-requests, the use of CUDA graphs (which the K2 run had disabled), and the absence of FlashInfer CUTLASS MoE autotune.

The Breakthrough: Patching the Autotune Gate

The first major intervention came when the assistant traced the FlashInfer autotune logic in SGLang's model_runner.py ([msg 671]). There, on lines 1837-1838, was a revealing comment:

# TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed.
# "flashinfer_cutlass",

The flashinfer_cutlass MoE backend had been explicitly excluded from the autotune list, with a TODO note warning of compilation errors. The assistant recognized that this gate, likely added for a different architecture or software version, might not apply to SM120 Blackwell. The decision to patch it was a calculated gamble — the autotune could crash the server, or it could unlock significant performance.

After a failed attempt with a multiline sed command ([msg 674]), the assistant succeeded with a targeted single-line substitution ([msg 677]):

sed -i '1838s|# "flashinfer_cutlass",|"flashinfer_cutlass",|' /root/sglang/python/sglang/srt/model_executor/model_runner.py

The verification in [msg 678] confirmed the patch had taken effect. The assistant then killed the running server and restarted with a dramatically different configuration: --max-running-requests 1024 (raised from 64), --disable-cuda-graph, and --disable-radix-cache ([msg 679]).

The results were immediate and stunning. At 256 concurrency, throughput jumped from ~880 tok/s to 1,950 tok/s — more than double the previous peak ([msg 695]). The assistant's excitement was palpable: "1,950 tok/s at 256 concurrency!! That's more than double what we had before (879 tok/s)!"

The Crash That Wasn't OOM

But triumph was short-lived. When the assistant pushed to 512 concurrency, the server crashed with a cryptic error: 'PrefillMetadata' object has no attribute 'page_table_1_flattened' ([msg 696]). This was not an out-of-memory error as initially suspected — it was a bug in the NSA (Native Sparse Attention) attention code path, triggered by chunked prefill with large batches when the radix cache (prefix sharing) was enabled.

The assistant's response was methodical. Rather than trying to fix the NSA attention bug directly — which would require deep understanding of a complex code path — the assistant chose a pragmatic workaround: disable the radix cache entirely ([msg 710]). This was a safe trade-off: the radix cache optimizes for shared prefixes across requests, but in a benchmark with random prompts, there are no shared prefixes to exploit. The feature was adding complexity and risk without benefit.

With the radix cache disabled, the server stabilized. The assistant then ran a systematic concurrency sweep: 256, 512, and finally 1,024 concurrent requests. The results told a clear scaling story:

| Concurrency | Total tok/s | Output tok/s | Peak Output | |-------------|-------------|--------------|-------------| | 256 | 1,950 | ~700 | ~1,300 | | 512 | 2,800 | ~1,240 | ~2,350 | | 1024 | 3,740 | ~1,240 | ~3,945 |

At 1,024 concurrency, the server achieved 3,740 total tokens per second with a peak output burst of nearly 4,000 tok/s ([msg 714]). This was a 4.25× improvement over the starting baseline. The assistant had every reason to celebrate.## The 87-Watt Revelation

But the celebration was premature. In [msg 716], the assistant ran a routine diagnostic check:

nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv,noheader

The output was devastating:

0, 0 %, 87.92 W
1, 0 %, 89.24 W
2, 0 %, 85.91 W
3, 0 %, 86.60 W
4, 0 %, 89.38 W
5, 0 %, 88.11 W
6, 0 %, 90.07 W
7, 0 %, 85.52 W

All eight GPUs were drawing approximately 87 watts each — barely 14.5% of their 600W TDP. GPU utilization was reported as 0%. The 3,740 tok/s was being achieved despite the hardware being almost completely idle. This was the first concrete evidence that the bottleneck was not in GPU compute at all, but somewhere in the data path between the CPU, the network, and the GPUs.

The user's response in [msg 723] sharpened the diagnosis dramatically. The user corrected the assistant's assumption that the RTX PRO 6000 had a 300W TDP — it was actually 600W. More importantly, the user introduced critical hardware knowledge: the RTX PRO 6000 uses the SM120 architecture, which differs from the datacenter Blackwell SM100 in fundamental ways. The most critical difference was shared memory: SM120 has only 100KB per SM, compared to 228KB on SM100. The user provided a research document (sm120-attention-fix-research.md) that catalogued the full range of SM120-specific issues, from attention kernel block sizes to MoE kernel layout requirements.

This was the moment the optimization problem transformed. It was no longer about tweaking server parameters — it was about understanding why the software stack, designed for datacenter Blackwell, was failing to engage consumer Blackwell hardware.

The Allreduce Fusion Gap

The assistant launched a subagent task ([msg 729]) to perform a comprehensive analysis of the GLM-5 forward pass. The subagent traced the entire compute path — from NVFP4 quantized MoE expert GEMMs through attention mechanisms to allreduce communication — and delivered a critical finding: FlashInfer's allreduce fusion was disabled on SM120.

Allreduce fusion is a technique that overlaps the allreduce communication operation (synchronizing gradients across GPUs) with ongoing computation. In a standard transformer forward pass, each of the ~78 layers requires two allreduce operations (one for attention, one for MoE routing), totaling ~156 allreduces per forward pass. Without fusion, the GPU's streaming multiprocessors sit idle during each PCIe transfer. With fusion, the communication runs in the background while compute continues.

The code gate was in communicator.py:

(_is_sm90_supported or _is_sm100_supported)

SM120 was simply not in the list. The assistant identified four fixes needed across two files ([msg 740]):

  1. communicator.py: Add is_sm120_supported import and cached variable
  2. communicator.py: Extend the allreduce fusion condition to include SM120
  3. server_args.py: Add SM120 to the allreduce fusion auto-enable logic
  4. server_args.py: Add SM120 to the MoE runner backend auto-selection The assistant applied the patches with surgical sed commands and restarted the server with --enable-flashinfer-allreduce-fusion explicitly set ([msg 751]).

The Crash That Defined the Boundary

The server never started. The user's report was a single word: "crashed" ([msg 753]).

The assistant's diagnostic grep ([msg 754]) revealed the root cause:

RuntimeError: No supported CUDA architectures found for major versions [9, 10].

This error originated from FlashInfer's JIT compilation module — specifically, flashinfer/jit/comm.py, which builds a TRT-LLM communication module at runtime. The compilation context only supported CUDA architectures SM90 (Hopper, e.g., H100) and SM100 (datacenter Blackwell, e.g., B200). SM120 — the architecture of the RTX PRO 6000 — was not in the list. The allreduce fusion kernels simply did not exist for this architecture.

This was a hard wall. The assistant's patches to the Python-level gates had removed the "do not enter" signs, but the underlying compiled kernels were never built for SM120. The gate existed for a reason: the TRT-LLM communication kernels that implement the actual allreduce fusion operation are architecture-specific CUDA code, and nobody has compiled them for SM120.

The assistant initially reverted the changes, concluding that "the SM90/SM100 gate was there because the kernels don't exist for SM120" ([msg 756]). The todo list was updated with a cancelled item: "Fix FlashInfer allreduce fusion for SM120 — BLOCKED (kernel not compiled for SM120)."

The Permission to Fork

Then came a pivotal intervention from the user. In [msg 768], the user wrote: "Please think big and don't be afraid to fork/modify code." This single sentence transformed the session from a configuration-tuning exercise into a deep systems hacking investigation. The assistant was no longer limited to working within the constraints of upstream dependencies — it could modify them directly.

This directive was the catalyst for everything that followed. The assistant re-examined the problem with fresh eyes, tracing the error from the Python-level crash back through the JIT compilation system and into the actual CUDA kernel source code.## Tracing the Gates: A Multi-Layer Investigation

What emerged was a systematic campaign of source-level investigation and patching across two repositories. The assistant discovered that the allreduce fusion was blocked not by one gate but by a cascade of them, each at a different layer of abstraction.

Layer 1: The JIT Compilation Gate. The first barrier was in flashinfer/jit/comm.py, where gen_trtllm_comm_module() called get_nvcc_flags_list(supported_major_versions=[9, 10]). This Python-level filter explicitly rejected SM120. In [msg 766], the assistant patched this to supported_major_versions=[9, 10, 12], adding SM120 to the allowed list.

Layer 2: The Python Runtime Gates. The SGLang server had two additional checks: communicator.py checked (_is_sm90_supported or _is_sm100_supported) before enabling fusion, and server_args.py had a similar gate for auto-detection. Both were patched to include _is_sm120_supported ([msg 767], [msg 769]).

Layer 3: The CUDA Preprocessor Gate. This was the most critical discovery. By reading the actual CUDA header files (<msg id=779-780>), the assistant found that trtllm_allreduce.cuh contained:

#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) && (__CUDA_ARCH__ < 1200))
  cudaGridDependencySynchronize();
#endif

The condition __CUDA_ARCH__ &lt; 1200 explicitly excluded SM120 (where __CUDA_ARCH__ equals 1200) from using cudaGridDependencySynchronize(), a cooperative grid synchronization primitive available on Hopper and datacenter Blackwell. The assistant analyzed the surrounding code and determined that this synchronization call was an optimization, not a correctness requirement — the core allreduce logic was identical regardless of the architecture gate. In [msg 781], the patch was applied: the condition was changed to simply __CUDA_ARCH__ &gt;= 900, removing the upper bound.

Layer 4: Verification of Related Headers. The assistant didn't stop with the primary kernel header. It systematically checked every related CUDA source file — trtllm_allreduce_fusion.cuh, trtllm_moe_allreduce_fusion.cuh, and the .cu source files — for additional SM120 exclusions. The .cu files contained zero SM-specific architecture guards ([msg 774]), confirming the kernels were written in portable CUDA C++. The fusion headers used &gt;= 800 and &gt;= 1000 guards which naturally included SM120 (arch 1200). No &lt; 1200 exclusions were found elsewhere.

Layer 5: The SGLang Python Integration Layer. Finally, the assistant checked flashinfer_comm_fusion.py — SGLang's Python wrapper for the fusion functionality — and confirmed it contained no SM-specific gates (<msg id=787-788>). The Python layer simply called into the compiled FlashInfer module without any architecture checks.

The Moment of Truth: Launch and Benchmark

With all gates patched and the JIT cache cleared ([msg 786]), the assistant launched the server in [msg 790] with --enable-flashinfer-allreduce-fusion and a carefully tuned set of environment variables encoding hard-won knowledge about the PCIe topology.

The initial monitoring was confusing. A bash loop in [msg 792] returned "CRASHED" after detecting a sigquit signal in the log. But when the assistant investigated further in <msg id=793-794>, it discovered the server was actually still loading — the "CRASHED" signal was a false positive caused by stale log entries from a previous server instance.

In [msg 795], the assistant confirmed that the allreduce IPC handles had been allocated successfully across all 8 ranks. The JIT compilation had worked. The TRT-LLM allreduce fusion had initialized with SM120 support for the first time. The server reached "Uvicorn running on http://0.0.0.0:8000" and began serving requests.

But the benchmark results in [msg 800] told a devastating story: total token throughput dropped to 236 tok/s — a catastrophic regression from the ~3,740 tok/s achieved without allreduce fusion. GPU power draw fell to ~125W, barely a fifth of the 600W TDP. The fusion was functional but catastrophically slow.

The Diagnosis: Why It Failed

The assistant's diagnosis in [msg 801] was prescient: the cudaGridDependencySynchronize() calls that were gated behind the &lt; 1200 exclusion were not merely optimizations — they were essential for efficient allreduce on multi-GPU systems. By removing the upper bound, the assistant had enabled these calls on SM120, but the consumer Blackwell architecture may not support cooperative grid dependencies with the same efficiency as datacenter Blackwell. The synchronization primitive that worked on SM100 was causing excessive serialization or stalls on SM120.

The assistant reverted the allreduce fusion changes in [msg 802] and pivoted to alternative optimization strategies: NCCL tuning, --num-continuous-decode-steps 4, and the flashinfer_trtllm MoE backend (<msg id=802-809>).

The Aftermath: Exploring Alternative Paths

With the allreduce fusion path closed, the assistant pivoted to alternative optimizations:

  1. NCCL tuning: Adjusting NCCL parameters to optimize PCIe communication
  2. --num-continuous-decode-steps 4: Batching multiple decode steps together to amortize overhead
  3. The flashinfer_trtllm MoE backend: An alternative MoE kernel implementation that might offer better SM120 support None of these yielded meaningful gains. The fundamental limitation remained: the inference stack was designed for datacenter Blackwell (SM100) and had not been adapted for the consumer Blackwell (SM120) architecture.

Lessons Learned

This episode teaches several enduring lessons about adapting ML inference stacks across hardware generations.

Architecture gates are usually there for a reason. When upstream developers explicitly exclude an architecture from a feature, it is rarely an oversight. The &lt; 1200 gate in trtllm_allreduce.cuh reflected real differences in how SM120 handles cooperative grid synchronization. The assistant's patches were technically correct — the kernel compiled and ran — but functionally incorrect, as performance was worse than without fusion.

JIT compilation success does not guarantee runtime success. The allreduce fusion initialized without errors, allocated IPC handles, and began processing requests. But the runtime behavior was pathological. This is a reminder that "it compiles" is a necessary but not sufficient condition for correctness in GPU programming.

Consumer and datacenter GPUs are diverging. The SM100 vs. SM120 distinction is not just about core count or memory bandwidth. Datacenter Blackwell GPUs have hardware features — like enhanced cooperative grid synchronization — that consumer Blackwell GPUs lack. Inference stacks designed for datacenter hardware cannot simply be "ported" to consumer hardware by removing architecture gates; they may depend on hardware capabilities that don't exist on the consumer SKUs.

The value of systematic verification. The assistant's methodical approach — tracing the error from Python through JIT compilation to CUDA preprocessor directives, verifying each layer, and checking every related file — is a textbook example of debugging a complex system. The grep commands, the sed patches, the cache clearing, and the monitoring loops all contributed to a disciplined investigation that, even when it led to a dead end, produced valuable knowledge about SM120's capabilities and limitations.

The Broader Lesson: Architecture-Specific Kernel Support

This session illuminates a critical challenge in the AI infrastructure landscape. The RTX PRO 6000 uses the SM120 architecture, which is a "consumer Blackwell" variant distinct from the datacenter-focused SM100 (B200/B100). While both share the Blackwell generation name, they have different compute capabilities, different shared memory sizes, and — crucially — different kernel support in the open-source ecosystem.

FlashInfer, SGLang, and the broader inference stack were primarily developed and tested on datacenter hardware (H100/SM90, B200/SM100). Consumer and workstation SKUs often lag behind in kernel coverage, creating a gap that must be bridged by either upstream contributions or creative workarounds. The assistant's approach of forking and patching both sglang and flashinfer code demonstrates a willingness to modify upstream dependencies to unlock performance, but the allreduce fusion attempt highlights the risk of using architecture-specific synchronization primitives without thorough validation.

The key findings from this session can be summarized as:

  1. The MoE autotune patch worked: Enabling flashinfer_cutlass autotune for SM120 unlocked a 4× throughput improvement, from ~880 to ~3,740 tok/s. This was a pure software gate that could be safely opened.
  2. The allreduce fusion patch failed: Adding SM120 support to the allreduce fusion gate caused a catastrophic performance regression because the underlying compiled CUDA kernels rely on synchronization primitives that are either absent or inefficient on SM120.
  3. GPU utilization remains the unsolved problem: Even at 3,740 tok/s, the GPUs draw only ~250W out of 600W TDP. The allreduce fusion gap is a significant contributor, but there may be other factors — PCIe topology, kernel efficiency, scheduling overhead — that also limit utilization.
  4. SM120 is not SM100: The assumption that all Blackwell GPUs have similar characteristics is incorrect. SM120 has less than half the shared memory of SM100 (100KB vs 228KB), different optimal kernel configurations, and missing support for certain features like TMA (Tensor Memory Accelerator) block layouts.

The Path Forward

The session ends with the assistant at a crossroads. The allreduce fusion path is closed unless someone ports the TRT-LLM communication kernels to SM120 — a significant engineering effort. The alternative paths — NCCL tuning, decode step batching, alternative MoE backends — have not yet yielded gains. The most promising remaining directions include:

Conclusion

This segment of the opencode session captures a complete arc of ML inference optimization: from frustration at a performance plateau, through a dramatic breakthrough enabled by a single-line code patch, to the sobering discovery that the breakthrough masked a deeper hardware utilization problem, and finally to the confrontation with a hard architectural boundary that no amount of Python-level patching could cross.

The 3,740 tok/s milestone was real and significant — a 4.25× improvement over the starting baseline. But it was also deceptive. The GPUs were capable of much more, and the software stack was the limiting factor. The allreduce fusion gap, rooted in the difference between SM100 and SM120 architectures, represents a fundamental challenge that will require kernel-level work to resolve.

For practitioners deploying large models on consumer or workstation Blackwell hardware, this session offers both hope and caution. The hope: significant performance gains are available through careful configuration and targeted code patches. The caution: the inference stack's support for SM120 is incomplete, and some optimizations that work on datacenter hardware may simply not be available. The gap between "it compiles" and "it runs efficiently" is wider than it appears, and bridging it requires not just engineering effort but a deep understanding of GPU architecture.

The Blackwell divide is real. But as this session demonstrates, it is not insurmountable — it just requires the right combination of diagnostic rigor, willingness to patch upstream code, and acceptance of hardware-imposed boundaries.## References

[1] "The Blackwell Divide: How a 4× Throughput Breakthrough Revealed the Hidden Gap Between Consumer and Datacenter GPU Inference" — Chunk article covering the initial throughput optimization and the discovery of GPU underutilization.

[2] "The Allreduce Fusion Odyssey: Patching Through Layers of Architecture Gates to Unlock SM120 Performance" — Chunk article covering the deep investigation into allreduce fusion gates across the software stack.

[3] SGLang PR #16975 — MoE kernel fix for SM120 using StridedLayout, non-persistent kernels, and reduced pipeline stages.

[4] FlashInfer SM120 Support Issues — GitHub issues #1147, #2166 tracking incomplete SM120 support in FlashInfer.

[5] vLLM SM120 PR #31089 — Pattern for MXFP4 support on SM120 using is_persistent = False and num_stages = 1.

[6] NVIDIA CUDA Programming Guide — Architecture-specific features including cudaGridDependencySynchronize() for cooperative grid synchronization.