The Blackwell Divide: How a 4× Throughput Breakthrough Revealed the Hidden Gap Between Consumer and Datacenter GPU Inference

Introduction

In the rapidly evolving landscape of large language model inference, the line between "it works" and "it works fast" is often drawn in the sand of hardware-specific kernel support. This article chronicles a single, intense optimization session spanning over a hundred messages in an opencode conversation, where an AI assistant and a human user pushed a 744-billion-parameter Mixture-of-Experts (MoE) model — GLM-5-NVFP4 — from a modest ~880 tokens per second to a blistering ~3,740 tok/s on eight NVIDIA RTX PRO 6000 Blackwell GPUs. But the story does not end with that triumph. The very act of achieving that throughput exposed a deeper, more troubling problem: the GPUs were barely breaking a sweat, drawing only ~250W out of their 600W thermal design power. The hardware was loafing while the software stack, designed for datacenter Blackwell (SM100), silently failed to engage the consumer Blackwell (SM120) architecture's full potential.

This is a story about the hidden fault lines in modern AI infrastructure — the assumptions baked into open-source frameworks about which GPU architectures matter, and the engineering detective work required to uncover and confront those assumptions when deploying on non-datacenter hardware.

The Starting Point: A Plateau at 880 tok/s

The session opened with a familiar frustration. The assistant had successfully deployed GLM-5-NVFP4 — a massive MoE model with 256 experts, NVFP4 quantization, and DeepSeek Sparse Attention — on eight RTX PRO 6000 GPUs in a Proxmox LXC container. The infrastructure migration from KVM to LXC had been arduous, involving NVIDIA driver reinstallation, CUDA initialization fixes, and P2P bandwidth validation. But the payoff was underwhelming: throughput plateaued at roughly 880 total tokens per second at 256 concurrency, and single-stream performance remained stuck at ~11 tok/s — identical to the virtualized setup they had left behind.

The user's observation was pointed: the GPUs were drawing far less power than their rated capacity, and PCIe bandwidth utilization was low. Something was starving the compute units. The assistant's todo list at this point identified three critical differences between the current GLM-5 deployment and a prior successful Kimi K2-Thinking run on the same hardware that had achieved 5,816 tok/s peak: a severely limited concurrency cap of 64 max-running-requests, the use of CUDA graphs (which the K2 run had disabled), and the absence of FlashInfer CUTLASS MoE autotune.

The Breakthrough: Patching the Autotune Gate

The first major intervention came when the assistant traced the FlashInfer autotune logic in SGLang's model_runner.py ([msg 671]). There, on lines 1837-1838, was a revealing comment:

# TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed.
# "flashinfer_cutlass",

The flashinfer_cutlass MoE backend had been explicitly excluded from the autotune list, with a TODO note warning of compilation errors. The assistant recognized that this gate, likely added for a different architecture or software version, might not apply to SM120 Blackwell. The decision to patch it was a calculated gamble — the autotune could crash the server, or it could unlock significant performance.

After a failed attempt with a multiline sed command ([msg 674]), the assistant succeeded with a targeted single-line substitution ([msg 677]):

sed -i '1838s|# "flashinfer_cutlass",|"flashinfer_cutlass",|' /root/sglang/python/sglang/srt/model_executor/model_runner.py

The verification in [msg 678] confirmed the patch had taken effect. The assistant then killed the running server and restarted with a dramatically different configuration: --max-running-requests 1024 (raised from 64), --disable-cuda-graph, and --disable-radix-cache ([msg 679]).

The results were immediate and stunning. At 256 concurrency, throughput jumped from ~880 tok/s to 1,950 tok/s — more than double the previous peak ([msg 695]). The assistant's excitement was palpable: "1,950 tok/s at 256 concurrency!! That's more than double what we had before (879 tok/s)!"

The Crash That Wasn't OOM

But triumph was short-lived. When the assistant pushed to 512 concurrency, the server crashed with a cryptic error: 'PrefillMetadata' object has no attribute 'page_table_1_flattened' ([msg 696]). This was not an out-of-memory error as initially suspected — it was a bug in the NSA (Native Sparse Attention) attention code path, triggered by chunked prefill with large batches when the radix cache (prefix sharing) was enabled.

The assistant's response was methodical. Rather than trying to fix the NSA attention bug directly — which would require deep understanding of a complex code path — the assistant chose a pragmatic workaround: disable the radix cache entirely ([msg 710]). This was a safe trade-off: the radix cache optimizes for shared prefixes across requests, but in a benchmark with random prompts, there are no shared prefixes to exploit. The feature was adding complexity and risk without benefit.

With the radix cache disabled, the server stabilized. The assistant then ran a systematic concurrency sweep: 256, 512, and finally 1,024 concurrent requests. The results told a clear scaling story:

| Concurrency | Total tok/s | Output tok/s | Peak Output | |-------------|-------------|--------------|-------------| | 256 | 1,950 | ~700 | ~1,300 | | 512 | 2,800 | ~1,240 | ~2,350 | | 1024 | 3,740 | ~1,240 | ~3,945 |

At 1,024 concurrency, the server achieved 3,740 total tokens per second with a peak output burst of nearly 4,000 tok/s ([msg 714]). This was a 4.25× improvement over the starting baseline. The assistant had every reason to celebrate.

The 87-Watt Revelation

But the celebration was premature. In [msg 716], the assistant ran a routine diagnostic check:

nvidia-smi --query-gpu=index,utilization.gpu,power.draw --format=csv,noheader

The output was devastating:

0, 0 %, 87.92 W
1, 0 %, 89.24 W
2, 0 %, 85.91 W
3, 0 %, 86.60 W
4, 0 %, 89.38 W
5, 0 %, 88.11 W
6, 0 %, 90.07 W
7, 0 %, 85.52 W

All eight GPUs were drawing approximately 87 watts each — barely 14.5% of their 600W TDP. GPU utilization was reported as 0%. The 3,740 tok/s was being achieved despite the hardware being almost completely idle. This was the first concrete evidence that the bottleneck was not in GPU compute at all, but somewhere in the data path between the CPU, the network, and the GPUs.

The user's response in [msg 723] sharpened the diagnosis dramatically. The user corrected the assistant's assumption that the RTX PRO 6000 had a 300W TDP — it was actually 600W. More importantly, the user introduced critical hardware knowledge: the RTX PRO 6000 uses the SM120 architecture, which differs from the datacenter Blackwell SM100 in fundamental ways. The most critical difference was shared memory: SM120 has only 100KB per SM, compared to 228KB on SM100. The user provided a research document (sm120-attention-fix-research.md) that catalogued the full range of SM120-specific issues, from attention kernel block sizes to MoE kernel layout requirements.

This was the moment the optimization problem transformed. It was no longer about tweaking server parameters — it was about understanding why the software stack, designed for datacenter Blackwell, was failing to engage consumer Blackwell hardware.

The Allreduce Fusion Gap

The assistant launched a subagent task ([msg 729]) to perform a comprehensive analysis of the GLM-5 forward pass. The subagent traced the entire compute path — from NVFP4 quantized MoE expert GEMMs through attention mechanisms to allreduce communication — and delivered a critical finding: FlashInfer's allreduce fusion was disabled on SM120.

Allreduce fusion is a technique that overlaps the allreduce communication operation (synchronizing gradients across GPUs) with ongoing computation. In a standard transformer forward pass, each of the ~78 layers requires two allreduce operations (one for attention, one for MoE routing), totaling ~156 allreduces per forward pass. Without fusion, the GPU's streaming multiprocessors sit idle during each PCIe transfer. With fusion, the communication runs in the background while compute continues.

The code gate was in communicator.py:

(_is_sm90_supported or _is_sm100_supported)

SM120 was simply not in the list. The assistant identified four fixes needed across two files ([msg 740]):

  1. communicator.py: Add is_sm120_supported import and cached variable
  2. communicator.py: Extend the allreduce fusion condition to include SM120
  3. server_args.py: Add SM120 to the allreduce fusion auto-enable logic
  4. server_args.py: Add SM120 to the MoE runner backend auto-selection The assistant applied the patches with surgical sed commands and restarted the server with --enable-flashinfer-allreduce-fusion explicitly set ([msg 751]).

The Crash That Defined the Boundary

The server never started. The user's report was a single word: "crashed" ([msg 753]).

The assistant's diagnostic grep ([msg 754]) revealed the root cause:

RuntimeError: No supported CUDA architectures found for major versions [9, 10].

This error originated from FlashInfer's JIT compilation module — specifically, flashinfer/jit/comm.py, which builds a TRT-LLM communication module at runtime. The compilation context only supported CUDA architectures SM90 (Hopper, e.g., H100) and SM100 (datacenter Blackwell, e.g., B200). SM120 — the architecture of the RTX PRO 6000 — was not in the list. The allreduce fusion kernels simply did not exist for this architecture.

This was a hard wall. The assistant's patches to the Python-level gates had removed the "do not enter" signs, but the underlying compiled kernels were never built for SM120. The gate existed for a reason: the TRT-LLM communication kernels that implement the actual allreduce fusion operation are architecture-specific CUDA code, and nobody has compiled them for SM120.

The assistant had to revert all three patches. The SM90/SM100 gate in communicator.py was restored. The auto-enable logic in server_args.py was reverted. The fundamental assumption — that allreduce fusion could be enabled for SM120 by simply changing a few conditional checks — was wrong.

The Aftermath: Exploring Alternative Paths

With the allreduce fusion path closed, the assistant pivoted to alternative optimizations. The chunk concludes with the assistant exploring:

  1. NCCL tuning: Adjusting NCCL parameters to optimize PCIe communication
  2. --num-continuous-decode-steps 4: Batching multiple decode steps together to amortize overhead
  3. The flashinfer_trtllm MoE backend: An alternative MoE kernel implementation that might offer better SM120 support None of these yielded meaningful gains. The fundamental limitation remained: the inference stack was designed for datacenter Blackwell (SM100) and had not been adapted for the consumer Blackwell (SM120) architecture.

The Broader Lesson: Architecture-Specific Kernel Support

This session illuminates a critical challenge in the AI infrastructure landscape. The RTX PRO 6000 uses the SM120 architecture, which is a "consumer Blackwell" variant distinct from the datacenter-focused SM100 (B200/B100). While both share the Blackwell generation name, they have different compute capabilities, different shared memory sizes, and — crucially — different kernel support in the open-source ecosystem.

FlashInfer, SGLang, and the broader inference stack were primarily developed and tested on datacenter hardware (H100/SM90, B200/SM100). Consumer and workstation SKUs often lag behind in kernel coverage, creating a gap that must be bridged by either upstream contributions or creative workarounds. The assistant's approach of forking and patching both sglang and flashinfer code demonstrates a willingness to modify upstream dependencies to unlock performance, but the allreduce fusion attempt highlights the risk of using architecture-specific synchronization primitives without thorough validation.

The key findings from this session can be summarized as:

  1. The MoE autotune patch worked: Enabling flashinfer_cutlass autotune for SM120 unlocked a 4× throughput improvement, from ~880 to ~3,740 tok/s. This was a pure software gate that could be safely opened.
  2. The allreduce fusion patch failed: Adding SM120 support to the allreduce fusion gate caused a server crash because the underlying compiled CUDA kernels do not support SM120. This was a genuine hardware compatibility limitation, not a conservative software gate.
  3. GPU utilization remains the unsolved problem: Even at 3,740 tok/s, the GPUs draw only ~250W out of 600W TDP. The allreduce fusion gap is a significant contributor, but there may be other factors — PCIe topology, kernel efficiency, scheduling overhead — that also limit utilization.
  4. SM120 is not SM100: The assumption that all Blackwell GPUs have similar characteristics is incorrect. SM120 has less than half the shared memory of SM100 (100KB vs 228KB), different optimal kernel configurations, and missing support for certain features like TMA (Tensor Memory Accelerator) block layouts.

The Path Forward

The session ends with the assistant at a crossroads. The allreduce fusion path is closed unless someone ports the TRT-LLM communication kernels to SM120 — a significant engineering effort. The alternative paths — NCCL tuning, decode step batching, alternative MoE backends — have not yet yielded gains. The most promising remaining directions include:

Conclusion

This chunk of the opencode session captures a complete arc of ML inference optimization: from frustration at a performance plateau, through a dramatic breakthrough enabled by a single-line code patch, to the sobering discovery that the breakthrough masked a deeper hardware utilization problem, and finally to the confrontation with a hard architectural boundary that no amount of Python-level patching could cross.

The 3,740 tok/s milestone was real and significant — a 4.25× improvement over the starting baseline. But it was also deceptive. The GPUs were capable of much more, and the software stack was the limiting factor. The allreduce fusion gap, rooted in the difference between SM100 and SM120 architectures, represents a fundamental challenge that will require kernel-level work to resolve.

For practitioners deploying large models on consumer or workstation Blackwell hardware, this session offers both hope and caution. The hope: significant performance gains are available through careful configuration and targeted code patches. The caution: the inference stack's support for SM120 is incomplete, and some optimizations that work on datacenter hardware may simply not be available. The gap between "it compiles" and "it runs efficiently" is wider than it appears, and bridging it requires not just engineering effort but a deep understanding of GPU architecture.

The Blackwell divide is real. But as this session demonstrates, it is not insurmountable — it just requires the right combination of diagnostic rigor, willingness to patch upstream code, and acceptance of hardware-imposed boundaries.