The Last Mile: Chasing 100 tok/s Through Allreduce-RMS Fusion on PCIe-Bound Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, every millisecond counts. When you're serving a 744-billion parameter model like GLM-5 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between 57 tokens per second and 100 tok/s isn't just a number—it's the line between a usable interactive experience and a frustratingly slow one. Message 2010 captures a pivotal moment in this optimization journey: the assistant, having already wrung substantial gains from CUDAGraph and NCCL protocol tuning, identifies allreduce-RMSNorm fusion as the "most promising remaining optimization" and dispatches a subagent to test it. What follows is a masterclass in systematic performance debugging, where a well-reasoned hypothesis meets the hard constraints of hardware topology.

The Message in Full

The subject message, authored by the assistant, reads:

The allreduce-RMS fusion is the most promising remaining optimization. Let me try it: [task] {"description":"Enable allreduce-RMS fusion","prompt":"SSH to container: ssh root@10.1.230.174\n\n## Context\nGLM-5 GGUF at 57 tok/s with NCCL_PROTO=LL. Want to try allreduce-RMS fusion which fuses allreduce+RMSNorm into a single flashinfer kernel.\n\n## Step 1: Try with CLI override first\n\n```...} task_id: ses_383dc0970ffeVP5iEStGpr6fgn

>

<task_result> --- ## Summary of All Findings

>

### Hardware Topology - 8× NVIDIA RTX PRO 6000 Blackwell Server Edition (compute capability 12.0) - 96 GB HBM3e per GPU, ~1458 GB/s HBM bandwidth - PCIe Gen5 x16 — no NVLink, no NVSwitch - 2 NUMA nodes: GPUs 0-3 on NUMA 0 (NODE connectivity), GPUs 4-7 on NUMA 1, cross-NUMA is SYS (PCIe + QPI/UPI)

>

### Allreduce-RMS Fusion Result: IMPOSSIBLE on ...

The message is deceptively short. It contains a single declarative sentence, a task dispatch to a subagent, and the subagent's result. But within this compact form lies the culmination of dozens of preceding messages of debugging, profiling, and optimization work.

Context and Motivation: Why This Message Was Written

To understand why the assistant wrote this message, we must trace the optimization path that led here. The journey began after the model was finally producing coherent text—itself a triumph after fixing two deep bugs in the Triton MLA attention backend and the GGUF dequantization shard ordering. With correct output achieved, the assistant turned to performance.

The initial single-request throughput was a dismal ~20 tok/s. Profiling revealed that approximately 42% of each decode step was consumed by NCCL allreduce calls—the collective communication operations needed to synchronize intermediate results across the eight GPUs in the tensor-parallel group. The first major optimization was enabling CUDAGraph, which captured the entire model execution into a compiled CUDA graph, batching kernel launches and eliminating CPU dispatch overhead. This doubled throughput to ~43 tok/s.

The next breakthrough came from tuning the NCCL protocol. By setting NCCL_PROTO=LL (low-latency protocol), the assistant reduced allreduce latency for the tiny 12 KB payloads that dominate the GLM-5's MLA (Multi-head Latent Attention) architecture. This yielded a 34% improvement, pushing throughput to ~57 tok/s.

But the target was 100 tok/s. The remaining ~17 ms per decode step still showed approximately 12 ms consumed by NCCL allreduce, with only ~5 ms of actual GPU compute. The assistant had systematically eliminated every other bottleneck and was now staring at the fundamental constraint: the allreduce itself. Message 2009 had dispatched a task to "Explore paths to 100 tok/s," and that task returned with a clear recommendation: allreduce-RMSNorm fusion was the most promising path forward.

The Decision-Making Process

The decision to pursue allreduce-RMS fusion was not made lightly. The assistant had already explored and eliminated several alternatives:

• Custom allreduce: Broken on PCIe-only topologies with more than 2 GPUs due to a C++ kernel bug in the shared-memory path.
• SymmMem (symmetric memory): Disabled because Blackwell's compute capability 12.0 was not in the supported list.
• Reducing tensor parallelism: Counterproductive—TP=4 would double the weight per GPU, increasing memory-bandwidth-bound compute time.
• Speculative decoding: Worked but was content-dependent and didn't address the fundamental allreduce bottleneck. Allreduce-RMS fusion promised to eliminate one of the 158 allreduce calls per decode step by combining it with the subsequent RMSNorm operation into a single flashinfer kernel. This would reduce both the number of kernel launches and the data movement between GPUs. The assistant's reasoning was sound: if you can't eliminate allreduces entirely, at least fuse them with adjacent operations to amortize the overhead.

Assumptions and Their Correctness

The message reveals several implicit assumptions, some of which proved incorrect:

Assumption 1: Allreduce-RMS fusion would work on Blackwell GPUs. The assistant assumed that the flashinfer kernel supporting this fusion would be compatible with compute capability 12.0. The task result would ultimately show this was wrong—the fusion required features or memory bandwidth characteristics only available with NVLink interconnects.

Assumption 2: A CLI override would suffice. The task prompt instructed the subagent to "Try with CLI override first," suggesting the assistant believed the fusion could be enabled through a command-line flag. In reality, the server's pass_config showed all fusion passes disabled, and enabling them would require code modifications to vLLM's model execution pipeline.

Assumption 3: The fusion would provide a meaningful speedup. Even if the fusion were technically feasible, the assistant assumed it would move the needle toward 100 tok/s. Given that each allreduce call takes approximately 127 μs, fusing one of 158 calls would save at most ~0.08% of the decode time—hardly the 1.75x improvement needed. The real value would come from the reduced kernel launch overhead and improved memory access patterns.

Input Knowledge Required

To fully understand this message, the reader needs:

1. Tensor parallelism fundamentals: How model weights and activations are sharded across GPUs, requiring allreduce operations to synchronize partial results at each transformer layer.
2. NCCL internals: Understanding of NCCL's allreduce implementation, including the difference between the default protocol (which optimizes for bandwidth) and the LL (low-latency) protocol (which optimizes for small messages).
3. CUDAGraph mechanics: How CUDA graph capture eliminates CPU-side kernel launch overhead by pre-recording the entire sequence of GPU operations.
4. Allreduce-RMSNorm fusion: The technique of combining the allreduce synchronization with the subsequent RMS normalization into a single fused kernel, reducing both kernel launch count and memory traffic.
5. Hardware topology implications: The critical difference between NVLink-connected GPUs (which provide high-bandwidth, low-latency direct GPU-to-GPU communication) and PCIe-connected GPUs (which must route through the CPU and system memory).
6. vLLM's pass system: The pass_config mechanism that controls which optimization passes (including fusion passes) are applied during model compilation.
7. Blackwell GPU architecture: Compute capability 12.0, HBM3e memory, and the specific constraints of the RTX PRO 6000 Server Edition.

Output Knowledge Created

This message generates several important pieces of knowledge:

1. Hard topology documentation: The subagent's investigation produces a precise hardware topology description—8 GPUs across 2 NUMA nodes, PCIe Gen5 x16 with no NVLink, cross-NUMA communication at SYS latency. This is essential context for all future optimization decisions.
2. Negative result: The finding that allreduce-RMS fusion is impossible on this hardware configuration. While negative, this result is valuable because it prevents wasted effort on dead-end optimization paths and forces a reassessment of the performance ceiling.
3. Bottleneck confirmation: The investigation confirms that the remaining performance gap is fundamentally hardware-bound. No amount of software optimization can eliminate the PCIe latency penalty—the only path to 100+ tok/s would require either NVLink hardware or a fundamentally different parallelism strategy (e.g., pipeline parallelism with reduced allreduce frequency).
4. Methodology demonstration: The message demonstrates a systematic approach to performance optimization: profile to identify the bottleneck, apply the most impactful fix, re-profile, and repeat. Each optimization is validated with benchmarks before moving to the next.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, visible in the chain of messages leading to this one, reveals a methodical and data-driven approach. After each optimization, the assistant:

1. Benchmarks the current performance
2. Breaks down the remaining time by component
3. Identifies the largest remaining bottleneck
4. Researches possible mitigations
5. Tests the most promising option
6. Evaluates the result and iterates This message sits at step 5 of that cycle for the allreduce-RMS fusion hypothesis. The assistant has done the research (step 4) via the previous task, identified this as the most promising path, and is now testing it. The confidence in the statement "the most promising remaining optimization" comes from having systematically eliminated every other option. The subagent dispatch itself reveals another layer of thinking: the assistant recognizes that testing this hypothesis requires hands-on work on the remote machine (SSH commands, checking logs, modifying configs), which is better delegated to a focused subagent than done inline. The task prompt is carefully structured with context, clear steps, and expected outcomes—a sign of the assistant's experience in delegating complex technical work.

Conclusion

Message 2010 is a snapshot of the moment when software optimization meets hardware reality. The assistant, having exhausted every software-level technique to reduce NCCL allreduce overhead, turns to the most sophisticated remaining option—kernel fusion—only to discover that the hardware topology makes it impossible. The message captures both the promise of the approach and its ultimate limitation.

What makes this message remarkable is not the result (a negative finding) but the process it represents. It is the product of dozens of preceding messages of debugging, profiling, and optimization, each building on the last. The assistant's systematic approach—measure, identify, fix, re-measure—transformed a broken model producing garbage output into a deployed service running at 57 tok/s. And while the 100 tok/s target remained elusive, the message established definitively that the bottleneck was hardware, not software—a finding that would guide all future decisions about the deployment.

In the broader narrative of the GLM-5 deployment, this message marks the end of the software optimization phase and the beginning of acceptance of hardware constraints. It is a lesson in the art of knowing when to stop optimizing and start deploying.