The Allreduce-RMS Fusion Postmortem: When Hardware Reality Meets Software Optimization

Introduction

In the course of deploying a massive 78-layer Mixture-of-Experts (MoE) language model (GLM-5) across eight NVIDIA RTX PRO 6000 Blackwell GPUs using vLLM, a critical performance optimization was attempted: enabling allreduce-RMS fusion. Message 139 in this opencode session is the final summary report from a subagent tasked with implementing this fusion pass. What makes this message remarkable is not what it accomplished, but what it definitively ruled out. The message is a masterclass in systematic performance investigation—a document that traces a clear path from hypothesis, through experimentation, to a hard hardware-imposed conclusion. It documents not only the failure of a specific optimization technique but also provides a comprehensive performance characterization of a complex distributed inference system, revealing that approximately 55–65% of inference time is consumed by allreduce communication overhead on PCIe-connected GPUs.

Context and Motivation: Why This Message Was Written

The message was produced as the terminal output of a subagent that had been dispatched with the task "Enable allreduce-RMS fusion (agent: general)." To understand why this task existed, we must look at the broader context of the session. The root session had been working through a complete ML environment setup on Ubuntu 24.04, culminating in the deployment of the GLM-5-NVFP4 model using vLLM. By the time the subagent was invoked (segment 16 of the root session), significant progress had already been made: garbage output had been resolved by fixing bugs in the Triton MLA attention backend and GGUF dequantization shard ordering, single-request decode throughput had been optimized from approximately 20 to 57 tok/s using CUDAGraph and NCCL tuning, and the model had been deployed as a systemd service.

The motivation for the allreduce-RMS fusion task was straightforward: the assistant had identified that each layer of the 78-layer model involves two allreduce operations (one after attention output, one after MoE aggregation), totaling 156 allreduces per decode step. If these allreduces could be fused with the subsequent RMS normalization, the overhead of launching separate kernels and transferring data between them could be eliminated. The vLLM codebase already had an enable_allreduce_rms_fusion option, but it was disabled for the GGUF model being used. The subagent was tasked with enabling it and measuring the performance impact.

The Investigation: What Was Actually Done

The message reveals a multi-pronged investigation that went far beyond simply toggling a configuration flag. The assistant first patched two files in the vLLM installation: vllm/config/vllm.py to add compute capability 120 (the Blackwell architecture's compute capability) to the list of supported devices for allreduce-RMS fusion, and allreduce_rms_fusion.py to add corresponding size entries for capability 120. This was a necessary prerequisite because the fusion pass had explicit device capability checks that excluded the Blackwell GPUs (compute capability 12.0).

When the patched configuration was tested, it failed with the error message [SymmDeviceMemory] Device does not support multicasting. This error is the critical finding of the entire investigation. The flashinfer allreduce fusion implementation relies on cudaMulticastCreate(), a CUDA API function that enables multiple GPUs to share a single physical memory allocation through hardware multicast. This capability requires NVSwitch or similar hardware-level support—it is fundamentally unavailable on PCIe-connected workstation GPUs, regardless of their compute capability.

The assistant then pivoted from the fusion attempt to a comprehensive NCCL tuning exploration. The message documents tests of seven different NCCL configurations:

Assumptions Made and Their Consequences

Several assumptions underlay this investigation, and the message implicitly documents their testing and refutation.

Assumption 1: Allreduce-RMS fusion is a software configuration issue. The initial assumption was that enabling the fusion pass was a matter of adding the correct compute capability identifier to the configuration files. This assumption was reasonable—the vLLM codebase already supported the fusion for compute capabilities 80 (Ampere), 89 (Ada Lovelace), and 90 (Hopper). The Blackwell GPUs (capability 120) simply hadn't been added to the list. However, this assumption failed because the fusion's dependency was not on compute capability but on hardware multicast support, which is a platform-level feature orthogonal to compute capability.

Assumption 2: The fusion pass would improve performance. Even if the fusion had been enabled, it's not clear it would have helped. The fusion combines the allreduce with RMS normalization, which reduces kernel launch overhead and memory traffic. But the fundamental bottleneck—PCIe traversal latency—would remain. The fused kernel would still need to move data across the PCIe bus; it would just do so with fewer kernel launches. Given that the compute time is estimated at only 6–8ms per token out of 17.4ms total, the kernel launch overhead is likely a small fraction of the total.

Assumption 3: NCCL environment variables can significantly impact performance. The assistant tested seven different NCCL configurations, and only NCCL_PROTO=LL showed any improvement. This systematic elimination of variables is a textbook example of performance debugging: change one thing at a time, measure, and compare. The conclusion that all other NCCL parameters are irrelevant for this workload is itself a valuable finding.

Assumption 4: The custom allreduce in vLLM could be patched to work. The assistant investigated the custom allreduce implementation and found that it was disabled for world_size > 2 on PCIe-only GPUs. The check is in is_fully_connected(), which uses NVML to query NVLink P2P status. Without NVLink, fully_connected returns False, and for 8 GPUs, custom allreduce is disabled. The assistant correctly assessed that force-enabling it would be risky—the implementation uses IPC shared memory buffers with P2P access, and on PCIe with more than 2 GPUs, P2P reads between non-adjacent GPUs must traverse the root complex, which introduces significant latency.

Input Knowledge Required to Understand This Message

To fully comprehend this message, the reader needs knowledge spanning several domains:

CUDA Architecture and GPU Topology: Understanding compute capabilities (12.0 for Blackwell), the difference between NVLink and PCIe connectivity, NUMA node topology, and the concept of GPU P2P communication. The message references cudaMulticastCreate(), which is a relatively obscure CUDA API function for hardware-level memory sharing across GPUs connected via NVSwitch.

NCCL Internals: Knowledge of NCCL protocols (LL, LL128, Simple), the ring allreduce algorithm, NCCL environment variables (NTHREADS, BUFFSIZE, ALGO, MAX_NCHANNELS), and how these affect allreduce latency and bandwidth. The message assumes the reader understands that NCCL_PROTO=LL reduces per-operation latency at the cost of bandwidth efficiency.

vLLM Architecture: Familiarity with vLLM's compilation passes, the fusion infrastructure, the allreduce_rms_fusion.py implementation, the custom allreduce module, and the GGUF model loading path. The message references the fact that GGUF models use custom_ops: ['none'], meaning all operations are Inductor-compiled rather than using vLLM's custom kernels.

MoE Model Architecture: Understanding that Mixture-of-Experts models use sparse activation (8/256 experts in this case), requiring expert aggregation via allreduce after each MoE layer. The message also references MLA (Multi-head Latent Attention) with specific dimensions (kv_lora_rank=512, qk_head_dim=256).

Performance Analysis Methodology: The ability to interpret throughput measurements, understand the relationship between latency and throughput, and recognize when a system is communication-bound versus compute-bound. The message's estimate that HBM bandwidth utilization is only ~14% is a sophisticated inference from the observed throughput and the known model characteristics.

Output Knowledge Created by This Message

The message generates several forms of new knowledge:

Definitive Hardware Constraint Documentation: The most important output is the clear documentation that allreduce-RMS fusion via cudaMulticastCreate() is impossible on PCIe-connected GPUs, regardless of their compute capability. This is a hardware-imposed limitation that no amount of software patching can overcome. For anyone deploying vLLM on workstation-class hardware with multiple GPUs, this is a critical constraint to understand.

Performance Baseline for 8× PCIe Blackwell GPUs: The message establishes a comprehensive performance baseline for this specific hardware configuration running a large MoE model. The key numbers—57.6 tok/s for single-request decode, 97.4 tok/s aggregate for 2 concurrent requests, 144.4 tok/s for 4 concurrent requests—provide reference points for future optimization attempts.

NCCL Tuning Guidance: The systematic testing of NCCL environment variables produces actionable guidance: for PCIe-connected GPUs running small allreduce operations (12KB per allreduce), NCCL_PROTO=LL is the only tuning parameter that matters. All other NCCL parameters tested had zero effect. This saves future practitioners from having to rediscover this through trial and error.

Bottleneck Characterization: The estimate that 55–65% of per-token time is allreduce latency (9–11ms out of 17.4ms) is a crucial insight. It means that even perfect optimization of compute kernels would only improve throughput by at most 40–45%. The theoretical ceiling of 100–140 tok/s with zero-cost allreduce provides a clear upper bound on what's achievable on this hardware.

Alternative Optimization Paths: The message identifies four remaining optimization options not tested: pipeline parallelism (PP=2×TP=4 to keep TP groups within a single NUMA node), speculative decoding, expert parallelism, and smaller GGUF quantization to fit in 4 GPUs. Each of these represents a different strategy for reducing or eliminating the allreduce bottleneck.

The Thinking Process: A Window into Systematic Debugging

The reasoning visible in this message reveals a methodical, hypothesis-driven approach to performance optimization. The assistant's thinking follows a clear pattern:

  1. Identify the bottleneck: Through timing analysis, determine that allreduce communication dominates per-token latency.
  2. Explore the most promising optimization: Attempt to enable allreduce-RMS fusion, which would combine two operations into one, reducing kernel launch overhead and memory traffic.
  3. Test the optimization: Patch the configuration, run the model, observe the failure.
  4. Understand the failure: The error message points to cudaMulticastCreate() and multicast support. Investigate what this requires (NVSwitch hardware).
  5. Pivot to secondary optimizations: Since the fusion is impossible, systematically explore NCCL tuning options.
  6. Eliminate variables systematically: Test each NCCL environment variable independently, measuring the same workload each time.
  7. Identify the single effective variable: NCCL_PROTO=LL provides a 22% improvement; everything else is noise.
  8. Quantify the remaining gap: Estimate compute time (6–8ms) vs. allreduce time (9–11ms) to understand the theoretical ceiling.
  9. Document remaining options: Identify four alternative approaches that could be explored in future work. This is textbook performance optimization methodology, and the message serves as an excellent case study in how to systematically investigate and characterize a complex system's performance limitations.

Conclusion

Message 139 is far more than a simple summary of failed optimization attempts. It is a comprehensive performance analysis of a state-of-the-art distributed inference system, documenting both the successes (NCCL_PROTO=LL providing a 22% improvement) and the hard limitations (allreduce-RMS fusion impossible on PCIe, allreduce consuming 55–65% of per-token time). The message's greatest value lies in what it definitively rules out: it saves future investigators from having to rediscover that allreduce-RMS fusion requires NVSwitch hardware, that NCCL tuning beyond NCCL_PROTO=LL is futile for this workload, and that the theoretical ceiling on this hardware is approximately 100–140 tok/s. For anyone deploying large language models on multi-GPU workstation-class hardware, this message is a valuable reference document—a map of the terrain that clearly marks which paths lead to improvement and which are dead ends.