The Custom Allreduce Wall: When Hardware Topology Thwarts Optimization

Introduction

In the high-stakes world of large language model (LLM) inference optimization, progress often comes in two flavors: the gradual climb of incremental tuning, and the sudden jolt of discovering a fundamental constraint. Message 129 of this opencode session belongs squarely in the second category. It is the moment when the assistant, after an exhaustive campaign of NCCL tuning, memory bandwidth analysis, and latency profiling, finally uncovers the root cause of a stubborn performance ceiling: vLLM's custom allreduce implementation is disabled on configurations with more than two PCIe-only GPUs.

This discovery is not merely a data point—it is a turning point. It reframes the entire optimization problem, shifting the assistant's focus from tuning parameters within the existing system to questioning whether the system's architecture can be modified at all. The message captures a moment of insight, verification, and the beginning of a new line of investigation.

The Context: An Optimization Plateau at 57.5 tok/s

To understand the significance of message 129, one must appreciate the journey that preceded it. The assistant had been tasked with deploying the GLM-5-NVFP4 model—a massive 402 GB Mixture-of-Experts (MoE) model with 78 layers, Multi-head Latent Attention (MLA), and 256 experts per layer—on a machine with 8 RTX PRO 6000 Blackwell GPUs. The deployment used vLLM with tensor parallelism across all 8 GPUs, and the goal was to maximize single-request decode throughput.

The preceding messages (113–128) document a systematic optimization campaign. The assistant tried different NCCL protocols (LL, Simple, SHM), experimented with CUDA memory copy settings (NCCL_SHM_USE_CUDA_MEMCPY, NCCL_SHM_MEMCPY_ASYNC), and tested various compilation levels (O1 vs O2). Each experiment returned the same result: approximately 57.5 tokens per second. The throughput was stubbornly flat.

The assistant then performed a rigorous bandwidth analysis. By measuring HBM bandwidth directly (finding ~1458 GB/s) and estimating the active weight read per token (~28 GB total, ~3.5 GB per GPU), the assistant calculated that the memory bandwidth utilization was only 13%. This ruled out the obvious suspect of memory bandwidth saturation. The inter-token latency was measured at 10–25 ms (consistent with 57.5 tok/s), and time-to-first-token (TTFT) was a negligible ~52 ms. Flashinfer autotuning was already active, CUDA graphs were capturing 51 sizes, and the fusion passes (fuse_allreduce_rms, fuse_norm_quant) were disabled due to GGUF's use of Inductor-compiled ops.

By message 128, the assistant had exhausted the conventional tuning space. Every NCCL knob had been turned, every obvious bottleneck investigated. The throughput was a wall, not a slope. This is the moment when the assistant pivots from "how do we tune this better?" to "what fundamental constraint are we hitting?"—and the answer lies in the communication layer.

The Subject Message: Discovery of the Custom Allreduce Constraint

Message 129 reads in full:

[assistant] There it is. vLLM's custom allreduce is disabled because it's not supported on more than 2 PCIe-only GPUs. So we're falling back to NCCL allreduce for every communication.

Let me check if the custom allreduce can be enabled on more GPUs by patching, or if there's a fundamental limitation: [bash] ssh -o StrictHostKeyChecking=no root@10.1.230.174 'grep -n "not supported on more than two PCIe\|PCIe.*only\|custom_all_reduce" /root/ml-env/lib/python3.12/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py | head -20' 109: "warning, specify disable_custom_all_reduce=True explicitly.", 156: " more than two PCIe-only GPUs. To silence this warning, " 157: "specify disable_custom_all_reduce=True explicitly." 168: "warning, specify disable_custom_all_reduce=True explicitly." 267: def custom_all_reduce(self, input: torch.Tensor) -> torch.Tensor | None:

The message is deceptively short, but it packs a tremendous amount of meaning. The opening phrase—"There it is."—carries the weight of a hard-won discovery. The assistant had been searching for the bottleneck, and now it has found it, not in compute kernels or memory bandwidth, but in the communication fabric connecting the GPUs.

Why This Message Was Written: Reasoning, Motivation, and Context

The assistant wrote this message because it had reached a dead end in the conventional optimization space and needed to understand why. The reasoning chain is visible across the preceding messages:

Exhaustion of the NCCL tuning space: Messages 113–119 systematically tested every NCCL environment variable and protocol. Each returned ~57.5 tok/s. The assistant concluded "The NCCL tuning space seems exhausted."
Ruling out memory bandwidth: Messages 119–121 performed detailed bandwidth calculations, measuring HBM at 1458 GB/s and finding only 13% utilization. The assistant concluded "We're NOT memory bandwidth bound."
Ruling out scheduling overhead: Messages 122–128 measured TTFT, inter-token latency, flashinfer autotuning, CUDA graph sizes, and fusion passes. None showed a bottleneck.
The pivot to communication: In message 128, the assistant wrote: "Let me try a more impactful direction. The NCCL tuning has plateaued. Let me check if there's a way to reduce the number of allreduces or use custom allreduce." This was the critical insight—if the per-layer compute is fast but the overall throughput is capped, perhaps the communication between GPUs (the allreduce operations after each attention and MoE layer) is the bottleneck.
The grep that revealed everything: The assistant ran a grep on the vLLM source code, searching for the custom allreduce implementation. The results showed warning messages about PCIe-only GPU limitations. The motivation was pure debugging: the assistant had a hypothesis (custom allreduce might be faster than NCCL) and needed to verify whether it was even available. The discovery that it was explicitly disabled was both an answer and a new question.

How Decisions Were Made

Several decisions are embedded in this message, both explicit and implicit:

The decision to investigate custom allreduce: This was the most important decision. After exhausting NCCL tuning, the assistant could have continued down other paths (e.g., trying different quantization formats, modifying the model architecture, or accepting the current throughput). Instead, the assistant chose to investigate the communication layer—a deeper, more architectural level of optimization. This decision reflects a sophisticated understanding of where bottlenecks hide in distributed LLM inference.

The decision to read source code: Rather than relying on documentation or logs, the assistant went directly to the vLLM source code (custom_all_reduce.py) and searched for the relevant error messages. This is a hallmark of expert debugging—when logs are insufficient, read the code that produces them.

The decision to check for patching feasibility: The second sentence of the message—"Let me check if the custom allreduce can be enabled on more GPUs by patching, or if there's a fundamental limitation"—shows that the assistant immediately began evaluating whether this constraint could be circumvented. This is not a passive discovery; it is an active assessment of the optimization landscape.

The decision to use a bash command for verification: The assistant chose to run grep on the remote machine, searching for specific strings in the Python source file. This is efficient—it extracts exactly the relevant lines without needing to read the entire file.

Assumptions Made by the User or Agent

The message and its context reveal several assumptions:

Assumption that custom allreduce would be faster: The assistant assumed that vLLM's custom allreduce implementation would be faster than NCCL for the small tensors involved in per-layer allreduces. This assumption is well-founded—custom allreduce uses shared GPU memory for P2P communication, which has lower latency than NCCL's kernel-based approach for small messages. However, the assumption was moot because the feature was disabled.

Assumption that the constraint was in the code, not the hardware: The assistant initially assumed that the limitation was a software check that could potentially be patched. The subsequent investigation (messages 130–134) revealed that the constraint is genuine—the custom allreduce uses IPC shared memory buffers with P2P access, and on PCIe with more than 2 GPUs, P2P reads between non-adjacent GPUs must traverse the root complex, which is slow. NVLink bypasses this, but these GPUs lack NVLink.

Assumption that NCCL was the fallback: The assistant assumed that when custom allreduce is disabled, vLLM falls back to NCCL allreduce. This is correct and is confirmed by the log message seen in message 128, which shows the engine initialization with standard NCCL-based communication.

Assumption that the 2-GPU limit was the relevant constraint: The assistant assumed that the "more than 2 PCIe-only GPUs" check was the one blocking custom allreduce. The grep results confirmed this—lines 156–157 contain the exact warning message about this limitation.

Mistakes or Incorrect Assumptions

While the message itself is accurate, there are subtle aspects worth examining:

The assumption that patching might be straightforward: The assistant's question—"Let me check if the custom allreduce can be enabled on more GPUs by patching, or if there's a fundamental limitation"—initially leans toward the possibility of a simple patch. The subsequent investigation (messages 130–134) reveals that the limitation is indeed fundamental: the custom allreduce relies on P2P IPC shared memory buffers, which don't scale well across PCIe without NVLink. The assistant's own analysis in message 134 concludes: "On PCIe with >2 GPUs, P2P reads between non-adjacent GPUs have to traverse the root complex, which is slow. NVLink bypasses this."

The implicit assumption that allreduce is the dominant bottleneck: The assistant discovered that custom allreduce is disabled, but this doesn't necessarily mean that enabling it would dramatically improve throughput. The allreduce operations might not be the dominant term in the per-token latency budget. The assistant's subsequent investigation (messages 135+) would need to quantify the allreduce overhead to determine whether this is truly the bottleneck or merely a contributing factor.

The assumption about the grep search pattern: The assistant searched for "not supported on more than two PCIe\|PCIe.*only\|custom_all_reduce" which found the warning messages but not the actual check logic. The assistant then needed additional grep commands (messages 130–131) to find the is_fully_connected check and the world_size > 2 and not fully_connected condition. This is not a mistake per se, but it shows that the initial search was incomplete—the assistant had to iterate to find the full picture.

Input Knowledge Required to Understand This Message

To fully grasp message 129, one needs knowledge spanning several domains:

Distributed computing concepts: Understanding what an allreduce operation is—a collective communication that sums tensors across all GPUs and distributes the result back—is essential. In tensor-parallel LLM inference, allreduce is used after attention and after the MoE layer to synchronize the partial results from each GPU.

vLLM architecture: Knowledge that vLLM has a custom allreduce implementation that bypasses NCCL for lower latency, and that this implementation uses shared GPU memory with P2P (peer-to-peer) access. Understanding the distinction between NCCL allreduce (kernel-based, general-purpose) and custom allreduce (IPC-based, optimized for small tensors).

GPU interconnect topology: Understanding the difference between PCIe (Peripheral Component Interconnect Express) and NVLink. PCIe connects GPUs through the CPU's root complex, which introduces latency and bandwidth limitations. NVLink provides direct GPU-to-GPU connections with higher bandwidth and lower latency. The concept of "fully connected" GPUs via NVLink is critical.

The specific hardware configuration: The session uses 8 RTX PRO 6000 Blackwell GPUs connected via PCIe, without NVLink. This is the root cause of the limitation.

The model architecture: The GLM-5 model uses MoE with 8 active experts out of 256 per layer, plus a shared expert. Each layer performs two allreduce operations (one after attention, one after MoE), and with 78 layers, that's 156 allreduce calls per token. At 57.5 tok/s, that's nearly 9,000 allreduce operations per second.

Python and bash debugging skills: Understanding how to use grep to search source code on a remote machine via SSH, and how to interpret the line numbers and context in the output.

Output Knowledge Created by This Message

Message 129 creates several valuable pieces of knowledge:

The primary finding: vLLM's custom allreduce is disabled on configurations with more than 2 PCIe-only GPUs. This is a concrete, verifiable fact about the system.

The code location: The relevant source file is /root/ml-env/lib/python3.12/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py, and the warning messages are at lines 109, 156–157, and 168.

The fallback behavior: When custom allreduce is disabled, vLLM falls back to NCCL allreduce for every communication. This means every allreduce in the model (156 per token) uses the NCCL kernel-based implementation.

The optimization landscape: The message implicitly defines the boundaries of what can be optimized. NCCL tuning has been exhausted, custom allreduce is unavailable, and the remaining options are either (a) patching vLLM to force-enable custom allreduce despite the limitation, (b) reducing the number of allreduce operations through fusion, or (c) accepting the current throughput.

A methodology for investigating similar issues: The message demonstrates a pattern—when tuning parameters plateau, look at the architecture. When logs are insufficient, read the source code. When a feature is disabled, check whether it can be enabled and whether the limitation is fundamental or arbitrary.

The Thinking Process Visible in Reasoning

The assistant's reasoning is laid bare in this message. The structure is:

Discovery: "There it is." — the moment of finding the answer.
Interpretation: "vLLM's custom allreduce is disabled because it's not supported on more than 2 PCIe-only GPUs." — translating the code output into a meaningful statement.
Implication: "So we're falling back to NCCL allreduce for every communication." — understanding what this means for the system.
Next question: "Let me check if the custom allreduce can be enabled on more GPUs by patching, or if there's a fundamental limitation." — immediately pivoting to the next investigation. The thinking is visible in the choice of grep pattern. The assistant searched for "not supported on more than two PCIe" (the exact warning text), "PCIe.*only" (a broader pattern to catch related messages), and "custom_all_reduce" (to find the function definition). This shows a systematic approach to code exploration: start with the specific error message, broaden to related patterns, and then find the implementation entry point. The fact that the assistant chose to run grep with head -20 (limiting to 20 lines) shows an expectation that the relevant lines would be few and clustered. This is efficient—if the search had returned hundreds of lines, the assistant would have needed to refine the pattern. The thinking also reveals a sophisticated understanding of the trade-offs involved. The assistant doesn't immediately conclude that patching is the answer. Instead, it asks whether the limitation is fundamental. This reflects an understanding that some software constraints exist for good reasons (hardware limitations, correctness guarantees) while others are arbitrary (configuration choices, conservative defaults).

The Broader Significance

Message 129 is significant beyond its immediate content because it represents a shift in the optimization strategy. Before this message, the assistant was operating within the system's existing configuration space—tuning NCCL parameters, adjusting compilation levels, measuring latencies. After this message, the assistant begins to question the system's architecture itself.

This shift is visible in the subsequent messages (130–134), where the assistant reads the is_fully_connected function, examines the NVLink detection logic, and evaluates whether the custom allreduce kernel could work on PCIe. The assistant ultimately concludes that the limitation is genuine—the custom allreduce uses IPC shared memory with P2P access, which doesn't scale across PCIe without NVLink.

However, this discovery also opens a new door. If the custom allreduce is unavailable and NCCL is the bottleneck, perhaps the solution is to reduce the number of allreduce operations through kernel fusion. The assistant had already noticed that fuse_allreduce_rms was disabled. This becomes the next line of investigation: can the allreduce be fused with the RMS normalization to reduce communication overhead?

Conclusion

Message 129 is a masterclass in systematic debugging. It captures the moment when an optimization campaign hits a fundamental constraint, and the assistant's response is not frustration but curiosity. The message demonstrates the value of reading source code, the importance of understanding hardware topology, and the art of asking the right next question.

The discovery that vLLM's custom allreduce is disabled on 8 PCIe-only GPUs is not the end of the story—it is the beginning of a new chapter. By understanding why the constraint exists, the assistant can now make informed decisions about whether to work around it, accept it, or change the approach entirely. This is the essence of deep performance optimization: not just turning knobs, but understanding the machinery behind them.