The 42% Bottleneck: Diagnosing NCCL Allreduce Overhead in Multi-GPU LLM Inference

In the high-stakes world of large language model deployment, achieving acceptable inference throughput is often a battle fought in microseconds. When the user of an opencode coding session asked why their freshly deployed GLM-5 model — a 744-billion-parameter Mixture-of-Experts model quantized to GGUF Q4_K_XL and spread across eight NVIDIA RTX PRO 6000 Blackwell GPUs — was crawling along at only 20 tokens per second for a single request, the assistant's response was not a guess. It was a surgical analysis grounded in detailed profiling data. Message 1987 represents a pivotal moment in the optimization journey: the moment when raw data was transformed into a clear, actionable hypothesis about where the performance was being lost.

The Performance Mandate

The context for this message is critical. The user had just received a comprehensive benchmark summary ([msg 1983]) showing that the model was producing correct output after two major bugs were fixed — an MLA output buffer disconnect in the Triton attention backend and a GGUF shard ordering bug in fused linear layers. But the single-request throughput was a disappointing 20 tok/s. The user's response was blunt: "20t/s is still really slow, gat you profile and try to improve to at least 100/s" ([msg 1984]). This set the stage for a performance optimization sprint.

The assistant's first move was to launch a detailed profiling task ([msg 1986]). The results of that profiling are the direct input to message 1987. The assistant received a comprehensive breakdown showing that each decode step took approximately 50.5 milliseconds, yielding 19.8 tok/s. More importantly, the profile revealed where those 50.5 milliseconds were going: 42% of the time was consumed by NCCL allreduce operations — 157 individual calls, each taking about 135 microseconds, totaling roughly 21 milliseconds per decode step. The remaining 57% was GPU compute, which the profile indicated was already near memory-bandwidth optimal.

The Message: A Turning Point in Reasoning

Message 1987 opens with a crisp summary of the profiling findings:

This is a very detailed profile. The bottleneck is clear: 42% of decode time is NCCL allreduce (157 calls × 135μs each = 21ms), and the remaining 57% is GPU compute that's near memory-bandwidth optimal.

This sentence does several things at once. It acknowledges the quality of the profiling data, declares the primary bottleneck with quantitative precision, and implicitly rules out one class of optimization targets — the GPU compute kernels are already performing near their theoretical limit. The assistant is telling the reader (and itself) that there is no low-hanging fruit in the compute kernels. The path to 100 tok/s must go through reducing the allreduce overhead.

The message then pivots to the key insight:

The key insight is that --enforce-eager prevents CUDA graph replay, which would batch all those 157 NCCL calls and 1500+ kernel launches. We already tested CUDAGraph and it produced garbage — that's likely related to our forward_impl direct-call fix bypassing the custom op. Let me see if we can make CUDAGraph work with our fix, since that's the single biggest optimization opportunity.

This is the heart of the message: a hypothesis about why CUDAGraph failed earlier, and a decision to invest effort in making it work. The reasoning chain is:

1. The dominant bottleneck is NCCL allreduce overhead (42% of decode time).
2. CUDAGraph would batch these 157 allreduce calls and 1500+ kernel launches into a single graph replay, dramatically reducing launch overhead.
3. CUDAGraph was tested earlier and produced garbage output.
4. The likely cause of the CUDAGraph failure is the MLA direct-call fix — the custom op bypass that was necessary to fix the output buffer disconnect bug.
5. Therefore, the path to 100 tok/s is to fix CUDAGraph compatibility with the MLA patch. This is a classic debugging and optimization reasoning pattern: identify the bottleneck, identify the tool that addresses that bottleneck, diagnose why that tool failed previously, and formulate a plan to fix the incompatibility.## Assumptions Embedded in the Analysis Every optimization decision rests on assumptions, and message 1987 is rich with them. The assistant assumes that the profiling data is accurate and representative — that the 50.5ms decode step time and the 42% NCCL allreduce proportion are stable measurements, not artifacts of a particular run. This is a reasonable assumption given the consistency noted in the profiling output ("Extremely consistent: 50.5ms per decode step, 19.7-19.8 tok/s"), but it is an assumption nonetheless. If the profile had captured an anomalous run, the optimization strategy would be misdirected. A deeper assumption is that the NCCL allreduce overhead is primarily a launch overhead problem rather than a communication bandwidth problem. The assistant implicitly classifies the 157 allreduce calls as something that CUDAGraph can batch away, which would only be effective if the overhead is in kernel launch latency rather than in the actual data transfer across PCIe. The profiling data supports this: 135 microseconds per allreduce call on PCIe-connected GPUs is consistent with launch latency dominating the per-call cost. If the bottleneck were instead PCIe bandwidth saturation, batching the calls would not help — the same total data would need to be transferred. The assistant also assumes that the CUDAGraph failure was caused by the MLA direct-call patch rather than by some other incompatibility. This is a plausible hypothesis — the forward_impl bypass changes how tensors are passed to the attention kernel, and CUDAGraph's tensor capture mechanism may not handle this correctly — but it is not yet proven. The message acknowledges this uncertainty implicitly by framing the next step as "see if we can make CUDAGraph work with our fix."

The Thinking Process Visible in the Message

Message 1987 is notable for what it reveals about the assistant's thinking process, even though it is a relatively short message. The structure of the reasoning is:

1. Accept the data: The profile is detailed and the bottleneck is clear. No time is wasted questioning the profiling methodology.
2. Classify the bottleneck: 42% NCCL allreduce, 57% GPU compute that's already near-optimal. This classification immediately rules out optimizing compute kernels and focuses attention on communication overhead.
3. Identify the tool: CUDAGraph is the obvious solution for batching kernel launches and NCCL calls. The assistant already knows from earlier testing that CUDAGraph exists and was attempted.
4. Diagnose the failure: The earlier CUDAGraph test produced garbage output. The assistant connects this to the MLA direct-call fix, which was the most invasive change made to the attention pipeline.
5. Formulate the plan: Fix CUDAGraph compatibility with the MLA patch. This is the single biggest optimization opportunity. The todo list at the bottom of the message reinforces this plan. The first item is marked completed (profiling), the second is in progress (fix CUDAGraph), and additional items are listed for investigation (custom allreduce, reducing TP). This shows that the assistant is thinking ahead: if CUDAGraph cannot be made to work, there are fallback options to explore.

Alternatives Considered and Implicitly Rejected

The message mentions two alternative optimization paths in the todo list: "Investigate custom allreduce for PCIe GPUs" and "Consider reducing TP from 8 to..." These are listed as pending, not in-progress, indicating that the assistant has prioritized CUDAGraph as the first line of attack.

The custom allreduce option is particularly interesting. vLLM has a custom allreduce implementation that can be faster than NCCL for certain topologies, but it is typically disabled for PCIe-connected GPUs (as opposed to NVLink-connected ones). The assistant notes this in the todo description: "currently disabled by vLLM." Enabling custom allreduce would be a software-only change, but it might not address the fundamental issue of 157 separate allreduce calls per decode step.

Reducing tensor parallelism (TP) from 8 to a lower number would reduce the number of allreduce calls but would increase the memory pressure on each GPU. With a 402GB model, reducing TP might cause out-of-memory errors. This option is likely a last resort.

The assistant also implicitly rejects the idea of optimizing the GPU compute kernels. The profile indicates they are already near memory-bandwidth optimal, so further optimization there would yield diminishing returns. This is a mature engineering judgment: know when to stop optimizing one component and shift focus to the actual bottleneck.

Input Knowledge Required to Understand This Message

To fully grasp message 1987, the reader needs several pieces of context:

• What NCCL allreduce is: In tensor parallelism, each GPU holds a shard of the model weights. After each layer's forward pass, the GPUs must synchronize their partial results via an allreduce operation. NCCL (NVIDIA Collective Communications Library) is the standard library for this. On PCIe-connected GPUs (without NVLink), allreduce must go over the PCIe bus, which has higher latency than NVLink.
• What CUDAGraph is: CUDA Graphs allow batching multiple kernel launches and CUDA API calls into a single graph object that can be replayed with minimal driver overhead. For models with many small kernel launches (like MoE models with many expert networks), CUDAGraph can dramatically reduce launch latency.
• The MLA direct-call fix: In a previous debugging session, the assistant fixed a bug where the Triton MLA attention backend's custom PyTorch op created a phantom output tensor. The fix bypassed the custom op and called forward_impl directly. This change is suspected to be incompatible with CUDAGraph's tensor capture mechanism.
• The hardware topology: Eight RTX PRO 6000 Blackwell GPUs connected via PCIe, not NVLink. This means all GPU-to-GPU communication goes through the PCIe bus, which has higher latency and lower bandwidth than NVLink. This constraint shapes all optimization decisions.
• The model characteristics: GLM-5 is a 744B MoE model quantized to GGUF Q4_K_XL. MoE models have many small linear layers (experts), each of which requires an allreduce in tensor parallelism. This explains the 157 allreduce calls per decode step — each expert activation must be synchronized across GPUs.

Output Knowledge Created by This Message

Message 1987 creates several important outputs:

1. A clear bottleneck identification: 42% of decode time is NCCL allreduce. This is now the primary optimization target.
2. A prioritized optimization plan: Fix CUDAGraph first, then explore custom allreduce and TP reduction as fallbacks.
3. A hypothesis about CUDAGraph failure: The MLA direct-call patch is the likely cause of the earlier CUDAGraph garbage output.
4. A decision to invest effort: The assistant commits to debugging CUDAGraph compatibility rather than pursuing other optimizations.
5. Updated task tracking: The todo list is revised to reflect the new priorities, with profiling marked complete and CUDAGraph fix marked in progress.

The Broader Significance

Message 1987 is a microcosm of the entire optimization process. It demonstrates how profiling data transforms vague performance complaints ("20t/s is still really slow") into precise engineering targets ("reduce NCCL allreduce overhead from 21ms to near zero"). It shows the importance of understanding the interaction between fixes — the MLA direct-call patch fixed one bug but may have broken CUDAGraph compatibility, creating a new problem to solve. And it illustrates the iterative nature of performance optimization: profile, identify bottleneck, hypothesize fix, test, repeat.

The message also reveals the assistant's engineering maturity. Rather than chasing speculative optimizations or tweaking random parameters, the assistant lets the data guide the decision. The profile says NCCL allreduce is 42% of the time; CUDAGraph is the tool that addresses NCCL launch overhead; the only question is whether it can be made compatible with the existing patches. This is evidence-based optimization at its best.

In the end, the assistant's reasoning in message 1987 would prove correct. Subsequent messages show that CUDAGraph was successfully made to work, boosting throughput from ~20 tok/s to ~43 tok/s, and further NCCL tuning with NCCL_PROTO=LL pushed it to ~57 tok/s. While the 100 tok/s target was not reached in this session, the 185% improvement was a direct result of the analysis and prioritization performed in this single, decisive message.