The Allreduce Bottleneck: A Deep Dive into PCIe-Limited LLM Inference at 57.6 tok/s

Introduction

In the high-stakes world of large language model (LLM) inference, every millisecond counts. When deploying a 400-billion-parameter model like GLM-5 across eight powerful RTX PRO 6000 Blackwell GPUs, achieving optimal throughput requires understanding exactly where time is spent inside each inference step. Message 136 of this opencode session captures a pivotal moment: the assistant, having already stabilized the environment, fixed critical bugs in Triton MLA attention and GGUF dequantization, and achieved a respectable 57.5 tok/s decode rate, now turns to a fundamental question—what is actually consuming the 17.28 milliseconds between each generated token?

This message is a masterclass in performance analysis through deductive reasoning, back-of-the-envelope calculation, and clever use of existing empirical data. Rather than deploying heavy profiling tools that would require server restarts, the assistant extracts maximum insight from the metrics already available and from carefully designed comparative experiments conducted earlier in the session. The result is a clear diagnosis: allreduce communication overhead across eight PCIe-connected GPUs accounts for 55–65% of the per-token latency, representing both the primary bottleneck and the greatest opportunity for optimization.

Context: The Road to 57.6 tok/s

To understand why message 136 was written, we must first appreciate the journey that preceded it. The broader session (segments 0–16 of the root conversation) had been a marathon of infrastructure setup and debugging. The team had:

• Installed NVIDIA drivers (590.48.01) and CUDA Toolkit 13.1 on Ubuntu 24.04
• Verified two (later eight) RTX PRO 6000 Blackwell GPUs
• Resolved flash-attn installation issues by reducing parallel compilation jobs from 128 to 20
• Built a compatible stack: PyTorch 2.9.1, flash-attn 2.8.3, vLLM 0.15.1→0.16.0rc2
• Fixed garbage output from two bugs in Triton MLA attention and GGUF dequantization shard ordering
• Optimized single-request decode throughput from ~20 tok/s to ~57 tok/s using CUDAGraph and NCCL tuning By message 136, the assistant had already established a stable baseline of ~57.5 tok/s. The NCCL "LL" (low-latency) protocol had been enabled, providing a 22% improvement over the default protocol. But the question remained: could the system go faster, and if so, what would need to change? Messages 120–135 form the immediate prelude to message 136. In those messages, the assistant systematically: 1. Measured HBM bandwidth (~1458 GB/s) and confirmed the system was not memory-bandwidth bound (only ~14% utilization) 2. Verified decode rate consistency across different token lengths (50.7–57.9 tok/s) 3. Measured TTFT (time to first token) at ~52ms, confirming it was negligible for long generations 4. Discovered that vLLM's custom allreduce was disabled for more than 2 PCIe-only GPUs, forcing fallback to NCCL allreduce 5. Examined the custom allreduce implementation and understood why it couldn't scale to 8 PCIe GPUs without NVLink 6. Checked available metrics and found no fine-grained timing instrumentation exposed This investigative groundwork sets the stage for message 136, where the assistant synthesizes everything into a coherent bottleneck analysis.

The Message: A Quantitative Deconstruction of 17.28ms

The subject message begins with a precise calculation drawn from vLLM's Prometheus metrics:

Average inter-token latency: 30.22/1749 = 17.28ms = 57.9 tok/s. Average time per output token: 0.2236/17 = 13.15ms per request (but this is the per-request average, not per-token).

The assistant immediately distinguishes between two metrics: the inter-token latency (the time between consecutive tokens within a single generation, 17.28ms) and the per-request average (13.15ms, which averages over the prefill phase and is therefore misleading for decode analysis). This distinction is crucial—the assistant correctly identifies that 17.28ms is the "hard floor" for decode throughput.

From here, the message unfolds in three distinct analytical movements.

Movement 1: Theoretical Allreduce Estimation

The assistant constructs a model of the per-token compute graph for a 78-layer MoE model with tensor parallelism of 8:

• Each layer performs approximately 2 allreduces: one after attention output aggregation, one after MoE expert aggregation
• With 78 layers, that's 156 allreduces per token
• Each allreduce communicates the hidden dimension (6144) in fp16: 12,288 bytes
• On PCIe Gen5 with 8 GPUs using ring allreduce and the LL protocol:
• Data transfer time is negligible (~0.1μs for 12KB)
• Per-hop latency is estimated at 5–10μs
• With 7 hops in an 8-GPU ring, per-allreduce latency is ~50–70μs
• Total allreduce time: 156 × 50–70μs = 7.8–10.9ms This estimate is derived from first principles, combining knowledge of the model architecture (78 layers, MoE routing, hidden dimension), the communication topology (PCIe ring), and the NCCL protocol characteristics. The assistant explicitly states the assumptions, including the per-hop latency estimate, which is based on typical PCIe P2P latencies rather than measured values. The conclusion is striking: allreduce accounts for 45–63% of the 17.28ms per token, leaving only 6–10ms for actual compute (dequantization, GEMM operations, attention, and routing).

Movement 2: Empirical Validation via Protocol Comparison

Rather than relying solely on theoretical estimates, the assistant cleverly reuses data from earlier experiments to validate the model. In previous messages, the assistant had compared throughput with and without the NCCL LL protocol:

• With LL protocol: 128 tokens in 2.22s → 57.6 tok/s → 17.36ms/token
• Without LL protocol: 128 tokens in 2.71s → 47.3 tok/s → 21.17ms/token
• Difference: 0.49s for 128 tokens → 3.83ms/token savings from LL The LL protocol primarily reduces allreduce latency (by eliminating certain protocol overheads like the "simple" protocol's extra synchronization steps). Therefore, the 3.83ms savings can be attributed entirely to allreduce optimization. Dividing by 156 allreduces gives 24.6μs savings per allreduce—a figure that aligns well with the expected reduction from eliminating protocol overhead. This cross-check is elegant: it uses the LL vs. non-LL comparison as a natural experiment to isolate allreduce time. If allreduce were not the dominant factor, switching protocols would show minimal improvement. The 22% improvement confirms that communication is indeed the bottleneck. From this, the assistant derives:
• Allreduce time with LL: ~9–11ms/token (consistent with the theoretical estimate)
• Compute time: ~6–8ms/token (the residual after subtracting allreduce)
• Allreduce as percentage of total: 55–65%

Movement 3: The Theoretical Ceiling

The final calculation answers the question that motivated the entire analysis: what would throughput be if allreduce were free?

Theoretical max with zero-cost allreduce: ~100–140 tok/s

This is derived by dividing the estimated compute time (6–8ms) into 1 second. If allreduce overhead were eliminated, the per-token time would drop from 17.28ms to ~7–10ms, yielding 100–140 tok/s—roughly double the current throughput.

This number serves as both a goal and a reality check. It tells the team that the current 57.6 tok/s is not a fundamental limit of the GPUs' compute capability but rather an artifact of the PCIe communication topology. It also implies that significant gains would require either:

• Reducing the number of allreduces (e.g., through operator fusion)
• Reducing allreduce latency (e.g., through NVLink or custom allreduce)
• Hiding communication behind computation (e.g., through overlapping)

The Thinking Process: A Window into Performance Analysis

What makes this message particularly valuable is the transparency of the assistant's reasoning. The thinking process reveals several hallmarks of expert-level performance analysis:

1. Triangulation from multiple evidence sources

The assistant doesn't trust any single measurement. The allreduce estimate is cross-checked three ways: theoretical calculation from first principles, empirical comparison of LL vs. non-LL protocols, and consistency checking against the known total per-token time. Each approach has different assumptions and error sources; their convergence on ~9–11ms for allreduce time strengthens confidence in the result.

2. Knowing what NOT to do

The assistant explicitly considers and rejects the option of enabling NCCL debug logging (NCCL_DEBUG=INFO), recognizing that it would require a server restart and add significant overhead. This is a mature engineering judgment—sometimes the best measurement is the one you already have, and adding instrumentation would change the behavior being measured.

3. Boundary analysis

By calculating the theoretical maximum with zero-cost allreduce (100–140 tok/s), the assistant establishes an upper bound that guides future optimization efforts. This is more useful than a simple bottleneck identification because it quantifies the opportunity: if allreduce could be eliminated, throughput would roughly double.

4. Attention to metric semantics

The assistant carefully distinguishes between inter-token latency (17.28ms) and per-request average latency (13.15ms), recognizing that the latter is contaminated by prefill timing. This attention to what each metric actually measures prevents incorrect conclusions.

Assumptions and Their Validity

The analysis in message 136 rests on several assumptions, some explicit and some implicit:

Explicit Assumptions

• 2 allreduces per layer: This assumes the standard transformer architecture where attention output and MoE output each require an allreduce. This is correct for the GLM-5 architecture with tensor parallelism.
• Per-hop latency of 5–10μs: This is a typical range for PCIe P2P on modern systems, but actual latency depends on the specific PCIe topology, root complex design, and NUMA configuration. The wide range (2×) reflects this uncertainty.
• 156 allreduces per token: This assumes all 78 layers participate equally, which is correct for a standard transformer decode pass.

Implicit Assumptions

• Allreduce operations are not overlapped with compute: The model assumes sequential execution (compute → allreduce → compute → allreduce...). In practice, NCCL may overlap communication with computation to some degree, which would reduce the effective allreduce overhead.
• All allreduces have equal cost: The analysis averages across all 156 allreduces, but in practice, the attention-output allreduce and MoE-output allreduce may have different sizes or timing characteristics.
• The LL vs. non-LL difference purely reflects allreduce optimization: This assumes no other aspect of the system changed between the two measurements, which is reasonable for a controlled experiment but worth noting.

Potential Mistakes or Incorrect Assumptions

The most significant potential issue is the sequential execution model. Modern NCCL implementations can overlap communication with computation using CUDA streams and asynchronous operations. If some allreduce operations overlap with subsequent compute, the effective allreduce overhead would be less than the sum of individual allreduce latencies. This would mean:

• The true allreduce contribution might be lower than 55–65%
• The compute time estimate (6–8ms) might be correspondingly higher
• The theoretical maximum with zero-cost allreduce might be lower than 100–140 tok/s However, the LL vs. non-LL comparison provides a strong empirical check. The 3.83ms savings from switching protocols represents a lower bound on allreduce time (since the LL protocol reduces but doesn't eliminate allreduce overhead). The fact that this alone is 22% of total time confirms that communication is a major factor, even if the exact percentage is uncertain. Another subtle issue: the assistant assumes that the LL protocol's 3.83ms savings are evenly distributed across all 156 allreduces. In reality, the LL protocol may have different effects on different allreduce sizes or patterns. The 24.6μs per-allreduce figure is an average that may not apply uniformly.

Input Knowledge Required

To fully understand message 136, the reader needs:

1. vLLM architecture knowledge: Understanding that tensor parallelism requires allreduce operations after attention and MoE layers, and that vLLM has both a custom allreduce (using IPC shared memory) and a fallback to NCCL.
2. NCCL protocol knowledge: The difference between LL (low-latency), Simple, and other NCCL protocols, and why LL reduces latency for small messages.
3. Transformer model architecture: The 78-layer structure of GLM-5, the MoE routing mechanism (8/256 experts), and the hidden dimension (6144).
4. Hardware topology understanding: PCIe Gen5 bandwidth characteristics, P2P latency between GPUs on the same PCIe root complex, and the role of NVLink in GPU-to-GPU communication.
5. Performance analysis methodology: The ability to decompose end-to-end latency into components, use comparative experiments to isolate variables, and estimate theoretical bounds.
6. Previous session context: The NCCL LL protocol experiment (messages 120–128), the custom allreduce investigation (messages 129–134), and the metric inspection (message 135).

Output Knowledge Created

Message 136 produces several valuable outputs:

1. A quantified bottleneck diagnosis: Allreduce communication is identified as the primary bottleneck, consuming 55–65% of per-token latency. This is not a vague "communication is slow" claim but a specific, quantified result.
2. A validated analytical model: The theoretical allreduce estimate (~7.8–10.9ms) is cross-validated against empirical data (~9–11ms from LL protocol comparison), creating a model that can be reused for other configurations.
3. A theoretical performance ceiling: The 100–140 tok/s estimate provides a clear optimization target and helps prioritize future work.
4. A methodological template: The approach of combining first-principles estimation with empirical cross-validation is a reusable pattern for performance analysis in similar systems.
5. A decision framework: The analysis implicitly evaluates several optimization strategies: - Operator fusion (e.g., fusing allreduce with RMS normalization): Could reduce the number of allreduces - Custom allreduce on PCIe: Would require solving the P2P scalability problem - NVLink upgrade: Would eliminate the bottleneck entirely but requires hardware changes - Communication-computation overlap: Could hide some allreduce latency

Why This Message Matters

Message 136 is the culmination of a sustained performance investigation spanning 16 messages. It represents the moment when scattered observations—HBM bandwidth measurements, protocol comparisons, code inspections, metric readings—coalesce into a coherent understanding of the system's behavior.

The message is also notable for what it doesn't do. It doesn't jump to conclusions, doesn't propose premature optimizations, and doesn't rely on a single data point. Instead, it builds a case methodically, using multiple independent lines of evidence that all point to the same conclusion. This is performance analysis at its best: rigorous, transparent, and actionable.

For the broader session, this analysis sets the stage for the next optimization frontier. The subagent task that spawned this investigation was to "enable allreduce-RMS fusion"—an optimization that would reduce the number of allreduces by combining the allreduce with the subsequent RMS normalization. The analysis in message 136 provides the quantitative justification for why this fusion matters: if it can eliminate even a fraction of the 156 allreduces per token, the impact on throughput would be directly measurable.

Conclusion

Message 136 demonstrates that deep performance understanding doesn't always require sophisticated profiling tools. Sometimes the most powerful insights come from combining first-principles reasoning with cleverly designed comparative experiments. By decomposing the 17.28ms per-token latency into its constituent parts—allreduce communication (~9–11ms) and compute (~6–8ms)—the assistant provides a clear diagnosis that guides all subsequent optimization efforts.

The analysis also serves as a cautionary tale about hardware topology: eight powerful GPUs, each capable of ~1.5 TB/s HBM bandwidth, can be significantly constrained by PCIe communication overhead. In the world of distributed LLM inference, the network between GPUs is often the bottleneck, not the compute itself. Message 136 captures this reality with precision and clarity, turning raw metrics into actionable insight.