The AllReduce Revelation: Decoding the Bottleneck in 8×Blackwell Inference
Introduction
In the high-stakes world of large language model inference on multi-GPU systems, performance analysis is rarely a straightforward affair. What appears to be a compute-bound problem may turn out to be a communication-bound one, and what looks like a single bottleneck often decomposes into a cascade of interacting constraints. Message [msg 2448] from a deep profiling session of the Kimi-K2.5 INT4 model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs captures a pivotal moment of insight: the moment when the assistant, having gathered comprehensive profiling data, realizes that the dominant bottleneck is not the matrix multiplications (GEMMs) but rather the collective communication operations—specifically, AllReduce—that synchronize tensor-parallel shards across GPUs. This message is a masterclass in diagnostic reasoning, combining quantitative analysis with architectural understanding to pinpoint the exact source of performance degradation.
The Message
The subject message reads as follows:
This is the definitive data. Now I see something very interesting: the CUDA time per step (21.7ms) is higher than the measured TPOT (12.1ms). This makes sense — the profiler counts all CUDA operations across all streams, and with CUDAGraph, some operations overlap. But the key finding is clear.
>
Let me also check how much of the allreduce comes from the vllm::all_reduce custom path (which uses IPC shared memory):
>
``bash # The vllm::all_reduce at 180ms / 160 calls = 1125us/call is interesting # 160 calls = ~1 per step, not 127/step — this might be lm_head or special allreduce # The 19680 NCCL calls / 155 steps = 127/step matches 61 layers × 2 allreduces + some extras echo "Analysis of allreduce pattern:" echo " NCCL AllReduce: 19680 calls / 155 steps = 127 calls/step" echo " → 61 layers × 2 (attention + MoE) = 122 expected" echo " → Extra 5 per step may be from: prefill overlap, sampler, etc." echo "" echo " vllm::all_reduce: 160 calls / 155 steps ≈ 1.03 calls/step" echo " → This is the custom IPC-based allreduce for small messages" echo " → 1125us per call is VERY slow for an allreduce!" echo " → Likely used for a single large allreduce (lm_head output?)" echo "" echo " NCCL per-call: 78us (at 14KB hidden_size)" echo " Custom per-call: 1125us — 14× slower!" ``
This message is a shell command that prints an analysis of the allreduce pattern observed in the torch.profiler data. It is not executing a new experiment but rather reasoning aloud about data that has already been collected, parsed, and summarized by a Python analysis script written in the preceding message ([msg 2446]).
Context: The Profiling Campaign
To understand the significance of this message, one must appreciate the extensive effort that preceded it. The session had been running for hours across multiple segments (segments 14–19 of the conversation), covering everything from deploying the GLM-5-NVFP4 model to pivoting to Kimi-K2.5 INT4, resolving FP8 KV cache incompatibilities on Blackwell SM120 GPUs, and achieving approximately 60 tok/s single-request throughput. The current segment (segment 19) was dedicated to a comprehensive profiling campaign to understand exactly where the remaining performance headroom lay.
The profiling campaign followed a three-phase plan. Phase 1 consisted of macro-level throughput and latency benchmarks against the running vLLM server, which had already shown throughput plateauing at approximately 1,536 tok/s at concurrency 128. Phase 2 involved micro-benchmarks of individual GEMM operations at exact Kimi-K2.5 dimensions, plus NCCL AllReduce burst measurements. Phase 3 was the crown jewel: a full torch.profiler capture of actual decode steps.
The torch.profiler capture (executed in [msg 2443]) required a careful orchestration: stopping the production vLLM service, restarting it with --profiler-config pointing to a torch profiler configuration, waiting through the 30+ minute model loading process (monitored via periodic GPU memory checks and log inspection), sending warmup requests, starting the profiler, dispatching five single-stream decode requests of 32 tokens each, and finally stopping the profiler. The result was a set of eight trace files (one per GPU rank) totaling over 600 MB of compressed JSON trace data, plus profiler summary text files.
The Python analysis script written in [msg 2446] parsed the profiler summaries and produced the critical breakdown shown in [msg 2447]: total leaf kernel time of 3.363 seconds across approximately 155 decode steps, yielding 21.7ms per step. The category breakdown revealed the bombshell: AllReduce consumed 51.5% of total CUDA time, far outstripping every other category. The GEMMs that the assistant had initially suspected as the bottleneck were relegated to a secondary role.
The Reasoning Process
The subject message reveals the assistant's thinking as it digests this unexpected finding. The first observation is a subtle but important measurement artifact: the profiler reports 21.7ms of CUDA time per decode step, but the measured time-per-output-token (TPOT) from the macro benchmarks was only 12.1ms. The assistant correctly identifies the reason: the profiler counts all CUDA operations across all GPU streams, and with CUDAGraph (which vLLM's V1 engine uses to overlap operations), some of those operations execute concurrently rather than sequentially. The 21.7ms figure is the sum of all kernel durations assuming no overlap, while the 12.1ms TPOT reflects the actual wall-clock time where many kernels run in parallel across different streams. This is a sophisticated understanding of GPU profiling semantics.
The assistant then drills into the AllReduce category specifically, recognizing that there are two distinct allreduce implementations at play: NCCL AllReduce (the standard NVIDIA Collective Communications Library path) and a custom vllm::all_reduce that uses IPC shared memory for small message sizes. By dividing the total CUDA time by the number of calls for each variant, the assistant extracts per-call latencies.
The NCCL AllReduce numbers tell a clear story: 19,680 calls across 155 decode steps yields 127 calls per step. The assistant maps this onto the model architecture: Kimi-K2.5 has 61 transformer layers, each requiring two allreduces per step (one for attention, one for the MoE feed-forward network), giving 122 expected calls. The extra 5 calls per step are attributed to prefill overlap, the sampler, or other auxiliary operations. The per-call latency of 78 microseconds for a 14KB message (the hidden state size of 7,168 elements at BF16 precision, divided across 8 GPUs) is consistent with PCIe-based allreduce on this hardware.
The custom vllm::all_reduce path is more puzzling. With only 160 calls across 155 steps—essentially one call per step—and a per-call latency of 1,125 microseconds, it is 14× slower than NCCL AllReduce. The assistant hypothesizes that this single call per step is used for a large allreduce operation, possibly the language model head (lm_head) output, which would involve a larger message size than the per-layer hidden states. The "VERY slow" annotation reflects the assistant's surprise at the magnitude of the disparity.
Assumptions and Knowledge Required
To follow the assistant's reasoning, the reader needs substantial background knowledge. First, one must understand tensor parallelism (TP): the model's parameters and activations are sharded across 8 GPUs, so after each transformer sublayer (attention, MoE), the partial results from each GPU must be summed via AllReduce to produce the complete output. Second, one must know the architecture of Kimi-K2.5: a Mixture-of-Experts model with 61 transformer layers, each containing both an attention mechanism and an MoE feed-forward network, hence the 2×61 = 122 allreduces per step. Third, one needs familiarity with vLLM's internals: the V1 engine's use of CUDAGraph for kernel overlapping, the custom IPC-based allreduce for small messages, and the distinction between NCCL and vLLM's own allreduce implementation.
The assistant also makes several assumptions. It assumes that the 155 decode steps captured by the profiler are representative of steady-state behavior. It assumes that the profiler's categorization of operations into "allreduce" is accurate and complete. It assumes that the hidden state size of 14KB (7,168 elements × 2 bytes per BF16 element ÷ 8 GPUs) is the correct size for per-layer allreduce messages. And it assumes that the extra 5 allreduce calls per step beyond the expected 122 are from auxiliary operations rather than a miscount.
One potential subtlety: the assistant's calculation of "78us per call" for NCCL AllReduce divides total NCCL allreduce time by 19,680 calls. But if some of those calls are for different message sizes (e.g., larger during prefill than decode), the average could be misleading. The assistant does not break down NCCL allreduce by message size, which could hide important variation.
Output Knowledge Created
This message creates several pieces of actionable knowledge. First, it definitively establishes that AllReduce is the dominant bottleneck, consuming 51.5% of decode time. This overturns the earlier hypothesis that tiny MoE expert GEMMs (caused by TP=8 sharding) were the primary performance limiter. Second, it quantifies the per-call latency of both allreduce implementations: 78 microseconds for NCCL and 1,125 microseconds for the custom IPC path. Third, it maps the allreduce call count onto the model architecture, confirming that the expected 2×61 = 122 allreduces per step is accurate and that the remaining calls are from auxiliary operations.
The message also implicitly creates a new research direction. With AllReduce identified as the bottleneck, the obvious question is: can anything be done about it? The hardware constraint is fundamental—these eight GPUs are connected only via PCIe, without NVLink or NVSwitch. PCIe bandwidth is shared and latency is high compared to NVLink. The assistant's subsequent pivot to investigating speculative decoding (mentioned in the segment summary) represents a recognition that the communication bottleneck is a hard physical constraint that cannot be optimized away through software tuning alone.
Significance and Implications
The subject message is significant not just for what it reveals about Kimi-K2.5 inference on Blackwell GPUs, but for what it demonstrates about the process of performance debugging in modern ML systems. The assistant began with a hypothesis (tiny GEMMs are the bottleneck), designed experiments to test it (micro-benchmarks), gathered comprehensive data (torch.profiler), and then let the data speak—even when it contradicted the initial hypothesis. The willingness to follow the evidence wherever it leads is the hallmark of effective performance analysis.
The finding also has broader implications for multi-GPU inference without NVLink. As models grow larger and tensor parallelism becomes more widespread, the communication overhead of AllReduce becomes an increasingly severe bottleneck on PCIe-only systems. This is not a problem that can be solved by faster GPUs or better kernels; it requires architectural changes such as expert parallelism (which avoids allreduce for MoE layers but introduces all-to-all communication), disaggregated prefill/decode, or algorithmic innovations like speculative decoding that reduce the number of decode steps per token.
In the messages immediately following this one ([msg 2449] onward), the assistant updates the benchmark document with the profiling results and begins investigating speculative decoding as the most promising software-only optimization path. The seed of that pivot is planted right here, in the realization that the bottleneck is not in the compute but in the communication—and that no amount of GEMM optimization can make PCIe AllReduce faster.