The NCCL AllReduce Benchmark: Uncovering the Hidden Bottleneck in Multi-GPU Inference

In the midst of an intensive profiling campaign targeting the Kimi-K2.5 INT4 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message (index 2426) captures the precise moment when the investigation pivots from micro-benchmarking individual compute kernels to measuring the communication fabric that binds the distributed inference system together. The message is deceptively simple — a bash command that copies a Python script to a remote server and executes it across all eight GPUs using PyTorch's torchrun launcher — but it represents the culmination of careful reasoning about where the true bottleneck might lie, and it sets the stage for the most surprising discovery of the entire profiling campaign.

Context and Motivation

To understand why this message matters, one must appreciate the journey that led to it. The assistant and user had been deploying and optimizing large language models on a system with eight RTX PRO 6000 Blackwell GPUs connected only via PCIe — no NVLink bridges. Earlier work with the GLM-5-NVFP4 model had revealed that dtype casting (the conversion from FP4 to BF16 during matrix multiplication) consumed 69% of decode time. That bottleneck was specific to the NVFP4 format and the FlashInfer CUTLASS kernel path used by SGLang.

The current model, Kimi-K2.5 INT4, uses a completely different kernel path: Marlin W4A16 kernels, which fuse INT4 dequantization directly into the GEMM operation. The assistant hypothesized that Marlin might eliminate the dtype-cast overhead entirely, potentially shifting the bottleneck to something else entirely. This hypothesis needed testing, which motivated the three-phase benchmarking plan laid out in earlier messages ([msg 2407]).

By message 2426, the first two phases were complete. Phase 1 (macro-level throughput benchmarks) had shown the system plateauing at approximately 1,536 tokens per second at concurrency level 128, with single-stream TPOT (time per output token) around 12.5 milliseconds. Phase 2 (micro-benchmarks of individual GEMM operations) had been executed after stopping the vLLM server to free GPU memory ([msg 2422]), measuring the exact Marlin W4A16 kernel latencies at Kimi-K2.5's dimensions. The micro-benchmarks confirmed that Marlin was indeed efficient — the INT4 GEMMs were fast, and the dtype-cast overhead that plagued the GLM-5 deployment was absent.

But this raised a troubling question: if the GEMMs were fast, why was single-stream TPOT still 12.5ms? Something else must be consuming the majority of decode time. The assistant's attention turned to NCCL AllReduce — the communication primitive used to synchronize gradients and activations across the eight tensor-parallel GPUs.

The Message Itself

The message contains a single bash command that chains two operations:

scp /home/theuser/glm-kimi-sm120-rtx6000bw/bench_k25_nccl.py root@10.1.230.174:/tmp/bench_k25_nccl.py && ssh root@10.1.230.174 '/root/ml-env/bin/torchrun --nproc_per_node=8 /tmp/bench_k25_nccl.py' 2>&1

First, it copies the NCCL benchmark script (bench_k25_nccl.py) from the development machine to the inference server at IP 10.1.230.174. Then, it executes the script across all eight GPUs using torchrun --nproc_per_node=8, which launches one Python process per GPU with distributed process group initialization. The 2>&1 redirects stderr to stdout so all output — including PyTorch's distributed runtime warnings — is captured in the response.

The output shown is truncated, but it begins with a warning from torch/distributed/run.py about setting OMP_NUM_THREADS=1 to avoid CPU oversubscription. This is a standard PyTorch distributed launcher message, not an error. The full output (visible in the subsequent message [msg 2427]) revealed the critical data: AllReduce burst of 122 operations completed in 6.81ms total at batch size 1, with All-to-All operations showing dramatic latency increases at larger batch sizes.

The Thinking Process Behind This Message

The assistant's reasoning, visible across the preceding messages, follows a clear investigative logic:

1. Hypothesis formation: The macro benchmarks showed a throughput ceiling (~1,536 tok/s) that couldn't be explained by GEMM compute alone, since Marlin kernels were efficient. The assistant suspected communication overhead.
2. Bottleneck isolation strategy: Rather than guessing, the assistant designed a targeted benchmark that would measure NCCL AllReduce in isolation, at the exact message sizes and operation counts that the Kimi-K2.5 model uses during inference. The model has 61 layers, each requiring two AllReduce operations (one for attention, one for MoE), totaling 122 AllReduce calls per decode step.
3. Experimental design: The benchmark script (bench_k25_nccl.py) was written specifically for this system — it measures AllReduce on hidden_size=7168 (BF16, so 14,336 bytes per rank) across 8 GPUs, and also measures All-to-All at various token counts per GPU. The script was designed to be run with torchrun for proper NCCL process group initialization.
4. Execution timing: The assistant deliberately stopped the vLLM service before running micro-benchmarks ([msg 2422]), because all 8 GPUs were occupied with the model (96.9GB each). The NCCL benchmark, like the GEMM micro-benchmarks, required exclusive GPU access.

Assumptions and Their Validity

The assistant made several assumptions in this message:

• The NCCL benchmark script correctly models inference communication patterns: The script assumes 122 AllReduce operations per decode step, each on a tensor of size hidden_size=7168 in BF16. This is accurate for Kimi-K2.5's architecture (61 layers × 2 AllReduces per layer). However, it assumes all AllReduces are independent and sequential — in practice, vLLM's CUDAGraph might overlap some communication with computation, which the isolated benchmark wouldn't capture.
• torchrun with 8 processes correctly initializes NCCL: This is a standard PyTorch distributed setup. The warning about OMP_NUM_THREADS is benign. The assumption holds.
• The benchmark results are representative of in-production behavior: The NCCL benchmark measures raw AllReduce latency without concurrent compute kernels. In actual inference, AllReduce may overlap with GEMM execution (if the model uses asynchronous communication), potentially hiding some of the latency. The isolated benchmark would show the worst-case communication cost, which is useful for identifying the upper bound of the bottleneck.
• The remote server is accessible and the script path is correct: The SCP and SSH commands use hardcoded paths (/root/ml-env/bin/torchrun) and IP addresses. These assumptions were validated by successful execution.

Input Knowledge Required

To fully understand this message, one needs:

1. Knowledge of the system architecture: Eight RTX PRO 6000 Blackwell GPUs connected via PCIe (no NVLink), running Ubuntu 24.04 with CUDA 12.8 and PyTorch 2.10.0.
2. Understanding of tensor parallelism (TP=8): The model is sharded across all 8 GPUs, meaning each GPU holds 1/8 of each weight matrix. After each layer's computation, AllReduce synchronizes the partial results across GPUs.
3. Familiarity with NCCL and torchrun: NCCL is NVIDIA's collective communication library; torchrun is PyTorch's distributed launcher that sets up the process group for multi-GPU communication.
4. Knowledge of Kimi-K2.5 INT4 model architecture: 61 layers, hidden_size=7168, MoE with intermediate_size=2048, using Marlin W4A16 kernels for quantized matrix multiplication.
5. Context from the earlier phases: The macro benchmarks showing throughput plateau, and the micro-benchmarks showing efficient Marlin GEMMs, which together motivated the NCCL investigation.

Output Knowledge Created

This message produced the NCCL AllReduce benchmark results, which were analyzed in the subsequent message ([msg 2427]). The key findings:

• AllReduce burst (122 ops, batch=1): 6.81ms total — approximately 55.8 microseconds per AllReduce operation. This is significant because the total decode step time was measured at approximately 12.5ms, meaning AllReduce alone could account for over half of the step time.
• All-to-All latency characteristics: At small token counts (1-8 tokens per GPU), All-to-All was very fast (19-25 microseconds), but it jumped dramatically at 16+ tokens per GPU, revealing a threshold behavior likely related to PCIe bandwidth saturation. These results directly informed the subsequent decision to run a full torch.profiler capture ([msg 2427] onward), which ultimately confirmed that AllReduce accounted for 51.5% of decode time (11.17ms per step) — the dominant bottleneck in the system.

Significance in the Broader Campaign

This message sits at a critical inflection point in the optimization journey. Before the NCCL benchmark, the team had been focused on GEMM optimization — exploring Marlin kernels, persistent fused kernels, L2 cache pinning, and column-major tile scheduling for Blackwell's SM120 architecture. The NCCL results forced a fundamental re-evaluation: the bottleneck was not compute but communication.

This discovery had profound implications. It meant that no amount of GEMM kernel optimization would significantly improve throughput — the PCIe bus was the limiting factor. It led the team to explore Expert Parallelism (EP) as an alternative to Tensor Parallelism (TP), but the user correctly noted that without NVLink, all-to-all communication over PCIe would face serialization issues at higher batch sizes. Precision-compromising approaches (like FP8 KV cache quantization) were declined. The investigation ultimately pivoted to speculative decoding as the most promising software-only optimization path — using a smaller draft model to generate multiple candidate tokens per step, effectively hiding the AllReduce latency behind additional compute.

Conclusion

Message 2426 appears, on the surface, to be a routine execution of a benchmark script. But it represents the moment when a carefully constructed hypothesis about system performance was put to the test, and the results would redirect the entire optimization strategy. The assistant's methodical approach — isolating individual components (macro throughput, micro GEMM, NCCL communication) and measuring each at the exact dimensions and operation counts of the target model — exemplifies disciplined performance engineering. The NCCL benchmark results didn't just quantify the AllReduce overhead; they revealed a fundamental architectural constraint of PCIe-only multi-GPU inference that no amount of kernel tuning could overcome. This message is thus a turning point, where the investigation shifted from "how do we make the GEMMs faster?" to "how do we work around the communication bottleneck?" — a question that would define the remainder of the session.