The Benchmark That Confirmed a Hardware Wall: NCCL Tuning on 8× Blackwell GPUs

Introduction

In the high-stakes world of deploying trillion-parameter language models, every millisecond counts. When you have eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe, the question isn't just which model to run — it's how fast can you push data across those PCIe lanes. Message 2373 in this opencode session captures a pivotal moment: the benchmark that definitively proved that no amount of software tuning could overcome the fundamental hardware bottleneck of PCIe-bound multi-GPU inference.

This message is deceptively simple — a single bash command running a benchmark script, followed by a table of numbers. But those numbers carry the weight of hours of debugging, model downloads, environment setup, and a systematic exploration of what makes large language models fast (or slow) on real hardware. Let's dive deep into what this message means, why it was written, and what it reveals about the art of deploying 1T-parameter models.

The Message

The subject message at index 2373 reads:

[assistant] [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py
Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C=   1 |   6/  6 ok (0 err) | wall=  37.5s |    81.9 tok/s | lat avg=  6.25 p50=  6.25 p99=  6.27
  C=   2 |   8/  8 ok (0 err) | wall=  30.6s |   133.9 tok/s | lat avg=  7.64 p50=  7.64 p99=  7.70
  C=   4 |  12/ 12 ok (0 err) | wall=  26.1s |   235.2 tok/s | lat avg=  8.70 p50=  8.72 p99=  8.73
  C=   8 |  16/ 16 ok (0 err) | wall=  19.9s |   411.0 tok/s | lat avg=  9.96 p50= ...

On the surface, these are excellent numbers: 81.9 tokens per second for a single concurrent request, scaling to 411 tok/s at 8 concurrent requests. But the story is in what these numbers don't show — any improvement from the NCCL tuning that preceded them.

Why This Message Was Written

To understand why this benchmark was run, we need to trace the chain of decisions that led here. The session had been a whirlwind tour of model deployment on 8× Blackwell GPUs. The team had already tried:

  1. GLM-5-NVFP4 — a custom model that required extensive patching of vLLM's GGUF loader and a custom Triton MLA sparse attention backend
  2. NVFP4 Kimi-K2.5 — a 540GB model that achieved ~61 tok/s single-stream but was bottlenecked by PCIe allreduce across the 61-layer MLA architecture
  3. MiniMax-M2.5 FP8 — a 230B GQA model that achieved up to 4,000 tok/s with Expert Parallelism, proving that architecture matters enormously
  4. Native INT4 Kimi-K2.5 — the current model, a 547GB beast that loaded in 36 minutes and achieved 81.4 tok/s baseline The user's instruction in message 2359 was clear: "Run benchmarks and try to get to single stream >40~50; If not there try NCCL LL alg and other safe-ish tricks." The assistant had already exceeded the 40-50 tok/s target with the baseline (81.4 tok/s), but the user specifically asked to try NCCL tuning. So the assistant killed the running vLLM server, cleaned up zombie processes that were holding GPU memory, and relaunched with a set of NCCL environment variables designed to optimize the allreduce communication pattern. The specific tuning parameters chosen were: - NCCL_PROTO=LL — Low Latency protocol, already used in the baseline - NCCL_ALGO=Ring — Ring allreduce, often better for PCIe than Tree - NCCL_P2P_LEVEL=SYS — System-level P2P (cross-PCIe) - NCCL_MAX_NCHANNELS=16 — More channels for parallelism - NCCL_BUFFSIZE=16777216 — 16MB buffers - CUDA_DEVICE_MAX_CONNECTIONS=1 — Reduce kernel launch overhead After waiting another ~36 minutes for the model to reload, the assistant ran the benchmark script to measure whether these tuning parameters made any difference.

What the Results Reveal

The benchmark results are almost indistinguishable from the baseline. Compare:

| Concurrency | Baseline (msg 2364) | Tuned (msg 2373) | Difference | |-------------|---------------------|-------------------|------------| | C=1 | 81.4 tok/s | 81.9 tok/s | +0.6% | | C=2 | 133.1 tok/s | 133.9 tok/s | +0.6% | | C=4 | 236.3 tok/s | 235.2 tok/s | -0.5% | | C=8 | 413.1 tok/s | 411.0 tok/s | -0.5% |

The differences are within measurement noise. The NCCL tuning had effectively zero impact. This is the message's true payload: a negative result that is more informative than a positive one.

Assumptions Tested and Refuted

The tuning experiment was built on several assumptions, each of which the results now challenge:

Assumption 1: NCCL algorithm choice matters for PCIe-bound MLA inference. The Ring allreduce algorithm is generally considered better than Tree for systems with uniform bandwidth, and PCIe typically fits this profile. But the results show no meaningful difference, suggesting that the allreduce operation is not algorithm-bound — it's bandwidth-bound. When the PCIe bus is saturated, no algorithm can create more bandwidth.

Assumption 2: More NCCL channels improve throughput. Increasing NCCL_MAX_NCHANNELS from its default (typically 2-4) to 16 was intended to parallelize the allreduce across multiple CUDA streams. But on a PCIe-bound system, adding more channels doesn't help when the physical link is already saturated. The bottleneck is the wire, not the software stack.

Assumption 3: Buffer size tuning can reduce overhead. The 16MB buffer size (NCCL_BUFFSIZE) was chosen to match typical layer sizes in the MLA architecture. But again, when the fundamental limit is PCIe bandwidth, buffer tuning is rearranging deck chairs on the Titanic.

Assumption 4: The bottleneck is in NCCL at all. This was the most fundamental assumption — that the allreduce operation was the primary throughput limiter. The results confirm this was correct (the bottleneck is communication), but they also show that within the NCCL framework, there's no further optimization to be had. The bottleneck is the PCIe Gen5 bandwidth itself, not the software layer on top of it.

The Deeper Lesson: Hardware-Aware Model Selection

The real significance of this message becomes clear when we look at the broader context of the session. Earlier, the team had deployed MiniMax-M2.5 FP8, a 230B model using Grouped Query Attention (GQA) with Expert Parallelism (EP). That model achieved nearly 4,000 tok/s at high concurrency — an order of magnitude faster than the Kimi-K2.5 INT4.

Why the difference? MiniMax uses GQA, which requires much less allreduce communication than MLA (Multi-head Latent Attention). With EP, the MoE experts are distributed across GPUs without requiring allreduce for every layer. The Kimi-K2.5, by contrast, uses MLA with 61 layers, each requiring an allreduce across all 8 GPUs. Every token generated requires 61 allreduce operations, each moving ~2GB of data across PCIe.

The benchmark in message 2373 confirmed that no amount of NCCL tuning could change this fundamental reality. The hardware wall was real, and the only way around it was to choose models with communication-efficient architectures.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Understanding of NCCL — NVIDIA's Collective Communications Library, used for multi-GPU communication. Knowledge of Ring vs Tree algorithms, LL protocol, and channel count tuning is essential.
  2. Knowledge of tensor parallelism — How large models are split across GPUs, requiring allreduce operations to synchronize intermediate results after each transformer layer.
  3. PCIe topology awareness — Understanding that 8 GPUs connected via PCIe switches share limited bandwidth to the CPU, and that allreduce across all 8 GPUs requires multiple PCIe hops.
  4. Model architecture differences — MLA (Multi-head Latent Attention) vs GQA (Grouped Query Attention) and how they affect communication patterns. MLA requires more allreduce because of its latent bottleneck structure.
  5. The benchmark methodology — The benchmark script measures end-to-end throughput including prompt processing and token generation, using multiple concurrent requests to measure scaling behavior.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Confirmed benchmark numbers for Kimi-K2.5 INT4 on 8× RTX PRO 6000 Blackwell GPUs: 81.9 tok/s single-stream, scaling to 411 tok/s at C=8.
  2. Evidence that NCCL tuning is ineffective for PCIe-bound MLA inference. The Ring algorithm, LL protocol, 16 channels, and 16MB buffers produced no measurable improvement.
  3. Validation that INT4 quantization improves throughput over NVFP4 (81.9 vs 61 tok/s) for the same model architecture, because INT4 reduces the weight data that must be moved during allreduce.
  4. Confirmation of the hardware bottleneck hypothesis — the PCIe allreduce is the fundamental limiter, and no software tuning can overcome it.

Conclusion

Message 2373 is a masterclass in negative results. It's easy to celebrate when tuning works and throughput improves. It's harder — and more valuable — to run the experiment, get the same numbers, and accept that the hardware is the limit. The assistant didn't try to fudge the results or claim marginal improvements. It ran the benchmark, got essentially identical numbers, and let the data speak.

For anyone deploying large language models on multi-GPU systems, this message carries a crucial lesson: understand your bottleneck before you tune. If the bottleneck is PCIe bandwidth, no amount of NCCL knob-twiddling will help. The solution is architectural — choose models with less communication overhead, use expert parallelism, or invest in NVLink-connected GPUs. Sometimes the most important optimization is knowing when to stop optimizing.