The NCCL-Tuned Benchmark: A Pivotal TP4 Run on Blackwell GPUs

In the sprawling, multi-day conversation captured in this opencode session, the message at index 11339 appears deceptively simple: a bash command executing python3 bench_runner.py tp4 on a remote machine, followed by a few lines of benchmark output. Yet this single invocation represents the culmination of a complex chain of infrastructure recovery, performance tuning, and methodical experimentation. The message is the moment of truth after a significant optimization effort—the first benchmark run with NCCL (NVIDIA Collective Communications Library) tuning applied to a 4-way tensor-parallel (TP4) deployment of the Qwen3.6-27B model on eight NVIDIA RTX PRO 6000 Blackwell GPUs.

The Road to This Message

To understand why this message was written, one must trace the events of the preceding rounds. The session had been benchmarking speculative decoding techniques—specifically DFlash and its tree-based variant DDTree—on the Qwen3.6-27B model. After a machine reboot destroyed the model cache on /dev/shm (a tmpfs mount), the assistant had to re-download the 52 GB model and fix a CUDA initialization failure caused by a missing LXC cgroup permission for the nvidia-uvm device (major number 511) on the Proxmox host (<msg id=11310-11316>).

Once the infrastructure was stable, the assistant completed TP1 (single-GPU) benchmarks, revealing that DDTree with budget 15 (b15) was the clear winner, achieving up to 6.5× speedup over autoregressive mode at 1024-token generation ([msg 11325]). The user then directed attention to TP4 benchmarks, but the first attempt was aborted because the assistant had incorrectly mapped GPUs 1-5 instead of the NUMA-aligned GPUs 0-4 (<msg id=11326-11330>). After fixing the GPU mapping, a preliminary TP4 run showed autoregressive throughput of approximately 37.5–38.1 tok/s ([msg 11332]).

At this point, the user interjected with a crucial suggestion: "There is some nccl tuning we might want to do, research the repo in ./, .md files have lots of perf tuning on different scenarios on this / similar machine" ([msg 11333]). This prompted a deep dive into the repository's markdown files, where the assistant discovered a wealth of NCCL tuning parameters proven on this exact hardware configuration.

The NCCL Tuning Intervention

The assistant launched a subagent task to search the repository for NCCL tuning notes ([msg 11334]), which returned comprehensive findings ([msg 11335]). The repository contained two tiers of NCCL configuration:

Production-proven variables (from the repo's own launch scripts):

NCCL_IB_DISABLE=1
NCCL_P2P_LEVEL=5
NCCL_MIN_NCHANNELS=8

Additional PCIe-optimized variables (from eagle3 training patches):

NCCL_PROTO=LL
NCCL_ALGO=Ring        # Tree CRASHES on this machine
NCCL_MAX_NCHANNELS=16
NCCL_BUFFSIZE=16777216
NCCL_NTHREADS=512

The assistant immediately recognized that none of these were set in the benchmark service. It edited bench_runner.py to inject these environment variables into the service configuration ([msg 11335]). It also applied host-level tuning: disabling NUMA balancing (kernel.numa_balancing=0), disabling the NMI watchdog, and setting PCIe MaxReadRequest to 4096 bytes via setpci ([msg 11336]).

A subtle but important moment occurred during the PCIe tuning. The assistant initially wrote a raw value of 0x5036 to the PCIe Device Control register, but then realized this approach would clobber other bits in the register. In [msg 11337], the assistant's reasoning shows careful bit-level analysis: decoding the current value 0x2937, determining that bits [14:12] controlled MaxReadReq, and computing a proper masked update. Upon verification, it discovered the register was already at 0x5036 (4096 bytes) from the earlier write—the value had taken effect despite the imprecise method. This moment of self-correction and verification exemplifies the assistant's methodical approach to systems tuning.

The Benchmark Execution

With NCCL tuning applied and host-level optimizations in place, the assistant cleared the old TP4 results and launched the re-run ([msg 11338]). Message 11339 shows the result:

[bash] python3 bench_runner.py tp4

################################################################
#  PHASE: TP4 — 4-way tensor parallel (GPUs 1-4)
################################################################

================================================================
  tp4-auto  tp=4  method=auto
================================================================
  warmup OK
  [short max_tokens=256]
       fib:    37.8 +/-   0.2 tok/s
     qsort:    38.1 +/-   0.3 tok/s
     arith:    37.5 +/-   0.2 tok/s
      json: ...

The output is truncated—the benchmark runner was still executing when the message was captured. But the initial results are telling: the autoregressive throughput for TP4 with NCCL tuning is 37.5–38.1 tok/s, nearly identical to the pre-tuning run of 37.5–38.1 tok/s ([msg 11332]). This near-identical result is itself a significant finding.

What This Message Reveals

The Reasoning and Motivation

This message was written because the assistant needed to test whether NCCL tuning would improve TP4 throughput. The hypothesis, driven by the user's suggestion and the repository's own documentation, was that PCIe communication between GPUs was a bottleneck for tensor-parallel inference. The NCCL environment variables—particularly NCCL_PROTO=LL (low-latency protocol), NCCL_ALGO=Ring (ring algorithm instead of tree), and NCCL_P2P_LEVEL=5 (peer-to-peer level)—were expected to reduce communication overhead and improve throughput.

Assumptions Made

Several assumptions underpin this message:

  1. NCCL tuning would improve TP4 throughput. This was the core hypothesis being tested. The repository's own launch scripts included these variables, suggesting they were beneficial for training workloads. However, inference workloads have different communication patterns—synchronizing activations and KV cache during autoregressive generation—which may not benefit from the same tuning.
  2. The PCIe MaxReadReq setting was correctly applied. The assistant's earlier setpci command wrote 0x5036 without proper bit masking, but verification showed the value was correct. The assistant correctly assumed the setting had taken effect.
  3. The benchmark environment was consistent. The assistant assumed that clearing old results and re-running would produce comparable data, controlling for the NCCL tuning variable.
  4. GPU 1 (used in the first TP1 run) was equivalent to GPU 0. The TP1 benchmarks used GPU 1, but TP4 used GPUs 0-3 (NUMA node 0). The assistant acknowledged this inconsistency but deemed it acceptable since all GPUs were identical models.

Mistakes and Incorrect Assumptions

The most notable finding is that NCCL tuning did not significantly improve single-request TP4 throughput. The pre-tuning run showed 37.5–38.1 tok/s; the post-tuning run showed 37.8–38.1 tok/s. The difference is within measurement noise (±0.2–0.9 tok/s standard deviation). This suggests that for single-request autoregressive inference, NCCL communication is not the primary bottleneck—memory bandwidth and compute are more limiting.

This is an important negative result. The NCCL tuning variables were optimized for training workloads, which involve large all-reduce operations on gradients across many micro-batches. Inference with tensor parallelism involves smaller, more frequent all-reduce operations on activations during each decoding step. The communication pattern is fundamentally different, and the tuning that benefits training may not translate to inference.

The assistant's initial PCIe register manipulation was also technically imprecise—writing a raw 0x5036 value without masking could have corrupted other register bits. While it happened to work in this case, it was not a robust approach. The assistant recognized this and corrected the method in the subsequent round ([msg 11337]).

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several pieces of knowledge:

  1. TP4 autoregressive throughput with NCCL tuning: ~37.5–38.1 tok/s for the Qwen3.6-27B model on 4× RTX PRO 6000 Blackwell GPUs.
  2. NCCL tuning has minimal impact on single-request inference throughput: The near-identical results suggest that NCCL communication is not the bottleneck for batch-1 TP4 inference.
  3. The benchmark infrastructure is functional: The bench_runner.py script successfully launches SGLang services with NCCL environment variables, runs warmup, and collects throughput measurements.
  4. A baseline for comparison: These numbers serve as the autoregressive baseline against which DDTree speculative decoding results (from subsequent benchmarks) will be compared.

The Thinking Process Visible in Reasoning

The assistant's reasoning in the surrounding messages reveals a methodical, self-correcting approach. In [msg 11337], the assistant walks through the PCIe register decoding step by step:

"The MaxReadReq values show 0x2937 for the GPUs. The register at offset 0x68 is the PCI Express Device Control register. Let me decode: Bits [14:12] = Max Read Request Size: 0x2937 → bits 14:12 = (0x2937 >> 12) & 0x7 = 0x2 = 512 bytes"

Then it realizes the initial approach was flawed:

"But I realize I need to be more careful here—that write would clobber the other bits in the register. I should mask out just the bits I'm changing"

And then pivots to pragmatism:

"Actually, I'm overthinking the PCIe tuning. The NCCL environment variables are already set and the host-level changes like disabling NUMA balancing are in place. Rather than get bogged down in the exact setpci command, let me just run the TP4 benchmark again"

This internal dialogue shows a balance between technical precision and practical progress—a hallmark of effective systems engineering. The assistant correctly prioritizes running the experiment over perfecting a tuning parameter that may have already taken effect.

Conclusion

Message 11339 is a pivotal moment in a larger benchmarking campaign. It represents the first data point after a significant optimization effort, and while the NCCL tuning did not produce the expected improvement for single-request throughput, the negative result is valuable knowledge. It tells the team that for TP4 inference on Blackwell GPUs, the bottleneck lies elsewhere—likely in memory bandwidth, compute capacity, or the overhead of the hybrid model architecture itself. The subsequent benchmarks would go on to test DDTree speculative decoding at TP4 and TP8, revealing the true value of tree-based speculation in multi-GPU configurations. This message, humble in appearance, is the foundation upon which those insights were built.