The 18.5% Boost: How NCCL Tuning Unlocked DDTree's Concurrency Potential on 8× Blackwell GPUs

In the high-stakes world of large language model inference benchmarking, a 3% improvement is respectable, 7% is noteworthy, and 18.5% is the kind of number that makes engineers sit up and pay attention. Message [msg 11340] captures exactly such a moment: the assistant reports the results of applying NCCL (NVIDIA Collective Communications Library) tuning to a TP4 (4-way tensor parallel) inference setup for the Qwen3.6-27B model running on 8× RTX PRO 6000 Blackwell GPUs, then pivots to launching TP8 benchmarks. The message is simultaneously a progress report, a validation of a tuning hypothesis, and a decision point that shapes the remainder of the benchmarking session.

The Road to This Message

To understand why this message exists, we must trace the chain of events that led to it. The session had been a grueling exercise in infrastructure recovery and methodical benchmarking. Earlier messages document a cascade of problems: a machine reboot that wiped the model from /dev/shm (tmpfs), a missing nvidia-uvm device cgroup permission (major number 511 was blocked by LXC cgroup v2 policies, preventing CUDA initialization), and the subsequent re-download of the 52 GB Qwen3.6-27B model. The assistant had already completed TP1 (single-GPU) benchmarks, which established DDTree with budget 15 (b15) as the clear winner — achieving up to 6.5× speedup over autoregressive at 1024-token outputs.

When the assistant moved to TP4 benchmarks, the user intervened with a critical observation: the GPUs were being assigned incorrectly across NUMA domains. The machine has two NUMA nodes: NUMA 0 (GPUs 0–3) and NUMA 1 (GPUs 4–7). The assistant had configured TP4 to use GPUs 1–4, which straddled both NUMA domains — a configuration that forces PCIe traffic across the inter-NUMA interconnect for every collective communication operation. The user's correction to use GPUs 0–3 (all on NUMA 0) was essential for meaningful results.

Then came the pivotal user instruction: "There is some nccl tuning we might want to do, research the repo in ./, .md files have lots of perf tuning on different scenarios on this / similar machine." This directive sent the assistant on a targeted search of the repository's markdown files, which yielded a rich set of NCCL environment variables and host-level tuning parameters that had been empirically validated on this exact hardware configuration.

What the NCCL Tuning Entailed

The assistant's research task (executed as a subagent) uncovered two categories of tuning parameters. First, production-proven NCCL environment variables from the repository's own launch scripts:

NCCL_IB_DISABLE=1       # Disable InfiniBand (PCIe-only system)
NCCL_P2P_LEVEL=5        # P2P level for PCIe topology
NCCL_MIN_NCHANNELS=8    # Minimum communication channels

Second, PCIe-optimized variables derived from EAGLE-3 training patches:

NCCL_PROTO=LL           # Low-latency protocol
NCCL_ALGO=Ring          # Ring algorithm (Tree crashes on this machine)
NCCL_MAX_NCHANNELS=16   # Maximum channels
NCCL_BUFFSIZE=16777216  # Buffer size
NCCL_NTHREADS=512       # Thread count

Additionally, the assistant applied host-level tuning: disabling NUMA balancing (kernel.numa_balancing=0), disabling the NMI watchdog (kernel.nmi_watchdog=0), and setting PCIe MaxReadReq to 4096 bytes on all GPU devices. The MaxReadReq setting controls how much data the GPU can request in a single PCIe read transaction — larger values reduce the number of round-trips for bulk data transfers, which is critical for tensor-parallel inference where model shards must be synchronized across GPUs after every layer.

The Results: A Tale of Two Scalings

The assistant's message presents a clean before/after comparison table that tells a nuanced story:

| Config | Before NCCL tune | After NCCL tune | Delta | |---|---|---|---| | tp4-linear 256 | 135.9 | 140.1 | +3% | | tp4-linear 1024 | 151.5 | 154.8 | +2% | | tp4-b15 256 | 194.2 | 208.7 | +7.5% | | tp4-b15 1024 | 272.1 | 281.1 | +3.3% | | tp4-b15 C=8 agg | 1072.9 | 1270.8 | +18.5% |

The pattern is revealing. The autoregressive baseline (tp4-auto) barely budged — a 0.8% improvement from 37.5 to 37.8 tok/s, essentially noise. Linear DFlash speculative decoding showed modest gains of 2–3%. But DDTree with budget 15 showed a 7.5% improvement at short (256-token) outputs and a 3.3% improvement at longer (1024-token) outputs.

The standout number is the concurrency sweep result: 1270.8 tok/s aggregate at 8 concurrent requests with DDTree b15, an 18.5% improvement over the pre-tuning baseline of 1072.9 tok/s. This is the kind of gain that transforms a benchmark from "interesting" to "deployable."

Why DDTree Benefits More from NCCL Tuning

The differential improvement across configurations is not random — it reveals something fundamental about the communication patterns of different inference methods.

Autoregressive decoding is compute-bound and memory-bandwidth-bound: each step generates one token by loading the entire model from GPU memory and running a forward pass. The communication overhead is minimal because there's nothing to parallelize across GPUs except the model shards themselves. NCCL tuning can only improve the all-reduce operations that synchronize gradients (not relevant here) or the point-to-point transfers between shards during the forward pass. Hence the negligible improvement.

Linear DFlash speculative decoding generates a fixed tree of candidate tokens in a single forward pass, then verifies them. The tree structure is static and predictable, so the communication pattern is regular. NCCL tuning helps, but the gains are modest (2–3%) because the baseline was already reasonably optimized for this pattern.

DDTree, however, uses a dynamically constructed tree that adapts to the model's confidence distribution. This creates a more complex and irregular communication pattern: the tree structure changes per step, the number of branches varies, and the verification phase requires more sophisticated synchronization. In this environment, NCCL tuning has more room to make a difference — reducing latency on each collective operation compounds across the irregular communication graph, producing the outsized 7.5% gain at short outputs and the remarkable 18.5% gain under concurrency.

The concurrency result deserves special attention. At 8 concurrent requests, the system is operating at the intersection of two scaling dimensions: tensor parallelism (splitting each request across 4 GPUs) and request-level batching (running 8 requests simultaneously). The NCCL tuning reduces the latency of each collective operation, which directly improves the throughput of the batch scheduler. When 8 requests are competing for GPU compute and PCIe bandwidth, every microsecond saved on communication translates into more tokens generated per second across the entire batch.

The Implicit Decision: Proceed to TP8

After presenting the TP4 results, the assistant launches TP8 benchmarks with a single bash command: python3 bench_runner.py tp8. This is a decision made without explicit deliberation, but it carries significant weight. The TP4 results validated that:

NCCL tuning works and provides measurable benefit
DDTree b15 remains the optimal configuration
The system is stable and producing reproducible numbers TP8 (8-way tensor parallel across all GPUs) is the natural next step, but it's not guaranteed to be faster than TP4. In fact, earlier benchmarks in the session had shown TP4 outperforming TP8 for single requests due to PCIe cross-NUMA overhead — when all 8 GPUs are used, half of them are on NUMA 1 and must communicate with NUMA 0 across the inter-NUMA interconnect. The NCCL tuning might mitigate this overhead, but it cannot eliminate the fundamental latency penalty of crossing NUMA boundaries. The assistant is implicitly testing this hypothesis by running TP8 with the new tuning applied.

Assumptions and Potential Pitfalls

Several assumptions underpin this message that deserve scrutiny. First, the NCCL environment variables were sourced from training configurations (EAGLE-3 training patches) and applied to inference. While NCCL is used for similar collective operations in both contexts, the communication patterns differ — training involves gradient synchronization across all GPUs after every backward pass, while inference involves activation transfer between shards during the forward pass. The optimal settings may not be identical.

Second, the host-level PCIe MaxReadReq setting was applied with a potentially incorrect command. In the message immediately following ([msg 11341]), the assistant realizes that its initial setpci command may have clobbered other bits in the PCIe Device Control register. The correction uses bitwise masking ((0x$cur & 0x8FFF) | 0x5000) to preserve the existing register state while modifying only the MaxReadReq bits. This is a subtle but important fix — writing arbitrary values to PCIe configuration registers can cause unpredictable behavior, including system instability.

Third, the assistant assumes that the improvements seen at TP4 will generalize to TP8. This is not guaranteed — TP8 introduces additional communication overhead (more GPUs, more all-reduce operations, cross-NUMA traffic) that may diminish the relative benefit of NCCL tuning. The TP8 results (which follow in subsequent messages) will tell this story.

The Thinking Process Visible in the Message

The assistant's reasoning is evident in the structure of the message itself. It presents data before interpretation, showing the raw numbers before drawing conclusions. The table format — with before/after columns and delta percentages — reflects a methodical, scientific approach to performance analysis. The explicit highlighting of the 18.5% concurrency improvement (bolded in the original) signals that the assistant recognizes this as the most significant finding.

The decision to proceed directly to TP8 without waiting for user confirmation reflects an understanding of the session's momentum. The user had already approved the NCCL tuning investigation and was expecting benchmark results. By presenting the TP4 improvements and immediately launching TP8, the assistant maintains the workflow's cadence — a pattern of "measure, tune, measure again, scale up" that characterizes effective performance engineering.

Conclusion

Message [msg 11340] is a microcosm of the entire benchmarking session: it combines infrastructure recovery (the NCCL tuning was only possible after fixing the cgroup issue and re-downloading the model), data-driven validation (the before/after table), and forward-looking execution (launching TP8). The 18.5% concurrency improvement from NCCL tuning is the headline result, but the deeper insight is that the value of communication optimization scales with the complexity of the speculative decoding algorithm. DDTree's dynamic tree construction creates more opportunities for NCCL tuning to make a difference, and under concurrency pressure, those improvements compound dramatically. This message marks the transition from "does it work?" to "how fast can it go?" — the inflection point where a research benchmark becomes a production deployment candidate.