The 34% Optimization: How NCCL_PROTO=LL Unlocked 57 tok/s on PCIe-Bound Blackwell GPUs

Introduction

In the high-stakes world of large language model inference, every millisecond counts. When deploying a 744-billion-parameter model like GLM-5 across eight RTX PRO 6000 Blackwell GPUs connected solely by PCIe, the difference between a usable deployment and a frustratingly slow one often comes down to a single environment variable. Message [msg 2007] in this opencode session captures exactly that moment — the discovery that NCCL_PROTO=LL yields a 34% throughput improvement, pushing single-request decode performance from 43 tok/s to 57 tok/s. But this brief message, barely a paragraph long, represents the culmination of an intensive multi-day optimization journey spanning custom kernel debugging, CUDAGraph integration, speculative decoding experiments, and a deep analysis of NCCL's internal protocols. This article unpacks the reasoning, context, and significance of that single message.

The Message

The subject message reads in its entirety:

Great find — NCCL_PROTO=LL gives a 34% improvement by using the low-latency protocol for the tiny 12KB allreduce payloads. 57 tok/s is a significant step up from 43.

Following this is a structured todo list marking several items as completed: profiling the decode bottleneck (87% NCCL allreduce), enabling CUDAGraph (43 tok/s), the NCCL_PROTO=LL optimization (57 tok/s, +34%), and noting that custom allreduce was broken on PCIe configurations with more than two GPUs.

The message is deceptively short. It contains no bash commands, no code patches, no elaborate analysis — just a celebratory summary and a status update. Yet this brevity itself is meaningful: it signals that the heavy investigative work has already been done in the preceding subagent task (launched in [msg 2006]), and the assistant is now synthesizing the result and updating its mental model of the system's performance landscape.

The Long Road to 57 tok/s

To understand why this message matters, one must trace the optimization path that preceded it. The GLM-5 GGUF deployment on vLLM had already survived a grueling debugging phase. Earlier in the same segment ([msg 1992]), the model was producing incoherent garbage output despite loading successfully. Two bugs were identified and fixed: a Triton MLA attention backend issue where a custom PyTorch op created a phantom tensor disconnected from the output buffer, and a GGUF dequantization shard ordering bug for fused projections. These fixes restored correct model output but left the team with a performance problem.

Initial single-request decode throughput was approximately 20 tok/s with --enforce-eager mode. Profiling revealed that roughly 42% of the decode step time was spent on NCCL allreduce calls — the synchronization overhead required to aggregate gradients and activations across the eight GPUs in the tensor parallelism group. Enabling CUDAGraph (vLLM's CUDA graph capture mechanism) doubled throughput to ~43 tok/s by batching kernel launches and eliminating CPU-side dispatch overhead for the repetitive decode step. But even with CUDAGraph, the NCCL allreduce calls still consumed significant time during graph replay because the actual PCIe data transfers were unavoidable.

A subsequent deep-dive analysis ([msg 1998]) quantified the problem with precision: 158 NCCL allreduce calls per decode step, each carrying approximately 12 KB of data, totaling roughly 20 ms of the ~23 ms decode step. That's 87% of the time spent on communication, not computation. The GPU memory bandwidth was massively underutilized at only 12% — the GPUs could read the 4.4 GB of active weights in just 3 ms of pure compute, but spent the rest waiting for synchronization.

The team explored several avenues. Custom allreduce (vLLM's optimized shared-memory allreduce for NVLink-connected GPUs) was tried but found broken on PCIe-only topologies with more than two GPUs due to a C++ kernel bug. Speculative decoding with ngram matching showed promise but was content-dependent, delivering variable speedups. Allreduce-RMS fusion was considered but deemed incompatible with the PCIe-only hardware. Each dead end narrowed the search space, progressively pointing toward a single remaining lever: the NCCL communication protocol itself.

What NCCL_PROTO=LL Actually Does

The NCCL (NVIDIA Collective Communications Library) library supports multiple internal protocols for performing collective operations like allreduce. The default protocol, NCCL_PROTO=Simple, uses a straightforward ring-based algorithm that works well for large message sizes but incurs significant overhead for small payloads. The LL (Low Latency) protocol, by contrast, uses a different approach optimized specifically for small messages: it employs a flag-based synchronization mechanism that reduces the per-call latency by eliminating unnecessary memory copies and synchronization steps.

For the GLM-5 deployment, each of the 158 allreduce calls per decode step carries only ~12 KB of data — the per-tensor gradients and activations distributed across the eight GPUs. At this message size, the overhead of the Simple protocol's data movement and synchronization dominates the actual communication time. The LL protocol cuts through this overhead, reducing each call from approximately 127 μs to something substantially smaller, yielding the observed 34% end-to-end improvement.

This optimization is particularly elegant because it requires no code changes, no kernel patches, and no architectural modifications. It is a pure configuration change — setting an environment variable before launching the vLLM server. The fact that such a simple change yields a 34% improvement underscores how thoroughly the system was already optimized in every other dimension: the model weights were efficiently quantized (Q4_K_XL), the attention backend was correctly implemented, CUDAGraph was capturing the full computation graph, and the tensor parallelism configuration (TP=8) was optimal for the memory-bandwidth-bound workload.

Assumptions and Decisions

The message reveals several implicit assumptions. First, the assistant assumes that the NCCL_PROTO=LL protocol is safe to use with this hardware configuration — that it won't introduce numerical instability or correctness issues. This is a reasonable assumption given that NCCL's LL protocol is a well-tested, production-grade feature, but it is nonetheless an assumption worth noting. The task result from [msg 2006] explicitly confirmed that the output remained "coherent and correct" with the LL protocol, validating this assumption post-hoc.

Second, the assistant assumes that the 12 KB allreduce payload size is the key factor making LL beneficial. This is consistent with NCCL's documented behavior: LL is designed for small messages where per-call latency dominates bandwidth. The assistant's reasoning here is sound and grounded in the profiling data from the earlier analysis.

Third, the assistant implicitly decides that further optimization efforts should focus elsewhere. By marking the NCCL_PROTO=LL item as completed and noting that custom allreduce is broken on PCIe, the message signals a shift in strategy. The remaining ~17 ms per decode step (at 57 tok/s, each step takes ~17.5 ms) is now dominated by pure GPU compute — the memory-bandwidth-bound matmul operations through 78 transformer layers. Further improvements would require either reducing the number of allreduce calls (through kernel fusion), increasing effective memory bandwidth (through better quantization or model architecture changes), or accepting the hardware bottleneck as fundamental.

Knowledge Flow

Input knowledge required to understand this message includes: familiarity with NCCL's protocol options (Simple vs LL), understanding of tensor parallelism and allreduce in distributed inference, knowledge of the GLM-5 model architecture (744B parameters, MoE with 40B active), awareness of the PCIe-only hardware topology (8x Blackwell GPUs without NVLink), and comprehension of the earlier profiling results showing 87% NCCL overhead.

Output knowledge created by this message includes: the confirmed performance ceiling of ~57 tok/s for single-request decode on this hardware configuration, the empirical validation that NCCL_PROTO=LL is effective for small-message allreduce in PCIe-bound settings, the documented incompatibility of custom allreduce with >2 GPU PCIe configurations, and a clear priority ordering for future optimization work (kernel fusion > protocol tuning > architectural changes).

The Thinking Process

The message's reasoning structure is visible in its concise formulation. The assistant begins with an evaluative statement ("Great find") that frames the discovery positively, establishing the significance of what follows. The causal explanation ("by using the low-latency protocol for the tiny 12KB allreduce payloads") demonstrates a clear mental model of why the optimization works — the assistant has connected the specific property of the workload (small payloads) to the specific capability of the tool (LL protocol's low-latency optimization). The quantitative framing ("34% improvement," "57 tok/s is a significant step up from 43") shows the assistant thinking in terms of relative and absolute gains, situating this optimization within the broader trajectory from 20 tok/s (baseline) to 43 tok/s (CUDAGraph) to 57 tok/s (NCCL_PROTO=LL).

The todo list that follows reveals the assistant's prioritization framework. Items are marked completed in a logical sequence: first profile the bottleneck, then apply the highest-impact fix (CUDAGraph), then apply the next-highest (NCCL_PROTO=LL), then document dead ends (custom allreduce broken). This structured approach — measure, optimize, measure again, iterate — is the hallmark of systematic performance engineering.

Conclusion

Message [msg 2007] captures a pivotal moment in a complex optimization journey. It is the point where the low-hanging fruit has been picked, where the easy wins (CUDAGraph, NCCL protocol tuning) have been exhausted, and where the fundamental hardware bottleneck — PCIe bandwidth for inter-GPU communication — becomes the irreducible constraint. The 57 tok/s achieved here represents the practical ceiling for single-request decode on this particular hardware configuration, barring more invasive changes like kernel fusion or model architecture modification. The message is simultaneously a celebration of progress and a sober assessment of limits — a fitting milestone in the art of production ML inference optimization.