The Pivot Point: Systematic NCCL Tuning for a 402B Parameter MoE Model

The Message

O1 + LL: ~57.7 tok/s — same as O2 + LL. The compilation level makes no difference for this workload.

>

Now let me try something more impactful — what about NCCL_NET_GDR_LEVEL or disabling GDR since we're on PCIe? And also NCCL_P2P_LEVEL=NVL vs SYS:

>

``bash ssh -o StrictHostKeyChecking=no root@10.1.230.174 'pkill -9 -f "vllm.entrypoints" 2>/dev/null; sleep 2; nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -r kill -9; sleep 5' ``

Introduction

Message 113 of this opencode session is a deceptively brief turning point in a systematic performance optimization campaign. At first glance, it appears to be a simple status update followed by a cleanup command. But beneath the surface, this message represents a critical moment of scientific reasoning: the closure of one experimental branch, the confirmation of a null hypothesis, and the deliberate pivot to a new set of independent variables. The assistant has been engaged in a methodical search through the high-dimensional configuration space of vLLM, NCCL, and GPU topology parameters, and message 113 marks the moment when one entire dimension of that space — the vLLM optimization level — is definitively ruled out as a lever for performance improvement.

The Context: A Performance Optimization Campaign

To understand message 113, one must appreciate the journey that led to it. The session is focused on deploying and optimizing the GLM-5 model, a massive 402 GB GGUF-quantized Mixture-of-Experts (MoE) model, across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. The assistant has been systematically benchmarking single-request decode throughput (tokens per second) while varying configuration parameters.

The optimization trajectory before message 113 tells a story of narrowing focus:

• Message 99 established a baseline: with default NCCL settings (no NCCL_PROTO override), the model achieved approximately 47.3 tok/s.
• Message 100 tested NCCL_PROTO=LL (Low Latency protocol) and found a dramatic 22% improvement to 57.6 tok/s. The assistant correctly identified this as "the most significant tuning parameter."
• Message 105 tested NCCL_PROTO=LL128, a variant designed for slightly larger message sizes. It performed worse than even the baseline at 45.5 tok/s, and also exhibited an early-stopping bug (generating only 54 tokens instead of the requested 128).
• Message 110–112 tested NCCL_PROTO=LL combined with --optimization-level 1 (a vLLM flag that controls the aggressiveness of CUDA graph compilation and torch.compile optimization). The result: 57.7 tok/s — essentially identical to the O2 + LL configuration. This last result is what the assistant reports in message 113. The phrasing is precise: "The compilation level makes no difference for this workload." This is a genuine discovery, not an assumption. The assistant had hypothesized that reducing the optimization level might reduce CUDA graph capture overhead or compilation time, thereby improving throughput. The data disproved that hypothesis.

The Reasoning: Why Compilation Level Doesn't Matter

The assistant's conclusion that optimization level is irrelevant is supported by the architecture of the workload. The GLM-5 model uses GGUF quantization with a Triton MLA (Multi-Head Latent Attention) backend. As noted in earlier messages ([msg 92]), vLLM was already using "PIECEWISE" CUDAGraph mode because FULL mode is not supported with the Triton MLA backend. In PIECEWISE mode, only certain segments of the model execution are captured into CUDA graphs, while others fall back to eager-mode PyTorch or torch.compile regions.

The optimization level flag in vLLM primarily affects:

1. Whether torch.compile is used at all
2. How aggressively CUDA graphs are captured
3. Whether certain fusion optimizations are applied For a model already constrained by GGUF dequantization overhead, PCIe-bound allreduce communication, and Triton-based attention kernels, the compilation strategy has limited headroom to affect. The bottleneck lies elsewhere — in the communication fabric and the dequantization-computation pipeline — not in the Python-to-CUDA graph capture efficiency. The assistant correctly reads this signal and moves on.

The Pivot: New Independent Variables

Having closed the optimization-level branch, the assistant pivots to two new NCCL parameters that target a fundamentally different part of the performance stack:

NCCL_NET_GDR_LEVEL

GDR stands for GPU Direct RDMA. This NCCL parameter controls whether GPUs can directly read from and write to remote GPU memory over the network fabric (e.g., InfiniBand or NVLink), bypassing host CPU memory. On systems with NVSwitch or NVLink interconnects, GDR can dramatically reduce latency. However, the assistant notes "since we're on PCIe" — the eight GPUs in this system are connected via PCIe Gen5 slots, not through NVLink or NVSwitch. In such a topology, GDR may actually add overhead because the PCIe fabric doesn't support the same direct peer-to-peer semantics as NVLink. Disabling GDR could force NCCL to use a simpler, more efficient communication path through host memory.

NCCL_P2P_LEVEL=NVL vs SYS

This parameter controls which peer-to-peer communication channels NCCL considers valid. NVL restricts communication to NVLink-connected peers, while SYS allows system-level (PCIe) paths. The assistant had been using NCCL_P2P_LEVEL=SYS in all prior experiments (explicitly set in the server launch commands at [msg 97], [msg 103], and [msg 110]). Now the assistant is considering testing NVL — but this is likely a misstep, as the GPUs are not NVLink-connected. Setting NCCL_P2P_LEVEL=NVL on a PCIe-only system would likely cause NCCL to fail to find valid communication paths, or fall back to extremely slow alternatives. The assistant's reasoning here reveals an assumption that NVL might still be worth testing, perhaps because some PCIe topologies with P2P support can masquerade as NVLink-like paths.

Assumptions and Potential Missteps

The assistant makes several assumptions in this message that deserve scrutiny:

1. That GDR is harmful on PCIe: This is a reasonable hypothesis, but not guaranteed. Some PCIe implementations with ACS (Access Control Services) and P2P support can benefit from GDR-like optimizations. The assistant is treating this as a testable hypothesis, which is correct scientific practice.
2. That NCCL_P2P_LEVEL=NVL is a viable alternative: This is more questionable. On a system with eight GPUs connected only via PCIe (no NVSwitch, no NVLink bridges), NVL would restrict NCCL to an empty set of valid peers. The assistant may be confusing the topology — earlier messages ([msg 92]) showed the assistant considering NUMA nodes and cross-socket communication, suggesting a dual-socket system where GPUs on the same socket might share a PCIe root complex. But NVLink is a different technology entirely.
3. That the optimization level experiment was conclusive: The assistant ran only three trials per configuration, which is sufficient for a coarse comparison but not for statistical significance. The variance between trials was low (~0.01s), so the conclusion is likely robust, but a more rigorous analysis would include confidence intervals.
4. That the kill command is sufficient cleanup: The assistant uses a multi-step kill sequence: pkill by process name, then nvidia-smi to find remaining compute apps, then xargs -r kill -9. This is thorough, but the sleep 5 after killing may not be enough for GPU memory to fully drain on all eight GPUs, especially if there are lingering CUDA contexts.

Input Knowledge Required

To fully understand message 113, the reader needs knowledge of:

• NCCL (NVIDIA Collective Communications Library): The communication library used by vLLM for tensor-parallel allreduce operations across GPUs. Understanding the protocol variants (LL, LL128, Simple) and the P2P level system is essential.
• vLLM optimization levels: The --optimization-level flag controls compilation strategy, CUDA graph capture, and fusion passes. Level 0 disables most optimizations, level 1 enables basic torch.compile, level 2 (default) enables aggressive CUDA graph capture.
• GPU topology and interconnects: The distinction between PCIe, NVLink, and NVSwitch, and how each affects communication latency and bandwidth. GDR (GPU Direct RDMA) is only beneficial when the fabric supports direct GPU-to-GPU memory access.
• GGUF quantization and MoE architecture: The GLM-5 model uses Q4_K quantization stored in GGUF format, which requires dequantization during inference. The Mixture-of-Experts architecture means only a subset of parameters are active per token, but allreduce still requires communication across all GPUs.
• The Triton MLA backend: A custom attention implementation that constrains which CUDA graph modes are available.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. Empirical result: vLLM optimization level has no measurable effect on decode throughput for this specific model+hardware combination. Future optimization efforts should not waste time on this parameter.
2. Refined search space: The assistant narrows the remaining high-impact variables to NCCL_NET_GDR_LEVEL and NCCL_P2P_LEVEL, both targeting the communication layer rather than the computation layer.
3. Experimental methodology: The message demonstrates a systematic approach to performance tuning: hold the best-known configuration constant (NCCL_PROTO=LL), vary one parameter at a time (optimization level), measure with sufficient trials (3), and pivot decisively when a parameter shows no effect.
4. Topology-aware reasoning: The assistant's comment about being "on PCIe" shows an understanding that NCCL tuning parameters are not universally beneficial — their effect depends on the physical interconnect topology.

The Thinking Process

The assistant's reasoning in message 113 is compact but reveals a structured thought process:

First, the result is reported with precision: "~57.7 tok/s — same as O2 + LL." The use of the tilde indicates awareness of measurement variance, and the direct comparison to the previous configuration shows the assistant is maintaining a mental model of all experimental conditions.

Second, the conclusion is stated explicitly: "The compilation level makes no difference for this workload." This is not merely an observation but an inference — the assistant is ruling out optimization level as a causal factor.

Third, the pivot is announced with a rationale: "Now let me try something more impactful." This reveals the assistant's prioritization framework. Having eliminated one variable, it moves to variables that target a different part of the performance stack — the communication layer rather than the compilation layer.

Fourth, the specific variables are named with a reasoning qualifier: "since we're on PCIe." This shows the assistant is applying domain knowledge about NCCL internals to generate hypotheses about which parameters might matter.

Finally, the cleanup command is issued. This is not just housekeeping — it's a deliberate step to ensure clean experimental conditions for the next test. The assistant has learned from earlier mistakes where residual processes interfered with new server launches (see <msg id=93-96> where processes had to be killed multiple times).

Conclusion

Message 113 is a masterclass in systematic performance optimization. It demonstrates the scientific method applied to systems tuning: form a hypothesis (optimization level affects throughput), design an experiment (compare O1+LL vs O2+LL), collect data (57.0, 57.6, 57.7 tok/s), interpret the results (no difference), and pivot to the next hypothesis (GDR and P2P level matter on PCIe). The message is brief, but it encapsulates a complete cycle of the experimental method. In the broader narrative of the session, it marks the transition from exploring vLLM's compilation stack to probing NCCL's communication topology — a shift that would ultimately lead to further throughput gains.