The Moment of Truth: Testing Custom Allreduce on PCIe-Connected Blackwell GPUs

In the high-stakes world of large language model inference optimization, every millisecond counts. For the engineer wrestling with an 8× RTX PRO 6000 Blackwell GPU system connected via PCIe, the challenge was stark: the EAGLE-3 speculative decoding verify step was performing 122 NCCL allreduce operations per iteration, consuming roughly 30 milliseconds and rendering speculation slower than the baseline. The search for a faster allreduce strategy had become the central bottleneck. Message [msg 5197] captures the precise moment when one of the most promising optimization paths—the custom allreduce kernel forced onto PCIe topology—was put to the test, and the data that would ultimately bury that approach began to arrive.

The Context: A Systematic Optimization Campaign

To understand why message [msg 5197] was written, one must appreciate the experimental dead ends that preceded it. The assistant had been methodically testing every available allreduce optimization for the Blackwell (SM120) architecture. FlashInfer allreduce fusion had failed because its JIT compiler does not support SM120. Torch symmetric memory had failed because SM120 is not in its architecture lookup table. Expert Parallelism with the flashinfer A2A backend had hit assertion errors and out-of-memory conditions. Each approach had been systematically eliminated, leaving the custom allreduce kernel as the next candidate.

The custom allreduce kernel in SGLang is designed for low-latency small-tensor allreduce operations. It uses IPC (Inter-Process Communication) shared memory to allow direct GPU-to-GPU reads, bypassing the NCCL library entirely. The idea is elegant: instead of routing allreduce operations through NCCL's ring algorithm (which pipelines data sequentially from neighbor to neighbor), the custom kernel lets every GPU read from every other GPU's memory simultaneously in a single stage. For NVLink-connected GPUs, this approach can dramatically reduce latency. But for PCIe-connected GPUs, the implications were unknown—and that is precisely what this experiment aimed to discover.

The Message: Launch and Wait

Message [msg 5197] is deceptively brief. It contains just three elements: a confirmation that the benchmark script exists ("Good, it's there"), a bash command that sleeps for 660 seconds (11 minutes) before grepping the server log, and the resulting log output showing the custom allreduce being enabled on multiple tensor-parallel ranks.

sleep 660 && ssh root@10.1.230.174 'grep "FORCE_CUSTOM_AR\|KV Cache\|Capture cuda\|server ready\|Not enough\|avail mem.*Load weight end\|allreduce\|mem_fraction" /data/eagle3/synth_100k/logs/custom_ar_pcie_v3.log | tail -30'

The 660-second sleep is not arbitrary. Loading the Kimi-K2.5 model across 8 GPUs with tensor parallelism takes significant time—the weights must be distributed, CUDA graphs must be captured, and the KV cache must be allocated. Earlier experiments had shown that the full startup sequence could take 5–10 minutes. The assistant's approach is to fire the command, wait the full duration, and then inspect the log to see whether the server started successfully and what configuration was applied.

The output confirms that the custom allreduce is indeed being activated:

[2026-02-27 13:07:12 TP5] SGLANG_FORCE_CUSTOM_AR_PCIE=1: Enabling custom allreduce on PCIe topology (8 GPUs). This uses IPC shared memory for low-latency small-tensor allreduce.
[2026-02-27 13:07:12 TP0] SGLANG_FORCE_CUSTOM_AR_PCIE=1: Enabling custom allreduce on PCIe topology (8 GPUs). This uses IPC shared memory for low-latency small-tensor allreduce.
[2026-02-27 13:07:12 TP3] SGLANG_FORCE_CUSTOM_AR_PCIE=1: Enabling custom allreduce on PCIe topology (8 GPUs). This uses IPC shared memory for lo...

The log lines from TP5, TP0, and TP3 (and presumably all 8 ranks, though the output is truncated) confirm that the SGLANG_FORCE_CUSTOM_AR_PCIE=1 environment variable took effect and the custom allreduce path was entered. This is the critical input knowledge: the assistant knows that the custom allreduce has a special PCIe mode that can be force-enabled via environment variable, and that this mode uses IPC shared memory. The log output validates that the mechanism is working as designed.

The Reasoning and Assumptions

The assistant's decision to test the custom allreduce on PCIe rested on several assumptions. First, that the custom kernel's single-stage all-to-all pattern might outperform NCCL's multi-stage ring algorithm for the small tensors used in the EAGLE-3 verify step. The verify step involves many small allreduce operations (122 per iteration), and the custom kernel was specifically designed for low-latency small-tensor communication. Second, that the PCIe topology, despite being a shared bus, might still benefit from the reduced number of communication stages. Third, that the IPC shared memory mechanism would provide lower latency than NCCL's buffered approach.

These assumptions were reasonable given the available information. The custom allreduce kernel had been shown to work well on NVLink-connected systems. The PCIe variant was explicitly implemented and documented. The theoretical advantage of a single-stage all-to-all over a multi-stage ring is well-understood in parallel computing. However, the assistant was about to discover a critical flaw in this reasoning.

The Outcome: A Devastating Result

The result, which arrives in the messages immediately following [msg 5197], was unambiguous. The server started with only 0.90 GB of KV cache (compared to 10.42 GB in the NCCL baseline), because the IPC buffer allocations consumed most of the available GPU memory. And the benchmark told the real story: 38.3 tok/s, compared to the 82 tok/s NCCL baseline. The custom allreduce was more than 2× slower.

The assistant's analysis in [msg 5202] identified the root cause: "The 1-stage allreduce requires every GPU to read from all 7 other GPUs simultaneously via PCIe, creating massive bus contention. With NCCL Ring, the traffic is pipelined (each GPU only reads from one neighbor)." For 8 GPUs on PCIe, the all-to-all pattern creates 56 simultaneous cross-GPU reads, saturating the PCIe switch. NCCL Ring's sequential neighbor-to-neighbor design is specifically optimized to minimize this contention.

This is a beautiful example of a theoretical advantage being defeated by a real-world hardware constraint. In theory, fewer communication stages should mean lower latency. In practice, the PCIe bus cannot sustain 56 simultaneous cross-device reads, so the "single stage" becomes a bottleneck rather than an optimization. The NCCL Ring algorithm, despite requiring more stages, actually completes faster because each stage only uses a fraction of the bus bandwidth.

Input and Output Knowledge

To fully understand this message, the reader needs several pieces of input knowledge. They need to understand the SGLang server architecture, including tensor parallelism across 8 GPUs and the role of allreduce in the EAGLE-3 verify step. They need to know about the custom allreduce kernel's PCIe mode and its IPC shared memory mechanism. They need to understand the PCIe topology of the system and the implications of bus contention. They need to recognize the log patterns that indicate successful server startup versus failure. And they need to know the benchmark methodology that will be applied after the server starts.

The output knowledge created by this message is the confirmation that the custom allreduce PCIe mode is functioning—the environment variable is being read, the kernel path is being entered, and the server is starting without crashing. This is a non-trivial result: earlier attempts with the custom allreduce had caused out-of-memory errors due to incorrect mem_fraction_static settings. The fact that the server starts cleanly is itself a validation of the configuration fix applied in the preceding messages.

The Thinking Process

The thinking visible in this message is subtle but important. The assistant is operating in a systematic experimental framework: formulate a hypothesis (custom allreduce might outperform NCCL on PCIe), set up the experiment (launch server with SGLANG_FORCE_CUSTOM_AR_PCIE=1), wait for results (sleep 660 seconds), collect data (grep the log), and evaluate. The "Good, it's there" acknowledgment shows that the assistant is checking prerequisites before committing to the long wait—the benchmark script must be present because without it, the experiment cannot be evaluated.

The choice of grep patterns is also revealing. The assistant is looking for FORCE_CUSTOM_AR (to confirm the kernel is active), KV Cache and Capture cuda (to confirm successful initialization), server ready (the final readiness signal), Not enough (to catch memory errors early), avail mem.*Load weight end (to measure memory consumption), allreduce (for any allreduce-related diagnostics), and mem_fraction (to verify the memory configuration). This comprehensive pattern set reflects the assistant's experience with the many failure modes encountered in previous experiments.

The Larger Arc

Message [msg 5197] sits at a turning point in the optimization campaign. It is the moment when the custom allreduce approach is tested and found wanting. The assistant will mark it as a "DEAD END" in the todo list ([msg 5203]) and pivot to other approaches—first MSCCL++ and torch symmetric memory, then ultimately the decision to upgrade CUDA to version 13 to unblock Blackwell-native optimizations. But the data from this experiment is invaluable: it confirms that NCCL Ring remains the best allreduce strategy for PCIe-connected multi-GPU systems, and it provides a clear physical explanation (bus contention) that guides future optimization efforts away from all-to-all patterns and toward approaches that reduce the number of allreduce operations rather than accelerating individual ones.

In the end, the custom allreduce experiment failed—but it failed informatively, teaching the team something fundamental about the hardware they were working with. That is the mark of good engineering: not avoiding failure, but ensuring that every failure produces knowledge.