The Allreduce Overlap Mirage: A Systematic Investigation into Communication-Computation Overlap for Large MoE Inference on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference optimization, few bottlenecks are as stubborn and performance-critical as AllReduce — the collective communication operation that synchronizes hidden states across GPUs during tensor-parallel inference. When profiling revealed that AllReduce consumed a staggering 51.5% of decode time for the Kimi-K2.5 INT4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 without NVLink, the optimization path forward seemed clear: find a way to hide this latency by overlapping communication with computation.

This article synthesizes a comprehensive subagent research session that investigated whether the two leading open-source LLM serving engines — vLLM and SGLang — support running two model replicas simultaneously on the same set of GPUs to overlap allreduce with compute. The investigation, spanning thirteen messages and dozens of file reads and grep searches, ultimately delivered a counterintuitive conclusion: the allreduce bottleneck was a mirage, and the real problem lay elsewhere.

The Context: A System Hitting PCIe Walls

The deployment consisted of 8× NVIDIA RTX PRO 6000 Blackwell GPUs (96GB each, ~768GB total VRAM) connected via PCIe Gen5 in a dual-socket NUMA topology. The GPUs were arranged in PIX (direct PCIe switch) pairs: GPU0↔1, GPU2↔3, GPU4↔5, GPU6↔7. Critically, there was no NVLink between them. This meant all inter-GPU communication — including the allreduce operations fundamental to tensor parallelism — had to traverse the PCIe fabric.

The model under investigation was Kimi-K2.5 INT4, a massive 744B-parameter Mixture-of-Experts model with 256 experts (8 active per token). The root session's profiling campaign had identified a brutal reality: AllReduce consumed 11.17ms per decode step, accounting for 51.5% of total decode time. This was not a minor overhead; it was the dominant cost.

The team had already explored numerous optimization avenues documented in the repository's glb5improvement-*.md files: MSCCLPP one-shot allreduce, FlashInfer allreduce fusion, Single Batch Overlap (SBO), and expert parallelism. Most had yielded disappointing results — the comprehensive findings document showed that "all communication optimizations (MSCCLPP, SBO, allreduce fusion) yielded 0-2% improvement." The TP4+PP2 experiment had been particularly devastating: halving the allreduce data while doubling per-GPU compute made performance worse, not better.

This led to a creative question: instead of trying to make allreduce faster, could the engine run two model replicas simultaneously on the same 8 GPUs to overlap allreduce with compute? The idea was that while one replica waited for its allreduce to complete, the other replica could be doing useful computation, effectively hiding the communication latency.

The Research Methodology: A Systematic Deep-Dive

The subagent's approach to this research question was methodical and multi-layered. Rather than immediately answering from general knowledge, the assistant executed a comprehensive investigation spanning multiple phases.

Phase 1: Surveying Prior Art

The assistant began by orienting itself within the local repository ([msg 1]), listing the directory structure to understand what resources were available. It discovered a rich collection of prior analysis: improvement documents (numbered 01 through 12+), a comprehensive findings document (FINDINGS.md and glm5findings.md), source code patches, server logs, and benchmarking scripts.

In messages 2 and 3, the assistant read the most directly relevant improvement documents: the DP2+TP4 data parallelism proposal ([msg 2]), the Expert Parallelism proposal, the Single Batch Overlap (SBO) proposal, the allreduce fusion proposal, the MSCCLPP one-shot allreduce proposal, and the Opportunistic Expert Activation (OEA) document. This phase established the team's existing knowledge and identified what had already been tried.

Phase 2: Targeted Grep Searches

With the context established, the assistant executed systematic grep searches across the repository ([msg 4]). Four searches targeted specific feature classes:

  1. Disaggregated prefill patterns — found 10 matches in a log file sglang-tp4pp2.log, suggesting previous experimentation with TP=4, PP=2 configurations.
  2. Async allreduce and communication overlap — returned no matches, a significant negative result.
  3. Data parallelism flags — found 13 matches, with the most informative being references in glb5improvement-10-dp2-tp4.md, which explicitly stated that "SGLang supports data parallelism via --dp-size."
  4. Multi-replica configurations — returned no matches, confirming that running two model replicas on the same GPUs had not been attempted or documented.

Phase 3: Deep Reading of Source Files and Logs

The assistant then read the server_args_sm120.py file ([msg 5]) to find all relevant SGLang flags, and systematically read the remaining improvement documents (06 through 11) to ensure comprehensive coverage ([msg 6]). This phase revealed the full configuration surface of SGLang's server arguments.

In message 7, the assistant conducted more targeted searches, finding the critical benchmark results in glm5findings.md that showed SBO yielding only 0-2% improvement. The line "All communication optimizations (MSCCLPP, SBO, allreduce fusion) yielded 0-2% improvement" was a bombshell — it directly contradicted the expected impacts listed in the individual improvement documents.

Phase 4: The Comprehensive Findings Document

Message 8 marked a pivotal moment. The assistant pivoted from reading individual proposal documents to reading the master findings document glm5findings.md, which consolidated all experimental results from the optimization campaign. This document contained the actual benchmark data, performance measurements, and conclusions — the empirical evidence the assistant needed.

Phase 5: Exhaustive Local Search

In messages 9 and 10, the assistant checked remaining secondary sources — the memory bandwidth optimization research, the decode gap analysis, and the patch_allreduce.py file. The patch file revealed that vLLM had three explicit barriers preventing custom allreduce from functioning on Blackwell GPUs (compute capability 12.0) connected via PCIe: a world_size > 2 PCIe gate, missing compute capability entries in size tables, and missing Blackwell support in symmetric memory tables.

Phase 6: Empirical Verification from Server Logs

Message 11 represented a critical methodological shift: the assistant turned from searching source code and documentation to examining actual server logs for ground-truth evidence of command-line flags. Two grep searches across sglang-pr.log and sglang-k2-nvfp4-pr16975.log confirmed that flags like dp_size, enable_single_batch_overlap, and disaggregated appeared in actual server configurations.

The Comprehensive Answer: Message 12

Message 12 was the culmination of this research — a detailed report organized into six major sections, each addressing a specific aspect of the user's question.

Disaggregated Prefill: A False Lead

The assistant correctly identified that SGLang's disaggregation_mode parameter is a multi-node disaggregated serving feature that splits prefill and decode across separate GPU groups using Mooncake or InfiniBand to transfer KV cache between them. This is fundamentally different from what the user asked about — it does not run two replicas on the same GPUs, nor does it overlap allreduce with compute. Disaggregated prefill is about spatial separation (different GPUs for different phases), not temporal overlap (hiding communication behind computation on the same GPUs).

Data Parallelism: The Memory Wall

SGLang has a --dp-size flag (confirmed in ServerArgs dumps). The configuration --dp-size 2 --tp-size 4 would launch two independent model replicas, each using 4 GPUs. Replica A would use GPUs 0-3 (on NUMA node 0), and Replica B would use GPUs 4-7 (on NUMA node 1). Each replica handles independent requests, with zero cross-NUMA communication.

However, the assistant immediately identified the fatal flaw: memory capacity. The analysis in glb5improvement-10-dp2-tp4.md showed that with TP4, each GPU needs approximately 102GB for model weights alone, exceeding the 96GB VRAM per GPU. This is a fundamental physical constraint — no amount of software optimization can fit 102GB of weights into 96GB of memory.

The assistant also correctly noted that vLLM does not have a --data-parallel-size or --dp-size flag. vLLM's parallelism is limited to --tensor-parallel-size and --pipeline-parallel-size. To run multiple vLLM replicas, one would need to launch separate server processes with different CUDA_VISIBLE_DEVICES restrictions and an external load balancer.

The critical insight was that neither engine supports running two replicas that time-multiplex the same physical GPUs. DP2+TP4 gives each replica a disjoint subset of GPUs. This is a crucial distinction: the user asked about running two replicas on the same set of GPUs to overlap allreduce with compute, but the only available mechanism (DP) partitions GPUs rather than sharing them.

Communication-Compute Overlap: The Closest Match

The assistant examined SGLang's Single Batch Overlap (SBO) feature, which is the closest thing to what the user asked about. SBO works by splitting Streaming Multiprocessors (SMs) on the GPU: 32 SMs are dedicated to communication (allreduce), while the remaining SMs (~96 on SM120 Blackwell) perform computation simultaneously.

The actual benchmark results from glm5findings.md were devastating:

| Concurrency | Baseline | SBO | Change | |---|---|---|---| | 1 | 9.17 | 9.14 | ~0% | | 10 | 38.03 | 38.22 | ~0% | | 256 | 352.79 | 357.57 | +1.4% | | 1024 | 1,520.55 | 1,539.19 | +1.2% |

SBO, the feature specifically designed to overlap allreduce with computation, delivered negligible to slightly negative improvement. The assistant's explanation was precise: "communication is not the bottleneck — the workload is memory-bandwidth-bound on small MoE expert GEMMs, not allreduce-bound."

This was the key insight that reframed the entire problem. The allreduce overhead that profiling showed at 51.5% of decode time was not an independent bottleneck that could be hidden; it was a consequence of the small-GEMM inefficiency that dominated the forward pass. When the GPU is already underutilized due to memory-bandwidth-bound operations, overlapping communication doesn't help because there's no compute to hide it behind.

Expert Parallelism: The Real Solution

The assistant identified Expert Parallelism (EP) as the most impactful option — though one with its own challenges. SGLang supports EP via --ep-size 8, with related flags for the all-to-all backend (--moe-a2a-backend flashinfer) and dense layer TP control (--moe-dense-tp-size). EP fundamentally changes the communication pattern: instead of allreduce (which touches ALL data across all GPUs), it uses all-to-all (which only moves tokens to their assigned expert GPUs). Each GPU holds num_experts / ep_size complete (unsharded) experts, making each expert GEMM 8x larger (N=2048 instead of N=256).

The tested results, however, were mixed: OOM crashes at 256+ concurrency due to A2A buffer requirements, and 10-17% slower at low concurrency due to A2A overhead on PCIe. The assistant noted that EP needs retry with conservative memory settings (--mem-fraction-static 0.75 --max-running-requests 512).

The Fundamental Answer

The assistant provided a definitive answer to the user's core question: No. Neither vLLM nor SGLang supports running 2 model replicas simultaneously on the SAME set of 8 GPUs to hide allreduce latency.

Two comprehensive tables summarized what each engine does support, organized by feature, flag, and whether it overlaps allreduce. The SGLang table listed seven features (DP, SBO, TBO, EP, MSCCLPP, Allreduce Fusion, Disaggregated Prefill) with their flags and allreduce-overlap status. Only SBO was marked as actually overlapping allreduce with compute.

Practical Options

The assistant presented three actionable options:

Option A: SGLang's --enable-single-batch-overlap — the easiest to try, but with proven negligible impact (+1-2%). The explanation was crucial: "On this workload (MoE with 256 experts on PCIe), allreduce accounts for <5% of total time. The bottleneck is memory-bandwidth-bound expert GEMMs, not communication."

Option B: Expert Parallelism (--ep-size 8) — the most impactful in theory, but requiring careful memory management. The assistant identified the specific configuration changes needed (--mem-fraction-static 0.75 --max-running-requests 512) and acknowledged the untested status at high concurrency.

Option C: Do nothing about allreduce — a counterintuitive but well-supported recommendation. The assistant cited the TP4+PP2 experiment as conclusive proof: "halving the allreduce data while doubling per-GPU compute made things 2x worse, not better."

The core insight was delivered with clarity: "For large MoE models on PCIe GPUs, the problem is not allreduce latency — it is that per-expert GEMMs are too small to utilize the GPU's compute capacity. The real solutions are Expert Parallelism (larger GEMMs), L2 cache pinning (faster weight access), and persistent fused kernels (less memory traffic) — none of which involve running multiple replicas on the same GPUs."

The Deeper Lesson: When the Bottleneck You See Is Not the Bottleneck You Should Fix

The most profound insight from this investigation is not about any particular feature — it is about the fundamental nature of the bottleneck. When the assistant uncovered the line "halving the allreduce data while doubling per-GPU compute made things 2x worse, not better," it revealed a truth about the hardware: on Blackwell GPUs connected via PCIe, the compute is memory-bandwidth-bound for FP4 MoE operations. The GPUs are waiting on memory, not on each other. Overlapping AllReduce with compute doesn't help because the compute is already stalled.

This has far-reaching implications. It means that communication-computation overlap is a dead end for this hardware configuration. Expert parallelism — which avoids AllReduce entirely for MoE layers — becomes the primary optimization target. Data parallelism with TP4+DP2 is unlikely to help because it increases per-GPU compute without addressing the memory bandwidth bottleneck. The optimization effort should shift toward reducing memory bandwidth pressure (e.g., L2 cache pinning, persistent grouped GEMM kernels, structured sparsity) rather than hiding communication latency.

Conclusion

The subagent's investigation into overlapping allreduce with compute across vLLM and SGLang is a masterclass in systematic technical research. The assistant's methodology — surveying prior art, executing targeted searches, reading source files and logs, examining empirical evidence, and synthesizing findings into actionable recommendations — demonstrates a sophisticated understanding of both the research process and the underlying technology.

The core lesson is both humbling and liberating: sometimes the most important optimization is recognizing that you're optimizing the wrong thing. The team spent significant effort investigating allreduce overlap — disaggregated prefill, DP2+TP4, SBO, MSCCLPP, allreduce fusion — only to discover that allreduce was never the bottleneck. The real problem was small-GEMM inefficiency, and the real solutions lay in Expert Parallelism, L2 cache pinning, and persistent fused kernels.

This is the kind of insight that only comes from systematic investigation, willingness to challenge assumptions, and the discipline to let evidence override intuition. The allreduce overlap mirage is just one example of a broader truth: in complex systems, the bottleneck you see is rarely the bottleneck you should fix.