When Allreduce Isn't the Bottleneck: A Deep Dive into Multi-GPU Inference Optimization

In the high-stakes world of large language model serving, every millisecond counts. When you're running a 1-trillion-parameter model like Kimi-K2.5 across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected only by PCIe, the question of how to squeeze out more throughput becomes an obsession. In message [msg 2400] of a lengthy coding session, the AI assistant delivers a masterclass in performance analysis that upends a seemingly intuitive optimization strategy. The user had asked a reasonable question: if allreduce communication is the bottleneck, why not run two inferences simultaneously—letting one compute while the other communicates—to overlap the latency? The assistant's response reveals a far more complex reality, one where assumptions about bottlenecks are overturned by empirical data, and where the path forward requires rethinking the entire parallelism strategy.

The Question That Sparked the Investigation

The conversation leading up to this message had been an exhaustive benchmarking campaign. The team had deployed Kimi-K2.5 INT4, a 547GB Mixture-of-Experts (MoE) model with 256 routed experts and a Multi-head Latent Attention (MLA) architecture, across eight Blackwell GPUs. They had achieved a respectable 82 tokens per second single-stream throughput, but the user was hungry for more. A comprehensive profiling campaign using torch.profiler had revealed that AllReduce operations consumed a staggering 51.5% of decode time—11.17 milliseconds per step. This seemed like an obvious target for optimization.

The user's question in [msg 2398] was elegant in its simplicity: "If allreduce is so slow it seems like it would make sense to run two inferences at the same time, one does allreduce while the other computes. Is that something vllm or sglang support?" The idea was to time-share the GPUs between two independent inference processes, overlapping the communication of one with the computation of the other. It's the kind of creative thinking that comes from staring at profiling numbers and asking "what if?" The assistant recognized the cleverness of the idea and launched a research task ([msg 2399]) to investigate what frameworks actually supported.

The Assistant's Response: A Nuanced Rebuttal

The subject message opens with a direct, honest assessment: "Not exactly what you're describing, but close alternatives exist." This framing is crucial—it acknowledges the user's insight while immediately grounding it in practical constraints. The assistant then systematically demolishes the premise while offering constructive alternatives.

The core problem is stated bluntly: "You can't have two copies of a 547GB model on the same 768GB of VRAM." This is the hard physical constraint that makes the user's elegant idea infeasible. Even with INT4 quantization, the model consumes 547GB. With 768GB total across eight GPUs, that leaves only 221GB for KV cache, activation memory, and framework overhead. A second replica would require another 547GB—an impossibility. The assistant doesn't stop at "no," however. It proceeds to catalog every existing mechanism that approximates the user's vision, explaining why each falls short.

SGLang's Single Batch Overlap (SBO)

The closest existing implementation is SGLang's --enable-single-batch-overlap flag, which partitions the GPU's Streaming Multiprocessors (SMs). Roughly 32 SMs handle allreduce on one CUDA stream while the remaining SMs begin computing the next layer on another stream. This is precisely the overlapping of communication and computation that the user envisioned, but applied within a single batch rather than across two independent inferences.

The assistant then drops a bombshell: "We already tested this on the GLM-5 workload earlier. Result was +1-2% improvement." The reason is devastating to the user's premise: "allreduce isn't actually the bottleneck—it's only ~5-8% of per-token time." This directly contradicts the earlier profiling that showed AllReduce at 51.5% of decode time. The assistant is effectively saying: you're optimizing the wrong thing.

SGLang's Two Batch Overlap (TBO)

The --enable-two-batch-overlap flag is even closer to the user's idea—it runs a prefill batch and a decode batch concurrently on different CUDA streams. This is genuinely two different batches overlapping their execution. The assistant notes this hasn't been tested yet, leaving the door open for future experimentation.

vLLM's Data Parallelism

vLLM recently added --dp-size support, which for MoE models with MLA can combine data parallelism for attention layers with tensor parallelism and expert parallelism for expert layers. A configuration like --dp-size 2 --tp-size 4 would place two replicas on disjoint GPU subsets (GPUs 0-3 and 4-7). But again, the memory constraint bites: "Kimi-K2.5 INT4 at ~547GB won't fit in 4×96GB = 384GB per replica."

Dual-Model Deployment

The assistant briefly considers running two different models simultaneously—MiniMax-M2.5 on GPUs 0-3 and Kimi-K2.5 on GPUs 4-7. But Kimi doesn't fit in four GPUs, so this avenue is also closed.

The Uncomfortable Truth and a Tension in the Data

This is where the message reveals its deepest insight. The assistant states: "All communication optimizations we tested (MSCCLPP, SBO, allreduce fusion, even TP4+PP2 which halves allreduce data) yielded 0-2% improvement." The profiling showed allreduce is only 5-8% of total time. The real bottleneck is that with 256+ experts and TP=8, each expert GEMM is tiny (N=256) and can't saturate the GPU's compute units. The GPU spends most of its time doing memory-bandwidth-bound small matrix multiplies, not waiting on allreduce.

This creates a fascinating tension with the earlier segment summary, which claimed AllReduce was 51.5% of decode time. How can both be true? The most likely explanation is that the 51.5% figure came from a specific profiling run using torch.profiler that captured a particular configuration—perhaps one where the allreduce was particularly expensive due to NCCL tuning parameters or batch size effects. The 5-8% figure, by contrast, may have come from earlier experiments with the GLM-5 model or from a different measurement methodology. Alternatively, the assistant might be conflating two different measurements: the fraction of wall-clock time spent in NCCL allreduce kernels versus the fraction of per-layer time that could be overlapped. The distinction matters because NCCL allreduce operations can be asynchronous—the GPU can issue the allreduce and then context-switch to other work while the PCIe transfer completes in the background. The profiler might attribute the wait time for allreduce completion to the allreduce kernel, even though the GPU could theoretically do other work during that wait.

This tension is not necessarily an error—it reflects the genuine complexity of profiling distributed GPU workloads. Different profiling tools, different batch sizes, different model architectures, and different NCCL configurations can produce wildly different bottleneck breakdowns. The assistant's key point is that empirically, all the explicit communication-overlapping optimizations they tried produced negligible gains, which suggests that the communication is not the primary bottleneck in practice, regardless of what the profiler says.

The Real Solution: Expert Parallelism

Having systematically closed off the user's proposed path and several alternatives, the assistant pivots to what would actually help: Expert Parallelism (EP). The --enable-expert-parallel flag changes the game entirely. Instead of sharding each expert across all 8 GPUs (which requires allreduce to combine partial results), each GPU holds complete experts. Tokens are routed via all-to-all communication to the GPU that holds the relevant expert. This eliminates MoE allreduce entirely and, crucially, makes each expert GEMM 8× larger—transforming memory-bandwidth-bound tiny matrix multiplies into compute-bound operations that can actually saturate the GPU's tensor cores.

The assistant points to a proven result: MiniMax-M2.5 with TP=8+EP achieved ~4000 tok/s, compared to ~84 tok/s single-stream with TP=4 alone. For Kimi-K2.5, combining vLLM's new --dp-size with --enable-expert-parallel could be transformative—though memory constraints remain a concern.

Assumptions, Knowledge, and Decision-Making

The message operates on several key assumptions. First, it assumes the user's goal is throughput optimization for a single model instance, not architectural experimentation. Second, it assumes the hardware configuration is fixed—eight PCIe-connected GPUs without NVLink—and that no hardware changes are forthcoming. Third, it assumes that the empirical results from earlier experiments (SBO yielding 1-2%, all communication optimizations yielding 0-2%) are representative and reproducible.

The input knowledge required to fully understand this message is substantial. One needs to understand tensor parallelism (sharding model layers across GPUs with allreduce to combine results), expert parallelism (distributing experts across GPUs with all-to-all routing), the distinction between compute-bound and memory-bandwidth-bound operations, the role of NCCL in GPU communication, the memory constraints of large model serving, and the specific architectures of MoE and MLA models. Without this background, the assistant's argument that "tiny expert GEMMs are the real bottleneck" would be incomprehensible.

The output knowledge created by this message is equally significant. It provides a decision framework for when to pursue communication-overlapping optimizations versus parallelism-strategy changes. It establishes that for models with very large expert counts (256+) and high tensor parallelism degrees (TP=8), the expert GEMMs become the dominant bottleneck due to their small size, not the allreduce communication. This is a non-obvious insight that could only be discovered through careful profiling and experimentation.

The Thinking Process

The assistant's reasoning reveals a methodical approach to performance analysis. It begins by validating the user's creative idea, then immediately grounds it in physical constraints (memory capacity). It systematically evaluates each available mechanism, explaining not just what exists but why each one fails for this specific workload. Crucially, it brings empirical evidence to bear—the 1-2% improvement from SBO testing—rather than relying on theoretical analysis. When the empirical data contradicts the profiling numbers, the assistant trusts the experiments over the profiler.

The final pivot to Expert Parallelism demonstrates a deep understanding of the underlying architecture. The assistant recognizes that the problem isn't communication latency per se, but the interaction between communication strategy and compute efficiency. By eliminating the allreduce for expert layers, EP simultaneously solves the communication problem and the compute efficiency problem (by making GEMMs 8× larger). This is the kind of insight that separates superficial optimization from architectural understanding.

Conclusion

Message [msg 2400] stands as a remarkable example of performance analysis in the age of distributed LLM inference. It takes a user's creative but infeasible idea, explains why it won't work, validates the underlying intuition, and redirects toward a solution that addresses the actual bottleneck. The tension between the 51.5% profiling number and the 5-8% empirical finding is not a flaw but a feature—it illustrates the gap between what profilers measure and what actually limits throughput in practice. The message ultimately teaches that in distributed ML inference, the bottleneck is rarely where you think it is, and the best optimization often comes from rethinking the parallelism strategy rather than trying to squeeze more performance out of a fundamentally constrained communication path.