The Allreduce Bottleneck and the Question That Changed the Investigation

"If allreduce is so slow it seems like it would make sense to run two inferences at the same time, one does allreduce while the other computes. Is that something vllm or sglang support?"

This message, sent by the user at index 2398 of a marathon coding session deploying and benchmarking large MoE language models on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, represents a pivotal moment of creative problem-solving. It is deceptively short — a single sentence and a follow-up question — but it encapsulates the user's deep understanding of the system's bottleneck, their willingness to think architecturally about solutions, and their practical orientation toward existing tools. To understand why this message was written, one must trace the thread of discoveries that led to it, and to appreciate its impact, one must follow the investigation it launched.

The Context That Produced This Question

The message did not emerge from a vacuum. It was the culmination of an extensive profiling campaign that had consumed the preceding hours of the session. The assistant had deployed the Kimi-K2.5 INT4 model — a 1-trillion-parameter Mixture-of-Experts model with Multi-head Latent Attention (MLA) architecture — across 8 GPUs connected only by PCIe Gen5, with no NVLink. After exhaustive NCCL tuning experiments (trying Ring and LL protocols, varying channel counts and thread configurations) and even a --compilation-config '{"level": 3}' flag that proved ineffective, the assistant had finally run a full torch.profiler capture. The results were stark: AllReduce accounted for 51.5% of decode time, consuming 11.17 milliseconds per step. The GEMM operations that everyone assumed would be the bottleneck were not the problem — the communication was.

The assistant had delivered this diagnosis in message 2386, presenting a comprehensive benchmark comparison table and concluding: "The bottleneck is physical — PCIe bandwidth for allreduce across 8 GPUs, 61 MLA layers each requiring multiple allreduce operations." The user's first attempt to respond (message 2394) was cut off mid-sentence: "If allreduce is so slow it seems like it wo..." — they were already forming the idea. But before they could finish, they asked for a summary, and the assistant produced an enormous recap of the entire session's discoveries (message 2397). Only after digesting that summary did the user complete their thought in message 2398.

The Reasoning Behind the Question

The user's proposal is elegant in its simplicity. If the GPUs spend 51.5% of their time waiting for AllReduce to complete — that is, waiting for data to travel across the PCIe bus and be aggregated — then those cycles are effectively wasted. What if, instead of idling during communication, the GPUs could be computing something else? Specifically, what if two independent inference executions were overlapped in time: while inference A waits for its allreduce to finish, inference B uses the compute units to do its forward pass, and vice versa?

This is essentially a form of pipeline parallelism across independent model replicas — not the traditional pipeline parallelism where different layers of the same model are spread across devices, but rather a temporal overlap where the communication phase of one request is hidden behind the computation phase of another. The user recognized that the standard tensor-parallelism execution model is synchronous: all GPUs compute their shard of the result, then all GPUs participate in an allreduce to combine those shards, then all GPUs compute the next layer. The compute and communication are strictly sequential within a single forward pass. But across different forward passes (different requests), there is no inherent reason they must be sequential — one request's compute could fill the gap left by another request's communication.

The user's assumption — a reasonable one — is that the GPU's compute resources (the Tensor Cores, the SM units) are idle during the allreduce operation. Allreduce is a communication primitive: it involves each GPU sending its shard to the others, receiving their shards, and performing a reduction (typically summation). While the NCCL library uses GPU kernels for the reduction step, the bulk of the latency is the PCIe transfer time. During those microseconds, the GPU's SMs could theoretically be running compute kernels for a different batch of data. The user is essentially asking: "Can we hide communication latency by increasing occupancy with independent work?"

What the User Got Right

The question demonstrates sophisticated systems thinking. The user correctly identified that:

  1. AllReduce is the dominant bottleneck — not GEMMs, not attention, not the MoE routing. The profiling data was unambiguous, and the user accepted it and moved directly to mitigation strategies rather than questioning the methodology.
  2. The bottleneck is structural, not tunable — NCCL protocol variations (Ring vs. LL), channel counts, thread configurations, and compilation flags had all been tried and produced negligible improvements. The user understood that when the bottleneck is PCIe bandwidth, no amount of software tuning will make it go away. The solution must be architectural.
  3. Overlapping communication with computation is a classic HPC technique — this is the same principle behind asynchronous MPI operations, CUDA streams with overlapping kernel execution and data transfer, and the "hide latency with more work" strategy used in GPU computing for decades. The user was applying a well-known pattern to a new domain.
  4. The question is practical — rather than asking "can we implement this ourselves?", the user asked "Is that something vllm or sglang support?", showing they wanted a deployable solution, not a research project.

Potential Misconceptions

However, the question also contains some implicit assumptions that deserve examination. First, it assumes that the GPU's compute resources are genuinely idle during allreduce. In practice, NCCL's allreduce implementation on modern GPUs uses the GPU's SMs to perform the reduction kernel — the GPUs are not purely "waiting" during communication. The PCIe transfer is a DMA operation that can overlap with kernel execution on the GPU, but the reduction kernel itself consumes SM cycles. Whether those cycles could be repurposed for independent inference work depends on the GPU's scheduling capabilities and whether the CUDA streams can be kept sufficiently independent.

Second, the question assumes that two independent inferences can coexist on the same set of GPUs without memory conflicts. The Kimi-K2.5 INT4 model is 547GB — it barely fits across 8 GPUs with 96GB each (768GB total). Running two simultaneous inferences would require either sharing the same model weights (which is feasible — they're read-only) but maintaining separate KV caches and intermediate activations, which could exceed available memory. The user may not have fully considered the memory implications of their proposal.

Third, the question implicitly assumes that the allreduce latency is the same regardless of what else the GPU is doing. In reality, if the GPU's compute resources are occupied with another inference, the NCCL kernels may be delayed, potentially increasing the allreduce latency rather than hiding it.

Input Knowledge Required to Understand This Message

To fully grasp what the user is asking, one needs to understand:

Output Knowledge Created by This Message

This message triggered a significant research effort. The assistant responded by launching a comprehensive investigation into:

  1. Disaggregated prefill in both vLLM and SGLang — a technique where prefill (the initial prompt processing) and decode (the token-by-token generation) are separated onto different GPUs or different instances.
  2. Data parallelism across model replicas — running two independent copies of the model on the same set of GPUs, with the scheduler dispatching requests to whichever replica is currently idle.
  3. Pipeline parallelism — the traditional approach of splitting model layers across devices, but applied to the specific problem of overlapping communication with computation.
  4. SGLang's disaggregation_mode parameter and vLLM's multi-instance GPU support. The research revealed that SGLang had a disaggregation_mode parameter and that vLLM had experimental support for multi-instance GPU (MIG) partitioning, but neither framework directly supported the user's specific proposal of overlapping allreduce across independent inferences on the same GPU set. The assistant discovered a GitHub issue in the SGLang project ([Proposal] Overlap allreduce in tensor parallelism #8728) that was exactly about this idea — someone else had independently proposed the same concept. The issue was closed as inactive, suggesting the feature had not been implemented.

The Thinking Process Visible in the Message

The user's thinking is remarkably compressed into this single sentence. The structure reveals their reasoning chain:

  1. Observation: "If allreduce is so slow" — acceptance of the profiling diagnosis.
  2. Logical consequence: "it seems like it would make sense to run two inferences at the same time" — the insight that idle cycles during communication could be filled with other work.
  3. Refinement: "one does allreduce while the other computes" — the specific mechanism: temporal overlap of communication and computation across independent requests.
  4. Practical check: "Is that something vllm or sglang support?" — the critical question of feasibility and existing implementation. This is a model of concise technical communication. The user does not explain why allreduce is slow (that was established earlier), does not justify why overlapping makes sense (the reader can infer it), and does not propose implementing a custom solution. They state the idea, propose the mechanism, and ask about existing support. The entire thought process is visible in the logical flow from premise to question.

Significance Within the Session

This message marks a turning point. Before it, the session had been focused on measuring and characterizing the bottleneck — profiling, benchmarking, NCCL tuning. After it, the session pivots to exploring architectural solutions: disaggregated prefill, data parallelism, speculative decoding, and the possibility of running multiple model instances. The user's question reframes the problem from "how do we make allreduce faster?" to "how do we work around the fact that allreduce is slow?" — a fundamentally different engineering approach.

The question also reveals something about the user's relationship with the assistant. The user is not just issuing commands; they are thinking alongside the assistant, processing the profiling results, and proposing creative solutions. The assistant's response — launching parallel research tasks, searching GitHub issues, exploring SGLang documentation — shows that the assistant treats the user's question as a serious technical proposal worth investigating thoroughly.

In the end, the investigation would lead to speculative decoding as the most promising software-only optimization path, and the user's question about overlapping allreduce would remain partially unanswered — a testament to the difficulty of the problem and the hard physical constraints of PCIe-only multi-GPU inference. But the question itself, brief as it is, captures a moment of genuine insight: the recognition that when you cannot make the bottleneck faster, you must learn to live around it.