When Pipeline Parallelism Falls Short: Diagnosing a Thundering Herd on 8 GPUs

The Message

Much worse than expected, maybe some sglang perf issue? Cross gpu bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream, maybe we're dispatching somehow in a way that wastes batching potential - e.g batches doing thundering herds, not evenly split on GPUs? Can we profile?

This single sentence, sent by a user in the middle of an intense optimization session for Kimi K2.6 inference on 8× RTX PRO 6000 Blackwell GPUs, is a masterclass in applied systems intuition. It arrives at a moment of disappointment—benchmark results that should have been good but weren't—and immediately pivots from frustration to hypothesis generation. The user does not simply say "this is slow." They propose a specific mechanism for why it is slow, rooted in a deep understanding of pipeline parallelism, GPU interconnect topology, and the SGLang serving framework's scheduling behavior.

The Context That Produced This Message

To understand why this message was written, we must trace the events of the preceding hour. The session had been wrestling with pipeline parallelism (PP) for the Kimi K2.6 model, a massive Mixture-of-Experts architecture. The initial PP8 deployment ([msg 11478]) used --disable-cuda-graph and no async batching depth, producing abysmal results: ~24 tok/s single-stream, with all GPUs showing 0% utilization and ~85W power draw (idle). The user immediately identified the core issues ([msg 11479]): CUDA graphs were disabled, and without async micro-batch depth, the pipeline was running naively—one GPU active, seven waiting.

The assistant responded by studying SGLang's PP scheduling code ([msg 11482]–[msg 11488]), discovering the --pp-async-batch-depth parameter, and reconfiguring the service. The new config removed --disable-cuda-graph and set --pp-async-batch-depth 8, allowing up to 8 micro-batches to be in-flight simultaneously across the 8-stage pipeline. The service restarted in 90 seconds ([msg 11490]), and the assistant ran a fresh benchmark ([msg 11491]).

The results were disappointing. Single-stream throughput improved from ~24 to ~34 tok/s—a 40% gain, but far below what 8 GPUs should deliver. Concurrent throughput at C=8 was 81.7 tok/s (barely 2.4× single-stream), and at C=32 it reached 248.1 tok/s (only 7.3× single-stream). The user aborted the benchmark mid-run and sent the target message.

The User's Reasoning: A Mental Model of Pipeline Scheduling

The message reveals a sophisticated mental model of how pipeline-parallel inference should behave. The user breaks the problem into three observations:

"Cross GPU bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream." This is a bandwidth-versus-throughput calculation. At ~34 tok/s per stream, each token requires only a single layer's activations to be sent across the PCIe bus between GPUs. For a Mixture-of-Experts model like K2.6, the intermediate activations per layer are on the order of megabytes, not gigabytes. PCIe Gen5 x16 offers ~64 GB/s bidirectional bandwidth. The user correctly reasons that the interconnect should not be the bottleneck—the data volume is simply too small.

"Maybe we're dispatching somehow in a way that wastes batching potential." This is the key insight. The user suspects that the scheduler is not distributing micro-batches evenly across pipeline stages. In an ideal pipeline-parallel system, each stage should process roughly the same number of micro-batches per unit time. If the scheduler sends batches in bursts—all arriving at GPU 0 simultaneously, then all moving to GPU 1 simultaneously—the pipeline "bubbles" (idle stages) dominate.

"Batches doing thundering herds, not evenly split on GPUs." The "thundering herd" metaphor is borrowed from systems where many processes wake simultaneously to contend for a shared resource. Here, the user hypothesizes that SGLang's scheduler releases all waiting requests at once, creating a wave that propagates through the pipeline stages. Stage 0 processes a batch, sends it to stage 1, then sits idle because no new requests have arrived yet. Stage 1 receives the batch, processes it, sends to stage 2, sits idle. The result is exactly the 1/8 utilization pattern they observed.

Assumptions and Potential Blind Spots

The user's hypothesis is compelling, but it rests on several assumptions worth examining. First, it assumes that SGLang's PP scheduler can distribute batches evenly but is configured incorrectly. The alternative—that the scheduler fundamentally cannot pipeline efficiently for this model architecture—would require deeper changes. Second, it assumes the bottleneck is scheduling logic rather than compute. The user dismisses cross-GPU bandwidth as a factor, but PCIe topology on a dual-socket server can introduce NUMA effects: GPUs on different sockets communicate through the CPU interconnect, which may have asymmetric bandwidth. Third, the user implicitly assumes that CUDA graphs are working correctly. The assistant enabled them by removing --disable-cuda-graph, but CUDA graph capture for pipeline-parallel execution is notoriously fragile—graphs may fall back to eager mode for certain operations, silently negating the benefit.

The user also does not consider the possibility that the model's Mixture-of-Experts routing creates inherent load imbalance. In an MoE model, different tokens activate different experts, and if the expert distribution across pipeline stages is uneven, some stages will systematically do more work. This is a structural issue that scheduling cannot fix.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several concepts. Pipeline parallelism in deep learning inference divides a model across GPUs by layer: GPU 0 processes layers 1–N, GPU 1 processes layers N+1–2N, and so on. Each token must traverse all stages sequentially, so throughput is limited by the slowest stage. "Micro-batching" (or "async batch depth") allows multiple tokens to be in different stages simultaneously, filling the pipeline. CUDA graphs capture a sequence of GPU kernel launches into a reusable graph object, eliminating Python-to-CUDA launch overhead—critical for the small, frequent kernel launches typical of transformer inference. "Thundering herd" describes a synchronization pattern where many processes contend for a resource simultaneously, causing inefficient bursts of activity followed by idle periods. The user also references SGLang's internal scheduler architecture, which uses a pp_loop_size computed as pp_size + pp_async_batch_depth to determine how many micro-batch slots are available.

Output Knowledge Created

This message creates actionable diagnostic direction. The user's "can we profile?" request sets the agenda for the next phase of work. The assistant will need to instrument the pipeline stages to measure per-GPU utilization over time, track micro-batch dispatch timing, and verify whether the scheduler is indeed creating thundering herds. The specific hypotheses—uneven batch distribution, scheduler burstiness, wasted batching potential—provide testable predictions. If profiling shows that all GPUs have similar utilization but low absolute values, the bottleneck is elsewhere (perhaps CUDA graph fallback or kernel efficiency). If it shows one GPU saturated while others idle, the thundering herd theory is confirmed.

The message also implicitly defines a success metric: even batch distribution across GPUs, with each stage processing micro-batches continuously rather than in bursts. This is a more nuanced goal than simply maximizing tok/s—it reflects an understanding that smooth pipeline utilization is the mechanism by which high throughput is achieved.

The Thinking Process Visible in the Message

What makes this message remarkable is the compression of reasoning. The user does not describe the benchmark results—they were present in the aborted output. They do not restate the problem. Instead, they leap directly to a causal model: "Cross GPU bandwidth is tiny → layer sends should be cheap → therefore the bottleneck must be scheduling, not bandwidth." This is abductive reasoning—inferring the most likely cause from observed symptoms.

The phrase "maybe we're dispatching somehow in a way that wastes batching potential" is particularly telling. It acknowledges uncertainty ("maybe," "somehow") while still proposing a specific mechanism. The user is thinking aloud, treating the assistant as a collaborator who can validate or refute the hypothesis. The final "Can we profile?" is both a request and a test: it asks for data that will confirm or deny the theory, keeping the investigation grounded in measurement rather than speculation.

This style of reasoning—combining systems intuition, quantitative bounds-checking, and a willingness to be wrong—is characteristic of experienced ML infrastructure engineers. The user does not need to know SGLang's scheduler code to predict its failure modes; they understand the general class of pipeline-parallel systems and can reason about where any implementation is likely to break down. The message is a blueprint for how to think about performance debugging: identify the constraint, check if it should be binding, and if not, look elsewhere.