Diagnosing Pipeline Parallelism Underutilization: GPU Bubbles and Batching Behavior in SGLang

Subject Message: "gpu util at 150/200 out of 600, makes no sense, seems like we're not properly pushing data into the pipeline such that all layers are properly utilised? Do we have cuda graphs? Also when we have bottleneck on a layer, work buffering at it, will sglang take all requests that are ready to process and process them in a batch through that set of layers on the gpu? Like, work arrives at layers 1-8 gpu0, is done, sent to gpu1 which is busy longer, then gpu0 finishes another batch, then gpu1 is free and picks up the doubled up batch to compute at once"

Introduction

This message represents a critical diagnostic moment in a high-stakes machine learning deployment session. The user and assistant had been working to deploy Kimi K2.6, a 548 GB Mixture-of-Experts (MoE) model with 384 experts, across 8× RTX PRO 6000 Blackwell GPUs. After the user's insightful suggestion to try Pipeline Parallelism (PP8) instead of Tensor Parallelism (TP8)—reasoning that PP would keep expert computation local to each GPU and avoid expensive PCIe AllReduce operations—the assistant had just benchmarked the PP8 configuration. The results were underwhelming: 24.7 tok/s for single requests, scaling to only 95.6 tok/s at C=8 concurrency. The user aborted the benchmark before higher concurrency levels completed, and this message captures their immediate diagnostic reaction.

The message is dense with technical intuition. The user is not merely reporting a number; they are actively reasoning about why the GPU utilization is low, proposing hypotheses about pipeline feeding, questioning whether CUDA graphs are active, and theorizing about SGLang's internal batching mechanics. This is the kind of message that separates a passive observer from an active debugger—someone who understands the architecture well enough to form testable hypotheses on the spot.

The Context: Why PP8 Was Expected to Win

To understand the user's frustration, we must appreciate why PP8 was attempted in the first place. The model, Kimi K2.6, is an MoE architecture with 384 experts per layer, 8 selected per token. With TP8, every MoE layer requires an AllReduce across all 8 GPUs over PCIe—a notorious bottleneck for PCIe-connected systems. The user's insight was that Pipeline Parallelism would keep expert dispatch entirely local to each GPU: each GPU would own a contiguous block of layers (roughly 7–8 layers per GPU with 61 total layers), and only activations would need to pass between stages. This eliminates the AllReduce bottleneck entirely.

However, PP introduces its own problem: pipeline bubbles. When a single token must traverse GPU0→GPU1→...→GPU7 sequentially, the pipeline is only as fast as its slowest stage. At low concurrency, most GPUs sit idle waiting for work to arrive from the previous stage. The benchmark confirmed this: 24.7 tok/s single-request throughput was abysmal compared to the TP8 baseline of 98 tok/s at C=1 (from earlier benchmarks in the session).

Decoding the GPU Utilization Observation

The user reports "gpu util at 150/200 out of 600." This almost certainly refers to GPU power draw in watts as reported by nvidia-smi. The RTX PRO 6000 Blackwell has a thermal design power (TDP) of 300W per GPU, so 150–200W represents roughly 50–67% utilization—far below the expected near-100% for a compute-bound workload. The "out of 600" is likely the user's approximation of the total system power budget or a misremembered figure; what matters is the qualitative observation that the GPUs are significantly underutilized.

This observation is the key diagnostic signal. If the GPUs were compute-bound, they would be drawing near their TDP. The fact that they're at half power suggests one of two things: either the workload is memory-bandwidth-bound (where compute units stall waiting for data), or the pipeline is not being kept busy—GPUs are spending significant time idle, waiting for activations to arrive from the previous PP stage. The user correctly identifies the latter as the likely culprit.

The CUDA Graphs Question

The user asks "Do we have cuda graphs?" This is a crucial technical question. CUDA Graphs (also known as CUDA Hyper-Graphs or stream-ordered graphs) allow the GPU to capture a sequence of kernel launches into a compiled graph object, eliminating kernel launch overhead and enabling the GPU to pipeline execution more efficiently. In the PP8 service configuration shown in the context, the assistant explicitly set --disable-cuda-graph. The user may not have known this, but their intuition that CUDA graphs could help is correct—graphs can reduce the latency between pipeline stages by minimizing host-side launch overhead.

However, CUDA graphs are not a silver bullet for pipeline bubbles. Graphs primarily reduce launch latency and can improve occupancy, but they cannot eliminate the fundamental idle time caused by an unbalanced pipeline where one stage takes longer than others. The user's question reveals an assumption that CUDA graphs might be the missing piece, but the deeper issue is architectural.

The Batching Hypothesis: A Sophisticated Mental Model

The most impressive part of this message is the user's detailed mental model of how pipeline batching should work:

"when we have bottleneck on a layer, work buffering at it, will sglang take all requests that are ready to process and process them in a batch through that set of layers on the gpu? Like, work arrives at layers 1-8 gpu0, is done, sent to gpu1 which is busy longer, then gpu0 finishes another batch, then gpu1 is free and picks up the doubled up batch to compute at once"

This describes a form of asynchronous pipeline batching or micro-batch coalescing. In an ideal pipeline scheduler, when GPU1 is the bottleneck (taking longer per micro-batch than GPU0), GPU0 should continue processing new requests and queue them. When GPU1 finishes its current work, it should drain the accumulated queue by processing multiple micro-batches in one go. This is analogous to how modern CPU pipelines handle back-pressure with buffered stages.

The user is essentially asking whether SGLang implements this kind of dynamic batching across pipeline stages. This is a deep question about SGLang's runtime architecture. The answer depends on whether SGLang uses a synchronous pipeline schedule (where all stages must complete a micro-batch before moving to the next) or an asynchronous schedule with micro-batch buffering. The assistant's subsequent investigation would need to examine SGLang's --pp-max-micro-batch-size and --pp-async-batch-depth parameters, which were visible in the help output but not configured in the PP8 service.

Assumptions Embedded in the Message

The user makes several assumptions worth examining:

  1. That low GPU utilization is abnormal for PP. In fact, pipeline bubbles are a well-known characteristic of PP at low concurrency. With only 7–8 layers per GPU and sequential execution, a single request cannot keep all GPUs busy simultaneously. The user's intuition that "we're not properly pushing data into the pipeline" is correct, but this is an inherent property of PP rather than a configuration bug.
  2. That CUDA graphs would help. While CUDA graphs could reduce launch overhead, the primary bottleneck is pipeline imbalance, not kernel launch latency. The assistant had disabled CUDA graphs to avoid known stability issues (the earlier TP8 benchmarks used --disable-cuda-graph as well).
  3. That SGLang might dynamically batch across pipeline stages. This is a reasonable hope but may not match SGLang's actual implementation. Many inference engines use a simple 1F1B (one-forward-one-backward) schedule for PP, which does not accumulate batches at bottleneck stages.
  4. That the pipeline is unbalanced. The user assumes one layer is a bottleneck. With MoE layers, this is plausible—different experts may have different compute requirements, and the routing pattern can create load imbalance.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message creates several valuable outputs:

  1. A clear diagnostic signal: Low GPU utilization confirms the pipeline is not being kept busy, ruling out compute-bound or memory-bound explanations.
  2. A testable hypothesis about CUDA graphs: Whether enabling CUDA graphs would improve throughput is now a concrete experiment to run.
  3. A deep question about SGLang's batching behavior: The user's mental model of micro-batch coalescing sets up a specific investigation into SGLang's pipeline scheduler implementation.
  4. A prioritization signal: The disappointing PP8 results suggest that for this workload on PCIe hardware, the pipeline bubble problem outweighs the AllReduce elimination benefit. This informs the decision to explore other parallelism strategies (EP4, EP8) that were subsequently benchmarked.

The Thinking Process Revealed

The user's reasoning process is visible in the structure of the message. They start with an observation (low GPU utilization), immediately identify it as anomalous ("makes no sense"), propose a root cause ("not properly pushing data into the pipeline"), ask about a specific feature that might help (CUDA graphs), and then build a detailed mental model of how the system should work to mitigate the problem. This is textbook diagnostic reasoning: observe → identify anomaly → hypothesize cause → propose intervention → model ideal behavior.

The message also reveals the user's deep understanding of the system architecture. They correctly identify that pipeline parallelism creates a sequential dependency between GPUs, that bottleneck stages create back-pressure, and that intelligent batching could smooth out the imbalance. This level of understanding is rare and speaks to the user's expertise in distributed inference systems.

Conclusion

This message captures a pivotal moment in the optimization journey. The PP8 experiment had failed to deliver on its promise, and the user was already forming the next set of hypotheses to test. The diagnostic path laid out in this message—investigating pipeline feeding, CUDA graphs, and dynamic batching—would ultimately lead the team to abandon PP8 in favor of Expert Parallelism (EP8), which dramatically improved throughput by eliminating AllReduce on MoE layers while avoiding pipeline bubbles. The user's ability to rapidly diagnose and pivot was instrumental in finding the winning configuration.