The Pipeline That Wouldn't Fill: Diagnosing SGLang Pipeline Parallelism on 8× Blackwell GPUs

Introduction

In any high-performance inference deployment, the gap between theoretical peak throughput and measured reality is where the most valuable engineering work happens. This article examines a single message from an opencode coding session — message index 11495 — in which an AI assistant performs a deep diagnostic analysis of a pipeline-parallel (PP8) deployment of the Kimi K2.6 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. The message sits at a critical inflection point: the team has just discovered that their carefully tuned PP8 configuration is delivering only 248 tok/s at high concurrency, less than half the throughput of the simpler TP8 (tensor parallelism) configuration they had previously benchmarked. The assistant must now figure out why.

This message is a masterclass in systems-level debugging under uncertainty. The assistant has no single obvious culprit — CUDA graphs are active, the service is running, the model loads correctly — yet the numbers are catastrophically wrong. The reasoning section of the message walks through multiple competing hypotheses, cross-references log evidence, constructs a mental model of SGLang's internal pipeline scheduler, and ultimately converges on a diagnosis: the pipeline stages are operating in lockstep rather than in true asynchronous overlap, and the request admission mechanism is capping throughput far below what the hardware should support. The message then dispatches a targeted bash command to extract the specific configuration parameters that would confirm or refute the diagnosis.

The Message in Full

The message begins with the assistant's reasoning block, which is worth quoting in its entirety because it reveals the full arc of the diagnostic process:

Agent Reasoning

>

Very interesting findings:

>

1. CUDA graphs ARE being used (cuda graph: True) -- good 2. Only 8 running-req per PP stage -- this is the problem. With 32 concurrent API requests, only 8 are actually being processed. The rest are queued. 3. Each PP stage sees the same 8 requests -- they're processing the same batch through all stages sequentially 4. Throughput shows ~422 tok/s -- but this is the per-stage metric, the actual output throughput is ~248 tok/s because of pipeline bubbles

>

5. GPU utilization fluctuates between 40-100% and power consumption maxes out around 310W, both pointing to the same bottleneck -- the pipeline isn't keeping the GPUs fed consistently.

>

The real problem is that with pp_async_batch_depth=8, all 32 requests are getting squeezed into just a handful of micro-batches instead of being distributed across the full 16 available slots, so each micro-batch crawls through all 8 stages one after another rather than overlapping.

>

Looking at the timestamps in the logs, every stage is reporting the same #running-req: 8 at identical times with matching token counts, which means the micro-batches are moving through the pipeline in lockstep instead of being staggered. The async depth should be filling the pipeline with multiple micro-batches at different stages simultaneously, but it's not happening — either the scheduler isn't actually splitting into 16 micro-batches, or the batches are too small to overcome the per-batch overhead, or the scheduling logic itself isn't properly interleaving the work. Examining the loop structure that cycles through 16 micro-batch slots, where each slot should progress through a different pipeline stage in a staggered fashion. But the logs show all stages processing the same batch size simultaneously, suggesting the micro-batches aren't actually being split up—instead, one large batch is moving through all stages together. The real constraint might be that the scheduler admits up to 64 requests total, but with pipelining, each micro-batch needs to be smaller to allow true parallelism across stages. Wait, that's exactly what the user is asking about—with async depth, GPU0 should start the next micro-batch while GPU1 processes the first one, which is classic pipeline parallelism. The issue might be that with only 8 requests and 8 decode tokens per step, each micro-batch is tiny, so pipeline overhead from NCCL communication between stages could be dominating the actual computation. Let me look at the actual performance numbers: PP8 with a single request gets 34.3 tok/s (30% faster than TP8's 26.3), but PP8 with 32 concurrent requests only reaches 248 tok/s compared to TP8's 577—that's a huge gap that suggests something's wrong with how the pipeline is scaling.

>

The real problem is that the logs show #running-req: 8 even when I set C=32, which means the max-running-requests 64 setting isn't being used. There's probably a pipeline-specific limit on concurrent requests. With pp_async_batch_depth=8 and 8 PP stages, there should be 16 micro-batch slots available, but if only 8 requests are running total, that might mean just one request per micro-batch, which would be terrible for throughput.

>

Let me look at the actual numbers: if each stage processes 422 tok/s with 8 requests and the pipeline has 8 stages with async depth 8, the output should theoretically be around 422 tok/s with temporal overlap, but we're only seeing 248 tok/s—that's roughly 40% bubble overhead. The simpler explanation is that the scheduler is artificially capping requests at 8 when using PP8, possibly tied to pp_async_batch_depth or the micro-batch size configuration.

>

Key findings from logs: - CUDA graphs are active (cuda graph: True) - Only 8 running-req per stage despite 32 concurrent API requests - All PP stages report same batch at same timestamp — lock-step, not truly pipelined - GPU util 40-100% with wild variance — pipeline bubble pattern

>

The #running-req: 8 is suspicious with C=32. Let me check what's limiting admission and how micro-batching actually works:

The message then issues a bash command that queries the SGLang service logs for three specific pieces of information: the auto-computed pp_max_micro_batch_size, the CUDA graph capture batch sizes, and the KV cache / token pool limits that might be capping concurrent requests.

Context and Motivation

To understand why this message was written, we need to reconstruct the chain of events that led to it. The session is deep into a multi-week effort to deploy the Kimi K2.6 model — a Mixture-of-Experts architecture — across various hardware configurations. Earlier in the segment, the team had benchmarked TP8 (tensor parallelism across 8 GPUs) and found it delivered 26.3 tok/s single-stream and 577.8 tok/s at C=32. The user then hypothesized that pipeline parallelism (PP8) would be significantly better for MoE models on PCIe, because PP avoids the expensive AllReduce operations that TP requires on every MoE layer — each GPU in a TP configuration must synchronize expert routing decisions across all 8 GPUs, and on PCIe this synchronization is bandwidth-bound.

The assistant implemented PP8 with two key changes: removing --disable-cuda-graph (which had been set due to earlier SM120 compatibility issues) and adding --pp-async-batch-depth 8 to enable asynchronous micro-batching. The theory was sound: with async depth 8, the pipeline would have 16 micro-batch slots (8 PP stages + 8 async depth), allowing multiple micro-batches to be in flight simultaneously. GPU0 could start processing micro-batch N+1 while GPU1 was still working on micro-batch N, filling the pipeline and achieving high utilization.

The benchmark results (msg 11491) were devastating: PP8 achieved 34.3 tok/s single-stream (only 30% better than TP8's 26.3) and just 248.1 tok/s at C=32 (less than half of TP8's 577.8). The user's response (msg 11492-11493) was pointed: "Much worse than expected, maybe some sglang perf issue? Cross gpu bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream, maybe we're dispatching somehow in a way that wastes batching potential."

The assistant then ran a profiling script (msg 11494) that sent sustained load while sampling GPU utilization and checking SGLang's internal logs. This profiling produced the raw data that msg 11495 now analyzes: GPU utilization oscillating wildly between 40-100%, power consumption capped at ~310W (far below the 600W TDP), and — crucially — every PP stage reporting exactly #running-req: 8 at identical timestamps.

The Diagnostic Reasoning: A Step-by-Step Analysis

The reasoning section of msg 11495 is not a linear narrative; it is a live thinking process that circles, refines, and converges. Let me trace its arc.

Step 1: Establish the Ground Truth

The assistant begins by listing five empirical findings from the profiling run. The first finding — "CUDA graphs ARE being used" — is important because it eliminates an entire class of hypotheses. Earlier in the conversation, the team had worried that CUDA graphs might be silently disabled by the triton attention backend, or that the SM120 architecture might not support them. The logs confirm that graphs are active, so the performance problem must lie elsewhere.

The second finding — "Only 8 running-req per PP stage" — is the smoking gun. The assistant has sent 32 concurrent requests, but only 8 are being processed. The remaining 24 are queued. This immediately suggests an admission control problem: something in the SGLang scheduler is limiting the number of requests that can be in-flight through the pipeline.

The third finding — "Each PP stage sees the same 8 requests" — is even more damning. If the pipeline were truly asynchronous, different stages would be working on different micro-batches at different points in the pipeline. Stage 0 might be processing micro-batch 3 while Stage 5 is processing micro-batch 1. But the logs show all stages reporting the same batch at the same time. This is the signature of lockstep execution — the pipeline is operating in "naive sequential" mode despite the async depth setting.

Step 2: Construct a Mental Model of the Pipeline Scheduler

The assistant then builds a theory about what should be happening. With pp_async_batch_depth=8 and pp_size=8, the internal loop size (pp_loop_size) should be 16. The scheduler should cycle through 16 micro-batch slots, each at a different pipeline stage, creating a staggered pipeline where multiple micro-batches are in flight simultaneously. This is classic pipeline parallelism: while GPU1 is computing the attention for micro-batch 0, GPU0 should already be computing the embedding for micro-batch 1.

But the logs contradict this model. The assistant considers three possible explanations:

  1. The scheduler isn't actually splitting into 16 micro-batches. Perhaps the micro-batch splitting logic has a bug, or the batch size auto-computation produces a value of 1 (one request per micro-batch), which would mean 8 micro-batches of 1 request each — not enough to fill the pipeline.
  2. The batches are too small to overcome per-batch overhead. Even if the scheduler creates 16 micro-batches, if each micro-batch contains only 1-2 requests, the NCCL communication overhead between pipeline stages might dominate the computation. The assistant notes that with only 8 requests and 8 decode tokens per step, each micro-batch is "tiny."
  3. The scheduling logic isn't properly interleaving. This is the most subtle possibility: the async depth parameter might be accepted syntactically but ignored semantically, or there might be a secondary condition (like a missing flag or a version-specific bug) that prevents actual interleaving.

Step 3: Quantify the Gap

The assistant performs a revealing calculation. The per-stage throughput is ~422 tok/s, but the end-to-end throughput is only ~248 tok/s. The ratio (248/422 ≈ 0.59) implies a pipeline bubble overhead of roughly 40%. In an ideal 8-stage pipeline with async depth 8, the bubble overhead should be close to zero — the pipeline should be fully saturated. A 40% bubble suggests that the pipeline is operating closer to a naive sequential model, where each micro-batch must traverse all 8 stages before the next micro-batch can enter.

The assistant then compares PP8 to TP8 numerically. PP8's single-stream advantage (34.3 vs 26.3 tok/s, a 30% improvement) confirms that the per-request compute is more efficient — this is the expected benefit of avoiding AllReduce on MoE layers. But the concurrent throughput collapse (248 vs 577 tok/s) means that PP8 is failing to scale with concurrency. Something is preventing the pipeline from absorbing additional requests.

Step 4: Formulate the Admission Control Hypothesis

The assistant converges on a specific hypothesis: the scheduler is artificially capping concurrent requests at 8 when using PP8. This cap might be tied to pp_async_batch_depth (which is also 8) or to the auto-computed pp_max_micro_batch_size. If the scheduler interprets pp_async_batch_depth=8 as "admit at most 8 requests total" rather than "admit up to 8 micro-batches worth of requests," then the pipeline would never have more than 8 requests in flight — and with 8 PP stages, that means roughly 1 request per stage, which is exactly the condition that produces lockstep execution.

This hypothesis elegantly explains all the observed symptoms:

Assumptions and Their Validity

The assistant's reasoning rests on several assumptions, some explicit and some implicit.

Assumption 1: The pp_async_batch_depth parameter is the primary control for pipeline filling. This is a reasonable assumption based on the SGLang source code examined in earlier messages (msg 11482-11484), where pp_loop_size = pp_size + pp_async_batch_depth. However, the assistant implicitly assumes that setting this parameter to 8 is sufficient to enable true asynchronous execution. In reality, there may be additional prerequisites — for example, the scheduler might require a minimum number of requests per micro-batch before it activates async mode, or there might be a separate flag for enabling micro-batch splitting.

Assumption 2: The max-running-requests 64 setting should allow 64 concurrent requests. The assistant notes that "the max-running-requests 64 setting isn't being used" and suspects a "pipeline-specific limit." This assumption is reasonable but may be wrong in an important way: perhaps the max-running-requests limit is being applied per pipeline stage rather than globally, which would cap each stage at 64 but still allow the pipeline to fill. Or perhaps the limit interacts with the micro-batch size in unexpected ways.

Assumption 3: The per-stage throughput of 422 tok/s represents the compute capacity of a single GPU. This is used to calculate the expected end-to-end throughput. But 422 tok/s might itself be limited by the pipeline structure — for example, if each stage is waiting for input from the previous stage, the per-stage metric might include idle time. The assistant acknowledges this implicitly by noting that "the actual output throughput is ~248 tok/s because of pipeline bubbles."

Assumption 4: The lockstep execution pattern is visible in the log timestamps. The assistant states that "every stage is reporting the same #running-req: 8 at identical times with matching token counts." This is a strong claim that depends on the granularity of the log timestamps. If the logs are sampled at 1-second intervals, micro-batches moving through the pipeline in 100ms would appear simultaneous even if they are actually staggered. The assistant does not question this assumption, but it's worth noting that the lockstep conclusion depends on the temporal resolution of the logging.

Potential Mistakes and Blind Spots

While the assistant's reasoning is thorough, there are several potential blind spots worth examining.

The NCCL communication overhead hypothesis is underdeveloped. The assistant briefly considers that "pipeline overhead from NCCL communication between stages could be dominating the actual computation" but does not pursue this line. Given that the GPUs are connected via PCIe (not NVLink), inter-GPU communication is indeed a potential bottleneck. However, the user had specifically argued that "cross gpu bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream," suggesting that the data volume per transfer is small. The assistant accepts this premise without independent verification.

The assistant does not consider a CUDA graph capture failure mode. CUDA graphs are confirmed active, but there is a difference between "CUDA graphs are enabled" and "CUDA graphs are working correctly for PP." It's possible that the graph capture succeeded but produced suboptimal graphs for the pipeline setting — for example, graphs that don't properly capture the NCCL communication pattern between stages, forcing a fallback to eager-mode launches for inter-GPU transfers.

The micro-batch size auto-computation is treated as a black box. The assistant's bash command queries pp_max_micro_batch_size but the reasoning does not speculate about what value it might return. If the auto-computation produces a very small value (e.g., 1), that would explain the lockstep behavior: with micro-batch size 1 and 8 requests, you get 8 micro-batches of 1 request each, which is exactly what the logs show. But if the auto-computation produces a larger value (e.g., 8), then the hypothesis shifts to an admission control bug.

The assistant does not consider that the problem might be in the benchmark client, not the server. The profiling script sends 64 requests via a ThreadPoolExecutor with 32 workers. If the client-side HTTP connection pooling or request serialization is introducing bottlenecks, the server might never see more than 8 concurrent requests simply because the client isn't sending them fast enough. The assistant assumes the server-side admission control is the bottleneck, but the client-side pipeline is equally plausible.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

Pipeline Parallelism Concepts: Understanding how pipeline parallelism works — the distinction between naive sequential execution (one micro-batch at a time through all stages) and asynchronous execution (multiple micro-batches in flight simultaneously) — is essential. The concept of "pipeline bubbles" (idle stages waiting for work) and the formula for bubble overhead in an N-stage pipeline are central to the analysis.

SGLang Architecture: The reader needs to know that SGLang is an inference engine for large language models, that it supports multiple parallelism strategies (TP, PP, EP), and that it has a pipeline scheduler with configurable async depth. The specific parameters --pp-async-batch-depth, --pp-max-micro-batch-size, and --max-running-requests are referenced throughout.

CUDA Graphs: Understanding that CUDA graphs capture a sequence of GPU kernel launches into a single executable object, eliminating CPU launch overhead for repeated computation patterns. The assistant's relief that "CUDA graphs ARE being used" reflects the importance of this optimization for PP throughput.

MoE Model Architecture: The Kimi K2.6 model uses Mixture-of-Experts layers, where each token is routed to a subset of expert sub-networks. This architecture creates a specific communication pattern: in TP mode, every MoE layer requires an AllReduce to synchronize expert routing across GPUs, which is expensive on PCIe. PP mode avoids this by keeping each GPU responsible for a contiguous set of layers, with only activations (not expert weights) communicated between stages.

NVIDIA GPU Hardware: The RTX PRO 6000 Blackwell GPUs have a 600W TDP, and the observed 310W power consumption indicates significant underutilization. The PCIe interconnect between GPUs (as opposed to NVLink) has limited bandwidth, which constrains the communication patterns that are efficient.

Output Knowledge Created

This message produces several forms of knowledge:

A confirmed diagnosis of lockstep pipeline execution. The assistant establishes that the PP8 deployment is not achieving true asynchronous pipeline parallelism. The evidence — identical #running-req: 8 across all stages, matching timestamps, 40% bubble overhead — forms a coherent diagnostic picture.

A specific hypothesis about admission control. The assistant identifies the #running-req: 8 cap as the root cause and hypothesizes that it is caused by a scheduler-level limit tied to pp_async_batch_depth or pp_max_micro_batch_size. This hypothesis is testable: the bash command at the end of the message will reveal the actual values of these parameters.

A methodology for diagnosing pipeline parallelism issues. The message demonstrates a reusable diagnostic pattern: (1) establish ground truth about what features are active, (2) measure per-stage vs end-to-end throughput to quantify bubble overhead, (3) examine log timestamps for lockstep signatures, (4) formulate competing hypotheses about the bottleneck, and (5) design targeted queries to test each hypothesis.

A quantitative baseline for PP8 performance. The specific numbers — 34.3 tok/s single-stream, 248 tok/s at C=32, 422 tok/s per-stage, 310W power, 40-100% GPU util variance — serve as a baseline that future optimizations can be measured against.

The Thinking Process: A Window into Systems Debugging

What makes this message particularly valuable is the visible thinking process. The assistant does not present a polished diagnosis; it works through the problem in real time, with false starts, self-corrections, and iterative refinement.

Notice the sentence that begins "Examining the loop structure that cycles through 16 micro-batch slots..." and then trails into "But the logs show all stages processing the same batch size simultaneously." This is the assistant talking itself through the mental model, then confronting it with evidence. The "Wait, that's exactly what the user is asking about" moment is particularly revealing — the assistant realizes that the user's earlier question about "work buffering at a bottleneck layer" was prescient, and that the async depth mechanism is supposed to provide exactly that buffering.

The repetition in the reasoning section — the assistant states the same core insight ("only 8 running-req is the problem") in multiple ways across multiple paragraphs — is characteristic of real-time debugging. Each restatement refines the hypothesis slightly: first it's "all 32 requests are getting squeezed into just a handful of micro-batches," then it's "the scheduler isn't actually splitting into 16 micro-batches," then it's "there's probably a pipeline-specific limit on concurrent requests." The final formulation — "the scheduler is artificially capping requests at 8 when using PP8, possibly tied to pp_async_batch_depth or the micro-batch size configuration" — is the most precise and actionable.

The assistant also demonstrates a healthy skepticism about its own theories. It considers and rejects the NCCL overhead hypothesis (too little data per transfer), the micro-batch size hypothesis (possible but doesn't explain the cap), and the scheduling bug hypothesis (possible but harder to test). The admission control hypothesis wins because it's the simplest explanation that accounts for all the evidence, and because it's directly testable with a log query.

Conclusion

Message 11495 captures a pivotal moment in a complex inference deployment effort. The assistant has just received disappointing benchmark results, has run a profiling session, and is now synthesizing the evidence into a coherent diagnosis. The message is notable for its intellectual honesty — the reasoning section shows the assistant grappling with uncertainty, weighing competing explanations, and ultimately converging on a testable hypothesis.

The deeper lesson of this message is about the gap between configuration and behavior in complex distributed systems. The assistant set --pp-async-batch-depth 8 with the expectation that it would enable asynchronous pipeline execution. But the system's actual behavior — lockstep execution with an 8-request admission cap — bore little resemblance to the configured intent. The diagnostic process required not just reading logs, but building a mental model of the scheduler's internal logic, comparing observed behavior against theoretical expectations, and designing targeted queries to isolate the root cause.

For anyone deploying large language models on multi-GPU hardware, this message offers a template for diagnosing pipeline parallelism issues. The specific findings — the 8-request cap, the lockstep timestamps, the 40% bubble overhead — are likely specific to this SGLang version and model configuration. But the methodology — establish ground truth, quantify the gap, examine temporal patterns, formulate hypotheses, test with targeted queries — is universal.