The Pipeline Parallelism "Aha" Moment: Debugging GPU Utilization in SGLang's PP8 Scheduler

In the middle of an intense optimization session for Kimi K2.6 inference on 8× RTX PRO 6000 Blackwell GPUs, a single message captures the moment when a frustrating performance mystery finally clicks into place. Message 11485 is not flashy—it contains a brief reasoning paragraph and a simple grep command—but it represents the critical insight that unlocks the entire pipeline parallelism (PP) performance problem. This article examines that message in detail: what led to it, what it reveals, and how it reshaped the deployment strategy.

The Context: A Pipeline That Wouldn't Perform

The session leading up to message 11485 had been a rollercoaster. The assistant had successfully deployed Kimi K2.6 with tensor parallelism 8 (TP8), achieving ~98 tok/s at concurrency 1 and ~1291 tok/s aggregate throughput ([msg 11469]). But the user wanted to explore pipeline parallelism (PP8) as an alternative, hoping it might avoid the AllReduce bottlenecks that plagued TP8 on PCIe-connected GPUs.

The first PP8 attempt crashed with an IndexError: list index out of range ([msg 11470]). The assistant traced this to a bug in SGLang's triton attention backend: during initialization, the code probed layer 0's value buffer using get_value_buffer(0), but on non-zero PP stages where start_layer > 0, this produced a negative index (0 - start_layer < 0), causing the out-of-bounds error ([msg 11472]). The assistant patched the bug by replacing get_value_buffer(0) with get_value_buffer(start_layer) ([msg 11476]).

After the fix, PP8 started successfully and loaded faster than TP8 (parallel weight loading across stages). But the benchmark results were devastating: only 24.7 tok/s for single requests and 95.6 tok/s aggregate at C=8 ([msg 11478]). The user aborted the benchmark and pointed out the obvious problem: GPU utilization was at 150–200W out of 600W maximum, meaning the GPUs were mostly idle ([msg 11479]).

The user's intuition was sharp: "seems like we're not properly pushing data into the pipeline such that all layers are properly utilised." They asked whether SGLang's PP scheduler could buffer work at pipeline stages—allowing GPU0 to start processing a new batch while GPU1 is still working on the previous one, rather than forcing all 8 GPUs to serialize their work.

The Investigation: Two Root Causes

The assistant checked the service configuration and found the smoking gun: --disable-cuda-graph was explicitly set ([msg 11480]). Without CUDA graphs, every kernel launch had to go through the full Python dispatch path, adding enormous overhead between pipeline stages. The assistant also noted the absence of --pp-async-batch-depth, which controls how many micro-batches can be in-flight simultaneously in the pipeline.

The assistant then dove into SGLang's PP scheduling code (<msg id=11481-11484>), reading the scheduler_pp_mixin.py file to understand the pipeline mechanics. This investigation led directly to message 11485.

Message 11485: The Discovery

The target message begins with a reasoning paragraph that shows the assistant processing a critical piece of information:

I see -- pp_loop_size = pp_size + pp_async_batch_depth. So with pp_async_batch_depth=8 and pp_size=8, there would be 16 micro-batch slots. This allows multiple micro-batches to be in-flight simultaneously, so when GPU0 finishes one batch and sends it to GPU1, it can immediately start the next batch.

This is the "aha" moment. The assistant had just read line 523 of scheduler_pp_mixin.py in the previous message ([msg 11484]):

self.pp_loop_size: int = self.pp_size + self.server_args.pp_async_batch_depth

This formula reveals the fundamental design of SGLang's pipeline scheduler. The pipeline loop size is the sum of the pipeline depth (number of stages) and the async batch depth (number of extra micro-batches that can be in-flight). With pp_async_batch_depth=0 (the default), the loop size equals the pipeline depth—meaning only one micro-batch can be in the pipeline at a time. Each GPU processes its portion, then waits for the next GPU to finish before proceeding. This creates the classic "pipeline bubble" problem: at any given moment, only one GPU is active while the other 7 sit idle, yielding roughly 1/8 utilization.

With pp_async_batch_depth=8, the loop size becomes 16. This means up to 8 additional micro-batches can be queued in the pipeline. When GPU0 finishes processing a micro-batch and sends it to GPU1, it can immediately start working on the next micro-batch from the queue. This keeps all GPUs busy simultaneously—GPU0 works on batch N+1 while GPU1 works on batch N, GPU2 works on batch N-1, and so on. The pipeline becomes a filled assembly line rather than a serial conveyor belt.

The reasoning then pivots to the second issue: CUDA graphs.

Now, about CUDA graphs: the --disable-cuda-graph was originally because of SM120 issues. But we now have CUDA 13 toolkit. Let me check if CUDA graphs work with the triton backend on K2.6.

This shows the assistant connecting two threads of investigation. Earlier in the session, CUDA graphs had been disabled because the Blackwell GPUs (SM120) had compatibility issues with SGLang's CUDA graph capture. But the environment had since been upgraded to CUDA 13 toolkit ([msg 11469]). The assistant is now questioning whether that flag is still necessary—and whether removing it could provide the second half of the performance fix.

The reasoning continues:

The key thing is whether that flag actually prevents CUDA graph capture, and whether removing it would be compatible with triton attention on K2.6. I need to understand what the default CUDA graph behavior is when the flag isn't set.

This is a crucial methodological point. The assistant recognizes that it needs to verify its assumptions before making changes. It doesn't assume that removing --disable-cuda-graph will automatically enable CUDA graphs—there might be other conditions that disable them (e.g., the attention backend choice, or model-specific constraints). The assistant plans to check the server_args code to understand the default behavior.

The Bash Command: Checking Defaults

The assistant then executes a straightforward grep command to check the default values of pp_async_batch_depth and pp_max_micro_batch_size in SGLang's server arguments:

ssh -o ConnectTimeout=5 root@10.1.2.200 "
# Check defaults for pp_async_batch_depth and pp_max_micro_batch_size
grep -n 'pp_async_batch_depth\|pp_max_micro_batch_size' /root/venv_sglang211/lib/python3.12/site-packages/sglang/srt/server_args.py | head -10
"

The output reveals:

391:    pp_max_micro_batch_size: Optional[int] = None
392:    pp_async_batch_depth: int = 0
4772:            default=ServerArgs.pp_max_micro_batch_size,
4778:            default=ServerArgs.pp_async_batch_depth,
6940:            self.pp_max_micro_batch_size is None or self.pp_max_micro_batch_size >= 1
6942:            "pp_max_micro_batch_size must be a positive integer or None (for auto-compute). "
6943:            f"Got: {self.pp_max_micro_batch_size}"

This confirms the assistant's suspicion. pp_async_batch_depth defaults to 0, meaning no async batching is enabled by default. Without explicitly setting this parameter, the PP pipeline runs in pure serial mode—one micro-batch at a time, with all the pipeline bubble inefficiency that entails.

The output also shows pp_max_micro_batch_size defaults to None, which triggers auto-computation. This is a separate parameter that controls how many sequences are batched together within a single micro-batch, as opposed to how many micro-batches are in-flight.

What This Message Achieves

Message 11485 is a turning point. Before this message, the assistant had two separate pieces of information:

  1. The PP8 performance was terrible (24.7 tok/s)
  2. The scheduler code had a formula pp_loop_size = pp_size + pp_async_batch_depth In this message, the assistant connects these pieces. The formula explains why performance was terrible: with pp_async_batch_depth=0, the pipeline has no overlap, and each GPU spends most of its time waiting. The assistant now has a clear, actionable hypothesis: setting pp_async_batch_depth=8 should dramatically improve throughput by filling the pipeline. The message also sets up the next investigation: checking whether CUDA graphs can be safely enabled. The assistant correctly identifies that CUDA graphs and async batching are complementary fixes—graphs reduce kernel launch overhead within a stage, while async batching keeps all stages busy simultaneously.

Assumptions and Knowledge

The message makes several implicit assumptions that are worth examining. First, the assistant assumes that pp_async_batch_depth=8 (matching the pipeline depth) is the right value. This is a reasonable starting point—it ensures there are enough micro-batches to fill all pipeline stages—but the optimal value might differ based on the workload characteristics and the relative speeds of different pipeline stages.

Second, the assistant assumes that CUDA graphs will work with the triton attention backend on K2.6 now that CUDA 13 is installed. This is an empirical question that requires testing. The earlier SM120 issues might have been caused by bugs in SGLang's CUDA graph capture code that are independent of the CUDA toolkit version.

Third, the assistant assumes that the --disable-cuda-graph flag is the only thing preventing CUDA graph capture. The subsequent investigation (in messages after 11485) would need to verify that no other conditions (like the attention backend choice or model architecture) force-disable graphs.

The input knowledge required to understand this message includes:

The Thinking Process

The reasoning in this message reveals a methodical debugging approach. The assistant starts with a specific observation (the pp_loop_size formula), immediately connects it to the performance problem (pipeline bubbles with pp_async_batch_depth=0), and formulates a concrete fix (set pp_async_batch_depth=8). It then pivots to the second issue (CUDA graphs), but instead of blindly removing the flag, it plans to verify the default behavior first.

This two-pronged approach is characteristic of good debugging: identify all the contributing factors, verify your understanding of each one, then apply fixes systematically. The assistant doesn't rush to change the service configuration—it first confirms the default values to ensure its mental model is correct.

The message also shows the assistant working within the constraints of the remote environment. It uses SSH to execute commands on the CT200 host, reads source code from the installed Python package, and interprets the results in real-time. The thinking is done "out loud" in the reasoning section, making the debugging process transparent.

Aftermath

The next message ([msg 11489]) acts on the insights from message 11485. The assistant checks whether CUDA graphs are auto-disabled for the triton backend (they are not), then creates a new PP8 service configuration with --pp-async-batch-depth 8 and without --disable-cuda-graph. The service starts successfully and is ready within 90 seconds ([msg 11490]).

The subsequent benchmarking would reveal whether these two changes—async batching and CUDA graphs—finally unlock PP8's potential on the Blackwell GPUs. But regardless of the final benchmark numbers, message 11485 represents the critical insight that made those changes possible. It's a reminder that in systems debugging, understanding the scheduler's fundamental formula is often the key to unlocking performance.