The Smoking Gun: Tracing a Pipeline Parallelism Bottleneck in SGLang

In the high-stakes world of large language model inference, every microsecond counts. When deploying the massive Kimi K2.6 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a developer expects pipeline parallelism to deliver a significant throughput advantage over tensor parallelism—especially for Mixture-of-Experts (MoE) architectures that suffer from expensive AllReduce operations across PCIe links. But when the benchmarks came in, the results were confounding: pipeline parallelism (PP8) achieved a respectable 34.3 tok/s on single requests (30% better than TP8's 26.3 tok/s), yet at 32 concurrent requests it collapsed to 248 tok/s—less than half of TP8's 577 tok/s. Something was fundamentally wrong with how the pipeline was scaling.

This is the story of a single message in the debugging session—message index 11498—that cracked the case wide open. It is a deceptively simple grep command, but it reveals the precise moment when a hidden auto-computation formula was uncovered, explaining why the pipeline was silently starving itself of work.

The Context: A Pipeline That Wouldn't Fill

The debugging session leading up to this message had been a classic exercise in performance forensics. The assistant had deployed Kimi K2.6 with SGLang's pipeline parallelism across 8 GPUs, configured with --pp-async-batch-depth 8 to allow multiple micro-batches to be in flight simultaneously, --max-running-requests 64 to admit plenty of concurrent work, and CUDA graphs enabled for optimized kernel execution. On paper, this should have been a winning configuration.

But the benchmarks told a different story. At 32 concurrent requests, aggregate throughput plateaued at 248 tok/s. GPU utilization fluctuated wildly between 40% and 100%, with power draw maxing out around 310W per GPU—well below the thermal limit. The assistant's profiling revealed a critical anomaly: the logs consistently showed #running-req: 8 across all pipeline stages, even though 32 API requests were in flight. Only 8 requests were being processed at any given time; the remaining 24 were queued, waiting for admission.

The assistant had already ruled out several suspects. CUDA graphs were confirmed active (cuda graph: True). KV cache utilization was only 10%, so memory pressure wasn't the issue. The disable_cuda_graph flag was False. But the persistent 8-request cap demanded an explanation. In message 11497, the assistant had checked the server logs and found that pp_max_micro_batch_size=None (auto-compute mode). The suspicion crystallized: the auto-computed micro-batch size was the bottleneck.

The Message: A Targeted Grep for the Root Cause

Message 11498 is a single bash command executed over SSH on the remote inference server (CT200). It performs two targeted grep searches through SGLang's Python source code:

ssh -o ConnectTimeout=5 root@10.1.2.200 "
# Check the get_next_batch_to_run function - this is where batch size is determined
grep -n 'def get_next_batch_to_run\|pp_max_micro_batch\|max_running' /root/venv_sglang211/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py | head -15
echo '==='
# Check where pp_max_micro_batch_size is actually used
grep -rn 'pp_max_micro_batch_size' /root/venv_sglang211/lib/python3.12/site-packages/sglang/srt/ --include='*.py' | head -10
" 2>&1

The first grep targets scheduler.py, specifically the get_next_batch_to_run function—the heart of SGLang's batch admission logic. The assistant is looking for where the scheduler decides how many requests to admit into a single batch. The second grep performs a recursive search across all Python files in the SGLang source tree for any usage of pp_max_micro_batch_size, the parameter that controls how many requests can fit in a single micro-batch within the pipeline.

This is not a random fishing expedition. It is a precisely targeted investigation based on a well-formed hypothesis: the auto-computed micro-batch size is capping the pipeline's throughput.

The Discovery: An Auto-Computation Formula

The output reveals the smoking gun. Lines 729–731 of scheduler.py contain the auto-computation logic:

if not get_global_server_args().pp_max_micro_batch_size:
    get_global_server_args().pp_max_micro_batch_size = max(
        self.max_running_requests // self.pp_size, 1

This is the critical formula. When pp_max_micro_batch_size is left as None (auto-compute mode), SGLang computes it as:

pp_max_micro_batch_size = max(max_running_requests // pp_size, 1)

With the deployed configuration of max_running_requests=64 and pp_size=8, this evaluates to:

pp_max_micro_batch_size = max(64 // 8, 1) = max(8, 1) = 8

The micro-batch size was being auto-computed to 8 requests per micro-batch. This is the exact number that appeared in the logs as #running-req: 8. The mystery was solved.

Why This Matters: The Pipeline Parallelism Math

To understand why this is devastating for throughput, consider how pipeline parallelism works. In SGLang's implementation, the pipeline has pp_size stages (8 GPUs), and with pp_async_batch_depth=8, there are pp_loop_size = pp_size × pp_async_batch_depth = 64 micro-batch slots. These slots allow multiple decode steps to overlap across pipeline stages—while step N processes through stages 1→8, step N+1 can begin processing at stage 1.

However, each micro-batch slot is limited to pp_max_micro_batch_size requests. With a cap of 8 requests per micro-batch, the scheduler admits only 8 requests into the pipeline at a time. The remaining concurrent requests queue up and wait their turn. The pipeline stages process the same 8 requests in lockstep, producing the "pipeline bubble" pattern observed in the GPU utilization logs (40–100% variance).

The auto-computation formula is designed to prevent any single micro-batch from monopolizing the pipeline, but in this configuration it was too conservative. The formula assumes that all max_running_requests could end up in a single micro-batch, so it divides by pp_size to spread them across stages. But with 64 max running requests and 8 pipeline stages, the formula produces 8—which exactly matches the observed cap. The pipeline was running at 12.5% of its potential batch capacity.

The Thinking Process: A Chain of Diagnostic Reasoning

The assistant's reasoning leading up to this message is a textbook example of systematic performance debugging. The chain went as follows:

  1. Observe the symptom: PP8 throughput at C=32 is 248 tok/s vs TP8's 577 tok/s—a 2.3× deficit despite 30% better single-request performance.
  2. Form initial hypotheses: The pipeline isn't filling properly. Possible causes include micro-batch splitting creating too many tiny batches, the scheduler not accumulating work efficiently, or CUDA graph issues across pipeline stages.
  3. Profile to gather evidence: GPU utilization shows 40–100% variance with pipeline bubble patterns. Logs show only #running-req: 8 across all stages despite 32 concurrent API calls. All stages report identical batch sizes at identical timestamps—lockstep execution, not true pipelining.
  4. Eliminate red herrings: CUDA graphs are confirmed active. Memory is not the bottleneck (10% KV cache utilization). The disable_cuda_graph flag is False.
  5. Narrow the suspect: The #running-req: 8 is suspiciously close to max_running_requests // pp_size = 64 // 8 = 8. The pp_max_micro_batch_size parameter is set to None (auto-compute). This must be the limiter.
  6. Verify with source code: Grep the scheduler to find the auto-computation formula. Confirm that pp_max_micro_batch_size = max(max_running_requests // pp_size, 1). This is not guesswork. Each step is grounded in empirical observation and a clear understanding of SGLang's architecture. The assistant correctly identified that the scheduler's admission logic was the bottleneck, formulated a specific hypothesis about the auto-computation formula, and verified it with a targeted source code search.

Assumptions and Their Validity

The assistant operated under several key assumptions in this message:

The bottleneck is in the scheduler's batch admission logic, not in kernel execution or communication. This assumption was validated by the profiling data: CUDA graphs were active, GPU utilization was moderate, and the pipeline stages were running in lockstep. The bottleneck was clearly upstream of the actual computation.

The auto-computation formula for pp_max_micro_batch_size would be found in scheduler.py. This was a reasonable assumption given that batch scheduling logic typically lives in the scheduler module. The grep confirmed this.

The pp_max_micro_batch_size parameter, when set to None, would be auto-computed somewhere in the source tree. The assistant had already confirmed from logs that the parameter was None, and the recursive grep across all Python files was designed to find where it gets resolved.

One earlier assumption proved incorrect: the assistant had assumed that setting --pp-async-batch-depth 8 and --max-running-requests 64 would be sufficient to fill the pipeline. The hidden auto-computation of pp_max_micro_batch_size silently overrode this intent. The assistant had not anticipated that SGLang would independently compute a micro-batch cap that was independent of the async depth setting.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with SGLang's pipeline parallelism architecture, understanding of micro-batching in pipeline-parallel inference, knowledge of the relationship between pp_size, pp_async_batch_depth, max_running_requests, and pp_max_micro_batch_size, and proficiency with grep and SSH for remote source code inspection.

Output knowledge created by this message is the critical auto-computation formula: pp_max_micro_batch_size = max(max_running_requests // pp_size, 1). This formula explains the observed behavior and provides a clear path to a fix: either set pp_max_micro_batch_size explicitly to a larger value (e.g., matching max_running_requests), or increase max_running_requests to raise the auto-computed value. The formula also reveals a design assumption in SGLang: that micro-batch capacity should be evenly divided across pipeline stages, which may be too conservative for workloads where the pipeline is not uniformly loaded.

The Broader Significance

This message is a microcosm of what makes performance debugging in distributed ML systems so challenging. The system had multiple interacting parameters (pp_size, pp_async_batch_depth, max_running_requests, pp_max_micro_batch_size), each with its own auto-computation logic and default behavior. The assistant had to trace through layers of abstraction—from benchmark numbers to GPU utilization logs to scheduler source code—to find a single line of Python that was silently throttling the entire pipeline.

The discovery also highlights a design tension in inference frameworks like SGLang. Auto-computation of parameters is convenient for users who don't want to tune every knob, but it can produce counterintuitive results when parameters interact. The formula max_running_requests // pp_size makes sense as a safety limit to prevent any single micro-batch from consuming all available capacity, but it backfires when the pipeline has sufficient memory and compute bandwidth to handle larger batches. The fix—setting pp_max_micro_batch_size explicitly—is trivial once the formula is known, but finding it required hours of systematic investigation.

In the messages that follow, the assistant would apply this discovery, reconfiguring the service with an explicit micro-batch size and watching the throughput numbers climb. But message 11498 remains the turning point: the moment when a hidden formula was exposed, and the pipeline's silent bottleneck was finally named.