The Pipeline Parallelism Benchmark That Revealed Deeper Problems

Introduction

In the high-stakes world of large language model deployment, every tok/s counts. When you're running an 8-GPU inference server for a massive MoE model like Kimi K2.6, the difference between good and great throughput can mean hours saved on generation runs. Message [msg 11491] captures a pivotal moment in an optimization journey: the moment when a well-reasoned hypothesis about pipeline parallelism meets the cold reality of empirical measurement. The assistant had just reconfigured an 8-GPU SGLang deployment, enabling CUDA graphs and asynchronous micro-batch pipelining—two changes that should have dramatically improved throughput. Instead, the benchmark results told a more complicated story, one that would send the team back to the drawing board to investigate deeper issues in SGLang's pipeline parallel implementation.

The Road to This Benchmark

To understand message [msg 11491], we must first understand what came before it. The team was deploying Kimi K2.6, a 1.6-trillion-parameter Mixture-of-Experts model, across 8× RTX PRO 6000 Blackwell GPUs connected via PCIe. The initial deployment used pipeline parallelism (PP8) with tensor parallelism disabled (TP1), meaning each GPU hosted one stage of the 8-stage pipeline. The first benchmark, in message [msg 11478], revealed disappointing results: approximately 24.7 tok/s for single requests and only 95.6 tok/s aggregate at C=8 concurrency. All eight GPUs were barely sipping power at ~85W each—a far cry from their 600W thermal design power.

The user immediately identified the core problem in message [msg 11479]: GPU utilization was abysmal, and they suspected two root causes. First, CUDA graphs were disabled (--disable-cuda-graph), which eliminated the GPU's ability to capture and replay entire kernel launch sequences without CPU intervention. Second, there was no mechanism for work buffering at pipeline boundaries—when one GPU finished its pipeline stage and sent data to the next GPU, it sat idle rather than picking up the next micro-batch. The user's intuition was precise: in a naive pipeline, only one GPU is active at any moment, giving you roughly 1/N utilization for an N-stage pipeline.

The assistant confirmed these suspicions in messages [msg 11480] and [msg 11481], discovering that the service was running with --disable-cuda-graph and without any --pp-async-batch-depth setting. Digging into SGLang's source code, the assistant found the scheduler_pp_mixin.py file which implements pipeline scheduling, and discovered that pp_loop_size = pp_size + pp_async_batch_depth. With pp_async_batch_depth=0, the pipeline had exactly 8 micro-batch slots—one per stage—meaning no micro-batch could be queued behind a busy GPU. The fix seemed clear: set --pp-async-batch-depth 8 to double the micro-batch capacity to 16, allowing multiple micro-batches to be in-flight simultaneously, and remove --disable-cuda-graph to enable CUDA graph capture.

Message [msg 11489] shows the assistant implementing these changes, rewriting the systemd service file and restarting the SGLang server. The service came up quickly in message [msg 11490], thanks to cached JIT compilations from the previous run.

The Benchmark Itself

Message [msg 11491] is the assistant's benchmark of this newly configured service. The message contains a Python script that tests both single-request and concurrent throughput, followed by the output. Let me quote the key results:

=== K2.6 PP8 async+cudagraph - single request ===
  34.3 tok/s | 3617 tok in 105.5s
  35.6 tok/s | 2009 tok in 56.5s
  33.9 tok/s | 4096 tok in 120.8s

=== Concurrent throughput ===
  C= 4: agg=   64.0 tok/s  wall= 184.1s  avg=2945tok/req
  C= 8: agg=   81.7 tok/s  wall= 248.6s  avg=2538tok/req
  C=16: agg=  156.4 tok/s  wall= 305.8s  avg=2989tok/req
  C=32: agg=  248.1 tok/s  wall= 378.7s  avg=2935tok/req

The benchmark script itself is a well-structured piece of test infrastructure. It defines an api() function that sends a chat completion request to the SGLang OpenAI-compatible endpoint, measures wall-clock time, and returns tokens-per-second. The script performs three warmup calls with a trivial "Say OK" prompt—these are critical because they trigger CUDA graph capture. Without warmup, the first real request would pay the cost of graph capture. The script then tests three different prompts (quicksort implementation, quantum entanglement explanation, Redis data structures guide) to get representative results across different generation patterns. Finally, it tests concurrent throughput using a ThreadPoolExecutor across concurrency levels 4, 8, 16, 32, with plans to go to 48 and 64 (the user aborted before those completed).

What the Numbers Reveal

The single-request results show a clear improvement: ~34.6 tok/s average compared to ~24.6 tok/s in the previous configuration. That's a 40% improvement, validating that CUDA graphs and async batching do help reduce per-request latency. The CUDA graph capture eliminates the Python-to-CUDA kernel launch overhead for each decode step, and the async batching allows the pipeline to stay slightly more filled even with a single request stream.

However, the concurrent throughput results tell a more complex story. At C=4, the new configuration achieves 64.0 tok/s, which is actually worse than the previous configuration's 68.3 tok/s. At C=8, the gap widens: 81.7 tok/s vs 95.6 tok/s. This is deeply counterintuitive. If CUDA graphs and async batching improve single-request performance, why would they hurt concurrent performance?

The answer lies in the dynamics of pipeline parallelism. With async batching enabled, the scheduler allows more micro-batches to be queued at each pipeline stage. But if the underlying issue is not pipeline bubbles but rather uneven distribution of work across GPUs—or, as the user later hypothesized in message [msg 11492], "thundering herds" where batches arrive at GPUs in uneven bursts—then async batching could actually exacerbate the problem. More queued micro-batches mean more work piling up behind a slow GPU while other GPUs finish their work and wait. The pipeline bubble shifts from being a scheduling problem to a load-balancing problem.

The higher-concurrency results (C=16 at 156.4 tok/s, C=32 at 248.1 tok/s) suggest that the configuration does scale better with more concurrent requests. The aggregate throughput continues to climb as concurrency increases, unlike the previous configuration which seemed to plateau around C=8. But the absolute numbers are still far below what the hardware should be capable of. Eight RTX PRO 6000 Blackwell GPUs, each with substantial compute and memory bandwidth, should be able to deliver significantly more than 248 tok/s for a model of this size.

Assumptions and Their Failure Modes

The assistant made several assumptions in designing this benchmark, some of which proved incorrect.

Assumption 1: CUDA graphs would work correctly with the triton attention backend on K2.6. The assistant had previously disabled CUDA graphs due to SM120 compatibility issues with the Blackwell GPUs. After installing CUDA 13.0, the assistant assumed these issues were resolved. The fact that the service started and produced valid output suggests CUDA graphs are functional, but the modest improvement (40% on single requests) raises questions about whether graph capture is working optimally for all kernel types in the pipeline.

Assumption 2: Async batch depth of 8 would be sufficient to fill the pipeline. The assistant set --pp-async-batch-depth 8, matching the PP size. The reasoning was that this would double the micro-batch capacity from 8 to 16, allowing each GPU to have work queued. However, this assumes the bottleneck is purely about having enough micro-batches in flight. If the real bottleneck is NCCL communication overhead across PCIe, or if SGLang's scheduler has other serialization points, then increasing micro-batch capacity won't help.

Assumption 3: The warmup calls would successfully trigger CUDA graph capture for all relevant batch sizes. The benchmark does three warmup calls with a trivial prompt. CUDA graphs are typically captured at specific batch sizes, and if the warmup doesn't exercise the same batch sizes as the real benchmark, the graphs may not be fully captured. This could explain why single-request performance improved (the warmup matches single-request batch sizes) but concurrent performance didn't (the warmup didn't prepare graphs for larger batch sizes).

Assumption 4: The benchmark script from the previous run would provide a fair comparison. The assistant reused the same benchmark script structure, which is good for comparability. However, the previous benchmark was aborted after C=8, so there's no data for C=16 and C=32 from the old configuration. We can't know whether the old configuration would have scaled better or worse at higher concurrency.

The Deeper Problem

The user's response in message [msg 11492] cuts to the heart of the issue: "Much worse than expected, maybe some sglang perf issue? Cross gpu bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream, maybe we're dispatching somehow in a way that wastes batching potential."

This is a crucial insight. With only ~35 tok/s per stream, the amount of data being transferred between pipeline stages is tiny. Each decode step produces a small hidden state vector, not a full batch of activations. The PCIe bandwidth between GPUs should be more than sufficient for this data volume. The fact that throughput doesn't scale linearly with concurrency suggests the bottleneck is not in the data transfer but in the scheduling logic itself.

The user's "thundering herd" hypothesis is particularly insightful. If SGLang's scheduler dispatches all ready requests to the first pipeline stage in a burst, they'll all arrive at GPU0 simultaneously, creating a queue. GPU0 processes them one by one, sending each to GPU1. But GPU1 might be idle while waiting for GPU0, then suddenly receive a flood of micro-batches. This uneven arrival pattern means GPUs spend more time waiting than computing. The async batching feature might actually make this worse by allowing more micro-batches to pile up at each stage.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

Pipeline Parallelism (PP): Understanding that PP splits a model across GPUs by layer ranges, with each GPU computing a subset of layers. The pipeline bubble problem—where only one GPU is active at a time in naive implementations—is central to interpreting these results.

CUDA Graphs: Knowledge that CUDA graphs capture a sequence of GPU kernel launches and can replay them without CPU involvement, reducing launch latency. This is particularly important for small decode steps where kernel launch overhead dominates.

SGLang Architecture: Familiarity with SGLang's server arguments (--pp-async-batch-depth, --disable-cuda-graph, --pp-size, --tp-size) and how they interact. Understanding that SGLang uses a scheduler with micro-batch slots and that pp_loop_size = pp_size + pp_async_batch_depth.

Kimi K2.6 Model Characteristics: Knowledge that this is a 1.6T MoE model with MLA (Multi-head Latent Attention) and that its architecture affects how it maps to pipeline stages.

Blackwell GPU Architecture: Understanding the RTX PRO 6000's capabilities (SM120 compute capabilities, 600W TDP, PCIe connectivity) and the implications of PCIe vs NVLink for inter-GPU communication.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. Empirical validation that CUDA graphs + async batching improve single-request PP throughput by ~40% (24.7 → 34.6 tok/s) for K2.6 on 8× Blackwell GPUs.
  2. Evidence that these improvements do not translate to concurrent throughput at low concurrency levels (C=4 and C=8 actually regressed), suggesting the bottleneck is not pipeline bubbles but something else.
  3. A scaling curve showing that aggregate throughput continues to increase with concurrency (64 → 81.7 → 156.4 → 248.1 tok/s at C=4/8/16/32), but with diminishing returns that suggest an upper bound far below hardware capability.
  4. A diagnostic data point that helps narrow down the root cause: since inter-GPU bandwidth should be sufficient for the observed throughput, the bottleneck is likely in SGLang's scheduling logic rather than in the communication fabric.
  5. A reusable benchmark harness that can be applied to future configuration changes to measure improvement or regression.

The Thinking Process

While message [msg 11491] doesn't contain explicit reasoning text (it's purely a benchmark execution), the structure of the benchmark reveals the assistant's thinking process.

The assistant begins with "Fast startup with cached JIT," noting that the service initialization was quick because the Triton JIT compilations from the previous run were cached. This is an important operational detail—it means the benchmark is measuring steady-state performance, not cold-start performance.

The warmup phase is explicitly labeled "Warmup (cuda graph capture)," showing that the assistant understands the mechanics of CUDA graph capture and is deliberately triggering it before the real measurements. The three warmup calls with a trivial prompt are designed to ensure the graphs are captured and cached.

The benchmark structure mirrors the previous run exactly, showing that the assistant wants a direct apples-to-apples comparison. The same prompts, the same concurrency levels, the same API function. This methodological consistency is good science—it isolates the configuration change as the only variable.

The concurrency levels (4, 8, 16, 32, 48, 64) are chosen to probe different regimes of pipeline behavior. C=4 and C=8 test whether the pipeline can be filled with a small number of concurrent requests. C=16 and C=32 test medium concurrency where async batching should help. C=48 and C=64 (which weren't reached) would test saturation behavior.

The fact that the user aborted the benchmark after C=32 suggests the results were disappointing enough that continuing wasn't useful. The user's subsequent message confirms this: "Much worse than expected."

Conclusion

Message [msg 11491] is a classic example of the scientific method in ML infrastructure engineering. The assistant formed a hypothesis (CUDA graphs + async batching will fix PP throughput), implemented the changes, and tested empirically. The results were mixed—partial validation at the single-request level, regression at low concurrency, and modest scaling at higher concurrency. This sent the team back to first principles, questioning whether the problem is in SGLang's scheduler rather than in the pipeline configuration.

The benchmark results in this message became the foundation for the next phase of investigation: profiling SGLang's dispatch logic, understanding NCCL communication patterns, and ultimately considering alternative parallelism strategies (expert parallelism, which would later prove dramatically more effective). Sometimes the most valuable result of an experiment is not the confirmation of a hypothesis but the clear evidence that the hypothesis was incomplete.