The Micro-Batch Hypothesis: When a Well-Reasoned Fix Fails to Move the Needle

Introduction

In the high-stakes world of large language model deployment, every token per second counts. When a team deploys a massive MoE model like Kimi K2.6 across 8 RTX PRO 6000 Blackwell GPUs, the choice of parallelism strategy can mean the difference between a responsive service and a sluggish one. This article examines a single pivotal message—message 11501 in an opencode coding session—where an assistant runs a benchmark to test a carefully reasoned hypothesis about pipeline parallelism performance in SGLang. The results are surprising and instructive: even a fix grounded in direct code analysis can fail to improve throughput, revealing that the true bottleneck lies elsewhere.

The Context: A Pipeline Parallelism Mystery

The story begins with the deployment of Kimi K2.6 using SGLang's pipeline parallelism (PP8) across 8 GPUs connected via PCIe. Pipeline parallelism splits the model across GPUs: each GPU handles a subset of layers, and requests flow through the pipeline stage by stage. In theory, PP should outperform tensor parallelism (TP) for Mixture-of-Experts models on PCIe, because PP avoids the expensive AllReduce communication that TP requires for every MoE layer. The user's intuition was sound: with PCIe bandwidth being the scarce resource, PP's per-layer communication should be much cheaper than TP's all-to-all synchronization across all 8 GPUs.

Yet the initial benchmarks told a different story. TP8 (tensor parallelism across 8 GPUs) achieved 26.3 tok/s for a single request and 577.8 tok/s at 32 concurrent requests. PP8, despite its theoretical advantages, managed only 34.3 tok/s for a single request (a modest 30% improvement) but collapsed to 248.1 tok/s at 32 concurrent requests—less than half of TP8's throughput. Something was fundamentally wrong.

The user's diagnosis was sharp and insightful: "Cross gpu bandwidth is tiny, layer sends should be really cheap with this few tok/s/stream, maybe we're dispatching somehow in a way that wastes batching potential—e.g batches doing thundering herds, not evenly split on GPUs?" This observation set the stage for a deep investigation into SGLang's pipeline scheduling internals.

The Investigation: Finding the Smoking Gun

The assistant began profiling the PP8 deployment with a series of diagnostic commands. GPU utilization fluctuated wildly between 40% and 100%, power consumption maxed out around 310W per GPU (well below the ~450W thermal limit), and—most critically—the SGLang logs showed only 8 running requests per pipeline stage despite 32 concurrent API calls. With max_running_requests=64 configured, only 8 requests were actually being processed at any given time. The rest were queued, waiting for admission into the pipeline.

Digging into SGLang's source code, the assistant traced the issue to the scheduler's auto-computation logic. In the file sglang/srt/managers/scheduler.py, line 730, the code read:

pp_max_micro_batch_size = max(max_running_requests // pp_size, 1)

With max_running_requests=64 and pp_size=8, this yielded pp_max_micro_batch_size = 8. Each micro-batch—the fundamental unit of work that traverses the pipeline—could hold at most 8 requests. With 16 micro-batch slots (8 from async depth plus 8 pipeline stages), the total theoretical capacity was 128 request slots, but the bottleneck seemed clear: each individual micro-batch traversing the pipeline could only process 8 requests at a time. This appeared to be the smoking gun.

The assistant's reasoning was straightforward: if each micro-batch is limited to 8 requests, then the pipeline stages are underutilized. With only 8 requests per micro-batch, the GPU compute capacity isn't fully exploited, and the pipeline overhead—NCCL communication, kernel launch latency, synchronization barriers—dominates the per-request computation. The fix was to explicitly set --pp-max-micro-batch-size 64, matching the max_running_requests limit, so that each micro-batch could contain up to 64 requests.

The assistant modified the systemd service file, stopping the old service and starting a new one with the explicit parameter:

--pp-size 8 --pp-async-batch-depth 8 --pp-max-micro-batch-size 64 --max-running-requests 64

After waiting 120 seconds for the model to load and CUDA graphs to compile, the service was ready. The moment of truth had arrived.## The Benchmark: A Well-Designed Test

Message 11501 contains the benchmark that tests this hypothesis. The assistant writes a Python script that sends HTTP requests to the SGLang endpoint and measures throughput. The script is carefully structured: it performs two warmup calls (short 16-token generations to trigger CUDA graph capture), then tests single-request throughput with two different prompts, and finally measures aggregate throughput across concurrency levels of 8, 16, 32, 48, and 64 concurrent requests.

The benchmark design reveals several assumptions. First, the assistant assumes that the fix is correct—that pp_max_micro_batch_size=64 will allow the pipeline to fill with more requests per micro-batch, increasing GPU utilization and throughput. Second, the assistant assumes that the bottleneck is indeed the micro-batch size limit, not some other constraint like KV cache capacity, NCCL communication overhead, or the scheduler's admission policy. Third, the assistant assumes that the CUDA graphs captured at the previous batch sizes (up to bs=64) will work correctly with the new configuration—a reasonable assumption since the graphs were already captured for batch sizes up to 64.

The script also makes pragmatic choices: it uses only two prompts for single-request testing (to save time), and it limits concurrent testing to 64 requests (matching max_running_requests). The timeout of 900 seconds per request is generous, reflecting the assistant's expectation that each request might take several minutes to complete at the lower throughput levels.

The Results: A Disappointing Confirmation

The benchmark output is stark:

=== K2.6 PP8 (micro_batch=64) - single request ===
  36.1 tok/s | 1343 tok in 37.2s
  36.0 tok/s | 1490 tok in 41.4s

=== Concurrent throughput ===
  C= 8: agg=   83.2 tok/s  wall= 267.2s  avg=2778tok/req
  C=16: agg=  159.5 tok/s  wall= 301.8s  avg=3009tok/req
  C=32: agg=  250.9 tok/s  wall= 367.4s  avg=2881tok/req
  C=48: agg=  316.2 tok/s  wall= 443.1s  avg=2919tok/req

Comparing with the previous PP8 results (before the fix):

| Configuration | Single (tok/s) | C=8 | C=16 | C=32 | C=48 | |---|---|---|---|---|---| | PP8 (old, micro_batch=8 auto) | 34.3 | 83.2 | 159.5 | 248.1 | — | | PP8 (new, micro_batch=64) | 36.1 | 83.2 | 159.5 | 250.9 | 316.2 |

The single-request throughput improved marginally from ~34.3 to ~36.0 tok/s—a 5% gain that could be attributed to warmup effects or measurement variance. The concurrent throughput at C=32 went from 248.1 to 250.9 tok/s—essentially unchanged. The new configuration did allow testing at C=48 (which wasn't tested before), reaching 316.2 tok/s, but this is still far below TP8's 577.8 tok/s at C=32.

The micro-batch size fix had essentially no effect. The hypothesis was wrong.

Why the Fix Failed: Deconstructing the Reasoning

The assistant's reasoning was internally consistent but made several incorrect assumptions about how SGLang's pipeline parallelism actually works.

Assumption 1: Micro-batch size limits throughput. The assistant assumed that a larger micro-batch would allow more requests to be processed simultaneously, increasing GPU utilization. In reality, the pipeline's throughput is limited by the slowest stage, not by micro-batch size. Even with 64 requests per micro-batch, if the pipeline has 8 stages and each stage processes its micro-batch sequentially, the total throughput is still limited by the per-stage processing time. The async depth of 8 allows two consecutive decode steps to overlap (step N on GPU 1 while step N+1 starts on GPU 0), but this overlap only helps if the per-step computation is large enough to amortize the pipeline bubble overhead.

Assumption 2: The bottleneck is in the scheduler's admission policy. The assistant focused on the #running-req: 8 log line as evidence that only 8 requests were being processed. But this may have been a measurement artifact: the log might report the number of requests in the current micro-batch, not the total in-flight requests across all pipeline stages. With async depth 8, there could be up to 16 micro-batches in flight simultaneously (8 from the async depth plus 8 from the pipeline stages), each containing up to 8 requests, for a total of 128 requests. The real bottleneck might not be admission at all.

Assumption 3: CUDA graphs would work seamlessly. The assistant verified that CUDA graphs were captured for batch sizes up to 64, but CUDA graph replay is a fragile optimization. If the graph capture doesn't perfectly match the actual execution pattern—different tensor shapes, different memory addresses, different kernel launch order—the graph replay can silently fall back to eager mode or produce incorrect results. The benchmark doesn't verify that CUDA graphs are actually being used during the benchmark, only that they were captured during startup.

Assumption 4: The fix was the right variable to change. The assistant identified pp_max_micro_batch_size as the culprit based on a single line of code. But the real bottleneck might be elsewhere: NCCL P2P communication overhead on PCIe, the scheduler's request admission policy, the KV cache management, or even the model architecture itself (e.g., the MoE routing causing uneven compute distribution across pipeline stages). The assistant didn't profile NCCL communication time, check GPU kernel-level utilization, or measure the pipeline bubble ratio directly.

The Deeper Lesson: Pipeline Parallelism on PCIe

The failure of this fix reveals a deeper truth about pipeline parallelism on PCIe-connected GPUs. Pipeline parallelism was designed for high-bandwidth interconnects like NVLink, where the communication overhead is negligible. On PCIe, the per-layer communication cost is much higher, and the pipeline bubble—the time when a GPU is idle waiting for the next micro-batch to arrive—dominates the total execution time.

With 8 pipeline stages and async depth 8, the theoretical maximum throughput is limited by the pipeline bubble ratio: even with perfect overlap, the pipeline can never exceed 1/(1 + (pp_size-1)/async_depth) of the single-stage throughput. For pp_size=8 and async_depth=8, this is 1/(1 + 7/8) ≈ 53% of the ideal throughput. But the actual throughput is even lower because the per-stage computation is small (each stage processes only a fraction of the model's layers) and the NCCL communication time is non-trivial on PCIe.

The assistant's micro-batch fix addressed a symptom (low per-micro-batch request count) but not the root cause (pipeline overhead dominating small-batch computation). The real solution might involve reducing the pipeline depth (fewer stages, larger per-stage compute), increasing the async depth further, or switching to a different parallelism strategy entirely—such as expert parallelism (EP), which the assistant later explored and found to be dramatically better.

Input and Output Knowledge

To fully understand this message, the reader needs several pieces of input knowledge:

  1. SGLang's pipeline parallelism architecture: How PP splits the model across GPUs, how micro-batches traverse the pipeline, and how async depth enables overlap between consecutive decode steps.
  2. CUDA graphs: How they capture GPU kernel launches and replay them to reduce CPU launch overhead, and why they require careful batch-size matching.
  3. PCIe vs NVLink bandwidth characteristics: Why PCIe is a bottleneck for tensor parallelism but should theoretically favor pipeline parallelism for MoE models.
  4. The Kimi K2.6 model architecture: A Mixture-of-Experts model where MoE layers require AllReduce in tensor parallelism but not in pipeline parallelism.
  5. The previous benchmark results: TP8 achieving 577.8 tok/s at C=32, and the original PP8 achieving only 248.1 tok/s at C=32. The output knowledge created by this message is:
  6. Empirical evidence that micro-batch size is not the bottleneck: The explicit pp_max_micro_batch_size=64 produced essentially identical throughput to the auto-computed value of 8, disproving the hypothesis.
  7. A refined understanding of pipeline parallelism limits on PCIe: The results suggest that pipeline overhead—not micro-batch capacity—is the dominant constraint on this hardware.
  8. A data point for the broader parallelism strategy exploration: This benchmark helped rule out one potential optimization path, steering the investigation toward expert parallelism (EP) and other strategies that later proved successful.
  9. A cautionary tale about code-level fixes: Even when you find what looks like a smoking gun in the source code, the actual bottleneck may be more fundamental and require a different level of analysis.

Conclusion

Message 11501 captures a pivotal moment in a complex deployment effort: the moment when a well-reasoned hypothesis meets the unforgiving reality of benchmark results. The assistant's investigation was thorough—profiling GPU utilization, examining source code, identifying a suspicious auto-computation, and implementing what seemed like the obvious fix. Yet the fix failed, and the failure was informative.

The message demonstrates that in distributed ML inference, the bottleneck is rarely where you first look. The assistant's code-level analysis of pp_max_micro_batch_size was correct in isolation, but it addressed a secondary constraint while the primary constraint—pipeline overhead on PCIe—remained untouched. The benchmark results forced a reassessment, leading the assistant to explore expert parallelism in subsequent messages, which ultimately achieved dramatically better throughput.

This is the essence of systems optimization: form a hypothesis, test it rigorously, accept the results even when they contradict your reasoning, and use the new knowledge to refine your understanding. The assistant's willingness to run a time-consuming benchmark (the command was terminated after exceeding 1800 seconds, but enough data was collected) and report the results honestly, without rationalizing or cherry-picking, is a model of empirical discipline.

The message also highlights the gap between source code analysis and runtime behavior. SGLang's scheduler code suggested a clear bottleneck, but the actual runtime behavior revealed that the bottleneck was elsewhere. This is a reminder that in complex distributed systems, static analysis can only take you so far—you must measure, measure, and measure again.