The Concurrency Ceiling: Benchmarking Pipeline Parallelism Scaling for Kimi K2.6 on PCIe Blackwell

Introduction

In the high-stakes world of large language model deployment, finding the optimal parallelism strategy is the difference between a service that delivers acceptable throughput and one that wastes expensive GPU hardware. This message, drawn from an extensive benchmarking campaign for the Kimi K2.6 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe, captures a pivotal moment in that search. The assistant has just fixed a critical configuration bug in SGLang's pipeline parallelism (PP8) implementation—the pp_max_micro_batch_size was being auto-computed to a paltry 8, limiting each micro-batch to only 8 requests despite the system having capacity for far more. With the fix applied (--pp-max-micro-batch-size 64), initial benchmarks showed PP8 scaling respectably from 83 tok/s at C=8 to 316 tok/s at C=48. The user, seeing this upward trajectory, issued a simple but demanding directive: "Try higher concurrenties until it breaks."

The assistant's response—message 11503 in the conversation—is a seemingly straightforward benchmark script. But beneath its surface lies a rich tapestry of engineering decisions, hidden assumptions, and revealing data that illuminates the fundamental nature of pipeline parallelism on bandwidth-constrained hardware. This article unpacks that single message in depth, examining why it was written, how decisions were made, what assumptions proved problematic, and what the partial results reveal about the architecture being tested.

The Message in Full

The message contains the assistant's reasoning followed by a Python benchmark script executed via bash:

Agent Reasoning: The user wants me to push concurrency higher on the PP8 setup to find where it maxes out. The current results show it's still scaling at C=48 (316 tok/s). Let me push to C=64, 96, 128, etc. Also let me check GPU util during the run.

The script then defines an API client function, warms up the model, and runs a concurrency sweep across levels [32, 64, 96, 128, 192, 256] with max_tokens=2048. The output shows only two data points before the user aborted the command:

C= 32: agg=  187.7 tok/s  wall= 324.9s  avg=1905tok/req  ok=32 fail=0
C= 64: agg=  396.0 tok/s  wall= 313.2s  avg=1938tok/req  ok=64 fail=0

At first glance, this looks like a routine benchmark run that was cut short. But each element—the choice of concurrency levels, the reduced token limit, the discrepancy with previous results, and the early termination—tells a deeper story.

Why This Message Was Written: The Search for the Saturation Point

The immediate trigger is the user's request to push concurrency until the system breaks. But the deeper motivation is to characterize the throughput curve of PP8 and find its asymptotic limit. Understanding where a parallelism strategy saturates is critical for capacity planning: if PP8 peaks at, say, 500 tok/s, then serving a target workload of 1000 tok/s would require two instances or a different strategy. If it scales linearly to 2000 tok/s, then PP8 is the clear winner.

The assistant's reasoning explicitly states this goal: "find where it maxes out." The previous data point at C=48 (316 tok/s) showed no sign of saturation—throughput was still climbing. The assistant needed to push further to find the knee in the curve. This is classic benchmarking methodology: sweep a parameter until the metric stops improving, then you've found the bottleneck.

The choice of concurrency levels—32, 64, 96, 128, 192, 256—reflects an expectation that saturation would occur somewhere in this range. The doubling from 32 to 64, then more granular steps (96, 128), then aggressive jumps (192, 256) suggests the assistant expected the curve to start flattening somewhere around 96–128 and wanted finer resolution there, while also probing extreme values to confirm the ceiling.## The Critical Design Decision: Reducing max_tokens

One of the most telling details in this message is the reduction of max_tokens from 4096 (used in the previous benchmark at message 11501) to 2048. The assistant's reasoning doesn't explicitly mention this change, but the script comments reveal the motivation: "Shorter max_tokens=2048 to get results faster, push concurrency hard."

This is a classic tradeoff in LLM benchmarking. Longer generations produce more tokens per request, which increases the statistical significance of throughput measurements and better reflects real-world usage patterns. But they also take longer to complete—the previous PP8 benchmark at C=48 took 443 seconds (over 7 minutes). At C=256 with 4096 max_tokens, a single run could take an hour or more. By halving the generation length, the assistant hoped to get results in minutes rather than hours, enabling rapid iteration.

However, this decision carries hidden consequences. Shorter generations change the ratio of prefill to decode time. With 2048 tokens, the prefill phase (processing the input prompt) represents a larger fraction of total time than with 4096 tokens. Since prefill is compute-bound while decode is memory-bandwidth-bound, reducing max_tokens shifts the workload mix. The throughput numbers from this run are not directly comparable to the 4096-token runs from earlier messages. The assistant implicitly acknowledges this by noting the discrepancy: the C=32 result here shows 187.7 tok/s, while the previous run (with 4096 max_tokens) showed 250.9 tok/s at C=32. The 25% difference is attributable to the changed workload profile.

This is a subtle but important methodological point. The assistant is optimizing for benchmark iteration speed at the cost of comparability with prior results. It's a reasonable tradeoff for a quick sweep, but it means the absolute throughput numbers from this run should be interpreted cautiously.

The Missing GPU Utilization Check

The assistant's reasoning mentions wanting to "check GPU util during the run," but the script as written does not include any GPU monitoring. The previous PP8 investigation (message 11494) had used nvidia-smi queries via SSH to capture GPU utilization and power draw during load. That data was instrumental in diagnosing the pipeline bubble problem—GPU utilization fluctuated wildly between 40–100%, confirming that the pipeline wasn't keeping GPUs fed.

In this message, the assistant plans to check GPU utilization but never executes it. The script is purely a throughput benchmark with no monitoring component. This is a missed opportunity: without GPU utilization data, the assistant can't tell whether the throughput ceiling (when found) is due to compute saturation, memory bandwidth limits, or pipeline scheduling inefficiency. The user's abort prevented this from becoming an issue, but it reveals a pattern where the assistant's reasoning sometimes plans more than the implementation delivers.

Assumptions Embedded in the Benchmark Design

The benchmark makes several implicit assumptions worth examining:

1. Linear scaling assumption: The assistant assumes that throughput will increase monotonically with concurrency until some saturation point. This is generally true for LLM serving, but PP8 with its pipeline structure can exhibit non-monotonic behavior—adding more requests can actually decrease throughput if the pipeline stalls due to uneven micro-batch distribution or if the scheduler's admission policy creates pathological patterns. The previous investigation (messages 11494–11499) had already revealed that SGLang's PP scheduler was limiting running requests to 8 despite 32 concurrent API calls. The fix to pp_max_micro_batch_size may not have fully resolved this issue; the scheduler could still be creating suboptimal micro-batch distributions at higher concurrency.

2. Homogeneous request assumption: All requests use the same three prompts cycled. This is standard practice for reproducible benchmarks, but it means the benchmark doesn't capture the effects of variable prompt lengths, which can cause pipeline bubbles when one micro-batch has a much longer prefill than others. Real-world workloads have heterogeneous prompt lengths, and PP8's sensitivity to this variance is a key performance characteristic that this benchmark ignores.

3. Steady-state assumption: The script submits all requests simultaneously and waits for all to complete. This measures throughput under a burst load, not a steady stream. In production, a serving system typically handles a continuous arrival process. The burst model can overestimate throughput if the system has head-of-line blocking or underestimate it if the system benefits from sustained pipeline filling. The assistant doesn't discuss this distinction.

4. Network and client overhead assumption: The benchmark runs from a remote client (CT200 is at 10.1.2.200), measuring wall-clock time including network latency. At 300+ seconds per run, network overhead is negligible, but the concurrent connection handling of the Python urllib library could introduce client-side bottlenecks. The ThreadPoolExecutor with 256 threads creates 256 simultaneous HTTP connections, which may overwhelm the client OS's socket buffers or the server's connection handling before the GPU is actually saturated. This is a classic benchmarking pitfall: the client becomes the bottleneck before the server does.

The Partial Results and What They Reveal

The script produced only two data points before the user aborted:

| Concurrency | Aggregate Throughput | Wall Time | Avg Tokens/Req | |-------------|---------------------|-----------|----------------| | 32 | 187.7 tok/s | 324.9s | 1905 | | 64 | 396.0 tok/s | 313.2s | 1938 |

The C=32 result (187.7 tok/s) is notably lower than the previous PP8 run at C=32 with 4096 max_tokens (250.9 tok/s). As discussed, this is due to the shorter generation length changing the prefill/decode ratio. But the C=64 result (396.0 tok/s) shows near-perfect linear scaling: doubling concurrency from 32 to 64 roughly doubled throughput (187.7 → 396.0, a 2.11× increase). This is actually better than linear scaling, suggesting that the system was underutilized at C=32 and the additional requests helped fill pipeline bubbles.

This near-perfect scaling is remarkable for pipeline parallelism on PCIe-connected GPUs. The earlier investigation had shown that PP8 with the default micro-batch size was severely bottlenecked—only 8 running requests despite 32 concurrent calls. The fix to pp_max_micro_batch_size=64 appears to have unlocked the pipeline's true potential. The fact that throughput scales linearly from C=32 to C=64 suggests the pipeline is now filling properly, with micro-batches flowing through all 8 stages concurrently rather than in lockstep.

However, the user's abort means we never got to see the saturation point. The critical question—does PP8 plateau at C=96, C=128, or keep scaling to C=256?—remains unanswered. The abort may reflect impatience with the long run times (each data point took ~5 minutes), or the user may have seen enough to make a decision. In the broader context of the session, the user had already expressed skepticism about PP8's performance (message 11492: "Much worse than expected"), and the partial results may have confirmed their suspicion that PP8, even with the fix, would not match the throughput of expert parallelism (EP8) which had achieved ~1531 tok/s at high concurrency in earlier benchmarks.

The Thinking Process: A Window into Engineering Judgment

The assistant's reasoning section is brief but revealing. It shows the assistant processing the user's request and formulating a response plan:

  1. Interpretation: "The user wants me to push concurrency higher on the PP8 setup to find where it maxes out." This correctly interprets the user's intent, recognizing that "until it breaks" means "until throughput stops scaling," not "until the server crashes."
  2. Context integration: "The current results show it's still scaling at C=48 (316 tok/s)." The assistant references the previous benchmark result, showing awareness of the current state and the need to extend beyond it.
  3. Planning: "Let me push to C=64, 96, 128, etc." The assistant chooses a sweep range that extends well beyond the current data, with denser sampling in the expected saturation region.
  4. Secondary goal: "Also let me check GPU util during the run." This shows the assistant recognizes that throughput numbers alone are insufficient—understanding why throughput saturates requires GPU utilization data. However, as noted, this monitoring was not implemented in the actual script. The reasoning is concise and task-focused, characteristic of an experienced engineer who knows what data they need and how to get it. There's no deliberation about alternative approaches (e.g., using a different benchmark tool, measuring latency distributions, or profiling with nsys). The assistant defaults to the same Python-based benchmark harness used in previous messages, prioritizing consistency and speed over methodological innovation.

Input and Output Knowledge

Input knowledge required to understand this message includes:

Conclusion

Message 11503 captures a pivotal moment in a complex benchmarking campaign—the moment when a previously broken parallelism strategy starts showing its true potential. The fix to pp_max_micro_batch_size transformed PP8 from a severely bottlenecked configuration (only 8 running requests) to one that scales near-linearly with concurrency. The assistant's response is methodologically sound: it extends the concurrency sweep, accounts for run-time constraints by reducing generation length, and plans (if not executes) GPU utilization monitoring.

The partial results—cut short by the user's abort—leave the story unfinished. We never learn where PP8 saturates on PCIe Blackwell, or whether it can match the peak throughput of expert parallelism. But the message itself is a rich artifact of engineering practice: the careful balancing of benchmark speed vs. accuracy, the implicit assumptions embedded in test design, the gap between planned and implemented monitoring, and the iterative dance between user intuition and empirical measurement. It's a reminder that in the world of large-scale ML deployment, even a "simple" benchmark script carries the weight of dozens of prior investigations, hardware constraints, and architectural decisions—all of which must be understood to interpret the numbers correctly.