The TP4+PP2 Surprise: When Pipeline Parallelism Undermines Inference Throughput on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference optimization, every configuration change is a gamble. Engineers form hypotheses based on hardware topology, prior benchmarks, and theoretical models of computation versus communication costs. Sometimes those hypotheses pay off spectacularly; other times, they reveal unexpected bottlenecks. Message [msg 856] captures one such pivotal moment: the assistant has just benchmarked a TP4+PP2 (Tensor Parallelism 4 + Pipeline Parallelism 2) configuration for the GLM-5-NVFP4 model running on 8x RTX PRO 6000 Blackwell GPUs, and the results are unequivocally worse than the TP8 baseline. This short message—barely a few lines of analysis followed by a command to launch a higher-concurrency test—encapsulates a critical juncture in a much longer optimization journey, where a well-reasoned hypothesis collides with empirical reality.

The Message

64 concurrency: 104 output tok/s, 213 total tok/s. Compared to TP8 at 64: 200 output tok/s, 700 total tok/s. This is significantly worse — about half the throughput. Let me try 256 and 512 to see if it recovers at higher concurrency:

>

``bash ssh root@10.1.230.174 'source /root/ml-env/bin/activate && python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 8000 \ --model glm-5 \ --tokenizer lukealonso/GLM-5-NVFP4 \ --dataset-name random \ --random-input-len 128 --random-output-len 128 \ --num-prompts 256 --request-rate 999 \ 2>&1 | tail -30' ``

Context: The TP4+PP2 Hypothesis

To understand why this message matters, one must appreciate the hardware topology that motivated the experiment. The server has 8x RTX PRO 6000 Blackwell GPUs distributed across two NUMA (Non-Uniform Memory Access) nodes, with GPUs 0–3 on one socket and GPUs 4–7 on the other. Critically, there is no NVLink connecting these GPUs—they communicate over PCIe, which introduces significant latency and bandwidth penalties for cross-GPU communication. Earlier in the session ([msg 836]), the assistant had explained that with TP8, every single MoE (Mixture-of-Experts) layer requires an 8-way allreduce, and roughly half of those allreduce operations must traverse the slow NUMA boundary, incurring ~40 GB/s bandwidth instead of the ~54 GB/s available within a single NUMA node.

The TP4+PP2 configuration was designed to address this bottleneck. By splitting the model's 78 layers across two pipeline stages—PP0 running layers 0–38 on GPUs 0–3 and PP1 running layers 39–77 on GPUs 4–7—each stage's allreduce becomes a 4-way operation confined to a single NUMA node. The only cross-socket communication is the activation transfer between pipeline stages, which happens once per layer boundary rather than once per layer. Prior research documented in FINDINGS.md had shown that TP4+PP2 achieved 5,154 tok/s on the GLM-4.7-FP8 model, a 23% improvement over TP8's 4,180 tok/s at 384 concurrency. This prior success made the configuration a natural candidate for the newer GLM-5-NVFP4 model.

The Results: A Clear Regression

The numbers speak for themselves. At 64 concurrent requests with 128 input and 128 output tokens:

| Metric | TP8 (baseline) | TP4+PP2 | Ratio | |--------|---------------|---------|-------| | Output token throughput | 200 tok/s | 104 tok/s | 0.52× | | Total token throughput | 700 tok/s | 213 tok/s | 0.30× |

The output throughput is roughly halved, but the total throughput (input + output) is less than a third of the TP8 baseline. This discrepancy hints at a deeper issue: the prefill phase is disproportionately slower under TP4+PP2. With 128 input tokens, prefill should be compute-bound, and the pipeline bubbles introduced by PP—where PP1 sits idle while PP0 processes the prefill, and vice versa during decode—would effectively halve the utilization of the GPU fleet during both phases. The total throughput ratio of 0.30× suggests that the pipeline bubble overhead is even worse than the theoretical 2× penalty, possibly because the inter-stage activation transfers add latency that compounds with the existing PCIe bottlenecks.

The Decision to Escalate Concurrency

Despite the discouraging results at 64 concurrency, the assistant does not abandon the configuration. Instead, the message reads: "Let me try 256 and 512 to see if it recovers at higher concurrency." This decision reveals a nuanced understanding of how pipeline parallelism behaves under load.

Pipeline parallelism introduces a fundamental inefficiency known as the "pipeline bubble." In a two-stage pipeline, stage 1 must wait for stage 0 to finish its prefill before it can begin its own computation, and during decode, the stages alternate between active computation and idle waiting. At low concurrency, this bubble dominates: with a single request, the pipeline is essentially half-empty at any given moment. At high concurrency, however, multiple requests can be in flight simultaneously, allowing the pipeline stages to process different requests in parallel—stage 0 can prefill request B while stage 1 decodes request A. This effect, known as "micro-batching" or "inter-request pipelining," can amortize the bubble overhead.

The assistant's reasoning, though not explicitly stated in this message, is grounded in this principle. If the 64-concurrency result is bubble-dominated, then increasing concurrency should allow the pipeline to fill, potentially recovering throughput. The prior GLM-4.7-FP8 data supported this: at 384 concurrency, TP4+PP2 outperformed TP8. The assistant is testing whether the same crossover point exists for GLM-5-NVFP4.

Assumptions Under Test

This message operates on several key assumptions, some of which may prove incorrect:

Assumption 1: Pipeline bubbles are the dominant factor at 64 concurrency. This is plausible but unverified. The 0.30× total throughput ratio is far worse than the 0.5× that pure pipeline bubble theory would predict, suggesting other factors—such as memory bandwidth contention, KV cache pressure, or the inter-stage activation transfer overhead—may be amplifying the penalty.

Assumption 2: The crossover point exists within the tested range (256–512 concurrency). The GLM-4.7-FP8 crossover occurred at ~384 concurrency, but GLM-5-NVFP4 is a different model with different computational characteristics. The FP4 quantization, the NSA (Nested Sparse Attention) decode backend, and the larger model size may shift the crossover point to much higher concurrency—or eliminate it entirely.

Assumption 3: The allreduce savings from 4-way within-NUMA communication compensate for the pipeline overhead. The 4-way allreduce is indeed faster than 8-way, but the assistant may have underestimated the cost of the inter-stage activation transfers. With PCIe-based P2P, each activation transfer between PP0 and PP1 must traverse the slow cross-NUMA path, and with 78 layers of activations, the cumulative latency may negate the allreduce savings.

Assumption 4: The model fits within the available memory with PP. The GPU memory distribution shows PP1 using ~95.5 GB vs PP0's ~91.6 GB, leaving only ~2.3 GB free on PP1. This tight memory headroom could force the memory allocator into inefficient patterns, especially at higher concurrency where KV cache consumption grows.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several concepts:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A quantitative benchmark: TP4+PP2 achieves 104 output tok/s and 213 total tok/s at 64 concurrency for GLM-5-NVFP4 on this hardware.
  2. A comparative result: TP4+PP2 is approximately 2× slower than TP8 at this concurrency level, contradicting the hypothesis that reducing allreduce width would improve throughput.
  3. A scaling hypothesis: The assistant posits that higher concurrency may recover performance, establishing a testable prediction.
  4. A methodological precedent: The assistant demonstrates disciplined benchmarking by immediately comparing against the TP8 baseline, enabling clear interpretation of the results.

The Thinking Process

The message's structure reveals the assistant's analytical process in real time. First comes observation: the raw numbers from the benchmark. Then comes comparison: framing those numbers against the known TP8 baseline. Then comes diagnosis: "This is significantly worse — about half the throughput." Finally comes the experimental response: testing higher concurrency to probe the hypothesis that pipeline bubbles are the culprit.

The parenthetical "about half" is particularly telling. The assistant could have said "about one-third" for total throughput, but instead focuses on output throughput (104 vs 200), which is the more relevant metric for decode-heavy workloads. This choice reflects an understanding that output token throughput is the primary optimization target for interactive inference.

The decision to test both 256 and 512 concurrency (mentioned as "256 and 512") shows a systematic approach. Rather than jumping to the highest concurrency or giving up after one data point, the assistant plans to characterize the scaling curve across multiple points, building a complete picture of how TP4+PP2 behaves under load.

Potential Mistakes and Blind Spots

While the assistant's methodology is sound, several blind spots are worth noting:

The memory pressure issue is underappreciated. With only ~2.3 GB free on PP1, higher concurrency will rapidly consume the remaining KV cache capacity. At 256 concurrency with 128 output tokens each, the KV cache alone could consume 10+ GB, potentially causing out-of-memory errors or triggering memory swapping that destroys throughput.

The inter-stage transfer cost may be nonlinear. The assistant assumes that activation transfers between PP stages are "small" compared to allreduce, but with 78 layers of hidden states at 4× the per-GPU size (since TP4 means each GPU holds 1/4 of the hidden dimension), these transfers could be substantial. Each transfer must traverse the PCIe P2P path, which earlier diagnostics showed to be virtualization-limited.

The model difference from GLM-4.7-FP8 is significant. The prior success with TP4+PP2 was on a different model with different quantization (FP8 vs NVFP4) and different architecture. The NVFP4 quantization may have different compute-to-communication ratios that shift the optimal configuration.

Conclusion

Message [msg 856] captures a moment of intellectual honesty in the optimization process. The assistant had a well-reasoned hypothesis, tested it rigorously, and found it contradicted by evidence. Rather than discarding the configuration entirely, the assistant probes further, testing whether the failure is fundamental or merely a function of insufficient concurrency. This is the essence of empirical systems optimization: forming hypotheses, testing them, and iterating based on data. Whether the higher-concurrency tests vindicate TP4+PP2 or confirm its inferiority, the process itself—meticulous, comparative, and hypothesis-driven—is the real story.