The Pivot Point: How a Negative Benchmark Proved the GLM-5-NVFP4 is Compute-Bound

In the long arc of optimizing GLM-5-NVFP4 inference across 8 RTX PRO 6000 Blackwell GPUs, there comes a moment where a carefully crafted hypothesis meets the cold reality of benchmark numbers — and shatters. That moment is message [msg 855], where the assistant deploys a TP4+PP2 (Tensor Parallelism 4 + Pipeline Parallelism 2) configuration, benchmarks it, and discovers it performs roughly half as well as the simpler TP8 configuration. This negative result is not a failure; it is the single most important diagnostic data point in the entire optimization journey, because it fundamentally reframes the problem from a communication bottleneck to a compute bottleneck.

The Hypothesis: Allreduce as the Suspect

To understand why TP4+PP2 was attempted, we need to reconstruct the reasoning that led to it. The assistant and user had been battling with the GLM-5-NVFP4 model — a massive 405B-parameter Mixture-of-Experts model quantized to NVFP4 — on a Proxmox virtualized environment with 8 RTX PRO 6000 Blackwell GPUs. Earlier in the session (<msg id=832–851>), they had established a working TP8 (Tensor Parallelism 8) configuration that achieved reasonable throughput, but the GPUs were only drawing about 235W out of 600W TDP, suggesting something was limiting performance.

The prime suspect was allreduce communication overhead. In TP8, every layer requires an allreduce operation across all 8 GPUs. In a virtualized environment with GPUs spread across multiple PCIe root complexes — each GPU on its own IOMMU group — cross-GPU communication is forced through system memory rather than direct P2P DMA. Earlier investigation ([msg 836]) had shown that P2P bandwidth between GPUs on different NUMA nodes was only ~40 GB/s (SYS topology) compared to ~54 GB/s within the same NUMA node (NODE topology). The hypothesis was that these cross-NUMA allreduces were the primary bottleneck.

The proposed solution was TP4+PP2: split the model's 78 layers across two pipeline stages (PP2), each running on 4 GPUs (TP4) within a single NUMA socket. This would mean all allreduces stayed within the fast NUMA-local interconnect, and the only cross-socket communication would be a single activation transfer between pipeline stages per forward pass — dramatically reducing communication pressure.

The user asked a clarifying question in [msg 835]: "Can we slice the model up such that TP is 4, on each socket, and we infer half of the layers on one socket, half of the layers on the second socket?" The assistant confirmed this was exactly what PP does, and cited prior research from FINDINGS.md showing TP4+PP2 had achieved 5,154 tok/s on a similar model (GLM-4.7-FP8) — a 23% improvement over TP8. This gave the team confidence that the approach was worth pursuing.

The Execution: Deploying TP4+PP2

Deploying TP4+PP2 was not trivial. The assistant encountered several hurdles:

  1. Ambiguous argument parsing: The --pp flag was ambiguous (could match --pp-size, --pp-max-micro-batch-size, etc.), requiring the use of the full --pp-size flag name (<msg id=842–846>).
  2. Script wrapping: The assistant created a shell script (/root/run_tp4pp2.sh) to ensure proper environment variable handling ([msg 847]).
  3. Model loading: The 405B model took significant time to load, with PP0 (GPUs 0–3) using ~91.6 GB each and PP1 (GPUs 4–7) using ~95.5 GB each — the second stage having higher memory usage due to the lm_head/output embedding (<msg id=851–852>). After successful startup, the assistant verified the model was producing coherent output with a single completion request ([msg 852]), then ran a single-stream benchmark ([msg 854]) which yielded 5.6 output tok/s with a TPOT of 179ms.

The Subject Message: The Benchmark That Changed Everything

Message [msg 855] is where the critical data arrives. The assistant runs the 64-concurrency benchmark:

Total input text tokens:                 3918      
Total generated tokens:                  3731      
Total generated tokens (retokenized):    3728      
Request throughput (req/s):              1.78      
Input token throughput (tok/s):          109.22    
Output token throughput (tok/s):         104.00    
Peak output token throughput (tok/s):    228.00    
Peak concurrent requests:                64        
Total token throughput (tok/s):          213.22

The assistant's immediate reaction is telling: "Single stream: 5.6 tok/s output, TPOT 179ms. That's actually worse than TP8's ~8-11 tok/s (TPOT ~122ms). The pipeline bubble overhead hurts single-stream latency as expected. Let me run the higher concurrency benchmarks to see if throughput improves."

The assistant correctly identifies that single-stream performance suffers due to pipeline bubbles — PP2 means one stage is idle while the other computes, which hurts latency. But the hope was that at higher concurrency, with many requests in flight, the pipeline would fill and throughput would surpass TP8.

The 64-concurrency result of 104 output tok/s and 213 total tok/s tells a different story. Compared to TP8's ~200 output tok/s and ~700 total tok/s at the same concurrency, TP4+PP2 is roughly half the throughput. This is not a marginal difference; it is a dramatic underperformance that demands explanation.

The Reasoning: What This Result Implies

The assistant does not yet draw the full conclusion in this message — it simply notes the result and proceeds to test higher concurrency levels (256 and 512 in subsequent messages <msg id=856–857>). But the seed of the critical insight is planted here.

The key logical chain that emerges from this result is:

  1. If the bottleneck were allreduce communication, reducing the allreduce from 8-way to 4-way (and keeping it NUMA-local) should have improved throughput. The fact that throughput halved instead means communication is not the primary bottleneck.
  2. The 2× slowdown correlates with 2× larger GEMMs per GPU. In TP8, each GPU computes 1/8th of each matrix operation. In TP4, each GPU computes 1/4th. The per-GPU compute per layer doubles. If the model were compute-bound, doubling the per-GPU compute would roughly double the time per layer — exactly what we observe.
  3. Pipeline bubbles compound the problem. Even at high concurrency, PP2 introduces idle time as stages must wait for each other. In a compute-bound scenario, where each GPU is already struggling to keep its SMs busy, adding pipeline bubbles only makes things worse. The user immediately grasps this implication in the very next message ([msg 858]): "Doesn't the halving of perf sort of imply we're compute bound -> kernels not tuned to the GPU?" This is the pivotal insight that redirects the entire optimization effort.

Assumptions Made and Corrected

Several assumptions are visible in this message and its surrounding context:

Assumption 1: The model is communication-bound. This was the working hypothesis that motivated the TP4+PP2 experiment. The benchmark disproves it conclusively. The assumption was reasonable given the known PCIe topology limitations in the Proxmox VM, but it turned out to be incorrect.

Assumption 2: TP4+PP2 would improve throughput at high concurrency. This was based on prior results with GLM-4.7-FP8, a different model with different characteristics. The assumption did not account for the unique properties of NVFP4 quantization and SM120 microarchitecture.

Assumption 3: Pipeline bubbles would be amortized at high concurrency. While this is generally true for PP, the degree of amortization depends on the compute-to-communication ratio. In a compute-bound regime, the bubbles remain significant even at high concurrency.

Assumption 4: The benchmark tooling would work seamlessly. The assistant had to debug a tokenizer issue (<msg id=853–854>) where bench_serving couldn't resolve the served model name glm-5, requiring the explicit --tokenizer flag. This is a minor but real friction point.

Input Knowledge Required

To fully understand this message, one needs:

  1. The TP4+PP2 topology: Knowledge that the model is split into two pipeline stages of 4 GPUs each, with each stage handling half the layers and performing 4-way allreduce within its NUMA node.
  2. The TP8 baseline: Understanding that TP8 achieved ~200 output tok/s at 64 concurrency, ~622 at 256, and ~1,867 total tok/s at 256 — the numbers the assistant implicitly compares against.
  3. The hardware context: 8x RTX PRO 6000 Blackwell GPUs in a Proxmox VM, with known PCIe P2P limitations (GPUs on separate root complexes preventing direct P2P DMA).
  4. The model characteristics: GLM-5-NVFP4 is a 405B MoE model quantized to NVFP4, requiring specialized FP4 GEMM kernels that may not be optimized for the SM120 microarchitecture.
  5. The optimization history: Prior attempts including FlashInfer CUTLASS MoE autotune, various server parameter tunings, and the earlier TP8 benchmarks that showed low GPU power utilization (~235W out of 600W).

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. TP4+PP2 throughput numbers at 64 concurrency: 104 output tok/s, 213 total tok/s — a definitive data point that disproves the communication-bound hypothesis.
  2. Confirmation of pipeline bubble overhead: Single-stream TPOT of 179ms vs TP8's ~122ms, quantifying the latency penalty of PP2.
  3. A reframed optimization problem: The negative result implicitly proves the model is compute-bound, redirecting focus to kernel efficiency, GEMM tuning, and FP4 microarchitecture optimization.
  4. A methodological lesson: The value of testing hypotheses with controlled experiments, even when the result is negative. The TP4+PP2 benchmark took significant effort to set up (argument parsing fixes, script creation, model loading), but the knowledge gained was invaluable.

The Thinking Process

The assistant's reasoning in this message is concise but reveals a clear thought process:

  1. Compare to baseline: Immediately notes that 5.6 tok/s is worse than TP8's 8-11 tok/s, showing the assistant has the baseline numbers at hand and is actively comparing.
  2. Attribute the single-stream degradation: Correctly identifies pipeline bubble overhead as the cause of single-stream latency increase — "as expected" indicates this was anticipated.
  3. Reserve judgment for high concurrency: "Let me run the higher concurrency benchmarks to see if throughput improves" — the assistant is methodically testing the hypothesis that PP benefits appear at scale.
  4. Execute the benchmark cleanly: Uses the corrected --tokenizer flag from the previous message, showing attention to detail and learning from the earlier error. The assistant does not over-interpret the 64-concurrency result in this message. It simply reports the numbers and moves to test higher concurrency. This restraint is wise — a single data point at one concurrency level is not enough to draw firm conclusions. The pattern only becomes clear when the 256 and 512 concurrency results (in subsequent messages) show the same ~2× degradation relative to TP8.

The Broader Significance

This message is a classic example of a falsification event in the scientific method applied to systems optimization. The team had a hypothesis (communication is the bottleneck → reducing communication will improve throughput), designed an experiment (TP4+PP2), executed it carefully, and obtained a result that clearly falsified the hypothesis. The result was negative — performance got worse, not better — but the information gained was positive: the model is compute-bound.

This reframing unlocks an entirely new optimization strategy. Instead of trying to reduce communication (which was already minimal in TP8), the team now focuses on:

Conclusion

Message [msg 855] is the turning point of the entire optimization journey. A carefully designed experiment produced a clear negative result, and the team had the intellectual honesty to accept what the data was telling them. The TP4+PP2 benchmark did not just fail to improve throughput — it revealed the fundamental nature of the bottleneck, redirecting weeks of effort from communication optimization to compute optimization. In the art of systems performance engineering, there is no more valuable skill than knowing when your hypothesis is wrong, and this message captures that moment perfectly.