The TP4+PP2 Benchmark: A Pivotal Moment in GLM-5-NVFP4 Inference Optimization

Introduction

In the high-stakes world of large language model inference optimization, every decision carries weight. The choice of parallelism strategy—how to distribute a model across multiple GPUs—can make the difference between a system that barely keeps pace with demand and one that delivers breakthrough throughput. Message 852 in this opencode coding session captures a critical inflection point: the moment when a team deploys a new TP4+PP2 (Tensor Parallelism 4 + Pipeline Parallelism 2) configuration for the GLM-5-NVFP4 model on 8x RTX PRO 6000 Blackwell GPUs, and takes its first measurements. This message is a window into the reasoning, assumptions, and decision-making process of a team deep in the trenches of performance optimization.

The Full Message

Let us examine the message in its entirety:

Interesting — PP0 (GPUs 0-3) uses ~91.6 GB each while PP1 (GPUs 4-7) uses ~95.5 GB each. The second stage has more memory used — likely because it has the lm_head/output embedding in addition to its half of the layers. Memory is tighter on PP1 (~2.3 GB free vs ~6.2 GB on PP0).

>

Now let me benchmark. I'll start with single-stream to see if PP helps latency, then work up to higher concurrency:

>

[bash] ssh root@10.1.230.174 'curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d "{\"model\": \"glm-5\", \"prompt\": \"Explain quantum computing in simple terms:\", \"max_tokens\": 50, \"temperature\": 0.7}" | python3 -m json.tool 2>/dev/null | head -30'

>

{ "id": "4485d43becc1401cb16d9d2e1c47b02e", "object": "text_completion", "created": 1771510652, "model": "glm-5", "choices": [ { "index": 0, "text": " A comprehensive guide for developers and SMBs\nDiscover how to explain quantum computing in simple terms with this comprehensive guide for developers and SMBs. Learn the basics, key concepts like qubits and superposition, real-world applications, and how BytePlus Model", "logprobs": n... } ] }

At first glance, this appears to be a routine status check—the assistant confirms the server is running and issues a test request. But beneath this surface lies a dense web of technical reasoning, strategic decision-making, and implicit assumptions that reveal the true nature of the optimization challenge.

Why This Message Was Written: The Strategic Context

To understand why this message exists, we must trace the chain of events that led to it. The session had been engaged in a multi-day effort to maximize inference throughput for the GLM-5-NVFP4 model, a 405-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 precision. Earlier benchmarks with TP8 (Tensor Parallelism across all 8 GPUs) had achieved approximately 4,150 total tok/s at high concurrency, but the team suspected that cross-NUMA allreduce communication was a bottleneck.

The critical insight came from the hardware topology. The 8 RTX PRO 6000 Blackwell GPUs are split across two NUMA nodes (two CPU sockets), each with 4 GPUs. With TP8, every allreduce operation requires communication across all 8 GPUs, with roughly half of that traffic crossing the slow inter-socket link. The user's question in message 835—"Can we slice the model up such that TP is 4, on each socket, and we infer half of the layers on one socket, half of the layers on the second socket?"—sparked the exploration of Pipeline Parallelism.

The assistant's response in message 836 confirmed that this is exactly what PP does: PP stage 0 runs layers 0-38 on GPUs 0-3 (NUMA node 0) with TP4, and PP stage 1 runs layers 39-77 on GPUs 4-7 (NUMA node 1) with TP4. This configuration promised to shrink allreduce from 8-way to 4-way (all within a single NUMA node with ~54 GB/s P2P bandwidth), while only requiring cross-socket communication for inter-stage activation transfers—a single transfer per layer boundary instead of per layer.

After resolving a command-line argument ambiguity (the --pp flag being ambiguous, requiring --pp-size instead) and launching the server, message 852 represents the first moment of truth: is the server running, and what does the memory distribution tell us?

The Memory Distribution Analysis: A Detective's Reading

The assistant opens with an observation about GPU memory usage that reveals deep system understanding. PP0 (GPUs 0-3) shows approximately 91.6 GB used per GPU, while PP1 (GPUs 4-7) shows approximately 95.5 GB used per GPU—a difference of roughly 4 GB per GPU. The assistant immediately offers a hypothesis: "likely because it has the lm_head/output embedding in addition to its half of the layers."

This is a non-trivial inference. The lm_head (language model head) is the final linear projection that maps hidden states to vocabulary logits. In a standard transformer with pipeline parallelism, the first PP stage handles embedding and the initial layers, while the last PP stage handles the final layers plus the output projection. The assistant correctly identifies that PP1 (stage 1, the second half) bears this additional memory burden.

The practical consequence is immediately noted: "Memory is tighter on PP1 (~2.3 GB free vs ~6.2 GB on PP0)." This 4 GB memory imbalance could become a constraint for batch size scaling—if PP1 runs out of memory first, it limits the entire pipeline. The assistant flags this implicitly, storing it as a data point for future tuning decisions.

This analysis also serves as a sanity check. The memory distribution confirms that the model is indeed split across PP stages as expected. If the memory were evenly distributed, it would suggest the PP wasn't actually splitting the model. The observed imbalance is evidence that the parallelism is working correctly.

The Benchmarking Strategy: Methodical Decision-Making

Having confirmed the server is operational and understood the memory landscape, the assistant articulates a clear benchmarking plan: "I'll start with single-stream to see if PP helps latency, then work up to higher concurrency."

This decision reflects a methodical, layered approach to performance evaluation. Starting with single-stream (a single request with no concurrent load) isolates the fundamental latency characteristics of the TP4+PP2 configuration. If PP introduces pipeline bubbles—where PP1 sits idle waiting for PP0 to finish its computation, and vice versa—single-stream latency will reveal this directly. A high single-stream latency would be a warning sign that pipeline bubbles dominate, even if throughput at high concurrency might still benefit from reduced allreduce overhead.

The progression "then work up to higher concurrency" shows an understanding that parallelism strategies have different sweet spots. PP typically performs poorly at low concurrency (pipeline bubbles dominate) but can excel at high concurrency (where many requests fill the pipeline stages). By testing across the concurrency spectrum, the assistant aims to find the crossover point where PP4+PP2 might surpass TP8.

The actual test request is a simple completion with 50 max tokens, using a prompt about quantum computing. This is a reasonable choice—short enough to get quick results, but long enough to exercise both prefill and decode phases. The temperature of 0.7 adds slight randomness to the output, which is fine for a functional test.

The Result: A Working Server, But What Does It Mean?

The server responds with a completion, confirming that the TP4+PP2 configuration is functional. The output text is a coherent (if generic) explanation about quantum computing, indicating that the model weights are loaded correctly and the pipeline is passing activations between stages without corruption.

However, the message does not include timing information. The curl command does not measure latency, and the response is truncated at 30 lines. This is a functional test, not a performance benchmark—the assistant is verifying that the server works before investing time in formal benchmarking. The actual performance data would come in subsequent messages.

This restraint is noteworthy. The assistant could have immediately launched a full benchmark suite, but instead chose a lightweight test. This reflects an understanding that premature optimization measurement on a potentially broken configuration wastes time. Verify function first, then measure performance.

Assumptions Embedded in This Message

Every technical decision rests on assumptions, and message 852 contains several that deserve scrutiny.

Assumption 1: PP will improve throughput. The entire TP4+PP2 experiment is predicated on the belief that reducing allreduce from 8-way to 4-way will outweigh the cost of pipeline bubbles. This assumption is reasonable given the known NUMA topology, but it is not guaranteed. The chunk summary for segment 7 reveals that TP4+PP2 ultimately proved to be 2× slower than TP8, meaning this assumption was incorrect for this specific model and hardware combination. The pipeline bubbles and the overhead of inter-stage communication outweighed the allreduce savings.

Assumption 2: The memory imbalance is solely due to lm_head. While plausible, this is an inference that could be verified by examining the model architecture. The lm_head is typically a single linear layer mapping from hidden dimension to vocabulary size (~151,936 for GLM-5). At FP4 precision, this would be a relatively small memory footprint. The 4 GB per-GPU difference might also include other stage-specific allocations like KV cache partitioning or intermediate buffers.

Assumption 3: Single-stream latency is a useful diagnostic for PP. This is sound reasoning—pipeline bubbles manifest most clearly at low concurrency. However, it assumes that the pipeline scheduling is naive (one micro-batch at a time) rather than using interleaving or asynchronous scheduling, which could mask bubbles even at low concurrency.

Assumption 4: The server is stable. The assistant observes the server responding to a single request and assumes it is ready for benchmarking. In practice, large model servers can exhibit transient failures, memory fragmentation, or CUDA errors that only manifest under sustained load. The assistant implicitly trusts that the server's initial responsiveness indicates overall health.

Input Knowledge Required

To fully understand this message, a reader needs substantial background knowledge:

  1. Tensor Parallelism (TP): The model's parameters are sharded across GPUs, with each GPU holding a slice of each layer. Allreduce operations synchronize activations after each layer.
  2. Pipeline Parallelism (PP): The model's layers are split into stages, with each stage assigned to a group of GPUs. Activations flow sequentially from stage 0 to stage 1.
  3. NUMA topology: The 8 GPUs are split across two CPU sockets, with faster intra-socket P2P bandwidth (~54 GB/s) and slower inter-socket bandwidth (~40 GB/s).
  4. GLM-5-NVFP4 architecture: A 405B MoE model with 77 layers, quantized to NVFP4 precision using ModelOpt. The model uses NSA (Native Sparse Attention) and a Mixture-of-Experts feedforward network.
  5. lm_head: The final linear projection in a language model that maps hidden states to vocabulary logits. In PP, this lives on the last stage.
  6. Previous benchmark results: TP8 achieved ~4,150 total tok/s at high concurrency, establishing the baseline that TP4+PP2 must beat.
  7. The LXC container environment: The server runs in an LXC container on a Proxmox host, which was set up to bypass VFIO/IOMMU P2P limitations discovered in earlier segments.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Memory distribution confirmation: PP0 uses ~91.6 GB/GPU, PP1 uses ~95.5 GB/GPU. This provides a memory budget for tuning batch sizes and KV cache allocation.
  2. Functional verification: The TP4+PP2 server starts, loads the model, and responds to requests correctly. No CUDA errors, no NaN outputs, no crashes.
  3. Memory headroom quantification: PP0 has ~6.2 GB free, PP1 has ~2.3 GB free. This sets an upper bound on how much additional memory can be allocated for larger batch sizes or KV cache.
  4. lm_head placement confirmed: The memory imbalance validates the expectation that the output projection lives on PP1.
  5. Baseline single-stream behavior: While not a formal benchmark, the successful completion provides a sanity check that the pipeline is functioning end-to-end.

The Thinking Process: A Window into Optimization Methodology

The reasoning visible in this message reveals a disciplined optimization methodology. The assistant does not jump to conclusions or prematurely declare victory. Instead, it:

  1. Observes: Notes the memory distribution and draws an inference about its cause.
  2. Plans: Articulates a clear benchmarking strategy (single-stream first, then scale concurrency).
  3. Executes: Runs a lightweight functional test to verify the server is working.
  4. Defers judgment: Does not draw conclusions about PP's effectiveness from a single test. This pattern—observe, plan, execute, defer—is characteristic of effective performance engineering. The assistant avoids the common trap of making decisions based on insufficient data. It recognizes that a single functional test tells us nothing about throughput, and that only a systematic benchmark across concurrency levels can reveal the true performance characteristics of the TP4+PP2 configuration. The message also shows a healthy respect for the complexity of the system. The assistant does not assume that the server will perform well just because it starts successfully. It knows that real-world performance depends on a multitude of factors—pipeline scheduling, allreduce efficiency, memory bandwidth, kernel launch overhead—that can only be measured, not predicted.

The Broader Narrative: A Chapter in an Optimization Journey

Message 852 sits within a larger narrative arc spanning multiple segments. The team had already:

Conclusion

Message 852 is a masterclass in disciplined performance engineering. It demonstrates how a skilled practitioner approaches a complex optimization problem: by verifying the system works, understanding its resource utilization, planning a systematic evaluation, and deferring judgment until sufficient data is collected. The message's value lies not in any dramatic revelation—the server works, the memory is imbalanced, a single request succeeds—but in the methodological approach it embodies.

The assistant's observation about memory distribution, its inference about lm_head placement, and its decision to start with single-stream benchmarking all reflect deep system knowledge and strategic thinking. Even the assumptions that later proved incorrect (that PP would outperform TP8) were reasonable given the available information at the time. The message captures a moment of informed optimism before the data arrived to correct course.

In the end, TP4+PP2 did not deliver the expected gains. But the disciplined approach visible in message 852 ensured that this negative result was discovered quickly and reliably, allowing the team to pivot to more promising optimization strategies. That is the mark of effective engineering: not always being right, but always learning efficiently.