"Is This PP? Or Is PP Just Two Totally Separate Model Instances?"

A user's pivotal question about model parallelism, NUMA topology, and the allreduce bottleneck

The Message

In the midst of an intense optimization session for deploying the 744B-parameter GLM-5-NVFP4 mixture-of-experts model across 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the user posed a question that cut to the heart of distributed inference strategy:

"Can we slice the model up such that TP is 4, on each socket, and we infer half of the layers on one socket, half of the layers on the second socket? It this PP? Or is PP just two totally separate model instances?"

This single message — brief, technically precise, and deceptively simple — represents a critical turning point in the conversation. It is not merely a request for a definition. It is a hypothesis, a design proposal, and a test of conceptual understanding all rolled into one. To fully appreciate what this message accomplishes, we must examine the context that produced it, the reasoning it reveals, and the technical territory it navigates.

The Context: A Battle Against the Allreduce Bottleneck

By the time this message arrives ([msg 835]), the assistant and user have been deep in the trenches of GPU inference optimization for many rounds. The hardware is formidable: 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (SM120 architecture, 96 GB VRAM each, 600 W TDP), connected via PCIe Gen5 with no NVLink. The GPUs are split across two NUMA sockets of an AMD EPYC 9335 processor, with four GPUs per socket enjoying fast NODE-level P2P bandwidth (53.76 GB/s), while cross-NUMA communication drops to SYS-level (40.24 GB/s).

The model is GLM-5-NVFP4, a 744B-parameter MoE architecture with 256 experts (8 activated per token), 78 transformer layers, and all GEMM operations running in FP4→BF16 mode. The assistant has been running with TP8 (tensor parallelism across all 8 GPUs), achieving up to 3,740 total tok/s at 1024 concurrency — respectable, but far from the theoretical potential of the hardware.

The bottleneck has been identified with surgical precision: each forward pass requires 156 allreduce operations (2 per layer × 78 layers), and each allreduce must communicate across all 8 GPUs over PCIe. The GPUs draw only ~235–330 W out of 600 W TDP during inference, a clear sign that compute is stalling on communication. The CUTLASS FP4 GEMM kernels achieve merely 0.8–55 TFLOPS during decode (0.02–3% of peak) for per-expert batch sizes of 16–64 tokens, confirming that the GPUs are waiting on data, not computing.

Prior research in FINDINGS.md had already shown that TP4+PP2 outperformed TP8 for a similar model (Kimi K2-Thinking) on the same hardware, achieving 5,154 tok/s vs 4,500 tok/s. The user's question emerges directly from this finding, attempting to generalize it to the current model and understand the underlying mechanism.

What the User Is Actually Proposing

The user's proposal is elegantly simple: instead of spreading every layer across all 8 GPUs (TP8), split the model into two halves by layers. The first 39 layers run on socket 0's 4 GPUs using TP4, and the second 39 layers run on socket 1's 4 GPUs using TP4. Within each socket, the allreduce is fast (NODE-level P2P at 53.76 GB/s). Between sockets, only the activations at the layer boundary need to cross — a single transfer per forward pass instead of 156 allreduces.

This is, in fact, pipeline parallelism (PP) with a pipeline depth of 2 (PP2), where each pipeline stage uses tensor parallelism of 4 (TP4). The user correctly identifies the key characteristics: TP=4 per socket, layers split across sockets. But the question "Is this PP? Or is PP just two totally separate model instances?" reveals a subtle conceptual distinction they are working through.

The user's alternative interpretation — "two totally separate model instances" — would be data parallelism, where each instance processes a different batch of inputs independently. That is emphatically not what they are proposing. Their proposal keeps a single model, splits it by layers, and runs each layer group on a different set of GPUs. That is pipeline parallelism. The confusion is understandable: both PP and data parallelism involve partitioning work across devices, but PP partitions the model while data parallelism partitions the data.

The Reasoning and Motivation

This message reveals several layers of sophisticated reasoning:

First, the user has internalized the NUMA topology as a first-class constraint. They are not thinking of the 8 GPUs as a homogeneous pool. They see two distinct groups of 4, each with fast internal connectivity and slower cross-group links. The proposal to make TP=4 "on each socket" is a direct acknowledgment that the hardware topology should dictate the parallelism strategy.

Second, the user is reasoning about communication patterns at a deep level. They understand that TP8 requires all 8 GPUs to allreduce after every single layer — 156 times per forward pass. By contrast, TP4+PP2 confines allreduce to within each socket (4 GPUs), and the cross-socket communication is reduced to a single activation transfer at the pipeline boundary. This is a profound insight: the number of cross-socket communication events drops from 156 to 1.

Third, the user is testing their understanding of parallelism taxonomies. The question "Is this PP?" is not just asking for a label. It is asking: "Does the concept of pipeline parallelism capture what I'm describing, or is there a different concept I should be using?" The follow-up "Or is PP just two totally separate model instances?" shows they are actively comparing their proposal against alternative interpretations, trying to map their intuition onto established terminology.

Assumptions and Potential Misconceptions

The user makes several assumptions that are worth examining:

That splitting layers across sockets eliminates cross-socket allreduce. This is largely correct, but there are nuances. In PP2+TP4, each allreduce still involves 4 GPUs, but those 4 GPUs are within the same NUMA node with fast P2P. The cross-socket communication is the pipeline activation transfer, which is a single P2P send (not an allreduce). However, the pipeline introduces a new challenge: the second socket's GPUs must wait for the first socket to finish its layers before they can begin. This "pipeline bubble" can reduce utilization if not carefully managed with micro-batching.

That PP means splitting layers across devices. This is correct. Pipeline parallelism partitions the model by layers, assigning contiguous ranges of layers to different devices or groups of devices. The user's description — "infer half of the layers on one socket, half of the layers on the second socket" — is a textbook description of PP with depth 2.

That PP might mean "two totally separate model instances." This is the misconception the user is testing. Two separate model instances processing different inputs would be data parallelism (or, more precisely, replicated parallelism). The user correctly senses that this is different from what they're proposing, and they're seeking confirmation.

That TP=4 within each socket will work seamlessly. This assumption is reasonable but not trivial. TP4 requires that each GPU in the group holds a shard of every layer's parameters. In the proposed scheme, socket 0's GPUs would hold shards of layers 0–38, and socket 1's GPUs would hold shards of layers 39–77. This means the model is partitioned both by layers (PP) and within each layer (TP). The implementation complexity is higher than TP8, and the sglang framework must support this hybrid configuration.

Input Knowledge Required

To understand this message fully, one needs:

Output Knowledge Created

This message creates several forms of knowledge:

A concrete proposal for the next configuration to test. The user has essentially designed the experiment: launch with --tp 4 --pp 2 and benchmark against the TP8 baseline. This becomes the assistant's next task.

A clarification of parallelism terminology. The question forces an explicit discussion of what PP is and isn't, distinguishing it from data parallelism. This conceptual clarity is essential for the optimization work ahead.

A connection between hardware topology and parallelism strategy. The message articulates a principle: the NUMA boundary should be a parallelism boundary. This principle generalizes beyond this specific deployment to any multi-socket GPU server.

A demonstration of the user's technical depth. The message shows that the user is not merely directing the assistant but actively reasoning about the problem at their level, proposing solutions based on their understanding of the hardware and model architecture.

The Thinking Process Visible

The user's thinking process is remarkably clear in this short message. We can see:

  1. Pattern recognition: The user has seen that TP4+PP2 worked better for a similar model and is trying to understand why so they can apply the same reasoning.
  2. Hypothesis formation: "Can we slice the model up such that TP is 4, on each socket, and we infer half of the layers on one socket, half of the layers on the second socket?" — this is a specific, testable hypothesis about how to reduce cross-socket communication.
  3. Conceptual mapping: The user is trying to map their proposed approach onto established terminology ("Is this PP?"), which is a form of conceptual validation.
  4. Boundary testing: "Or is PP just two totally separate model instances?" — the user is testing the boundaries of the concept, checking whether their understanding aligns with the standard definition.
  5. Iterative refinement: The question is not asked in isolation. It builds on hours of prior conversation, benchmarking, and debugging. The user has absorbed the findings about allreduce bottlenecks, NUMA topology, and prior TP4+PP2 results, and is synthesizing them into a new proposal.

Conclusion

This message, for all its brevity, represents a moment of genuine technical insight. The user has identified the core performance bottleneck (cross-socket allreduce), understood the hardware topology that causes it (NUMA domains with PCIe P2P), and proposed a parallelism strategy that directly addresses it (TP4 within each socket, PP2 across sockets). The question "Is this PP?" is not a request for a dictionary definition — it is a scientist testing a hypothesis against established theory.

In the broader arc of the conversation, this message sets the stage for the next phase of optimization. The assistant will go on to benchmark TP4+PP2, confirm that it changes the compute-communication balance, and continue the deep investigation into FP4 GEMM kernel efficiency on SM120 that ultimately defines the session's most important findings. But the seed of that investigation — the recognition that parallelism strategy must be shaped by hardware topology — is planted right here, in this single, thoughtful question.