The Pivot to Pipeline Parallelism: A Topology-Aware Strategy for Blackwell Inference

Introduction

In the high-stakes world of large-scale ML inference optimization, progress often comes not from incremental parameter tuning but from a fundamental rethinking of how work is distributed across hardware. Message 836 of this opencode session captures exactly such a moment: the assistant responds to the user's query about splitting a 744B-parameter MoE model across two NUMA sockets, and in doing so, orchestrates a strategic pivot from software patching to topology-aware parallelism configuration. This message is a masterclass in translating hardware constraints into architectural decisions, and it reveals the deep interplay between parallelism strategies, memory topology, and PCIe communication patterns that defines cutting-edge inference optimization.

The message arrives at a critical juncture. The session has been battling a stubborn performance bottleneck: the 8-way allreduce operations required after every attention and MoE layer are saturating the PCIe Gen5 bus, keeping GPU utilization low (~330W out of 600W TDP) and throughput capped. Previous attempts to fix this through software—patching FlashInfer's allreduce fusion kernels for SM120 support—had failed catastrophically, producing 236 tok/s (down from 1,867 tok/s). The team was stuck. The user's question about NUMA-aware model splitting opens a new front: instead of making the communication faster, why not reduce the amount of communication that needs to happen?

The Message in Full

The assistant's response is both pedagogical and action-oriented:

Great question. Let me clarify how Pipeline Parallelism (PP) works vs what you're describing.

>

## PP is exactly what you're describing

>

Pipeline Parallelism (PP) splits the model by layers across groups of GPUs. With TP4+PP2 on your setup:

>

- PP stage 0 (GPUs 0-3, NUMA node 0): Runs layers 0-38, with TP4 across the 4 GPUs on that socket - PP stage 1 (GPUs 4-7, NUMA node 1): Runs layers 39-77, with TP4 across the 4 GPUs on that socket

>

This is not two separate model instances — it's one model where the forward pass flows sequentially through stage 0, then stage 1. Each stage only does allreduce within its own 4 GPUs (same NUMA node = fast 53.76 GB/s P2P), and the only cross-socket communication is passing activations between stages (one transfer per layer boundary, not per layer).

>

## Why this helps on your hardware

>

The key wins:

>

1. Allreduce shrinks from 8-way to 4-way — half the participants, and all within a single NUMA node (NODE topology, ~54 GB/s instead of mixed SYS ~40 GB/s) 2. Cross-NUMA traffic drops dramatically — instead of 156 allreduces across 8 GPUs (half crossing NUMA), you get 156 allreduces within 4 same-NUMA GPUs + only ~78 small inter-stage transfers 3. Latency per allreduce drops — 4-way ring allreduce is faster than 8-way

>

The downside: PP adds pipeline bubbles (stage 1 waits while stage 0 computes, and vice versa), which hurts single-stream latency. But for throughput at high concurrency, the allreduce savings should more than compensate.

>

## From our prior K2 research

>

FINDINGS.md shows TP4+PP2 achieved 5,154 tok/s (vs TP8's 4,180 tok/s) at 384 concurrency for short generations on GLM-4.7-FP8 — a 23% throughput improvement. For long generations, TP8 was better because the pipeline bubbles dominate.

>

## Let me try it

>

Let me start the server with TP4+PP2. There's one concern: the flashinfer MoE token dispatcher has a hardcoded pp_size=1 which could be a problem. Let me check and fix if needed, then launch.

>

[bash] ssh root@10.1.230.174 'grep -n "pp_size" /root/sglang/python/sglang/srt/layers/moe/token_dispatcher/flashinfer.py' 137: pp_size=1,

>

[bash] ssh root@10.1.230.174 'grep -rn "pp_size.==.1\|assert.pp_size\|pp_size.1" /root/sglang/python/sglang/srt/layers/moe/ | head -20' /root/sglang/python/sglang/srt/layers/moe/token_dispatcher/flashinfer.py:137: pp_size=1,

The Reasoning and Motivation

This message is fundamentally about unblocking progress. The session had been deadlocked: allreduce fusion on SM120 was broken, NCCL tuning had failed, and the team was running out of software-level fixes. The user's question—born from an intuitive understanding of the NUMA topology—provides an escape route.

The assistant's reasoning unfolds in three layers:

First, conceptual clarification. The user asks whether PP means "two totally separate model instances." This reveals a common confusion between Pipeline Parallelism and Model Parallelism (which would be two independent copies). The assistant carefully distinguishes: PP is a single model split by layers, with sequential forward pass through stages. This distinction is crucial because it means the model's memory footprint doesn't double—each GPU still holds only its shard of parameters.

Second, quantitative justification. The assistant doesn't just say "PP helps." It provides concrete numbers: allreduce drops from 8-way to 4-way, cross-NUMA traffic drops from 156 allreduces to ~78 small transfers, and P2P bandwidth improves from ~40 GB/s (cross-NUMA SYS) to ~54 GB/s (same-NUMA NODE). These numbers come from earlier benchmarking work documented in the session's discoveries. The assistant also cites prior results from the K2 model on the same hardware: 5,154 tok/s with TP4+PP2 versus 4,180 tok/s with TP8—a 23% improvement.

Third, pragmatic execution. The assistant doesn't stop at theory. It immediately moves to check for implementation barriers, specifically a hardcoded pp_size=1 in the FlashInfer MoE token dispatcher. This is the kind of real-world engineering detail that separates a theoretical discussion from actionable work. The grep commands reveal the problem exists at line 137 of flashinfer.py.

Decisions Made

This message makes several implicit and explicit decisions:

  1. Abandon the allreduce fusion fix (temporarily). The assistant doesn't say this explicitly, but the decision to pursue TP4+PP2 represents a strategic retreat from the failed SM120 allreduce fusion patching effort. Instead of trying to make 8-way allreduce faster, the architecture will be changed to need less allreduce.
  2. Prioritize throughput over latency. The assistant acknowledges that PP introduces pipeline bubbles that hurt single-stream latency. Given the session's targets (1k+ total tok/s, >100 tok/s single/dual stream), this is a calculated risk. The prior K2 data suggests the throughput win is substantial for short generations.
  3. Accept the implementation risk. The hardcoded pp_size=1 is a red flag. The assistant acknowledges it but proceeds anyway, signaling a willingness to patch the source code if needed. This aligns with the user's earlier instruction: "Think big and don't be afraid to fork/modify code."

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Explicit assumptions:

Mistakes and Incorrect Assumptions

The most significant potential mistake is the assumption that TP4+PP2 will outperform TP8 on this hardware. The prior K2 data showed this was true only for short generations; for long generations, TP8 was better. The GLM-5-NVFP4 model has a different architecture (256 experts vs 161, different attention mechanism with NSA/DeepSeek Sparse Attention), and the benchmark results may not transfer directly.

Additionally, the assistant assumes that the 4-way allreduce within a NUMA node is the primary bottleneck relief. However, the session's later analysis (visible in the chunk summary) reveals that the model is actually compute-bound rather than communication-bound—TP4+PP2 ends up being 2× slower than TP8. This means the assumption that allreduce latency was the primary bottleneck was incorrect. The real issue is FP4 GEMM kernel efficiency on SM120, where the GPUs only draw ~235W out of 600W TDP and achieve merely 0.8–55 TFLOPS (0.02–3% of peak) during actual decode.

The assistant also assumes that the hardcoded pp_size=1 is the only PP-related issue. In practice, PP requires coordinated scheduling across stages, proper microbatch handling, and careful management of the pipeline flush/fill phases. A single hardcoded constant is likely just the tip of the iceberg.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. NUMA topology knowledge: Understanding that the 8 GPUs are split across two NUMA nodes (0-3 on node 0, 4-7 on node 1), with same-NUMA P2P at ~54 GB/s and cross-NUMA at ~40 GB/s.
  2. Parallelism concepts: The difference between Tensor Parallelism (TP, splitting individual operations across GPUs) and Pipeline Parallelism (PP, splitting layers across groups). The assistant's explanation serves as a mini-tutorial.
  3. Allreduce mechanics: Understanding that each MoE layer requires two allreduce operations (after attention, after MoE), totaling 156 allreduces per forward pass for a 78-layer model.
  4. The session's history: The failed allreduce fusion attempt, the power analysis showing GPUs at only ~330W out of 600W TDP, and the prior K2 benchmarking results.
  5. The hardware constraints: PCIe Gen5 as the only interconnect (no NVLink), SM120's 100KB shared memory limit, and the Proxmox virtualization layer.

Output Knowledge Created

This message creates several forms of output knowledge:

  1. A concrete plan: The decision to try TP4+PP2 becomes the next action item, replacing the stalled allreduce fusion effort.
  2. A code issue identified: The hardcoded pp_size=1 in the FlashInfer token dispatcher is documented and located. This is actionable intelligence for the patching effort.
  3. A conceptual framework: The assistant's explanation of PP in the context of NUMA topology provides a reusable mental model for reasoning about parallelism on multi-socket GPU servers.
  4. A benchmark hypothesis: The prediction that TP4+PP2 will outperform TP8 at high concurrency (based on K2 data) sets up a testable hypothesis for the next round of benchmarking.
  5. Risk documentation: The downside of pipeline bubbles for single-stream latency is explicitly stated, providing a basis for evaluating the trade-off when results come in.

The Thinking Process

The assistant's thinking process, visible in the structure of the response, reveals a methodical engineering mind at work:

Step 1: Validate the user's intuition. The user asks "Is this PP?" The assistant immediately confirms "PP is exactly what you're describing." This validation is important—it tells the user they're on the right track and encourages further collaboration.

Step 2: Map the abstract concept to concrete hardware. The assistant doesn't just define PP; it instantiates it on the specific hardware: "PP stage 0 (GPUs 0-3, NUMA node 0): Runs layers 0-38." This grounding in physical reality is what makes the reasoning trustworthy.

Step 3: Quantify the benefits. Three numbered points with specific bandwidth numbers. This transforms a qualitative argument ("PP reduces communication") into a quantitative one ("allreduce shrinks from 8-way to 4-way, cross-NUMA traffic drops from 156 allreduces to ~78 transfers").

Step 4: Acknowledge the downside. The assistant doesn't oversell. It explicitly mentions pipeline bubbles and references the prior K2 data showing that TP8 wins for long generations. This balanced assessment builds credibility.

Step 5: Move to action. "Let me try it." But before launching, the assistant checks for implementation barriers. The grep commands are not just busywork—they represent a systematic risk assessment. The hardcoded pp_size=1 is found and flagged.

Step 6: Broaden the search. The second grep command searches more broadly for any other hardcoded PP assumptions in the MoE code, showing a thoroughness that prevents surprises later.

Conclusion

Message 836 represents a pivotal moment in this optimization journey. It's the point where the team pivots from trying to make bad communication faster to architecting the model to need less communication. The assistant's response demonstrates how deep hardware knowledge, parallelism theory, and practical engineering combine to produce strategic decisions. Even though later benchmarking would reveal that TP4+PP2 was actually 2× slower than TP8 (because the real bottleneck was compute, not communication), the reasoning in this message is sound and the approach was worth trying. In optimization work, disproving a hypothesis is just as valuable as confirming it—and the TP4+PP2 experiment provided crucial data that helped redirect attention to the true bottleneck: FP4 GEMM kernel efficiency on SM120.

The message also showcases the collaborative dynamic of this session: the user contributes an intuitive hardware insight, and the assistant validates, elaborates, quantifies, and operationalizes it. This is opencode at its best—a human-machine partnership where each party brings complementary strengths to bear on a complex engineering challenge.