The Green Light: Confirming Pipeline Parallelism Support and Launching TP4+PP2

In the long arc of optimizing inference throughput for the GLM-5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs, there comes a moment where careful investigation converges into decisive action. Message 839 is that moment. It is brief — barely a sentence of reasoning followed by a single cleanup command — yet it represents the culmination of a multi-message investigation into whether Pipeline Parallelism (PP) is viable on this hardware, and the first concrete step toward a new configuration that would ultimately reshape the entire optimization strategy.

The Context: A Bottleneck Diagnosed

The conversation leading up to this message had been wrestling with a fundamental performance problem. The eight GPUs in this Proxmox virtualized environment are spread across two NUMA sockets, with each socket containing four GPUs connected via fast NVLink P2P (approximately 54 GB/s), but cross-socket communication traversing slower PCIe paths (approximately 40 GB/s). Earlier benchmarking had shown that Tensor Parallelism (TP) across all eight GPUs (TP8) incurred significant allreduce overhead — every MoE layer required synchronizing all eight GPUs, with roughly half of those synchronizations crossing the slow NUMA boundary.

The user, in [msg 835], had asked a sharp question: could the model be sliced so that TP runs within each socket (TP4 per socket), with half the layers on one socket and half on the other? They wondered whether this was what Pipeline Parallelism (PP) actually does, or whether PP meant running two entirely separate model instances.

The assistant's response in [msg 836] confirmed that PP is exactly what the user described — splitting the model by layers across groups of GPUs. With TP4+PP2, GPUs 0–3 (NUMA node 0) would run layers 0–38, and GPUs 4–7 (NUMA node 1) would run layers 39–77. The forward pass flows sequentially through stage 0 then stage 1. Allreduce stays within the fast same-NUMA group, and cross-socket communication is reduced to a single activation transfer per layer boundary — a dramatic reduction from 156 cross-socket allreduces per forward pass.

The Investigation: Tracing Through the Code

But before launching, the assistant needed to verify that PP was actually supported in the SGLang codebase for this specific model. The investigation unfolded across three messages ([msg 836], [msg 837], [msg 838]).

The first concern was a hardcoded pp_size=1 found in the FlashInfer MoE token dispatcher at /root/sglang/python/sglang/srt/layers/moe/token_dispatcher/flashinfer.py, line 137. This looked like a potential blocker — if the token dispatcher hardcodes PP size to 1, then PP might not work correctly regardless of what the server launch arguments say.

The assistant dug deeper by reading the surrounding code context. The pp_size=1 parameter was inside a Mapping constructor call within the Expert Parallelism (EP) all-to-all dispatcher. The critical insight was that this code path is only exercised when ep_size > 1 — that is, when expert parallelism is enabled. With the default --ep-size 1, the EP dispatcher is never invoked, and the pp_size=1 value is irrelevant.

The assistant then verified the model architecture itself. In /root/sglang/python/sglang/srt/models/glm4_moe.py, the model uses make_layers with pp_rank and pp_size parameters — confirming that the GLM4 MoE model architecture properly supports pipeline parallelism at the layer-slicing level. The pp_group.world_size and pp_group.rank_in_group attributes were present and used to determine which layers each PP stage should run.

The Subject Message: Decision and Action

This brings us to message 839, the subject of this article. The assistant writes:

The GLM4 MoE model uses make_layers with pp_rank and pp_size — PP is properly supported in the model architecture. Let me launch with TP4+PP2: [bash] ssh root@10.1.230.174 'pkill -9 -f sglang 2>/dev/null; sleep 2; echo "cleaned"'

The first sentence is a conclusion statement — it synthesizes the findings from the code investigation into a clear, confident assertion. The assistant has traced through two separate code paths (the token dispatcher and the model architecture), identified a potential concern (pp_size=1), evaluated whether that concern applies to the current configuration (it does not), and confirmed that the model's make_layers infrastructure correctly handles PP. The word "properly" is significant — it signals that the implementation is not just present but correct, that the layer slicing, rank assignment, and communication patterns are all wired up as expected.

The second sentence is pure action. "Let me launch with TP4+PP2" transitions from investigation to execution. The command that follows — pkill -9 -f sglang followed by a sleep and echo — is a cleanup step. It forcefully terminates any lingering SGLang processes from previous runs, waits for them to fully die, and confirms the cleanup. This is a necessary precondition: launching a new server configuration requires a clean slate, especially when switching from TP8 to TP4+PP2, as the GPU memory allocation and NCCL communicator setup are fundamentally different.

The Thinking Process Visible

The reasoning in this message is compressed but dense. Several layers of thinking are implicit:

Confidence from code evidence: The assistant doesn't just say "PP should work" — it cites specific code evidence (make_layers, pp_rank, pp_size) to ground the claim. This reflects a systematic approach: verify the claim by tracing through the actual source code, not by relying on documentation or assumptions.
Risk assessment: The assistant had identified a potential blocker (the pp_size=1 in the dispatcher) and had to decide whether it was a real problem or a false alarm. The judgment that it's only relevant to EP (which isn't being used) is a risk assessment — the assistant is implicitly saying "the code path that could break PP is not exercised by our configuration."
Assumption about code paths: There's an assumption here that the EP dispatcher code is truly isolated and that no other component hardcodes pp_size. The assistant checked the model file and found proper PP support, but it did not exhaustively audit every file in the SGLang codebase. This is a reasonable scoping decision — you can't audit every line, so you check the most likely problem areas and proceed if they're clean.
Procedural awareness: The cleanup command reveals an understanding of the operational requirements. Switching from TP8 to TP4+PP2 isn't just a matter of changing command-line arguments — old processes must be killed to free GPU memory and NCCL resources. The sleep 2 shows awareness that process termination isn't instantaneous.

What Followed

The next message ([msg 840]) shows the actual server launch command with the full TP4+PP2 configuration: --tp 4 --pp 2 --mem-fraction-static 0.92 --max-running-requests 1024 and the full suite of backend flags (flashinfer attention, cutlass FP8 GEMM, trtllm NSA backends, flashinfer_cutlass MoE runner). This would become the configuration that the assistant would later benchmark and find to be 2× slower than TP8 — a surprising result that would trigger an even deeper investigation into whether the model was compute-bound or communication-bound.

Input Knowledge Required

To fully understand this message, one needs:

Pipeline Parallelism concepts: Understanding that PP splits layers across GPU groups, that it introduces pipeline bubbles but reduces allreduce communication.
Tensor Parallelism concepts: Understanding that TP splits individual operations across GPUs within a group, requiring allreduce after each layer.
Expert Parallelism concepts: Understanding that EP distributes MoE experts across GPUs, and that the FlashInfer token dispatcher has special handling for EP.
SGLang architecture: Familiarity with the make_layers function, the pp_group object, and how model parallelism is configured.
The hardware topology: Knowing that the 8 GPUs span two NUMA sockets, with fast P2P within each socket and slower cross-socket links.
The investigation history: Knowing that the assistant had previously identified allreduce overhead as a key bottleneck and was actively seeking configurations that reduce cross-socket communication.

Output Knowledge Created

This message produces:

A confirmed fact: PP is properly supported in the GLM4 MoE model architecture for SGLang.
A decision point: The assistant has chosen to proceed with TP4+PP2 as the next configuration to test.
A clean state: The old SGLang processes are killed, preparing for a fresh launch.
A precedent: The methodology of verifying code support before attempting a configuration change is reinforced.

Significance

Message 839 may appear unremarkable at first glance — a short confirmation followed by a cleanup command. But it represents a critical juncture in the optimization journey. The assistant had to weigh the potential benefits of TP4+PP2 (reduced allreduce overhead, better NUMA locality) against the risks (pipeline bubbles, potential code incompatibility). By methodically investigating the codebase, identifying and dismissing a false alarm, and then decisively acting, the assistant demonstrated a disciplined approach to performance optimization: understand the bottleneck, verify the fix is viable, then execute. This message is the hinge point between analysis and action.