Investigating Expert Parallelism for Blackwell GPUs on PCIe: A Diagnostic Deep Dive

Introduction

In the course of deploying the massive GLM-5-NVFP4 model (744B parameters, 256 MoE experts) across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the team encountered a stubborn performance ceiling. Despite successfully resolving a critical NaN crash during decode by selecting the correct NSA backends (trtllm), and tuning server parameters like memory fraction and CUDA graphs, throughput remained stubbornly capped at roughly 200–236 output tokens per second. The single-stream latency was even more telling: a single request generated only ~11 tokens per second, suggesting a fundamental communication bottleneck rather than a compute limitation.

This article examines a single pivotal message ([msg 246]) in the conversation where the assistant responds to the user's incisive question: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" The message represents a critical inflection point in the debugging journey — a moment where the team pivots from tuning existing parameters to questioning the fundamental parallelism strategy itself.

The Context: A Performance Mystery on Blackwell

To understand the significance of [msg 246], we must first appreciate the journey that led to it. The team had spent considerable effort deploying GLM-5-NVFP4 — a quantized (NVFP4, or NVIDIA FP4) mixture-of-experts model with 256 experts and 61 layers — on eight RTX PRO 6000 Blackwell GPUs. These GPUs, while individually powerful with 96GB of memory each, were connected only via PCIe Gen5, with no NVLink interconnect. The system was running inside a Proxmox virtual machine (KVM/QEMU), which added another layer of complexity: nvidia-smi topo -m reported NS (Not Supported) for GPU peer-to-peer access, meaning all cross-GPU communication had to traverse host memory through the CPU.

The current configuration used Tensor Parallelism of degree 8 (TP8), meaning every token's computation was split across all eight GPUs. For each of the 61 layers, this required an all-reduce operation to synchronize the partial results. With PCIe providing roughly 32 GB/s bidirectional bandwidth — compared to ~900 GB/s on NVLink — each all-reduce became a significant bottleneck. The single-stream result of ~11 tok/s was a dead giveaway: the model was spending most of its time waiting for data to traverse the PCIe bus, not computing.

The user's question in [msg 243] cut to the heart of this issue. Expert Parallelism (EP) offers a fundamentally different communication pattern: instead of splitting every operation across all GPUs and synchronizing with all-reduce, EP distributes the 256 MoE experts across GPUs (roughly 32 experts per GPU). Each token is routed only to the GPUs holding its 8 activated experts, using all-to-all communication. The hypothesis was that all-to-all for a small subset of experts would move less data than all-reduce of full hidden states across all GPUs.

The Message: A Systematic Investigation Begins

[msg 246] opens with the assistant acknowledging the user's insight: "You're right to flag this." This is not mere politeness — it signals a genuine recognition that the team may have been looking at the wrong level of optimization. The assistant then lays out a concise but powerful analysis:

Current TP8 problem on PCIe: Every decode step for every token requires all-reduce across 8 GPUs over PCIe (~32 GB/s bidirectional). That's 61 layers × all-reduce per token. The single-stream speed of ~11 tok/s screams communication-bound.

This quantification is crucial. By multiplying 61 layers by the all-reduce cost per token, the assistant frames the problem in concrete terms. The single-stream measurement of ~11 tok/s — previously just a data point — becomes a diagnostic signal that "screams communication-bound." This is the kind of insight that separates surface-level tuning from genuine performance engineering.

The assistant then outlines the EP8 counter-proposal:

EP8 potential: With EP, the MoE layers do all-to-all (token routing) instead of all-reduce. Since only 8/256 experts activate per token, each token only needs to reach ~8/256 × 8 GPUs ≈ most GPUs but with much less data per exchange. The attention layers (which are small — BF16 non-MoE) still need some form of parallelism.

The reasoning here is sound but makes an important assumption: that the all-to-all communication volume for routing tokens to 8 experts is indeed smaller than the all-reduce volume for synchronizing full hidden states. For a model with hidden dimension 7168 (typical for models of this scale), an all-reduce moves roughly 7168 × 2 bytes (BF16) × 8 GPUs ≈ 115 KB per layer per token. The all-to-all for expert routing moves token hidden states to the GPUs hosting the relevant experts — potentially similar or larger volumes depending on routing patterns. The assistant implicitly acknowledges this complexity by noting that attention layers "still need some form of parallelism."

The Pivot to Practical Verification

What distinguishes this message is that the assistant does not stop at theoretical analysis. Having laid out the reasoning, it immediately pivots to verification:

The key question is whether deepep (DeepSeek's EP implementation) works on SM120 / PCIe.

This is a critical insight. Expert parallelism in sglang relies on DeepSeek's deepep library (also known as deep_ep), which provides the efficient all-to-all kernel for expert routing. If this library is not installed, or if it does not support the SM120 architecture of the Blackwell GPUs, then EP is simply not an option regardless of its theoretical benefits.

The assistant then runs three bash commands in sequence:

  1. Check if deep_ep is installed: python3 -c "import deep_ep; print(deep_ep.__version__)" — returns ModuleNotFoundError: No module named 'deep_ep'.
  2. Search pip for any EP-related packages: pip list | grep -iE "deep.ep|deepep|mori" — returns empty.
  3. Check if the sglang EP module exists: python3 -c "from sglang.srt.layers.moe.ep_moe.token_dispatcher import DeepEPDispatcher" — returns ModuleNotFoundError: No module named 'sglang.srt.layers.moe.ep_moe.token_dispatcher'. These three checks form a systematic dependency audit. The first checks for the runtime library, the second checks for any installable package, and the third checks whether the sglang codebase even contains the EP integration code. The answer is negative on all three fronts: deep_ep is not installed, no EP-related packages are present in the environment, and the sglang version being used does not include the EP module path.

Assumptions and Their Implications

The message reveals several assumptions, some explicit and some implicit:

Assumption 1: EP would be faster on PCIe. The assistant's analysis suggests EP could be better because all-to-all for 8/256 experts moves less data than all-reduce of full hidden states. However, this assumption deserves scrutiny. In practice, all-to-all communication for expert routing can be surprisingly expensive, especially when the routing is unbalanced (a common issue with MoE models where certain experts are "popular" and receive many tokens). Moreover, on PCIe without NVLink, the all-to-all pattern may suffer from the same fundamental latency issues as all-reduce — the bottleneck is the PCIe bus itself, not the communication pattern. The assistant does not fully explore this nuance, though the subsequent dependency check renders it moot anyway.

Assumption 2: deepep is the only path to EP. The assistant implicitly assumes that EP requires DeepSeek's deepep library. While this is true for sglang's current implementation, it is worth noting that other frameworks implement EP differently. The assistant correctly focuses on what sglang actually supports rather than speculating about alternative implementations.

Assumption 3: SM120 compatibility is the main risk. The assistant flags whether deepep "works on SM120 / PCIe," correctly identifying that Blackwell's new architecture (compute capability 12.0) may not be supported by all CUDA kernels. This assumption proves prescient — even if deep_ep were installed, it might not run on SM120 GPUs.

Assumption 4: The EP module path exists in this sglang version. The third bash command tests whether sglang.srt.layers.moe.ep_moe.token_dispatcher exists. The negative result suggests either that this sglang build does not include EP support, or that the module path has changed. This is a valuable finding — it tells the team that even if they install deep_ep, they may need a different version of sglang or additional patches to enable EP.

The Thinking Process: From Theory to Dead End

The structure of [msg 246] reveals a clear thinking process:

  1. Acknowledge and validate the user's question — "You're right to flag this."
  2. Quantify the problem — "61 layers × all-reduce per token" and "~11 tok/s screams communication-bound."
  3. Propose the alternative — EP with all-to-all instead of all-reduce, less data per exchange.
  4. Identify the dependency — deepep as the enabling technology.
  5. Test the dependency — three bash commands that systematically rule out EP as a viable option.
  6. Implicit conclusion — EP is not available in the current environment, so the team must look elsewhere. This is a textbook example of disciplined debugging: don't just theorize about solutions, verify the prerequisites before committing to a path. The message could have ended with a lengthy theoretical debate about EP vs TP on PCIe, but instead it cuts to the chase with concrete dependency checks. The fact that all three checks fail is itself informative. It tells the team that: - They cannot simply flip an --ep-size flag and expect EP to work - They would need to install deep_ep (which may itself have build issues on SM120) - They may need to upgrade or rebuild sglang to include the EP module - The effort required to enable EP may not be justified if the performance gains are uncertain

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. EP is not currently viable: The deep_ep library is not installed, and the sglang EP module path does not exist in the current build.
  2. The communication bottleneck is confirmed: The single-stream measurement of ~11 tok/s, combined with the 61-layer all-reduce analysis, provides strong evidence that PCIe bandwidth is the primary constraint.
  3. A dependency chain is documented: The team now knows exactly what would be required to attempt EP (install deep_ep, potentially rebuild sglang, verify SM120 compatibility).
  4. A diagnostic methodology is demonstrated: The three-command pattern (check runtime library, check package manager, check codebase paths) is a reusable approach for auditing any framework dependency.

Conclusion

[msg 246] represents a turning point in the GLM-5-NVFP4 deployment effort. It is the moment when the team stops tuning within the existing TP8 paradigm and begins questioning whether the entire parallelism strategy is wrong for the hardware. The assistant's response is exemplary: it validates the user's insight, quantifies the problem, proposes an alternative, and then — crucially — tests the prerequisites before committing to the new direction.

The finding that EP is not available in the current environment is disappointing but valuable. It saves the team from spending hours trying to configure EP flags that don't exist, and it redirects attention to other avenues: perhaps adjusting the MoE runner backend, tuning the all-reduce algorithm, or investigating whether the virtualization layer introduces additional overhead that no parallelism strategy can fully mitigate.

In the broader narrative of the session, this message sets the stage for the next major investigation: the user's follow-up question about whether virtualization itself is the bottleneck. That line of inquiry will ultimately reveal that the Proxmox VM's lack of direct GPU peer-to-peer support forces all cross-GPU transfers through host memory, adding significant latency to every all-reduce operation. The EP investigation, while inconclusive in itself, was a necessary step — it forced the team to confront the fundamental communication architecture rather than continuing to tweak surface-level parameters.