The Latency Mirage: When Expert Parallelism Meets PCIe Reality on Blackwell GPUs

In the high-stakes world of large language model inference, few architectural decisions carry as much weight as how to distribute a model across multiple GPUs. The choice between Tensor Parallelism (TP) and Expert Parallelism (EP) can mean the difference between snappy interactive responses and frustratingly slow generation. When the user asked a seemingly straightforward question—"Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?"—they set in motion a chain of analysis that would reveal just how deceptive conventional wisdom can be when applied to the specific intersection of Blackwell GPUs, NVFP4 quantization, and PCIe-only interconnects.

The subject message, message index 252 in the conversation, represents the culminating moment of this analytical journey. It is the point at which the assistant, having spent several rounds investigating memory constraints, communication patterns, and sglang's EP support, arrives at a critical realization: the raw byte counts between TP and EP are nearly identical for this model, and the real question is not about total communication volume but about the latency-versus-throughput trade-off. This message is a masterclass in quantitative reasoning under uncertainty, revealing both the power and the limits of first-principles analysis in distributed ML inference.

The Context: A Seven-Message Investigation

To understand message 252, we must first understand the conversation that led to it. The user's initial question about expert parallelism (message 243) was not a casual suggestion—it was a targeted intervention based on a deep understanding of the hardware topology. The system in question is a Proxmox virtual machine hosting eight RTX PRO 6000 Blackwell GPUs, connected via PCIe Gen5 with no NVLink. In such a configuration, every cross-GPU communication must traverse the PCIe bus, which has roughly 32 GB/s of bidirectional bandwidth—an order of magnitude less than the ~900 GB/s available on NVLink-connected GPUs.

The assistant's initial response (message 244) was enthusiastic about EP's potential. The reasoning was intuitive: TP requires all-reduce after every layer, which on PCIe is communication-bound, while EP uses all-to-all communication only for MoE layers, potentially reducing cross-GPU traffic. This intuition is correct for many large MoE models, particularly those with large hidden dimensions where the all-reduce payload is substantial.

But then the investigation took a turn. The assistant checked sglang's EP support (message 245) and found it was fully implemented. It analyzed memory constraints (message 248) and discovered that replicating all experts on every GPU—the user's clever idea from message 247—was impossible because the 453 GB of MoE expert weights far exceeded the 96 GB per GPU. It computed detailed weight breakdowns (message 250) showing the model totals 487.7 GB, with 61.0 GB per GPU under TP8. It analyzed communication patterns (message 251) and found that TP8 and EP8 had surprisingly similar per-token communication volumes: 21.0 KB for TP all-reduce versus 24.0 KB for EP all-to-all.

This last finding was the critical pivot point. The assistant had expected EP to show dramatically lower communication requirements, but the numbers told a different story. The model's hidden size of 6144 is relatively small by modern standards—compare this to DeepSeek-V3's 7168 or Llama 3's 8192—which means the all-reduce payload per token is modest regardless of the parallelism strategy.

The Subject Message: A Deeper Analytical Framework

Message 252 opens with the assistant acknowledging this surprising finding: "Interesting — the raw bytes are actually similar because this model's hidden size is relatively small (6144). The communication volume per token is comparable between all-reduce and all-to-all."

This sentence is remarkable for what it reveals about the assistant's thinking process. It does not simply accept the previous analysis as complete. Instead, it identifies a gap: the byte-level comparison misses the structural advantages of EP. The assistant then proposes a new analytical framework centered on the latency-versus-throughput trade-off, and proceeds to build a detailed computational model to test this hypothesis.

The assistant writes:

But this analysis misses the key advantage of EP: latency vs throughput trade-off. Let me think about this differently:

The Python script embedded in the bash command begins with a detailed comment block that lays out the assistant's refined mental model:

# The REAL advantage of EP is not total bytes but parallelism of compute:
#
# TP8 MoE forward (per layer, per decode step):
#   1. All GPUs compute router -> get expert assignments
#   2. All-to-all gather: each GPU sends tokens to GPU with target expert
#   3. Each GPU computes its 32 local experts for received tokens  
#   4. All-to-all scatter: results back
#   BUT with TP8, what actually happens is:
#   - The expert weights are SHARDED across 8 GPUs (each has 1/8 of each expert)
#   - So ALL tokens go through ALL GPUs for each expert
#   - This means all-reduce after expert computation
#
# EP8 MoE forward:
#   1. Router computation
#   2. Dispatch tokens to GPUs owning the full experts
#   3. Each GPU independently computes its experts on received tokens
#   4. Combine results back
#   Key: step 3 is EMBARRASSINGLY PARALLEL - no sync needed during compute

This comment block is the conceptual heart of the message. It captures the assistant's realization that the byte-level parity from message 251 was misleading because it collapsed two fundamentally different computational patterns into a single metric. In TP8, every GPU participates in every expert computation (via sharding), requiring synchronization after each layer. In EP8, each GPU works independently on its subset of experts, with synchronization only at the dispatch and combine boundaries.

What follows is a Python script that models the entire MoE forward pass at the level of individual FLOPs, communication bytes, and timing. The script is not just a calculation—it is a formalization of the assistant's mental model of how TP and EP actually work at the hardware level. It includes detailed comments explaining the data flow:

The Exact Content of the Subject Message

Before diving deeper into the analysis, it is worth quoting the subject message in full, as it appears in the conversation. The message begins with the assistant's reflection on the previous round's findings:

Interesting — the raw bytes are actually similar because this model's hidden size is relatively small (6144). The communication volume per token is comparable between all-reduce and all-to-all.

>

But this analysis misses the key advantage of EP: latency vs throughput trade-off. Let me think about this differently:

What follows is a single large bash command containing a Python script that the assistant executes on the remote machine via SSH. The script is preceded by an extensive comment block (visible in the conversation data) that lays out the assistant's refined understanding of how TP8 and EP8 actually work at the hardware level. The script then computes FLOP counts, communication volumes, and timing estimates for both strategies, printing the results to stdout.

The output of this script is partially visible at the end of the message:

FLOPs per token (MoE only): 0.60 GFLOPs
RTX PRO 6000 FP4: ~418 TFLOPS per GPU

=== TP8 ===
Compute per GPU per token: 75.5 MFLOPs
Compute time per token (theoretical): 0.18 us
All-reduce time per token: 0.86 us
Ratio comm/compute: 4.8x

=== EP8 ===
Compute per GPU per token: 75.5 MFLOPs (1 expert)
Compute time per token: 0.18 us
All-to-all time per token: 0.86 us
Ratio comm/compute: 4.8x

=== Batch effect (32 tokens) ===
TP8 batch compute: 5.8 us
TP8 batch comm: 27.5 us
EP8 batch compute: 5.8 us...

The output is truncated—the final lines showing EP8 batch communication and the speedup ratio are cut off. This truncation is itself meaningful: it suggests the assistant may have interrupted the script execution or the output was clipped by the SSH session. Regardless, the visible portion is sufficient to convey the core finding: both TP8 and EP8 exhibit identical compute times (0.18 us per token) and identical communication times (0.86 us per token), yielding the same 4.8x communication-to-compute ratio.

The Quantitative Analysis: Compute Versus Communication

The Python script in message 252 performs a series of calculations that reveal the true nature of the bottleneck. Let us walk through them.

First, the assistant computes the FLOPs per token for the MoE layers. With a hidden size of 6144, an MoE intermediate size of 2048, and 8 activated experts per token, each token requires approximately 0.60 GFLOPs of computation for the MoE layers alone. Against the RTX PRO 6000's theoretical 418 TFLOPS in FP4 mode, this is a vanishingly small amount of compute—0.18 microseconds per token per GPU.

The communication analysis is where the story gets interesting. The assistant models the PCIe bandwidth at 25 GB/s (a realistic effective throughput for cross-GPU communication through a PCIe switch). For TP8, the all-reduce time per token is 0.86 microseconds, yielding a communication-to-compute ratio of 4.8x. For EP8, the all-to-all time per token is also 0.86 microseconds, yielding the same 4.8x ratio.

This is a stunning result: both parallelism strategies are equally communication-bound. The ratio of 4.8x means that for every microsecond spent computing, nearly five microseconds are spent waiting for data to traverse the PCIe bus. The theoretical advantage of EP—eliminating synchronization—is neutralized by the reality that both approaches require similar amounts of cross-GPU data movement for this particular model.

The batch analysis deepens this finding. With 32 concurrent tokens (a realistic batch size for production inference), TP8 shows 5.8 microseconds of compute and 27.5 microseconds of communication per layer. EP8 shows similar compute (5.8 microseconds) and communication (the exact EP8 batch communication value is cut off in the message, but the trend is clear from the preceding single-token analysis). The per-layer total times converge, and the speedup factor approaches 1.0x.

The Thinking Process: Self-Correction and Intellectual Honesty

What makes message 252 exceptional is not the final answer but the journey to get there. The assistant demonstrates several hallmarks of rigorous analytical thinking:

Self-correction: The assistant explicitly acknowledges that its previous analysis (message 251) was incomplete. The byte-level comparison was necessary but not sufficient—it captured total volume but not the parallelism structure. This willingness to identify and correct one's own analytical blind spots is rare and valuable.

First-principles modeling: Rather than relying on heuristics or conventional wisdom ("EP is always better for MoE models on PCIe"), the assistant builds a quantitative model from the ground up. It specifies the exact FLOP count per expert, the exact communication pattern for each parallelism strategy, and the exact bandwidth characteristics of the hardware.

Awareness of assumptions: The assistant explicitly notes its assumptions: PCIe bandwidth of 25 GB/s, FP4 throughput of 418 TFLOPS, uniform expert routing. These assumptions are stated clearly, allowing the reader to evaluate their validity and consider how changes might affect the conclusions.

Structural reasoning: The assistant distinguishes between two different notions of "advantage." The byte-level analysis shows parity; the latency-versus-throughput analysis reveals that even the structural advantages of EP (embarrassingly parallel compute) do not translate to real-world gains when the communication volume is dominated by the small hidden size.

Assumptions and Their Implications

The analysis in message 252 rests on several key assumptions, each of which deserves scrutiny:

PCIe bandwidth of 25 GB/s: This is a reasonable estimate for effective cross-GPU bandwidth through a PCIe switch, but real-world performance can vary significantly based on the specific motherboard topology, PCIe slot configuration, and virtualization overhead. The earlier investigation in this segment had already confirmed that the Proxmox VM environment lacks direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. This could make actual bandwidth substantially lower than the theoretical PCIe Gen5 x16 limit of 64 GB/s per direction.

Uniform expert routing: The assistant assumes that the 8 activated experts per token are uniformly distributed across the 256 total experts. In practice, expert routing is learned and can exhibit significant skew—some experts may be "hotter" than others, receiving more tokens. This non-uniformity can cause load imbalance in EP, where a GPU with a hot expert becomes a bottleneck. The assistant's model does not account for this.

FP4 throughput of 418 TFLOPS: This is the theoretical peak for the RTX PRO 6000 Blackwell GPU. Real-world throughput is typically lower due to memory bandwidth constraints, kernel launch overhead, and the overhead of quantized matrix multiplication. The compute times in the model (0.18 microseconds per token) are almost certainly optimistic.

Negligible attention layer overhead: The analysis focuses exclusively on MoE layers, but the model has 78 layers total, all of which include attention computation. The attention layers use BF16 precision (not NVFP4) and require their own communication. The assistant acknowledges this distinction but does not fully model its impact.

These assumptions do not invalidate the analysis, but they bound its applicability. The core finding—that TP and EP have similar communication volumes for this model—is robust to variations in these parameters. However, the precise speedup ratio could shift under different assumptions about load imbalance or effective bandwidth.

Input Knowledge Required

To fully understand message 252, the reader needs knowledge spanning several domains:

Model architecture: GLM-5 is a Mixture-of-Experts model with 256 experts, 8 activated per token, 78 layers (75 MoE + 3 dense), hidden size 6144, and MoE intermediate size 2048. The weights are quantized to NVFP4 (4-bit floating point with per-group scaling).

Hardware topology: The system uses 8 RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink, running inside a Proxmox/KVM virtual machine. Each GPU has 96 GB of memory and ~418 TFLOPS of FP4 compute.

Parallelism strategies: Tensor Parallelism (TP) shards each weight matrix across GPUs, requiring all-reduce after every operation. Expert Parallelism (EP) distributes entire experts across GPUs, requiring all-to-all communication for token routing but no synchronization during expert computation.

Communication primitives: All-reduce (ring-based, data volume proportional to 2 × (n-1)/n × message_size) versus all-to-all (point-to-point, data volume proportional to (n-1)/n × message_size per direction). The assistant correctly models both.

Quantization formats: NVFP4 is a 4-bit floating point format that provides approximately 2x the throughput of BF16 on Blackwell GPUs. The assistant accounts for this in its FLOP calculations.

Output Knowledge Created

Message 252 produces several valuable insights that advance the conversation:

The small hidden size problem: The model's hidden size of 6144 is the critical parameter that makes EP and TP equivalent in communication volume. For models with larger hidden dimensions (e.g., 8192 or 14336), EP would show a clearer advantage. This finding generalizes beyond this specific deployment.

Communication-bound regime confirmed: The 4.8x communication-to-compute ratio quantifies exactly how bottlenecked the system is. Even with the most optimistic compute assumptions, the GPUs spend nearly 80% of their time waiting for data. This shifts the optimization focus from compute efficiency to communication efficiency.

EP is not a silver bullet: The analysis provides rigorous evidence that expert parallelism does not automatically solve PCIe-bound inference. The specific model architecture and quantization scheme matter enormously. This is a cautionary result for practitioners considering EP deployments.

Framework for future analysis: The Python script in message 252 serves as a reusable template for evaluating parallelism strategies on new model-hardware combinations. It captures the essential parameters (hidden size, expert count, GPU count, bandwidth, FLOPs) and produces actionable comparisons.

The Broader Significance

Message 252 represents a pivotal moment in the conversation. Before this analysis, the team was considering a significant engineering effort to switch from TP8 to EP8, which would have required changes to the server configuration, potential code modifications in sglang, and extensive re-benchmarking. The analysis demonstrates that this effort would likely yield negligible performance gains—and might even degrade performance due to the additional complexity of all-to-all communication patterns on the virtualized PCIe topology.

More broadly, this message illustrates a pattern that recurs throughout the opencode session: the tension between intuitive reasoning and quantitative analysis. The intuition that EP should be faster on PCIe is sound for many models, but the specific parameters of GLM-5-NVFP4 (small hidden size, aggressive quantization, high expert count) invert the expected relationship. Only by building a detailed computational model and running the numbers could the team discover this counterintuitive result.

The message also demonstrates the value of the assistant's ability to execute code as part of its reasoning process. The Python script is not just a calculation tool—it is a thinking tool. By formalizing its mental model as executable code, the assistant can test its assumptions, explore edge cases, and arrive at conclusions that would be difficult to reach through verbal reasoning alone.

Conclusion: When the Numbers Speak

Message 252 is a testament to the power of quantitative reasoning in systems engineering. The assistant took a seemingly straightforward question about parallelism strategies and built a detailed analytical framework that revealed a counterintuitive truth: for this specific model on this specific hardware, expert parallelism offers no meaningful advantage over tensor parallelism.

The key insight—that the model's small hidden size makes communication volume nearly identical between the two strategies—could not have been discovered through intuition alone. It required careful modeling of FLOPs, communication patterns, bandwidth constraints, and batch effects. The assistant's willingness to question its own initial assumptions, formalize its reasoning as executable code, and present the results transparently is a model for how to approach complex engineering decisions.

In the end, the message does not just answer the user's question about expert parallelism. It provides a framework for thinking about distributed inference bottlenecks that will serve the team well as they continue to optimize this deployment. The numbers have spoken, and they tell a story that conventional wisdom would never have predicted.