When Intuition Meets Memory Physics: A User's Expert Replication Proposal for GLM-5 on 8 Blackwell GPUs

In the middle of an intensive performance-tuning session for the massive GLM-5-NVFP4 model running on eight RTX PRO 6000 Blackwell GPUs, the user poses a concise but technically loaded question:

"Can we have all experts on all gpus and dispatch each expert activation to one of the 8 gpus?"

This single sentence, appearing as message [msg 247] in the conversation, represents a moment of creative engineering intuition colliding with hard physical constraints. To understand why this question matters and what it reveals, we must reconstruct the context that led to it, the assumptions embedded within it, and the knowledge it ultimately produced.

The Context: A System Bottlenecked by PCIe

By the time the user asks this question, the assistant and user have already spent considerable effort deploying and debugging the GLM-5 model — a 744-billion-parameter Mixture-of-Experts (MoE) model with 256 routed experts per layer, quantized to NVIDIA's experimental NVFP4 format. The deployment environment is unusual: eight RTX PRO 6000 Blackwell GPUs (each with 96GB of VRAM) connected exclusively through PCIe Gen5, with no NVLink interconnect. Furthermore, the system runs as a KVM/QEMU virtual machine under Proxmox, adding virtualization overhead to every cross-GPU communication.

The assistant has already established baseline throughput: approximately 225 output tokens per second with 64 concurrent requests, and a single-stream latency of roughly 11 tokens per second. The bottleneck has been identified as communication-bound — every decode step requires all-reduce operations across all eight GPUs for each of the model's 78 layers, traversing PCIe at roughly 32 GB/s bidirectional instead of the ~900 GB/s that NVLink would provide.

The Expert Parallelism Discussion That Preceded It

Immediately before the user's question, the assistant had been investigating expert parallelism (EP) as a potential solution ([msg 244]). The reasoning was straightforward: standard tensor parallelism (TP8) requires all-reduce after every layer, which is latency-bound on PCIe. Expert parallelism distributes the 256 experts across GPUs (32 experts per GPU) and uses all-to-all communication instead of all-reduce, potentially reducing cross-GPU traffic.

However, the assistant's analysis revealed a counterintuitive result: for this particular model, EP8 and TP8 had nearly identical communication volume — approximately 107 MB per decode step for TP8 versus 113 MB for EP8+TP8 ([msg 251]). The reason was the model's relatively small hidden size of 6144. With such a small hidden dimension, the all-to-all dispatch and combine operations carried essentially the same byte count as the all-reduce they replaced. The assistant concluded that "the raw bytes are actually similar" and that EP offered at best a marginal improvement.

The User's Alternative: A Different Kind of Parallelism

This is where the user's question enters. The user proposes a fundamentally different strategy: instead of sharding experts across GPUs (standard EP) or sharding each expert's computation across GPUs (standard TP), replicate every expert on every GPU, then dispatch individual expert activations to specific GPUs for computation.

This is a creative insight. The user's mental model appears to be: if the bottleneck is all-reduce latency (which requires all GPUs to synchronize after every layer), then eliminate the all-reduce by having each GPU compute entire experts independently. In this scheme, when a token activates 8 experts, each of those 8 expert computations could be dispatched to a different GPU. GPU 0 computes expert A for all tokens that need it, GPU 1 computes expert B, and so on. The results are gathered with a simple all-to-all scatter/gather rather than a synchronous all-reduce.

The advantage would be genuine: expert computation becomes embarrassingly parallel, with no synchronization required during the compute phase. Each GPU works independently on its assigned experts, and only the final results need to be communicated.

The Assumption That Couldn't Hold

The user's proposal rests on a critical assumption: that replicating all 256 experts across all 8 GPUs is feasible within the 96GB memory budget per GPU. This assumption is reasonable on the surface — after all, the model fits within 488GB total across 8 GPUs with TP8, so perhaps each GPU could hold a complete copy?

The assistant's response ([msg 248]) immediately identifies the problem. Through a detailed weight analysis ([msg 250]), the assistant calculates that the MoE expert weights alone consume approximately 453GB in NVFP4 format. Replicating all 453GB of expert weights on each of the 8 GPUs would require 453GB per GPU — nearly five times the available 96GB. Even with aggressive memory optimization, full replication is physically impossible.

The assistant's follow-up analysis ([msg 250]) systematically evaluates the memory arithmetic:

The Deeper Insight: Why the User's Intuition Was Still Valuable

While the specific proposal was infeasible due to memory constraints, the user's intuition pointed at a real and important problem. The assistant's subsequent analysis revealed that the core issue was not the parallelism strategy per se, but rather the fundamental communication bottleneck imposed by the virtualized PCIe environment.

The assistant went on to investigate whether a hybrid approach — TP2 for attention layers combined with expert replication — could work, but even that required 228 GB per GPU ([msg 250]). The analysis ultimately confirmed that standard EP8 was the only feasible parallelism strategy, and even that offered marginal benefit due to the small hidden size.

More importantly, the user's question prompted the assistant to perform a detailed communication analysis that quantified exactly why the system was bottlenecked. The analysis showed that the compute-to-communication ratio was heavily skewed: compute time per token was approximately 0.18 microseconds per expert activation, while all-reduce time was approximately 0.86 microseconds — a 4.8× ratio favoring communication as the bottleneck ([msg 252]). This quantified what the team had suspected but hadn't measured.

Output Knowledge and Lessons

The user's question produced several concrete outcomes:

  1. A precise memory budget analysis for the GLM-5-NVFP4 model, establishing that expert weights dominate at 453GB and cannot be replicated across GPUs.
  2. Confirmation that the communication bottleneck is structural, not merely a matter of parallelism strategy. The small hidden size (6144) means that all-to-all and all-reduce have similar byte counts, limiting the benefit of EP.
  3. A deeper understanding of the virtualization overhead, which the assistant subsequently investigated by checking PCIe peer-to-peer status and running bandwidth tests (<msg id=253-254>), revealing that the KVM environment forced all cross-GPU transfers through host memory with dramatically reduced small-message bandwidth (~1 GB/s for typical all-reduce message sizes).
  4. The recognition that the real bottleneck is latency, not bandwidth — the virtualization layer adds significant latency to every cross-GPU synchronization point, and no amount of parallelism strategy optimization can fully compensate for that infrastructure limitation.

Conclusion

The user's question at [msg 247] is a textbook example of productive engineering intuition. It proposes a creative solution to a clearly identified problem (all-reduce latency on PCIe), makes a reasonable but ultimately incorrect assumption about memory feasibility, and in doing so, prompts a deeper investigation that reveals the true nature of the bottleneck. The exchange demonstrates how even "wrong" ideas can be valuable in the engineering process — they force precise quantification of constraints, clarify the boundaries of what's possible, and often lead to more fundamental insights about the system's actual limitations.

The message also illustrates a key principle in distributed inference: parallelism strategies cannot be evaluated in isolation. The interaction between model architecture (hidden size, number of experts, quantization format), hardware topology (PCIe vs NVLink, virtualized vs bare metal), and memory budgets creates a complex optimization landscape where intuition alone is rarely sufficient — precise measurement and analysis are essential.