The Memory Wall: Why You Cannot Replicate 453 GB of MoE Experts Across 8 GPUs

In the high-stakes world of large language model deployment, every millisecond of latency and every gigabyte of memory is accounted for. When you have eight NVIDIA RTX PRO 6000 Blackwell GPUs—each with 96 GB of VRAM—connected only by PCIe without the luxury of NVLink, every architectural decision becomes a careful calculus of memory versus communication. This is the story of a single message in an opencode coding session that cut through speculation with cold, hard numbers, answering a clever proposal with an undeniable arithmetic truth.

The message in question, <msg id=250>, is a turning point in a longer debugging and optimization session. The team had successfully deployed the GLM-5-NVFP4 model—a massive 744-billion-parameter Mixture-of-Experts (MoE) language model—across eight Blackwell GPUs using SGLang with tensor parallelism of 8 (TP8). After resolving a critical NaN crash during decode by selecting the correct NSA attention backends (trtllm), they had established baseline throughput: approximately 225 output tokens per second under concurrent load, but only about 11 tokens per second for a single request. This abysmal single-stream performance pointed to a clear bottleneck: the PCIe interconnect. Every decode step required an all-reduce operation across all eight GPUs for each of the model's 61 MoE layers, and PCIe Gen5, while fast at ~32 GB/s bidirectional, is no match for NVLink's ~900 GB/s.

The Spark of an Idea

The user, clearly knowledgeable about distributed inference architectures, posed a provocative question in <msg id=243>: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" This led the assistant to explore Expert Parallelism (EP), which distributes the 256 MoE experts across GPUs rather than splitting every computation across all GPUs. The key insight is that EP replaces the expensive all-reduce with a cheaper all-to-all communication pattern—only the 8 activated experts per token need to communicate, not the full hidden state.

But the user went further. In <msg id=247>, they proposed an even more intriguing idea: "Can we have all experts on all gpus and dispatch each expert activation to one of the 8 gpus?" This is a fundamentally different approach. Instead of partitioning experts across GPUs (standard EP), replicate every single expert on every GPU, and then route each token's expert computations to whichever GPU happens to be free. This would eliminate cross-GPU communication for expert computation entirely—each GPU would independently compute its assigned expert activations using locally stored weights. The only communication needed would be the initial dispatch of token hidden states and the final collection of results.

On paper, it's a beautiful idea. If each GPU holds all 256 experts, then any GPU can compute any expert for any token. You could load-balance across GPUs dynamically, potentially achieving near-perfect utilization. The all-reduce bottleneck vanishes because there's nothing to reduce—each GPU already has the full model.

The Cold Arithmetic

The assistant's initial response in <msg id=248> was cautious, suspecting a memory problem but lacking precise numbers. An attempt to read the model configuration failed with a KeyError because the config file used different field names than expected. After dumping the full config in <msg id=249> to discover the correct architecture parameters, the assistant was ready for the definitive calculation.

This brings us to the subject message, <msg id=250>. The assistant writes a comprehensive Python script that computes the exact memory footprint of every component of the GLM-5-NVFP4 model. The script is remarkable for its thoroughness—it doesn't just compute the total model size, but breaks down every architectural component with precise parameter counts and memory estimates.

Let us examine what this script calculates. The model has a hidden size of 6144, with 256 experts per MoE layer, each with an intermediate size of 2048. There are 78 layers total, with the first 3 being dense (standard MLP) and the remaining 75 being MoE layers. Each expert consists of three matrices—gate projection, up projection, and down projection—each mapping between the hidden size (6144) and the intermediate size (2048). That gives 3 × 6144 × 2048 = 37.7 million parameters per expert. In NVFP4 format (4-bit floating point with 2-byte scales per 16 parameters, totaling approximately 4.5 bits per parameter effective), each expert occupies about 24 MB.

The script then computes the attention mechanism, which uses Multi-head Latent Attention (MLA)—a parameter-efficient attention variant. With a query low-rank projection of 2048, a key-value low-rank projection of 512, 64 attention heads, and a head dimension of 256, the attention per layer comes to about 0.33 GB in BF16. The shared expert (a single non-routed expert per MoE layer) adds 75 MB per layer in BF16. The three dense layers at the start of the model use the larger intermediate size of 12288 (from the config's intermediate_size field), contributing about 0.453 GB each. Finally, the embedding layer with a vocabulary of 154,880 tokens adds about 1.90 GB.

The totals are stark:

The Verdict on Replication

The script then evaluates the user's proposal. If every GPU must hold all 453 GB of MoE experts, plus its share of attention and other components, the per-GPU memory requirement becomes:

What This Message Reveals

This message is far more than a simple "no" to a user's question. It is a masterclass in data-driven engineering analysis. Several aspects deserve close examination.

The reasoning process visible in the script shows a methodical decomposition of the model into its constituent parts. The assistant doesn't guess or approximate—it reads the actual architecture parameters from the session's earlier exploration (hidden size 6144, MoE intermediate 2048, 256 experts, 78 layers, 3 dense layers, MLA attention dimensions) and computes exact byte counts. It accounts for the nuances of NVFP4 quantization (4 bits per parameter plus 2 bytes of scale per 16 parameters, yielding ~4.5 effective bits), the difference between BF16 and FP4 storage, and the distinction between routed experts, shared experts, and dense layers.

The assumptions embedded in the calculation are worth noting. The assistant assumes that the shared expert uses BF16 precision (not NVFP4), which is reasonable given that shared experts are typically kept at higher precision. It assumes the attention mechanism uses the MLA variant with the specific low-rank dimensions discovered in the config dump. It assumes the embedding layer is stored in BF16. These are educated inferences based on the model configuration, and they produce a total (487.7 GB) that aligns well with the model's advertised size of approximately 744B total parameters with 40B active.

The output knowledge created by this message is substantial. Before this calculation, the team had only a rough sense that the model was "big" and that full replication "probably wouldn't fit." After this message, they have precise numbers: 453 GB of MoE experts, 24 MB per expert, 6.04 GB per layer of experts, 25.7 GB of attention, and a definitive answer that replication requires 457 GB per GPU—4.76× the available memory. This knowledge directly informs the next steps: the team must either use standard EP (distributing experts across GPUs, which requires installing the deep_ep library and testing SM120 compatibility) or continue with TP8 while exploring other optimizations like CUDA graphs, memory fraction tuning, or batch size adjustments.

The Broader Significance

This message illuminates a fundamental tension in deploying large MoE models on multi-GPU systems. The model's total parameter count (744B) is distributed across 256 experts per layer, creating a massive memory footprint that must be spread across GPUs. The choice of parallelism strategy—tensor parallel, expert parallel, or data parallel—determines how communication overhead trades off against memory pressure.

TP8 minimizes memory per GPU (each GPU holds 1/8 of every tensor) but maximizes communication (all-reduce at every layer). EP8 distributes experts across GPUs, replacing all-reduce with all-to-all but requiring each GPU to hold a full set of attention weights. Full expert replication (the user's proposal) would eliminate communication for MoE layers entirely but requires each GPU to hold the entire expert weight matrix—which, as we now know, is 453 GB.

The calculation also reveals something about the model's architecture. The MoE experts dominate the memory footprint at 93% of total weights (453 GB out of 487.7 GB). Attention, despite using the parameter-efficient MLA variant, still consumes 25.7 GB. The three dense layers at the start of the model, which use the larger intermediate size of 12288, contribute only 1.36 GB. This extreme imbalance—where experts are the overwhelming majority of parameters—is characteristic of modern MoE models and explains why expert parallelism is such an active area of research and development.

Input Knowledge Required

To fully understand this message, one needs familiarity with several concepts: Mixture-of-Experts architectures and how routed experts differ from dense layers; tensor parallelism and how it splits weights across GPUs; NVFP4 quantization and its effective bit width including scale factors; Multi-head Latent Attention (MLA) and its low-rank projections; and the practical memory constraints of GPU inference (model weights, KV cache, activations, and framework overhead all compete for VRAM). The message also assumes knowledge of the GLM-5 model family, including its 78-layer structure with 3 initial dense layers, 256 experts, and the specific hidden and intermediate dimensions.

Mistakes and Limitations

One potential limitation of the analysis is that it assumes the shared expert uses BF16 precision. If the shared expert is also quantized to NVFP4, the memory footprint would be slightly lower (about 1.4 GB instead of 5.7 GB for all shared experts). However, this would not change the conclusion—the MoE experts at 453 GB remain the dominant term.

Another subtle point: the calculation assumes that all 256 experts must be replicated on every GPU. In practice, one might use a hybrid approach where the most frequently activated experts are replicated on multiple GPUs while rarer experts are stored on a single GPU, using --ep-num-redundant-experts in SGLang. This would reduce the per-GPU memory requirement while still providing some of the communication benefits of replication. The assistant had mentioned this possibility in <msg id=248> but did not explore it numerically in this message.

Conclusion

Message <msg id=250> is a testament to the value of precise, data-driven reasoning in systems engineering. When faced with a creative architectural proposal, the assistant did not rely on intuition or hand-waving. It wrote a detailed memory model, computed exact byte counts for every component, evaluated multiple scenarios, and delivered a definitive answer supported by arithmetic. The user's idea—while clever—was simply impossible given the physical constraints of the hardware.

This message also serves as a microcosm of the larger challenge in deploying large language models: the tension between communication efficiency and memory efficiency. Every parallelism strategy makes a trade-off, and the optimal choice depends on the specific hardware topology, model architecture, and workload characteristics. For this team, with eight PCIe-connected Blackwell GPUs and a 744B MoE model, the path forward lies not in replicating experts but in carefully tuning the expert parallelism configuration, installing the necessary communication libraries, and measuring the real-world throughput impact.

The arithmetic is unforgiving, but it is also liberating. With precise numbers in hand, the team can stop wondering about hypothetical architectures and focus on what is actually achievable. That is the true value of this message: not just an answer, but a methodology for finding answers.