The EP8 Breakthrough: Expert Parallelism Takes Flight on Blackwell GPUs

Introduction

In the relentless pursuit of inference throughput for the massive GLM-5-NVFP4 mixture-of-experts (MoE) model, a single message in an opencode coding session marks a pivotal turning point. Message [msg 1057] is deceptively brief—a few lines of observation followed by a bash command to wait for server startup—but it represents the culmination of a systematic optimization campaign spanning dozens of prior messages, and the first concrete validation that Expert Parallelism (EP) is operational on an 8-GPU Blackwell system. This message is the moment where theory meets practice: after ruling out communication-side optimizations like MSCCLPP and Single Batch Overlap as marginal at best, and after being blocked entirely on Piecewise CUDA Graphs due to framework incompatibilities, the assistant finally sees evidence that a fundamentally different parallelism strategy is alive and working.

The Optimization Journey: Context and Motivation

To understand why this message matters, one must appreciate the journey that led to it. The assistant had been tasked with deploying and optimizing the GLM-5-NVFP4 model—a 5-trillion-parameter MoE architecture quantized to FP4—on a system with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The core challenge was that the model's MoE layers route each token through only a subset of "expert" sub-networks, and on Blackwell's SM120 architecture, the per-expert matrix multiplications (GEMMs) are tiny, making them memory-bandwidth-bound rather than compute-bound. The 99 KB shared memory limit and absence of TMEM (Tensor Memory) on SM120 further constrained what optimizations were viable.

The assistant had methodically worked through a prioritized list of optimization strategies, documented as "improvement docs" (glb5improvement-xx.md) in a local research repo. The Tier 1 optimizations were:

  1. Piecewise CUDA Graphs — BLOCKED due to torch.compile(fullgraph=True) incompatibility with FlashInfer's FP4 JIT code, even after heroic patching efforts.
  2. MSCCLPP (Microsoft Collective Communication Library) — tested and found to yield only ~2% improvement over baseline, confirming that allreduce communication was not the primary bottleneck.
  3. Single Batch Overlap (SBO) — tested and found to be essentially identical to baseline, further ruling out communication-side gains.
  4. Expert Parallelism (EP8) — the remaining hope, which distributes different experts across different GPUs rather than sharding all experts across all GPUs. By the time we reach [msg 1057], the assistant has already compiled a comprehensive comparison table showing that MSCCLPP and SBO produced negligible improvements across all concurrency levels (1, 10, 256, 1024 concurrent requests). The conclusion was stark: "Communication optimizations (MSCCLPP, SBO) have negligible impact. The entire bottleneck is the MoE expert GEMMs being memory-bandwidth-bound due to small per-expert matrix sizes. This makes EP8 the most important optimization to test."

What the Message Reveals

The message itself consists of two parts: an observation and a command. The observation reads:

Interesting! Notice TP0 EP0, TP1 EP1 — Expert Parallelism is active! Each GPU now shows EP rank. Also notice memory: 59.50 GB per GPU vs 60.89 GB in TP8 mode. Slightly less memory per GPU because each GPU holds fewer (complete) experts instead of all (sharded) experts.

This is the first confirmation that the EP8 configuration is working correctly. The server logs show TP0 EP0, TP1 EP1, etc., indicating that each GPU has both a tensor parallelism (TP) rank and an expert parallelism (EP) rank. In pure TP8 mode, all 8 GPUs collaborate on every operation, with each GPU holding a shard of every expert's weights. In EP8 mode, each GPU holds a complete set of a subset of experts, and tokens are routed to the GPU that holds their assigned expert. This is a fundamentally different distribution strategy.

The assistant also notes the memory difference: 59.50 GB per GPU in EP8 mode versus 60.89 GB in TP8 mode. This ~1.4 GB reduction per GPU is because each GPU no longer needs to store shards of all experts; instead, it stores complete copies of fewer experts. The reduction is modest because the model's parameters are only part of the memory footprint—KV cache, activations, and framework overhead consume the rest.

The second part of the message is a bash command to wait for the model to finish loading, followed by the server log output showing the model checkpoint loading with EP ranks visible. This is a routine "wait and check" operation, but it confirms that the server hasn't crashed during initialization—a non-trivial achievement given the complexity of the configuration.

The Reasoning Behind EP8

The decision to test EP8 was driven by a deep understanding of where the bottleneck actually lies. The assistant's earlier analysis had shown that:

  1. Allreduce is not the bottleneck — MSCCLPP and SBO barely moved the needle, proving that communication between GPUs during allreduce operations was not the limiting factor.
  2. Per-expert GEMMs are the bottleneck — Each expert in the MoE layer performs a matrix multiplication that is small (because only a fraction of tokens route to each expert). On SM120, these small GEMMs are memory-bandwidth-bound: the GPU spends more time moving data from HBM to SRAM than actually computing.
  3. EP changes the compute-to-communication ratio — By assigning entire experts to individual GPUs, EP eliminates the need for allreduce on expert outputs (since each GPU computes its experts independently). However, it introduces an all-to-all communication step to route tokens to the correct GPU before expert computation, and another all-to-all to gather results afterward. The key insight was that if the allreduce was not the bottleneck, then the all-to-all overhead of EP might be acceptable—especially if EP allows each GPU to work on larger, more compute-efficient batches of tokens for its assigned experts. With EP8, each GPU handles only 1/8 of the experts, so the per-expert batch size is larger (all tokens going to that expert are routed to that single GPU), making the GEMMs more compute-efficient.

Assumptions and Potential Pitfalls

The assistant made several assumptions in pursuing EP8:

  1. That the flashinfer_cutlass MoE backend supports EP — The assistant verified this by reading the server source code (server_args.py), confirming that moe_a2a_backend="flashinfer" automatically sets ep_size=tp_size and that the flashinfer_cutlass runner backend allows ep_size of either 1 or tp_size.
  2. That the all-to-all overhead would be less than the allreduce overhead it replaces — This was an open question. The assistant had already proven that allreduce was not the bottleneck, but all-to-all is a different communication pattern that could be more expensive.
  3. That the model weights would fit in GPU memory with EP8 — The slightly lower memory footprint (59.50 GB vs 60.89 GB) suggested this was viable, but the margin was thin given that each GPU has 96 GB of HBM.
  4. That the SGLang framework's EP implementation is stable — The assistant was using a nightly build of SGLang, and EP was a relatively new feature. Crashes under load were a real possibility (and indeed, as the segment summary reveals, EP8 would later crash under moderate load at 256 concurrent requests).

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. EP8 is operational on Blackwell GPUs — The server launched successfully with EP topology, proving that the SGLang framework's EP implementation works on SM120 hardware.
  2. Memory savings from EP are modest — ~1.4 GB per GPU reduction, confirming that the model's parameter memory is only a fraction of total GPU memory usage.
  3. The EP configuration is stable at startup — The model loaded successfully across all 8 GPUs without crashes during initialization.
  4. EP rank assignment matches TP rank — Each GPU has matching TP and EP ranks (TP0 EP0, TP1 EP1, etc.), indicating a 1:1 mapping between tensor parallelism and expert parallelism groups.

The Thinking Process

The message reveals a clear, methodical thinking process. The assistant begins by observing and interpreting the server log output—noting the EP rank annotations and immediately drawing the correct conclusion that EP is active. This is followed by a comparative analysis of memory usage between EP8 and TP8 modes, with a correct explanation of why EP uses less memory.

The assistant then proceeds to verify stability by waiting for the model to finish loading. This is a critical step: before running benchmarks, one must confirm that the server is fully initialized and stable. The subsequent log output confirms that checkpoint loading is progressing normally.

What's notable is what the assistant doesn't do in this message: it doesn't jump to conclusions about performance. The observation is purely about functionality and memory—the assistant is carefully checking that the configuration is valid before proceeding to benchmark. This disciplined approach—verify functionality first, then measure performance—is a hallmark of rigorous engineering.

The Broader Narrative

This message sits at a critical inflection point in the optimization campaign. The assistant had systematically eliminated every other Tier 1 optimization, narrowing the field to EP8 as the last remaining hope for significant throughput gains. The successful EP8 launch, confirmed in this message, represents the first positive signal after a series of dead ends.

However, the story doesn't end here. As the segment summary reveals, EP8 would later prove to be 10–14% slower at low concurrency and would crash under moderate load (256 concurrent requests). The crash would be attributed to autotuner failures for the M256 tile and NCCL errors during all-to-all communication. This outcome—EP8 working at startup but failing under load—underscores the complexity of distributed inference systems, where initialization success does not guarantee production stability.

Nevertheless, the message stands as a testament to methodical optimization: each hypothesis was tested, each result was documented, and each dead end was accepted before moving to the next approach. The EP8 launch was not the victory the assistant had hoped for, but it was a necessary step in the scientific process of understanding where the true bottleneck lies.

Conclusion

Message [msg 1057] is a brief but significant milestone in a complex optimization journey. It captures the moment when Expert Parallelism first comes to life on an 8-GPU Blackwell system, providing the first concrete evidence that a fundamentally different parallelism strategy is viable. The message embodies the engineering virtues of careful observation, comparative analysis, and disciplined verification. While EP8 would ultimately fail to deliver the hoped-for gains, the knowledge gained from this experiment—that the bottleneck is truly in the per-expert GEMMs and not in communication patterns—would shape the direction of all subsequent optimization efforts. In the end, the most valuable outcome of this message was not the successful EP8 launch, but the confirmation that the assistant's bottleneck analysis was correct, setting the stage for deeper investigations into FP4 GEMM kernel efficiency on SM120.