The EP8 Disappointment: When Expert Parallelism Fails to Deliver

Introduction

In the high-stakes world of large language model inference optimization, few techniques hold as much promise as Expert Parallelism (EP). For Mixture-of-Experts (MoE) architectures like GLM-5-NVFP4, EP distributes different experts across GPUs, allowing each GPU to compute larger, more efficient matrix multiplications on its subset of experts rather than performing the same sharded computation across all GPUs. The theory is elegant: by increasing per-expert matrix sizes, you improve arithmetic intensity and GPU utilization. But theory and practice often diverge, and message 1059 in this opencode session captures that divergence in vivid detail.

This message is the moment when the assistant, after successfully deploying an EP8 (Expert Parallelism with 8 GPUs) configuration for the GLM-5-NVFP4 model, receives the first benchmark results and confronts a sobering reality: EP8 is 10–14% slower than the baseline TP8 (Tensor Parallelism) configuration at low concurrency. Worse, when the assistant pushes to higher concurrency where EP should theoretically excel, the server crashes entirely. This single message encapsulates the entire arc of an optimization hypothesis being formulated, tested, and beginning to unravel—all within a few lines of analysis and two bash commands.

Context: The Road to EP8

To understand message 1059, one must appreciate the journey that led to it. The assistant had spent the preceding hours systematically testing a series of optimizations for the GLM-5-NVFP4 model running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The core bottleneck had been identified: the model's MoE layers use small per-expert matrix sizes, which are memory-bandwidth-bound on SM120's limited shared memory (99KB) and lack of TMEM support. The FP4 quantized GEMM kernels simply cannot achieve high throughput when each expert processes only a handful of tokens.

Earlier in segment 8, the assistant had already ruled out several Tier 1 optimizations:

The Subject Message: Analysis and Action

Message 1059 opens with the assistant processing the low-concurrency benchmark results that were launched in the previous message ([msg 1058]). The tone shifts immediately from the optimism of "EP8 server is UP and running!" to the more measured "Hmm, EP8 at low concurrency is slower than baseline."

The assistant presents two data points:

The Tool Calls: Testing the Hypothesis

The assistant issues two parallel bash commands to test this hypothesis:

  1. A benchmark with 256 prompts at near-infinite request rate (999 req/s)
  2. A benchmark with 1024 prompts at the same request rate Both use the same benchmark script (sglang.bench_serving) with identical parameters to the baseline tests, ensuring comparability. The assistant dispatches both commands simultaneously, following the pattern established throughout the session of running multiple concurrency levels in parallel to minimize waiting time. The 256-prompt benchmark appears to start normally, with the familiar HF Hub warning about missing HF_TOKEN and the benchmark_args output. But the 1024-prompt benchmark crashes immediately with a Python traceback:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/sglang/python/sglang/bench_serving.py", line 3415, in <module>
    run_benchmark(args)
  File "/root/sglang/python/sglang/bench_serving.py", line 2956, in run_benchmark
    return asyncio.run(
           ^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ...

The traceback is truncated in the message (the ... at the end), but the critical information is that the benchmark itself crashes, not just returns poor results. This is the first indication that something is seriously wrong with the EP8 server under load.

What Happens Next: The Crash Investigation

The following messages ([msg 1060] and [msg 1061]) reveal the full extent of the failure. The assistant checks the server log and finds that the server process died mid-stream, causing a ClientPayloadError: Response payload is not completed. The server log reveals the root cause: the FlashInfer autotuner failed when trying to profile the M256 tile size for the grouped GEMM kernel, with an error from TensorRT-LLM: "Failed to initialize cutlass TMA WS grouped gemm." This is a kernel compilation failure specific to the SM120 architecture—the M256 tile configuration is not supported, but the autotuner attempts it anyway and crashes the entire server process.

Assumptions and Their Validity

The assistant operated under several key assumptions in this message:

Assumption 1: EP8 would improve high-concurrency throughput. This was the central hypothesis driving the test. The assumption was well-reasoned: larger per-expert matrices should improve arithmetic intensity. However, the assumption implicitly discounted the all-to-all communication cost at scale. Even at high concurrency, the all-to-all overhead might remain significant, especially given that the GPUs are connected through a Proxmox VM with potential PCIe topology limitations (as explored extensively in segments 3-5 of the session). The crash prevented a full test of this assumption.

Assumption 2: The EP8 server was stable enough to handle high concurrency. The assistant had only verified that the server started and served low-concurrency requests. There was no intermediate load test or stress test before jumping to 256 and 1024 concurrency. This was a reasonable risk—the server had passed the startup smoke test—but it turned out to be incorrect. The autotuner failure at M256 suggests that certain kernel configurations are only triggered under higher load when larger batch sizes require different tile sizes.

Assumption 3: The benchmark infrastructure would handle errors gracefully. The 1024-prompt benchmark crashed rather than returning partial results or an error message. This is a limitation of the benchmark script itself—it doesn't handle server disconnection gracefully during a run. The assistant likely assumed that if the server failed, the benchmark would report an error rather than crash with a traceback.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Understanding of MoE architectures: How mixture-of-experts layers work, what expert parallelism is, and how it differs from tensor parallelism.
  2. Knowledge of all-to-all communication: The concept of distributing tokens to different GPUs based on expert routing decisions, and the associated communication overhead.
  3. Familiarity with the GLM-5-NVFP4 model: A quantized (FP4) MoE model with many small experts, making it particularly sensitive to per-expert matrix sizes.
  4. Awareness of the SM120 architecture: NVIDIA Blackwell's compute capability, with its 99KB shared memory limit and lack of TMEM support, which constrains FP4 GEMM kernel efficiency.
  5. Understanding of SGLang's parallelism modes: How --tp-size, --ep-size, and --moe-a2a-backend interact to determine the parallelism strategy.
  6. Context from the preceding optimization tests: That MSCCLPP and SBO had already been ruled out as transformative, making EP the last major untested optimization.

Output Knowledge Created

This message produces several valuable insights:

  1. EP8 regresses low-concurrency performance by 10-14% for this model and hardware configuration. This is a concrete, quantitative result that can inform deployment decisions.
  2. The all-to-all overhead dominates at low concurrency, providing a clear explanation for the regression. This is a general principle that applies to any MoE model considering EP.
  3. EP8 server stability is questionable under load, as evidenced by the crash at 1024 concurrency. The autotuner failure for M256 tile size on SM120 is a specific, actionable bug to report or work around.
  4. The baseline TP8 configuration remains the best option for low-to-moderate concurrency workloads, at least until the EP stability issues are resolved.
  5. The crash signature (autotuner failure for grouped GEMM at M256) provides a specific diagnostic direction for future investigation. The assistant later discovers this is due to SM120 not supporting certain CUTLASS TMA (Tensor Memory Accelerator) operations required for the grouped GEMM kernel at that tile size.

The Thinking Process: A Window into Scientific Reasoning

What makes message 1059 particularly interesting is the visible reasoning process. The assistant does not simply report results; it interprets them. The phrase "This makes sense at low concurrency" reveals an active cognitive process of connecting observations to underlying mechanisms. The assistant is building a mental model of the system's behavior and using that model to predict where EP should work better.

The conditional structure of the reasoning is noteworthy: "EP is worse at low concurrency, but it should be better at high concurrency." This is textbook hypothesis-driven experimentation. The assistant has a theory (all-to-all overhead vs. GEMM efficiency trade-off), derives a prediction (EP improves with concurrency), and designs a test (high-concurrency benchmarks). The fact that the test fails due to a server crash is itself informative—it reveals a stability boundary that wasn't previously known.

The assistant also demonstrates good experimental discipline. It runs the same benchmark suite (same input length, output length, request rate, model, tokenizer) as the baseline tests, ensuring comparability. The only variable changed is the server configuration (EP8 vs TP8). This controlled approach makes the 10-14% regression a reliable finding.

Mistakes and Incorrect Assumptions

While the assistant's reasoning is sound, there are subtle issues worth examining:

Over-reliance on the EP hypothesis. The assistant had invested significant effort in setting up EP8—reading server_args.py, checking restrictions, creating launch scripts, waiting for model loading. This investment may have created a confirmation bias, where the assistant was primed to expect EP to work. The low-concurrency results are correctly interpreted as a regression, but the assistant immediately pivots to "high concurrency where EP should shine" rather than considering that EP might fundamentally not work well for this model on this hardware.

Insufficient stress testing. The assistant jumped from concurrency 10 (which worked) to concurrency 256 and 1024 (which crashed). A more cautious approach might have tested concurrency 32, 64, and 128 first to find the stability boundary. However, given the time pressure of the session and the fact that baseline tests had run successfully at all four concurrency levels, this was an understandable shortcut.

The assumption that the autotuner would handle all tile sizes. The crash ultimately stems from the FlashInfer autotuner attempting an unsupported tile size (M256) on SM120. The assistant assumed that the autotuner would either skip unsupported configurations or handle failures gracefully. Instead, the failure crashes the entire server process. This is a bug in FlashInfer's autotuner, but the assistant's assumption that it was safe to run without pre-testing all tile configurations proved incorrect.

Conclusion

Message 1059 is a pivotal moment in the optimization journey for GLM-5-NVFP4 inference. It represents the collision of a well-reasoned hypothesis (Expert Parallelism will improve throughput) with the messy reality of distributed systems (the all-to-all overhead is significant, and the server crashes under load). The assistant handles this collision with scientific poise: acknowledging the regression, explaining it, formulating a conditional prediction, and testing it. The crash of the 1024-concurrency benchmark is not the end of the story—it becomes the starting point for the next investigation into autotuner stability on SM120.

In the broader narrative of the session, this message marks the point where the assistant realizes that no single optimization will unlock dramatic gains. The core bottleneck—small per-expert GEMMs on a shared-memory-limited architecture—is fundamental to the hardware. Each optimization (MSCCLPP, SBO, EP8) either fails to address the bottleneck or introduces new problems. The message thus serves as a turning point, shifting the investigation from "which optimization works?" to "what is the fundamental limit of this hardware, and can we work around it?"