The Moment of Truth: Benchmarking Expert Parallelism on Blackwell

In the high-stakes world of large language model inference optimization, few moments are as tense as the first benchmark of a newly deployed parallelization strategy. Message <msg id=1188> captures exactly such a moment: the assistant has just succeeded in launching an Expert Parallelism (EP8) configuration for the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, and is now running the critical comparison benchmarks that will determine whether weeks of effort—spanning driver installations, kernel rebuilds, memory tuning, and CUTLASS autotune debugging—have paid off.

The Road to EP8

To understand the significance of this message, one must appreciate the arduous journey that preceded it. The optimization campaign for GLM-5-NVFP4 on SM120 (Blackwell architecture) had been a methodical, evidence-driven process spanning multiple sessions. Earlier attempts at Expert Parallelism (EP8) had failed catastrophically—the server crashed under moderate load due to memory exhaustion. The assistant had diagnosed the root cause: the default memory configuration (--mem-fraction-static 0.9) left insufficient headroom for the all-to-all (A2A) communication buffers that EP requires.

The retry in this segment used a "memory-safe" configuration: --mem-fraction-static 0.75 --max-running-requests 512. This reduced the model's memory reservation from 90% to 75% of available GPU memory, leaving approximately 34 GB free per GPU for A2A buffers and other runtime allocations. The tradeoff was clear—less memory for KV cache meant lower maximum concurrency, but the hope was that EP's computational benefits would more than compensate.

The launch itself was dramatic. As shown in <msg id=1185>, the assistant initially detected a "CRASH DETECTED" signal—the CUTLASS MoE kernel was throwing runtime errors during warmup, attempting to use tile configurations like 128×256×128 that exceed SM120's 100 KB shared memory limit. But upon closer inspection in <msg id=1186>, the assistant discovered that the server had actually recovered: "EP8 is running! The 160 'errors' were just CUTLASS autotune failures for tiles that don't fit—the autotune handles this gracefully by falling back to working tiles." This resilience was itself a significant finding about the FlashInfer TRT-LLM MoE backend's fault tolerance.

The Message: Benchmarking Under Pressure

Message <msg id=1188> opens with the assistant analyzing the first results from EP8 at low concurrency:

EP8 at single stream: 9.79 tok/s, 99.02ms TPOT — that's actually close to baseline's 10.36 tok/s, 95.14ms. Only ~5% slower TPOT vs baseline at concurrency 1. At 2-stream: 18.28 tok/s, 103.90ms. Now the key question — does EP8 pull ahead at higher concurrency? Let me run 10, 64, 256, 512:

The tone is measured but the stakes are clear. The assistant is comparing EP8 against the TP8 (Tensor Parallelism with 8 GPUs) baseline that was established just moments earlier in <msg id=1171> and <msg id=1172>. The single-stream comparison is revealing: EP8 is slightly slower than TP8 at concurrency 1 (9.79 vs 10.36 tok/s, with TPOT of 99.02ms vs 95.14ms). This makes intuitive sense—Expert Parallelism introduces communication overhead from the all-to-all operations that distribute tokens across experts, and at single-stream there's no batching benefit to offset this cost.

The 2-stream results tell a similar story: EP8 achieves 18.28 tok/s with 103.90ms TPOT, while the baseline TP8 achieved 19.29 tok/s with 99.21ms TPOT (from <msg id=1172>). Again, EP8 is approximately 5% behind.

But the assistant correctly identifies that the real test is at higher concurrency. Expert Parallelism's theoretical advantage is that it distributes the MoE (Mixture-of-Experts) computation across GPUs, reducing the per-GPU compute burden for each token. At low concurrency, this benefit is drowned out by communication overhead. At high concurrency, when many tokens compete for GPU compute resources, EP should shine by allowing more tokens to be processed simultaneously through expert computation.

The Benchmark Results

The assistant then executes a comprehensive benchmark sweep across concurrency levels 10, 64, 256, and 512. The results begin to flow in:

At N=10: EP8 delivers 38.24 tok/s output throughput, with 101.09ms TPOT and 99.08ms ITL. The peak output reaches 88.00 tok/s. The TTFT (Time to First Token) is 1056.39ms—significantly higher than what TP8 would achieve, likely due to the A2A communication during prefill.

At N=64: Output throughput climbs to 175.66 tok/s, with peak at 395.00 tok/s. Total throughput reaches 360.1 tok/s (the message truncates here, but the trend is visible).

The message is cut off mid-output, but the trajectory is clear. The assistant is methodically building a performance profile across the concurrency spectrum, searching for the crossover point where EP8's advantages overcome its overheads.

The Reasoning and Decision-Making Process

What makes this message particularly interesting is the explicit reasoning structure. The assistant doesn't just run benchmarks blindly—it articulates a clear hypothesis: "Now the key question — does EP8 pull ahead at higher concurrency?" This framing reveals the underlying mental model:

  1. Low-concurrency regime: Communication overhead dominates → EP8 underperforms TP8
  2. High-concurrency regime: Compute bottleneck dominates → EP8's distributed expert computation should win The assistant is testing for a crossover point—the concurrency level at which EP8's benefits begin to outweigh its costs. This is classic optimization methodology: identify the regime where a technique should work, then benchmark to confirm or refute the hypothesis. The decision to benchmark at N=10, 64, 256, 512 is also strategic. These values span the concurrency range from light load (10 concurrent requests) to heavy load (512 concurrent requests), covering both the regime where TP8 is expected to lead and where EP8 should excel. The choice of 512 as the maximum reflects the memory constraint—with --max-running-requests 512, the server can handle up to 512 concurrent requests, so benchmarking at exactly that limit tests the system at its designed capacity.

Assumptions and Potential Pitfalls

Several assumptions underpin this benchmarking approach. First, the assistant assumes that the random dataset (128 input tokens, 128 output tokens) is representative of real-world workloads. This is a standard benchmarking practice, but it may not capture the non-uniform expert routing patterns that OEA (Opportunistic Expert Activation) was designed to exploit. The assistant's earlier OEA experiments (in prior chunks) showed near-zero average improvement on random data, suggesting that the benchmark dataset may be masking real-world benefits.

Second, the assistant assumes that the TP8 baseline from the previous benchmarks is directly comparable. However, the baseline benchmarks were run with different server parameters—specifically, the baseline used --mem-fraction-static 0.9 and --max-running-requests 1024, while EP8 uses 0.75 and 512. The assistant is careful to note the memory differences, but the comparison is still apples-to-oranges in terms of configuration.

Third, there's an implicit assumption that the CUTLASS autotune failures (160 of them logged in the EP8 server) don't affect performance. The assistant observed that the server recovered and ran, but the fallback tiles may be suboptimal. The autotune's "graceful" failure means EP8 is running with whatever tiles survived the shared memory constraint, which may not be the fastest possible configuration.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. EP8 single-stream performance on SM120: 9.79 tok/s with 99.02ms TPOT, establishing a baseline for EP efficiency at low concurrency.
  2. EP8 dual-stream performance: 18.28 tok/s with 103.90ms TPOT, showing near-linear scaling from 1 to 2 streams.
  3. EP8 performance at moderate concurrency (N=10, 64): 38.24 tok/s and 175.66 tok/s respectively, providing data points for the scaling curve.
  4. The ~5% overhead of EP vs TP at low concurrency: A quantified comparison that helps inform deployment decisions.
  5. Validation that EP8 can run stably on SM120 with memory-safe config: Despite 160 CUTLASS autotune errors during warmup, the server recovered and served requests correctly.

The Broader Context

This message sits at a critical juncture in the optimization campaign. The assistant has already tried and documented numerous approaches: Piecewise CUDA Graphs (blocked by SGLang limitations), MSCCLPP allreduce (minimal gains), Single Batch Overlap (modest improvement), OEA (near-zero gain on random data), and now EP8. Each approach has been systematically tested, benchmarked against a clean baseline, and either adopted or ruled out.

The user's next message (<msg id=1189>) pivots to a different question: "For this model on this machine, gen5 pcie, 2 sockets, what's the maximum possible perf in this model, in theory, for single stream?" This suggests that the EP8 results may not have been as transformative as hoped, prompting a shift from empirical optimization to theoretical analysis. The assistant's thorough benchmarking in <msg id=1188> provides the empirical data needed to ground that theoretical analysis.

Conclusion

Message <msg id=1188> exemplifies the scientific method applied to systems optimization: form a hypothesis, design an experiment, run controlled benchmarks, and let the data speak. The assistant's careful comparison of EP8 against TP8 across multiple concurrency levels, its explicit reasoning about where EP should excel, and its methodical execution of the benchmark suite all reflect a disciplined approach to performance engineering. Whether EP8 ultimately proves superior or not, the data generated in this message will inform the next iteration of the optimization campaign—and that, in the end, is the true value of rigorous benchmarking.