The Moment of Promise: Launching Expert Parallelism for GLM-5-NVFP4 on Blackwell GPUs

In the systematic pursuit of inference throughput for the GLM-5-NVFP4 mixture-of-experts (MoE) model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 1058] captures a pivotal moment of transition. After methodically testing and ruling out three Tier 1 optimizations—Piecewise CUDA Graphs (blocked by torch.compile incompatibility with FlashInfer's FP4 JIT code), MSCCLPP allreduce (yielding only ~2% improvement), and Single Batch Overlap (also ~2%)—the assistant finally deploys what it believes to be the most promising optimization: Expert Parallelism with EP8 topology. This message is the moment the EP8 server comes online and benchmarks begin, carrying the weight of accumulated evidence and the hope that this approach might finally unlock meaningful throughput gains.

The Reasoning Behind EP8

The decision to pursue Expert Parallelism was not arbitrary—it was the culmination of a rigorous diagnostic process spanning multiple sessions. Throughout segments 6 and 7, the assistant had established that the GLM-5-NVFP4 model's performance bottleneck was not in communication (allreduce, NCCL) but in the MoE expert GEMMs themselves. On Blackwell's SM120 architecture, with its 99KB shared memory limit and lack of TMEM support, the small per-expert matrix multiplications were memory-bandwidth-bound rather than compute-bound. Each GPU held sharded experts (via tensor parallelism), meaning every expert computation was split across all 8 GPUs, resulting in tiny matrix dimensions that could not fully utilize the GPU's compute capability.

Expert Parallelism offered a fundamentally different approach: instead of sharding each expert across all GPUs, EP distributes the experts themselves across GPUs. Each GPU holds a subset of complete experts, and the all-to-all communication pattern routes tokens to the GPUs that hold the relevant experts. This means each GPU computes fewer experts but at larger matrix sizes—potentially shifting the computation from memory-bandwidth-bound toward compute-bound territory. The assistant's analysis, documented in improvement documents 02–11 written earlier in this chunk, identified EP as the single optimization with transformative potential.

The Preparation: Verifying SGLang's EP Support

Before launching the EP8 server, the assistant conducted thorough due diligence. Messages [msg 1048] through [msg 1052] show a careful investigation of SGLang's source code to verify that Expert Parallelism was supported for this model configuration. The assistant grepped through /root/sglang/python/sglang/srt/server_args.py to examine the ep_size parameter, the moe_a2a_backend choices, and the validation logic at lines 2115–2180. It discovered that:

The Launch and Initial Observations

Message [msg 1054] and [msg 1055] show the server being launched. The assistant first kills any existing SGLang processes, then starts the EP8 server with the command:

python3 -u -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 \
  --reasoning-parser glm45 --tool-call-parser glm47 \
  --trust-remote-code --tp-size 8 --mem-fraction-static 0.92 \
  --max-running-requests 2048 --kv-cache-dtype auto \
  --quantization modelopt_fp4 --attention-backend flashinfer \
  --fp8-gemm-backend cutlass --nsa-decode-backend trtllm \
  --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass \
  --moe-a2a-backend flashinfer \
  --disable-cuda-graph --disable-radix-cache \
  --enable-mscclpp \
  --num-continuous-decode-steps 16 \
  --host 0.0.0.0 --port 8000

The critical addition compared to previous configurations is --moe-a2a-backend flashinfer, which triggers the EP8 topology. The server log in [msg 1056] shows the model loading with safetensors checkpoint shards, and in [msg 1057] the assistant makes a key observation: the log lines now show TP0 EP0, TP1 EP1, etc.—confirmation that Expert Parallelism is active. Each GPU has both a tensor parallelism rank and an expert parallelism rank, meaning the 8 GPUs are organized as a TP8+EP8 configuration (effectively 8-way tensor parallelism combined with 8-way expert parallelism).

The assistant also notes a subtle but important detail: memory usage dropped from 60.89 GB per GPU (in pure TP8 mode) to 59.50 GB per GPU. This makes sense because each GPU now holds fewer complete experts rather than sharded fractions of all experts. The freed memory translates into increased KV cache capacity: 517,632 tokens versus 490,880 tokens in the baseline—a 5.4% increase that could improve throughput at high concurrency by reducing preemption.

Message 1058: The Benchmark Dispatch

Message [msg 1058] itself is concise but action-packed. The assistant confirms the EP8 server is operational and immediately dispatches two benchmark commands in parallel:

  1. A single-prompt benchmark (--num-prompts 1 --request-rate 999) to measure baseline latency
  2. A 10-prompt benchmark (--num-prompts 10 --request-rate 999) to measure low-concurrency throughput The assistant's tone is notably optimistic: "EP8 server is UP and running!" The exclamation mark and the observation about increased KV cache capacity ("It even has slightly more KV cache capacity") convey a sense of anticipation. After days of marginal improvements (2% from MSCCLPP, 2% from SBO, a blocked CUDA graphs approach), EP8 represents the first genuinely different optimization strategy. The decision to start with low-concurrency benchmarks (1 and 10 requests) is methodologically sound. The assistant knows from the analysis in [msg 1047] that EP8's all-to-all communication overhead will hurt at low concurrency—when each expert processes very few tokens, the cost of routing tokens between GPUs dominates. But establishing the full profile requires measuring all concurrency levels. The assistant plans to follow up with 256 and 1024 concurrency benchmarks, where EP8's benefits should materialize.

Assumptions Embedded in This Message

Several assumptions underpin this message, some explicit and some implicit:

That EP8 is compatible with the full configuration stack. The assistant verified the code paths but did not test end-to-end before launching. The FP4 quantization, FlashInfer Cutlass MoE backend, MSCCLPP, and TRTLLM decode backends all need to coexist with EP8. The server starting successfully validates this assumption for the load phase, but runtime behavior remains untested.

That the all-to-all overhead will be offset at high concurrency. This is the core hypothesis driving the EP8 experiment. The assistant's earlier analysis concluded that small per-expert GEMMs are the bottleneck. EP8 makes each expert computation larger (fewer experts per GPU, each processing more tokens), but introduces communication overhead for the all-to-all routing. The assumption is that at sufficient concurrency, the compute benefit outweighs the communication cost.

That the server will remain stable under load. The EP8 configuration is more complex than pure TP8, involving additional NCCL communication patterns (all-to-all versus allreduce). The assistant implicitly assumes the server will handle the full benchmark suite without crashing.

That the benchmark methodology remains valid. The assistant reuses the same bench_serving script with identical parameters (random 128-in/128-out sequences, request rate 999) to ensure comparability with the baseline and previous optimization tests. This assumes that EP8 does not change the model's behavior in ways that invalidate the benchmark (e.g., different tokenization, different output distribution).

The Knowledge Flow

Input knowledge required to understand this message includes: understanding of MoE model architecture and the distinction between tensor parallelism (sharding each expert's weights across GPUs) and expert parallelism (distributing complete experts across GPUs); familiarity with SGLang's server configuration flags; awareness of the Blackwell SM120 hardware constraints (shared memory limits, FP4 GEMM characteristics); and knowledge of the previous optimization attempts and their results.

Output knowledge created by this message includes: the confirmation that EP8 is functional for GLM-5-NVFP4 with the flashinfer_cutlass MoE backend; the observation that EP8 reduces per-GPU memory usage and increases KV cache capacity; and the benchmark results at concurrency 1 and 10 (which, as revealed in subsequent messages, show EP8 is 10–14% slower than baseline at low concurrency).

The Broader Narrative

This message sits at a critical inflection point in the optimization journey. The assistant has systematically eliminated communication-side optimizations and is now testing the one approach that addresses the root cause: the inefficiency of small per-expert GEMMs on SM120. The message captures the optimism of a new experiment before the results come in—a moment of scientific anticipation.

What makes this message particularly interesting is what it reveals about the assistant's methodology. Rather than jumping to EP8 immediately, the assistant first ruled out simpler optimizations (CUDA graphs, MSCCLPP, SBO), established rigorous baselines at four concurrency levels, documented each finding in improvement documents, and only then moved to the more complex EP configuration. This systematic approach—hypothesize, test, measure, document, pivot—is the hallmark of effective performance engineering.

The subsequent messages ([msg 1059] onward) reveal that EP8 was 10–14% slower at low concurrency and crashed under moderate load (256 concurrent requests) with autotuner failures and NCCL errors. The promise of this message was not fulfilled—at least not in this iteration. But the methodology ensured that the failure was informative: it confirmed that the all-to-all overhead dominates at low concurrency, and it revealed stability issues that might be addressable with tuning. The assistant would go on to investigate the crash, adjust parameters, and continue iterating.

In the end, message [msg 1058] stands as a testament to the scientific approach to performance optimization: form a hypothesis based on evidence, implement the experiment, measure rigorously, and let the data speak—whether the news is good or bad.