The Moment of Commitment: Launching Expert Parallelism on GLM-5-NVFP4

ssh root@10.1.230.174 'nohup bash /root/run_tp8_ep8.sh > /root/sglang-server-ep8.log 2>&1 & echo "Server starting with PID: $!"'
Server starting with PID: 95407

At first glance, message [msg 1055] appears trivial — a single bash command dispatched over SSH to start a server, followed by a one-line confirmation of a process ID. But this message is anything but trivial. It represents the culmination of a systematic, multi-session optimization campaign spanning dozens of tool calls, multiple failed experiments, and a growing understanding of the fundamental hardware constraints imposed by NVIDIA's Blackwell architecture (SM120). The command launches an SGLang inference server configured with Expert Parallelism (EP8) for the GLM-5-NVFP4 model, and it is the single most consequential optimization attempt the assistant has made in this entire segment.

The Road to EP8: A Story of Systematic Elimination

To understand why this message matters, one must trace the path that led here. The assistant had been methodically working through a prioritized list of optimizations for the GLM-5-NVFP4 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The baseline performance had already been established at four concurrency levels (1, 10, 256, and 1024 requests), and the assistant had written eleven improvement documents (glb5improvement-01 through glb5improvement-11) to catalog potential approaches.

The Tier 1 optimizations had been tested one by one, and each had been found wanting:

• Piecewise CUDA Graphs (Tier 1.1) were blocked entirely. The torch.compile(fullgraph=True) requirement proved incompatible with FlashInfer's FP4 JIT code. Even after patching get_cuda_version to avoid subprocess calls and adding @torch.compiler.disable to fp4_quantize, the fullgraph constraint prevented graph breaks — a hard blocker that the assistant documented and moved on from.
• MSCCLPP (Tier 1.2) was tested and yielded only ~2% improvement across all concurrency levels. The assistant's own summary was blunt: "negligible." The peak output token rate at 1024 concurrency showed a 20.6% spike, but this was not sustained, and the overall throughput barely budged.
• Single Batch Overlap (SBO) (Tier 1.3) was tested in combination with MSCCLPP and produced results "essentially identical to baseline." The comparison table the assistant compiled showed output tok/s values of 9.14 vs 9.17 (concurrency 1), 38.22 vs 38.03 (concurrency 10), 357.57 vs 352.79 (concurrency 256), and 1539.19 vs 1520.55 (concurrency 1024) — differences well within noise. These results collectively confirmed a critical insight: the bottleneck was not in communication or allreduce operations, but entirely in the MoE expert GEMMs. The small per-expert matrix sizes on the GLM-5-NVFP4 model were memory-bandwidth-bound on the SM120 architecture, which has only 99KB of shared memory and lacks Tensor Memory (TMEM). No amount of communication optimization could fix a compute-bound bottleneck.

Why EP8 Was Different

Expert Parallelism was the most promising optimization because it directly addressed the identified bottleneck. By distributing experts across GPUs (EP8 means 8-way expert parallelism), each GPU would handle fewer experts, potentially increasing the per-expert matrix sizes and improving GEMM utilization. This was fundamentally different from MSCCLPP and SBO, which optimized the communication fabric between GPUs — a path the assistant had now conclusively ruled out as transformative.

The assistant had done its homework before launching this command. In messages [msg 1048] through [msg 1053], it investigated the SGLang codebase to understand the EP8 configuration requirements. It checked the server_args.py file to verify that --moe-a2a-backend flashinfer would automatically set ep_size to tp_size (8), and that the flashinfer_cutlass MoE runner backend supported this configuration. It confirmed the constraints: ep_size must be either 1 or equal to tp_size, and the quantization must be modelopt_fp4, modelopt_fp8, or bfloat16 — all satisfied by the current setup.

The launch script (run_tp8_ep8.sh) was carefully constructed in [msg 1053]. It preserved all the successful parameters from previous runs: TP8 (tensor parallelism across 8 GPUs), FlashInfer attention backend, CUTLASS FP8 GEMM backend, TRTLLM for NSA decode and prefill, MSCCLPP enabled, 16 continuous decode steps, and 2048 max running requests. The only new addition was --moe-a2a-backend flashinfer, which triggered the EP8 topology.

The Message Itself: A Study in Deliberate Action

The command in [msg 1055] is deceptively simple. It uses nohup to detach the server from the terminal, redirects output to a log file, and echoes the PID for tracking. But the real weight of this message lies in what it does not contain: hesitation, uncertainty, or branching. By this point, the assistant had already:

1. Ruled out three other Tier 1 optimizations through empirical testing
2. Verified the codebase support for EP8
3. Constructed the launch script with all necessary parameters
4. Killed any previous server processes (in [msg 1054])
5. Waited 3 seconds for cleanup The PID 95407 is the first data point of a new experiment. Everything that follows — the server startup logs, the benchmark results at four concurrency levels, the eventual crash under moderate load — will be traced back to this single moment of commitment.

Assumptions and Risks

The assistant made several assumptions when launching this command:

• That EP8 would improve throughput. This was the core hypothesis, grounded in the analysis that small per-expert GEMMs were the bottleneck. If the bottleneck were elsewhere (e.g., attention computation, KV cache management), EP8 could actually hurt performance by adding all-to-all communication overhead.
• That the flashinfer_cutlass MoE backend would handle EP8 correctly. While the codebase supported this configuration, the assistant had not verified that the FP4 quantization path was fully functional with expert parallelism on SM120. The FlashInfer FP4 JIT code had already caused problems with torch.compile.
• That the server would stay stable under load. The assistant had no way to know at this point whether EP8 would introduce new failure modes — OOM errors, NCCL communication failures, or autotuner crashes.
• That the hardware topology would support efficient all-to-all communication. With eight GPUs potentially spread across multiple PCIe root complexes (as earlier segments had discovered), the all-to-all operations required for expert parallelism could be bottlenecked by PCIe bandwidth rather than NVLink.

What This Message Creates

The launch of the EP8 server creates several forms of knowledge:

1. A testable hypothesis. The assistant can now run benchmarks at concurrency 1, 10, 256, and 1024 and compare against the established baseline.
2. A new log file. /root/sglang-server-ep8.log will contain the server startup sequence, any warnings or errors during initialization, and the runtime behavior under load.
3. A decision point. If EP8 succeeds, it becomes the new baseline for further optimization. If it fails or underperforms, the assistant must pivot to more aggressive approaches like L2 pinning, persistent kernels, or custom FP4 GEMM implementations.

The Thinking Process

The assistant's reasoning in the lead-up to this message reveals a disciplined experimental methodology. Each optimization was tested in isolation, with controlled variables (same model, same tokenizer, same benchmark parameters). Results were compiled into comparison tables and analyzed quantitatively. When an optimization showed negligible improvement, it was documented and set aside without agonizing.

The progression from "Piecewise CUDA Graphs blocked" to "MSCCLPP ~2%" to "SBO identical to baseline" to "EP8 is the most important optimization to test" shows a clear Bayesian updating process. Each negative result strengthened the conviction that the bottleneck was in the GEMMs, not the communication fabric. By the time the assistant reached [msg 1055], the decision to try EP8 was not a guess — it was the logical conclusion of a systematic elimination process.

What Came After

As the segment summary reveals, EP8 did launch successfully and showed the correct topology with slightly lower per-GPU memory usage. However, it was 10–14% slower at low concurrency and crashed under moderate load (256 concurrent requests), with server logs showing autotuner failures for the M256 tile and NCCL errors. The crash was likely due to OOM or NCCL communication failures during all-to-all operations.

But that outcome does not diminish the significance of [msg 1055]. In scientific experimentation, the decision to run an experiment — to commit resources and accept the risk of failure — is often more important than the result itself. This message captures that moment of commitment: the point at which hypothesis becomes test, and analysis becomes action.