The MoE Runner Gambit: Probing FP4 Kernel Efficiency on Blackwell GPUs

In the high-stakes world of deploying large language models on exotic hardware, progress often comes not in grand breakthroughs but in the meticulous substitution of one kernel implementation for another. Message 278 of this opencode session captures precisely such a moment: a carefully orchestrated server launch command that swaps out the MoE (Mixture-of-Experts) runner backend in an attempt to squeeze more performance from eight NVIDIA RTX PRO 6000 Blackwell GPUs running the GLM-5-NVFP4 model.

The Message: A Command-Line Microcosm

The message is a single, sprawling bash command:

ssh -f 10.1.230.175 'source ~/ml-env/bin/activate && export NCCL_IB_DISABLE=1 && export NCCL_P2P_LEVEL=PHB && export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 && export NCCL_MIN_NCHANNELS=8 && export OMP_NUM_THREADS=8 && export SAFETENSORS_FAST_GPU=1 && export CUDA_HOME=/usr/local/cuda-12.8 && nohup python3 -m sglang.launch_server --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 --reasoning-parser glm45 --tool-call-parser glm47 --trust-remote-code --tp 8 --mem-fraction-static 0.92 --max-running-requests 64 --kv-cache-dtype auto --quantization modelopt_fp4 --attention-backend flashinfer --fp8-gemm-backend cutlass --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --disable-custom-all-reduce --enable-flashinfer-allreduce-fusion --host 0.0.0.0 --port 8000 > ~/sglang-glm5.log 2>&1 &'

At first glance, this looks like any other complex server invocation. But each flag carries the weight of hours of debugging, profiling, and architectural reasoning. The command is the culmination of a diagnostic journey that began with NaN crashes during decode, moved through baseline benchmarking, and arrived at a nuanced understanding of where the true performance bottleneck lies.

The Reasoning: Why This Command Exists

To understand why this particular command was written, we must trace the reasoning that led to it. The session had already achieved a major milestone: deploying GLM-5-NVFP4 on 8x RTX PRO 6000 Blackwell GPUs with a working configuration that produced coherent output. The NaN crash during decode—a show-stopping bug that had plagued earlier attempts—had been resolved by selecting trtllm backends for both NSA prefill and decode (--nsa-prefill-backend trtllm --nsa-decode-backend trtllm). This was a significant victory, as the SM120 architecture (Blackwell) required special handling that the default attention backends did not provide.

With the model running stably, the assistant turned to performance tuning. Baseline benchmarks at 64 concurrent requests showed approximately 225 output tokens per second and 516 total tokens per second. The assistant then increased the static memory fraction to 0.92 and enabled CUDA graphs, which captured successfully without out-of-memory errors. Yet throughput remained stubbornly similar—around 210–247 output tokens per second.

The critical insight came from GPU profiling. During inference, all eight GPUs showed 100% utilization but drew only 328W out of a 600W TDP—barely 55% of their power budget. The SM clocks were at 2422 MHz (near the 2430 MHz maximum), and no thermal or power throttling was active. This combination of full utilization with low power draw is a telltale sign that the GPU's tensor cores are not being saturated. The workload is compute-bound, but the computations themselves are too small to fully occupy the hardware.

The root cause, the assistant reasoned, lies in the nature of single-token decode in Mixture-of-Experts models. Each decode step involves tiny matrix operations: a batch of one token multiplied through the hidden dimension for each activated expert. With tensor parallelism across 8 GPUs, each GPU handles only a fraction of each expert's weights. These small matrix multiplications cannot saturate the massive compute arrays of the Blackwell GPU, leading to high utilization (the GPU is "busy" launching kernels and moving data) but low power draw (the tensor cores are underutilized because the matrices are too small).

The MoE Runner Hypothesis

This is where the MoE runner backend enters the picture. The assistant had previously investigated the available MoE kernel code paths in SGLang and discovered five distinct FP4 execution backends. The current deployment used flashinfer_cutlass, which employs a CUTLASS-based FP4 fused kernel. But the assistant hypothesized that a different kernel implementation might achieve better hardware utilization for the specific dimensions of GLM-5's MoE layers.

The flashinfer_trtllm backend, which this command activates via --moe-runner-backend flashinfer_trtllm, uses a TRT-LLM-style FP4 kernel. TRT-LLM (TensorRT-LLM) is NVIDIA's inference optimization framework, and its kernel implementations are often finely tuned for specific GPU architectures. The hope was that this backend might use different block sizes, tile configurations, or scheduling strategies that better match the Blackwell SM120 architecture and the GLM-5 model dimensions (E=256 experts, N=256 intermediate size per expert at TP8).

The command also includes --enable-flashinfer-allreduce-fusion, which fuses the all-reduce communication with flashinfer operations. In a tensor-parallel setup, each GPU must communicate its partial results to all other GPUs after each MoE layer. Fusing this communication with computation can reduce kernel launch overhead and improve throughput—especially important when the computation itself is already small.

Environment Variables: The PCIe Battle

The NCCL environment variables in the command reveal another layer of the performance story. The system runs in a Proxmox VM (KVM/QEMU), and earlier investigation had confirmed that direct GPU peer-to-peer (P2P) transfers are not supported in this virtualized environment. All cross-GPU communication must go through host memory, creating a significant latency bottleneck.

Assumptions and Knowledge

The command makes several assumptions. First, it assumes that the flashinfer_trtllm MoE backend is compatible with the FP4 quantization format (modelopt_fp4) on SM120 GPUs. The prior investigation confirmed that this backend exists and supports FP4, but whether it works correctly with GLM-5's specific architecture is an open question. Second, it assumes that the NCCL environment variable settings are optimal for this virtualized environment—a configuration that was empirically derived rather than theoretically guaranteed.

The input knowledge required to understand this message is substantial. One must know:

The Output Knowledge Created

This message creates knowledge through its execution. If the server launches successfully and produces coherent output, it confirms that flashinfer_trtllm is a viable backend for this model and hardware combination. The subsequent benchmarks will reveal whether this backend improves throughput, power utilization, or both. If the server crashes or produces incorrect results, that too is valuable knowledge—it narrows the search space and may point to incompatibilities between the TRT-LLM-style kernel and the GLM-5 architecture.

More broadly, the message contributes to the growing understanding of how to deploy large MoE models on workstation Blackwell GPUs. The SM120 architecture is new, and the interaction between FP4 quantization, MoE routing, and tensor parallelism in virtualized environments is not well-documented. Each successful launch and benchmark run adds data points to this sparse landscape.

The Thinking Process

The reasoning visible in this message is one of systematic exploration. The assistant has already tried multiple configurations: different NSA backends, different memory fractions, CUDA graphs, and various MoE runner backends (flashinfer_cutlass, flashinfer_cutedsl). Each trial eliminated a hypothesis and narrowed the search. The flashinfer_trtllm backend is the next logical candidate because it represents a fundamentally different kernel implementation strategy.

Notably, the assistant did not jump directly to this configuration. It first investigated the prior Kimi K2 tuning work (message 271-272), discovered that those configs were for different model dimensions and FP8 rather than FP4, then systematically explored the available FP4 code paths (message 275). This methodical approach—investigate, understand, then experiment—characterizes the entire session.

The command also reveals what the assistant chose NOT to do. It did not attempt expert parallelism (EP), which had been evaluated and dismissed as unhelpful for this model due to the small hidden size and similar communication volume. It did not attempt --enable-torch-compile or --schedule-policy variants, which were recognized as unlikely to address the fundamental compute-bound bottleneck. It focused instead on the one lever that could improve kernel efficiency: the MoE runner backend.

Conclusion

Message 278 is a snapshot of the optimization process in action. It is not a final answer but a carefully reasoned experiment, built on hours of debugging, profiling, and analysis. The command encodes hypotheses about kernel efficiency, communication patterns, and hardware utilization. Its execution will produce new data that either validates or refutes those hypotheses, driving the next iteration of the optimization cycle.

In the broader narrative of deploying GLM-5-NVFP4 on Blackwell GPUs, this message represents a pivot from "making it work" to "making it fast." The NaN crash was resolved. The model produces coherent output. Now the question is: how close can we get to saturating those 600W TDPs?