The Final Restart: Settling on a Working Configuration After Systematic MoE Backend Exploration

In the middle of a grueling debugging session spanning dozens of messages, message [msg 305] arrives as a quiet pivot point. It is a single bash command — a server restart — but it represents the culmination of an exhaustive investigation into MoE runner backends, autotuning infrastructure, and the performance characteristics of the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message is deceptively simple: an SSH command that kills the existing SGLang server process and launches a new one with a carefully curated set of flags. But understanding why this particular configuration was chosen, and what the assistant learned to arrive at it, requires tracing the reasoning across the preceding twenty-plus messages.

The Context: A Session of Systematic Elimination

To appreciate message [msg 305], we must first understand what came before it. The assistant had been deployed on a machine with eight RTX PRO 6000 Blackwell GPUs (SM120 architecture) running inside a Proxmox VM. The core task was to deploy the GLM-5-NVFP4 model — a large Mixture-of-Experts model with 256 experts and 78 layers — using SGLang, and to achieve acceptable throughput.

The session had already overcome significant hurdles. Earlier, the assistant had resolved a critical NaN crash during decode by discovering that the trtllm NSA (Nucleus Sampling Attention) backends worked on SM120 GPUs while others did not. It had established baseline throughput of approximately 225 output tokens per second with 64 concurrent requests. It had tuned server parameters like --mem-fraction-static (increased to 0.92) and enabled CUDA graphs successfully. And it had tested various MoE runner backends — flashinfer_cutlass, flashinfer_cutedsl — finding them all within a similar performance band.

But the assistant was not satisfied. The GPUs showed 100% utilization but only 55% power draw, suggesting the bottleneck was not compute but something else — likely communication overhead from the many small kernel launches required by the MoE architecture (78 layers × multiple kernels per layer × 256 experts to route). The user had raised the possibility of expert parallelism to improve PCIe-bound performance, but the assistant had analyzed memory constraints and concluded it was infeasible. The user had then questioned whether cross-GPU latency from the Proxmox VM environment was a bottleneck, and the assistant had confirmed this: the system was a KVM/QEMU VM with no direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory.

The Immediate Preceding Messages: Exhausting the MoE Runner Options

In the messages immediately before [msg 305], the assistant had been exploring the FlashInfer autotuner infrastructure. It discovered that FlashInfer has SM120-specific CUTLASS kernels for FP4 GEMM operations, and that there are tuning config files for B200 and GB200 GPUs but not for the RTX PRO 6000 Blackwell Server Edition. When the config file doesn't exist, the autotuner falls back to default configurations — suboptimal but functional.

The assistant then tested the flashinfer_cutedsl backend (a CuteDSL-based MoE runner) as a final alternative. The results came in at [msg 302]: approximately 206 output tokens per second and 473 total tokens per second — comparable to the 195–225 tok/s range seen with flashinfer_cutlass. A single-stream latency test at [msg 303] showed approximately 11 output tokens per second, unchanged from previous runs.

This was the moment of insight captured in [msg 304]: "Same ~11 tok/s single-stream. The MoE runner backend doesn't change things — the bottleneck is elsewhere." The assistant had now tested every available MoE runner backend — flashinfer_cutlass, flashinfer_cutedsl, and indirectly flashinfer_trtllm (which had been ruled out earlier as SM100-only) — and found them all delivering essentially identical throughput. The MoE kernel implementation was not the limiting factor.

Message 305: The Command Itself

With the investigation complete, the assistant's next action is to restart the server with the best-known configuration:

ssh -f 10.1.230.175 'source ~/ml-env/bin/activate &&
  export NCCL_IB_DISABLE=1 &&
  export NCCL_P2P_LEVEL=PHB &&
  export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 &&
  export NCCL_MIN_NCHANNELS=8 &&
  export OMP_NUM_THREADS=8 &&
  export SAFETENSORS_FAST_GPU=1 &&
  export CUDA_HOME=/usr/local/cuda-12.8 &&
  nohup python3 -m sglang.launch_server
    --model lukealonso/GLM-5-NVFP4
    --served-model-name glm-5
    --reasoning-parser glm45
    --tool-call-parser glm47
    --trust-remote-code
    --tp 8
    --mem-fraction-static 0.92
    --max-running-requests 64
    --kv-cache-dtype auto
    --quantization modelopt_fp4
    --attention-backend flashinfer
    --fp8-gemm-backend cutlass
    --nsa-decode-backend trtllm
    --nsa-prefill-backend trtllm
    --moe-runner-backend flashinfer_cutlass
    --disable-custom-all-reduce
    --enable-flashinfer-allreduce-fusion
    --host 0.0.0.0
    --port 8000
    > ~/sglang-glm5.log 2>&1 &'

Every flag in this command tells a story of a decision made during the session.

Deconstructing the Decisions Embedded in the Flags

Environment variables: The NCCL flags (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=PHB, NCCL_ALLOC_P2P_NET_LL_BUFFERS=1, NCCL_MIN_NCHANNELS=8) configure the communication library for the virtualized environment. NCCL_IB_DISABLE=1 disables InfiniBand (not available in this VM). NCCL_P2P_LEVEL=PHB sets peer-to-peer to "parent hub bus" level — appropriate for a system without direct GPU peer-to-peer (NVLink), which the assistant had confirmed was the case. OMP_NUM_THREADS=8 limits OpenMP threads. CUDA_HOME=/usr/local/cuda-12.8 points to the secondary CUDA toolkit that was installed earlier to resolve flash-attn build issues.

--moe-runner-backend flashinfer_cutlass: This is the key decision. After testing flashinfer_cutedsl and finding identical performance, the assistant returns to flashinfer_cutlass. This is not because it's faster — it's because it's the best understood working configuration. The flashinfer_cutlass path uses SM120-specific CUTLASS fused MoE kernels that were confirmed to exist and function correctly. The assistant had also discovered that this path does not use the FlashInfer autotuner (which is only for the flashinfer_trtllm path), meaning there are no hidden tuning knobs to adjust.

--nsa-decode-backend trtllm and --nsa-prefill-backend trtllm: These flags were the critical discovery that resolved the NaN crash. Earlier in the session, the server had crashed during decode with NaN values when using other NSA backends. The trtllm backend was the only one that produced coherent output on SM120 GPUs.

--disable-custom-all-reduce and --enable-flashinfer-allreduce-fusion: These flags reflect the assistant's understanding of the communication bottleneck. Custom all-reduce is disabled because the virtualized environment lacks direct GPU peer-to-peer support — custom all-reduce typically relies on NVLink or P2P capabilities that don't exist here. FlashInfer all-reduce fusion is enabled as an alternative optimization.

--mem-fraction-static 0.92: Increased from the default to maximize GPU memory utilization without causing out-of-memory errors. The assistant had tested this and confirmed CUDA graphs captured successfully at this level.

--tp 8: Tensor parallelism across all 8 GPUs. This was necessary because the model's 453GB of MoE expert parameters far exceeds the 96GB per-GPU memory, making full replication impossible.

What the Assistant Assumed — Correctly and Incorrectly

The assistant made several assumptions in this message, some explicit and some implicit:

Correct assumption: That flashinfer_cutlass is a stable, working MoE runner backend for this model on SM120 GPUs. This was validated by earlier successful runs.

Correct assumption: That the NCCL and environment variable configuration is appropriate for the virtualized environment. The assistant had previously confirmed the system is a KVM/QEMU VM without direct GPU peer-to-peer support.

Implicit assumption: That the bottleneck is elsewhere and cannot be fixed by further MoE backend tuning. This was supported by the empirical evidence that all tested backends produced similar throughput.

Implicit assumption: That recording final results with this configuration would provide a meaningful baseline for future optimization work. This is a reasonable research methodology — establish a stable baseline before attempting deeper changes.

Potential incorrect assumption: That no other MoE runner backend or combination of flags could improve throughput. The assistant had tested the available options but did not explore more exotic configurations like expert parallelism (which was ruled out due to memory constraints) or custom kernel modifications.

Input Knowledge Required to Understand This Message

A reader needs considerable context to fully grasp what this message accomplishes:

  1. The model architecture: GLM-5-NVFP4 is a Mixture-of-Experts model with 256 experts, 78 layers, and FP4 quantization. Understanding MoE routing and tensor parallelism is essential.
  2. The hardware environment: Eight RTX PRO 6000 Blackwell GPUs (SM120 architecture) running inside a Proxmox VM without direct GPU peer-to-peer connectivity. The "SM120" designation refers to the Blackwell compute architecture.
  3. The software stack: SGLang as the serving framework, FlashInfer for attention and MoE kernels, NCCL for communication, and the specific quantization format (modelopt_fp4).
  4. The debugging history: The NaN crash during decode, the discovery that trtllm NSA backends work on SM120, the autotuner investigation showing no config file exists for this GPU, and the MoE backend comparison showing all options produce similar throughput.
  5. The communication constraints: NCCL configuration for virtualized environments, the implications of NCCL_P2P_LEVEL=PHB, and why --disable-custom-all-reduce is necessary.

Output Knowledge Created by This Message

This message produces several forms of knowledge:

  1. A reproducible configuration: The exact command serves as a recipe for deploying GLM-5-NVFP4 on similar hardware. Anyone with eight RTX PRO 6000 GPUs in a virtualized environment can use this as a starting point.
  2. A baseline for comparison: By settling on this configuration and recording throughput metrics, the assistant creates a reference point against which future optimizations can be measured.
  3. Documentation of working flags: The combination of --nsa-decode-backend trtllm, --nsa-prefill-backend trtllm, and --moe-runner-backend flashinfer_cutlass is now known to work on SM120 GPUs — knowledge that was not obvious before this session.
  4. Negative knowledge: The message implicitly documents that flashinfer_cutedsl offers no advantage over flashinfer_cutlass, that the FlashInfer autotuner doesn't apply to the CUTLASS path, and that MoE backend selection is not the primary bottleneck.

The Thinking Process Visible in This Message

Although the message itself contains only the bash command, the reasoning behind it is visible in the surrounding context. The assistant's thought process follows a clear arc:

  1. Hypothesis generation: "Maybe the MoE runner backend matters for throughput" — leading to testing flashinfer_cutlass, flashinfer_cutedsl, and investigating the autotuner.
  2. Hypothesis testing: Each backend is benchmarked with the same methodology (64 concurrent requests, 256 input/output tokens), producing comparable results.
  3. Hypothesis falsification: All backends produce similar throughput (~200-225 tok/s output), falsifying the hypothesis that MoE backend selection is the bottleneck.
  4. Reframing: The assistant shifts focus to other potential bottlenecks — GPU utilization patterns, communication overhead, virtualization effects.
  5. Settling on a baseline: Rather than continuing to search for optimizations in a space that has been thoroughly explored, the assistant returns to the best-known working configuration to establish a final baseline. This is a mature debugging methodology: systematically eliminate variables, recognize when a line of inquiry is exhausted, and document the result before moving on.

The Broader Significance

Message [msg 305] is interesting precisely because it is not a breakthrough. It is not the message where the assistant discovers a 2x speedup or resolves a critical bug. It is the message where the assistant accepts the current state of knowledge and prepares to move forward. In many debugging narratives, we focus on the moments of discovery — the "aha" moments. But the equally important skill is knowing when to stop searching in one direction and consolidate what has been learned.

The message also highlights the challenges of deploying large MoE models in virtualized environments. The RTX PRO 6000 Blackwell GPUs are powerful compute devices, but without direct GPU peer-to-peer connectivity, their collective performance is severely constrained by PCIe bandwidth and host memory latency. The assistant's investigation revealed that the MoE kernel implementation was not the bottleneck — the communication infrastructure was. This is a finding that could only be reached by systematically ruling out other possibilities.

In the end, message [msg 305] represents a moment of intellectual closure. The assistant has tested the MoE runner backends, understood the autotuner infrastructure, confirmed the virtualization limitations, and is now returning to a known-good configuration to record final results. It is not the end of the optimization journey — but it is the end of one chapter, and the beginning of the next.