The Pivot Point: From Reusing Configs to Understanding FP4 MoE Code Paths

In the middle of an intensive debugging session deploying the GLM-5-NVFP4 model on eight RTX PRO 6000 Blackwell GPUs, a single assistant message at index 275 marks a critical inflection point. The message is deceptively short — a few paragraphs of analysis followed by a task invocation — but it represents a fundamental shift in strategy. After hours of iterating through server parameters, attention backends, and benchmarking configurations, the assistant pauses, synthesizes the profiling data, and makes a crucial decision: instead of trying to adapt tuning configurations from a prior project, it will investigate the FP4-specific MoE kernel code paths from scratch. This decision, rooted in a careful reading of both the logs and the GPU utilization metrics, sets the stage for the next phase of the work.

The Context: A Long Debugging Journey

To understand why message 275 matters, we must first appreciate the journey that led to it. The conversation spans a multi-session effort to deploy GLM-5-NVFP4 — a 744-billion-parameter Mixture-of-Experts model — on a workstation equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. The earlier segments (segment 0 and segment 1) covered the grueling environment setup: installing NVIDIA drivers and CUDA Toolkit 13.1 on Ubuntu 24.04, resolving flash-attn build issues by reducing parallel compilation jobs from 128 to 20, and eventually getting the model to serve through SGLang.

But the deployment was plagued by a persistent NaN crash during decode — the model would generate coherent text for a few tokens, then collapse into numerical instability. The breakthrough came in segment 2 when the assistant discovered that using --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm resolved the crash on SM120 GPUs. With the model finally serving, the team turned to performance tuning.

The benchmarks showed disappointing throughput: approximately 11 tokens per second for single-stream requests and around 200–236 tokens per second for batched requests across 64 concurrent users. The initial assumption was that PCIe bandwidth was the bottleneck — after all, eight GPUs communicating across PCIe for every layer of a 78-layer model would naturally be communication-bound. But when the user questioned whether PCIe was truly the limiter, the assistant investigated and found that PCIe Gen5 x16 links were running at 64 GB/s capacity, with current throughput measuring in the hundreds of kilobytes per second — essentially idle. This was not a communication bottleneck.

The Critical Discovery: 100% Utilization at 55% Power

The profiling data from message 267 revealed something remarkable: all eight GPUs were running at 100% utilization with 52–53% memory bandwidth utilization, drawing approximately 325–330 watts each — but the RTX PRO 6000 Blackwell GPUs have a 600-watt TDP. The GPUs were only using 55% of their power budget, yet they reported being fully utilized. There was no thermal throttling, no hardware slowdown, and the SM clocks were running at near-max boost (2422 MHz out of 2430 MHz maximum).

This combination of metrics tells a specific story. When a GPU reports 100% utilization but draws only half its power budget, it means the compute units are busy but not doing heavy work. The tensor cores — the specialized hardware for matrix multiplication — are being underutilized. The assistant correctly diagnosed this: the kernels are running but limited by small matrix dimensions and kernel overhead per expert dispatch. In a Mixture-of-Experts model during decode, each token activates only a small subset of experts, and with batch sizes of one during single-stream inference, the matrix operations are tiny. The GPU spends more time launching kernels and moving data between dispatch units than actually computing.

The Mistaken Assumption and Its Correction

The immediate trigger for message 275 was the user's comment in message 270: "./ did some weird tuning configs, use an expert to figure out what that was exactly @README.md." The assistant dutifully read the README.md and discovered a directory of MoE tuning configurations — JSON files specifying kernel block sizes and tile dimensions — that had been generated for a prior project deploying Kimi K2 on the same hardware. These configs were created using a tuning_fused_moe_triton.py script that benchmarked thousands of kernel configurations to find optimal settings for the RTX PRO 6000 GPUs.

The natural instinct would be to try these configs with GLM-5-NVFP4. But the assistant paused and checked the sglang server logs, discovering something crucial: the current server was using flashinfer_cutlass as the MoE runner backend, which uses a CUTLASS-based FP4 kernel path, not the Triton-based fused MoE kernels that the tuning configs were designed for. The prior tuning work was for FP8 quantization with a completely different code path.

This is the moment captured in message 275. The assistant writes: "The log doesn't mention anything about MoE kernel configs. The current runner is flashinfer_cutlass which uses a CUTLASS-based FP4 MoE kernel path (not Triton). The prior tuning was for the triton MoE runner backend with FP8 — different code path entirely."

This is a critical correction of what could have been a costly detour. Without this analysis, the team might have spent hours trying to apply incompatible tuning configurations, or worse, switched to a Triton-based MoE runner that might not support NVFP4 quantization at all.

The Decision: Investigate FP4-Specific Code Paths

Having ruled out reusing the existing configs, the assistant faced a fork in the road. The GPUs were showing signs of inefficient kernel execution — 100% utilization at 55% power — which suggested the MoE kernels themselves were suboptimal. The question was: what could be done about it?

The assistant could have tried:

  1. Switching to a different MoE runner backend and hoping for better performance
  2. Running the tuning script to generate new configs for the current backend
  3. Manually adjusting kernel parameters based on heuristics
  4. Investigating what MoE runner backends are actually available for NVFP4 quantization The assistant chose option 4, and this is the core decision in message 275. Rather than making assumptions about which backend might be better, the assistant decided to first understand the landscape of available options. The task spawned in this message asks: "I need to understand the MoE runner code paths in sglang for NVFP4 (modelopt_fp4) quantization." It instructs the subagent to SSH into the remote machine, examine the sglang source code, and identify which backends support FP4, how they differ, and what the current flashinfer_cutlass backend actually does.

The Thinking Process: Reading Between the Lines

The reasoning visible in message 275 reveals a methodical analytical process. The assistant starts by stating what it has learned (the log doesn't mention MoE kernel configs), then interprets that finding (the current runner uses a different code path), then connects that to the profiling data (100% utilization at 55% power means tensor cores aren't saturated), and finally formulates a hypothesis (the kernels are limited by small matrix dimensions and kernel overhead per expert dispatch).

What's particularly impressive is the assistant's ability to synthesize information from multiple sources. The profiling data (message 267), the server logs (message 274), the README and config files (messages 271–272), and the user's request (message 270) are all integrated into a coherent picture. The assistant doesn't just report findings — it connects them into a narrative that explains why the current performance is suboptimal.

The language is measured and precise. The assistant uses phrases like "the key insight from the profiling is" and "that means" to signal its reasoning. It doesn't jump to conclusions — it acknowledges uncertainty with "Let me look at what MoE runner backends are actually available for NVFP4 and whether there's something better." This is the voice of an investigator gathering evidence before making a recommendation.

Input Knowledge Required

To fully understand message 275, the reader needs to know:

  1. The hardware configuration: Eight RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5, each with 96 GB of memory and a 600W TDP.
  2. The model architecture: GLM-5-NVFP4 is a 744B-parameter Mixture-of-Experts model with 256 experts, using NVFP4 (NVIDIA FP4) quantization.
  3. The serving software: SGLang, with its various MoE runner backends (flashinfer_cutlass, triton, etc.).
  4. The prior tuning work: A previous project deploying Kimi K2 on the same hardware had generated MoE kernel tuning configurations using a Triton-based tuning script.
  5. The profiling data: GPU utilization at 100%, power draw at 55% of TDP, SM clocks at near-max boost.
  6. The server log contents: The current MoE runner backend is flashinfer_cutlass.

Output Knowledge Created

Message 275 produces several important pieces of knowledge:

  1. The existing tuning configs are incompatible: The Triton-based FP8 configs from the Kimi K2 project cannot be applied to the CUTLASS-based FP4 code path used by GLM-5-NVFP4.
  2. The GPU is compute-bound, not communication-bound: The 100% utilization at 55% power indicates the tensor cores are underutilized due to small matrix sizes in decode, not due to PCIe bandwidth limitations.
  3. The bottleneck is kernel efficiency: The MoE kernels for FP4 on SM120 GPUs may be using suboptimal block sizes and tile dimensions.
  4. A systematic investigation is needed: Rather than guessing which backend might work better, the assistant initiates a code-level investigation of all available FP4 MoE paths.

The Task Result: What Came Next

The task spawned in message 275 returned a comprehensive analysis of the FP4 MoE code paths. The subagent discovered that there are five distinct FP4 MoE execution paths defined in the SGLang source code, each with different characteristics. The current flashinfer_cutlass backend uses CUTLASS templates for FP4 matrix multiplication, but there are also paths using Triton kernels, CUDA graphs, and NVIDIA's TensorRT-LLM library. This knowledge would enable the team to make informed decisions about which backend to optimize or whether to generate new tuning configurations.

Conclusion

Message 275 is a textbook example of effective debugging strategy. Faced with a complex performance problem and the temptation to reuse existing configurations, the assistant instead took a step back, analyzed the available evidence, and identified the correct path forward. The decision to investigate FP4-specific code paths rather than adapting Triton-based FP8 configs saved the team from a potentially time-wasting detour. More importantly, the assistant's synthesis of the profiling data — connecting 100% utilization with 55% power draw to diagnose underutilized tensor cores — demonstrates the kind of cross-domain reasoning that characterizes expert performance debugging.

This message represents the pivot point where the conversation shifts from "what parameters can we tune?" to "what kernels are actually running, and are they the right ones for this hardware?" It's the moment when surface-level tuning gives way to deep architectural understanding.