The MoE Kernel Hunt: Diagnosing GPU Inefficiency in FP4 Decode

In the high-stakes world of deploying massive language models on consumer-grade hardware, every watt and every clock cycle matters. Message 276 of this opencode session captures a pivotal moment of diagnosis and decision-making: the assistant has just received a detailed subagent report on the FP4 Mixture-of-Experts (MoE) kernel code paths in SGLang, and is now synthesizing that information to explain a puzzling performance anomaly and chart a course forward. The GPUs are running at 100% utilization but drawing only 55% of their power budget — a contradiction that points to fundamental inefficiencies in how the model's MoE layers execute on the RTX PRO 6000 Blackwell GPUs.

The Road to This Message

To understand message 276, we must trace the conversation that led to it. The session had been wrestling with deploying the GLM-5-NVFP4 model — a 744-billion-parameter Mixture-of-Experts model with 256 experts — on 8x RTX PRO 6000 Blackwell (SM120) GPUs connected via PCIe. After resolving a critical NaN crash during decode by selecting the correct NSA attention backends (trtllm), the assistant had established baseline throughput: approximately 225 output tokens per second at 64 concurrent requests, with single-stream throughput around 10-11 tok/s.

The user then raised a provocative question: if PCIe Gen5 x16 offers 64 GB/s per GPU, why was throughput so low? The assistant investigated and discovered something surprising — the GPUs were at 100% utilization but drawing only 328W of 600W TDP (55% power). The SM clocks were at 2422 MHz (near max boost of 2430 MHz). No thermal or power throttling was active. This meant the GPUs were fully occupied but not computationally saturated — the tensor cores weren't being fed enough work.

The user then pointed the assistant at a local research repository (glm-kimi-sm120-rtx6000bw) that contained MoE tuning configs from a prior deployment of Kimi K2 on the same hardware. The assistant investigated these configs (messages 271-272) and found they were for a different model (GLM-4.7-FP8 with E=161 experts) and a different MoE runner backend (Triton-based fused_moe). They were not directly applicable to the current setup, which uses flashinfer_cutlass as the MoE runner.

This led the assistant to spawn a subagent task (message 275) with the explicit goal of investigating the FP4 MoE code paths in SGLang. The subagent returned a comprehensive analysis of five distinct FP4 MoE execution paths, their kernel implementations, and their applicability. Message 276 is the assistant's response to that task result.

The Message Itself

Let us examine the message in full:

Excellent — this is very informative. The current setup uses flashinfer.fused_moe.cutlass_fused_moe which is a CUTLASS-based FP4 fused kernel. There are alternative backends we could try: flashinfer_trtllm (different kernel, uses TRT-LLM-style FP4) and flashinfer_cutedsl (CuteDSL grouped GEMM).

But more importantly — the FP4 allgather optimization is NOT active in our setup because it requires dp_attention which we don't have (we're using TP-only). So that flag isn't helping us.

Given that GPUs are at 100% utilization but only 55% power (330W/600W), the issue is likely small-batch MoE kernel efficiency. Let me try the alternative flashinfer_trtllm MoE runner (different kernel implementation) and also try the cutlass (sgl_kernel) backend:

[todowrite with completed items and presumably new items for testing alternative backends]

This compact message accomplishes several things simultaneously: it acknowledges the task result, extracts actionable information, rules out a dead end, forms a hypothesis about the root cause, and commits to a specific experimental plan.

The Reasoning Architecture

The assistant's thinking in this message reveals a sophisticated diagnostic process. Let us unpack the layers.

Layer 1: Information extraction. The subagent task result described five FP4 MoE execution paths. The assistant immediately identifies the current backend (flashinfer.fused_moe.cutlass_fused_moe) and notes two alternatives: flashinfer_trtllm (which uses a TRT-LLM-style FP4 kernel) and flashinfer_cutedsl (which uses CuteDSL grouped GEMM). This is a critical step — the assistant is mapping the abstract code path taxonomy onto concrete, actionable options.

Layer 2: Negative filtering. The assistant identifies that the FP4 allgather optimization — which might have improved communication efficiency — is not applicable because it requires dp_attention (data parallelism for attention). The current setup uses tensor parallelism (TP) only, without data parallelism. This is an important negative result: it prevents wasted effort trying to enable a flag that would have no effect.

Layer 3: Hypothesis formation. The assistant connects two observations — 100% GPU utilization and 55% power draw — to form a hypothesis about the root cause. In GPU computing, 100% utilization means the GPU's execution units are never idle, but low power draw relative to TDP indicates they are not performing computationally intensive work. The most likely explanation is that the GPU is spending its time on kernel launch overhead, memory accesses, and small matrix operations that don't saturate the tensor cores. This is characteristic of small-batch MoE decode, where each token activates only a handful of experts per layer, and each expert operation involves tiny matrices (batch_size=1 × hidden_size=6144).

Layer 4: Experimental commitment. Based on this hypothesis, the assistant decides to try alternative MoE runner backends. The reasoning is that different kernel implementations may have different efficiency characteristics for the small-matrix regime. The flashinfer_trtllm backend uses a different FP4 kernel path (inspired by TensorRT-LLM), and the cutlass backend from sgl_kernel may have different tiling and scheduling strategies.

Assumptions and Blind Spots

The assistant's reasoning rests on several assumptions that deserve scrutiny.

Assumption 1: The MoE runner backend is the primary lever. The assistant assumes that switching from flashinfer_cutlass to flashinfer_trtllm or cutlass could meaningfully improve throughput. This is plausible but not guaranteed — all three backends ultimately execute the same mathematical operations (FP4 matrix multiplications), and the differences may be marginal for the small-batch regime. The bottleneck might be architectural (the sheer number of expert dispatches per layer) rather than implementational.

Assumption 2: 55% power draw indicates inefficiency. While it's true that low power draw at 100% utilization suggests the tensor cores aren't fully utilized, this is not necessarily a problem that can be solved by switching backends. Small-batch decode in MoE models is inherently memory-bound and launch-latency-bound — the GPU spends most of its time loading weights and dispatching kernels, not doing arithmetic. This is a fundamental characteristic of the workload, not a bug in the kernel implementation.

Assumption 3: The FP4 allgather optimization is irrelevant. The assistant correctly notes that the allgather optimization requires dp_attention. However, it does not consider whether enabling data parallelism (even partially) might be worth investigating. The earlier analysis had concluded that expert parallelism (EP) was not beneficial due to the small hidden size, but data parallelism for attention (DP attention) is a different mechanism that could potentially improve throughput by allowing larger batch sizes.

Assumption 4: The todo list structure implies forward progress. The assistant appends a todo list showing completed items (trtllm NSA backends, baseline benchmarks, mem-fraction tuning, CUDA graphs, EP investigation) and implicitly adds new items for MoE backend testing. This creates a narrative of steady progress, but it risks masking the possibility that the MoE backend switch might be another dead end.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several domains:

Mixture-of-Experts (MoE) architecture: Understanding that MoE models have multiple "expert" sub-networks per layer, with a router selecting which experts to activate for each token. This explains why decode involves many small matrix operations rather than one large one.
FP4 quantization (NVFP4/modelopt_fp4): The model uses 4-bit floating point quantization for the expert weights, which requires specialized kernel support. Not all MoE runners support FP4.
SGLang's MoE runner backends: The flashinfer_cutlass, flashinfer_trtllm, flashinfer_cutedsl, and cutlass backends are different code paths for executing MoE computations. Each uses a different kernel library (CUTLASS, TensorRT-LLM, CuteDSL) with different optimization strategies.
Tensor parallelism vs. data parallelism: TP splits model layers across GPUs; DP splits batches across GPUs. The dp_attention flag enables a hybrid where attention is data-parallel while MoE layers remain tensor-parallel.
GPU utilization and power metrics: 100% utilization means the GPU's execution units are never idle (there's always work queued), but power draw indicates how intensively those units are working. Low power at high utilization suggests many small, low-intensity operations rather than large matrix multiplications.

Output Knowledge Created

This message generates several valuable pieces of knowledge:

Current backend identification: The active MoE runner is flashinfer.fused_moe.cutlass_fused_moe, a CUTLASS-based FP4 fused kernel.
Alternative backend candidates: Two specific alternatives are identified for testing: flashinfer_trtllm and cutlass (sgl_kernel).
Negative result on FP4 allgather: The FP4 allgather optimization is ruled out as irrelevant for the current configuration (TP-only, no dp_attention).
Diagnostic hypothesis: The root cause of low throughput is identified as small-batch MoE kernel efficiency, not PCIe bandwidth or power throttling.
Experimental plan: A clear next step is defined — restart the server with alternative MoE runner backends and re-benchmark.

The Thinking Process in Context

What makes message 276 particularly interesting is how it exemplifies the assistant's diagnostic methodology. Earlier in the session, the assistant had been focused on external bottlenecks: PCIe bandwidth, virtualization overhead, expert parallelism feasibility. Each of these investigations returned negative or inconclusive results. The PCIe links were fast (Gen5 x16). The virtualization overhead existed but wasn't the primary bottleneck. Expert parallelism offered no advantage.

The profiling data showing 100% GPU utilization at 55% power was the turning point — it forced a shift from external (communication) to internal (computation) bottlenecks. Message 276 is the first message where the assistant fully commits to this new diagnostic direction, armed with the subagent's code path analysis.

The todo list at the bottom of the message is also revealing. It shows the progression of completed work: fixing the NaN crash with trtllm NSA backends, establishing baseline benchmarks, tuning memory fraction and CUDA graphs, investigating expert parallelism. Each completed item represents a hypothesis that was tested and either validated or discarded. The MoE backend investigation is the next hypothesis in this iterative process.

Broader Significance

This message illustrates a fundamental challenge in deploying large MoE models on multi-GPU systems: the tension between model scale and hardware utilization. A 744B-parameter model with 256 experts requires enormous memory capacity (453GB for the MoE experts alone), forcing distribution across 8 GPUs. But the per-GPU expert count in decode is small (approximately 1-2 experts per token per GPU), resulting in tiny matrix operations that cannot saturate the tensor cores.

This is not a problem that can be solved by faster interconnects or more memory bandwidth — it is a fundamental property of the MoE architecture in the decode regime. The assistant's attempt to find a better MoE kernel backend is a reasonable engineering response, but the underlying issue may be architectural: MoE models with many small experts are inherently inefficient for single-token decode, regardless of kernel implementation.

The message also demonstrates the value of systematic diagnosis. Rather than randomly trying flags, the assistant builds a chain of evidence: observe the anomaly (100% util, 55% power), form a hypothesis (small-batch kernel inefficiency), investigate the code paths (subagent task), identify actionable options (alternative backends), and commit to an experiment. This methodology — observe, hypothesize, investigate, act — is the essence of effective performance debugging in complex ML systems.