The Pivot: From Micro-Benchmarks to Real-World Power Measurement in Blackwell FP4 Inference

Introduction

In the long arc of optimizing GLM-5-NVFP4 inference on eight NVIDIA RTX PRO 6000 Blackwell GPUs, there arrives a moment of synthesis—a message where the assistant steps back from a dense thicket of micro-benchmark results, draws a crisp conclusion, and pivots to an entirely new line of investigation. That message is [msg 876], and it marks the transition from asking "how fast are these FP4 GEMM kernels?" to asking "what is actually happening inside the GPUs during real inference?"

This article examines that single message in depth: its reasoning, its decisions, its assumptions, and the knowledge it both consumed and produced.

The Message in Full

The assistant writes:

Interesting — sgl_kernel.cutlass_scaled_fp4_mm errors on small batch sizes (M<256) and is actually slower than flashinfer CUTLASS at larger sizes. FlashInfer's CUTLASS implementation is the faster one for FP4 on this hardware.

>

Now the key question: the per-expert batch sizes during decode are tiny (16 tokens avg). At those sizes, the FP4 GEMM takes ~30-42 microseconds regardless of problem size — that's pure kernel launch + memory access overhead. The computation is negligible.

>

Let me check something critical — what's the actual power draw during a decode workload vs the micro-benchmarks: [bash] # Restart the TP8 server for proper benchmarking ssh root@10.1.230.174 'pkill -9 -f sglang 2>/dev/null; sleep 2 cat > /root/run_tp8.sh << "ENDSCRIPT" #!/bin/bash source /root/ml-env/bin/activate export PYTHONUNBUFFERED=1 export NCCL_IB_DISABLE=1 export NCCL_P2P_LEVEL=5 export NCCL_MIN_NCHANNELS=8 export OMP_NUM_THREADS=8 export SAFETENSORS_FAST_GPU=1 export CUDA_HOME=/usr/local/cuda-12.8

>

python3 -u -m sglang.launch_server \ --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 \ --reasoning-parser glm45 --tool-call-parser glm47 \ --trust-remote-code --tp-size 8 --mem-fraction-static 0.92 \ --max-running-requests 1024 --kv-cache-dtype auto \ --quantization modelopt_fp4 --attention-backend flashinfer \ --fp8-gemm-backend cutlass --nsa-decode-backend trtllm \ --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_cutlass \ --disable-cuda-graph --disable-radix-cache \ --host 0.0.0.0 --port 8000 ENDSCRIPT chmod +x /root/run_tp8.sh nohup /root/run_tp8.sh > /root/sglang-server.log 2>&1 & echo "PID=$!"'

The message is deceptively short. It contains three distinct movements: a conclusion about kernel performance, a critical insight about decode geometry, and a decisive action to restart the server for power measurement. Each deserves close examination.

Why This Message Was Written: The Reasoning and Motivation

The assistant had just spent several rounds (messages [msg 868] through [msg 875]) running an extensive battery of FP4 GEMM micro-benchmarks. These benchmarks swept across matrix sizes from M=16 (a single expert's batch during decode) all the way to M=16,384 (large batched prefill). The results painted a stark picture:

The Decisions Made in This Message

Though the message contains no explicit "I will now do X" declaration, several decisions are implicit in its structure.

Decision 1: Abandon sgl_kernel for FP4 GEMM. The assistant had been exploring sgl_kernel.cutlass_scaled_fp4_mm as an alternative FP4 backend, motivated by the earlier discovery that sgl_kernel contained a rich set of FP4-related functions ([msg 870]). After benchmarking both libraries across the full size sweep, the conclusion was unambiguous: FlashInfer's CUTLASS implementation was faster at all usable sizes, and sgl_kernel's failed entirely below M=256. This decision would propagate into subsequent server configurations, where --moe-runner-backend flashinfer_cutlass was already set.

Decision 2: Frame the problem as kernel-launch-overhead dominated. The observation that FP4 GEMMs at small batch sizes took "~30-42 microseconds regardless of problem size" was a critical diagnostic. In GPU computing, when kernel execution time is insensitive to problem dimensions, it means the kernel is not compute-bound—it is either launch-latency-bound or memory-access-bound. For a 16×2048×6144 GEMM, the arithmetic intensity is high enough that compute should dominate if the kernel were well-tuned. The fact that it didn't meant either the kernel implementation had high fixed overheads (launch, scheduling, tile setup) or the memory subsystem was the bottleneck. Either way, the conclusion was that "the computation is negligible"—a strong claim that would shape all subsequent optimization work.

Decision 3: Restart the server with a specific configuration for power measurement. The assistant chose to restart with TP8 (tensor parallelism across all 8 GPUs), a memory fraction of 0.92, FlashInfer attention and MoE backends, and the trtllm NSA backends. Notably, it included --disable-cuda-graph and --disable-radix-cache, which would be unusual for a production configuration but made sense for a diagnostic run: CUDA graphs can mask kernel launch overhead, and radix caching can distort memory access patterns. The goal was to measure a clean, representative decode workload.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit.

Assumption 1: Power draw is a reliable proxy for compute utilization. The assistant's plan to measure "actual power draw during a decode workload vs the micro-benchmarks" assumes that GPU power consumption correlates strongly with computational throughput. This is generally true for NVIDIA GPUs—higher SM utilization and memory bandwidth draw more power—but it is not a perfect proxy. A GPU can be memory-bandwidth-bound at high power (if memory is heavily exercised) or compute-bound at moderate power (if the kernel mix is inefficient). The assistant implicitly assumed that if the GPUs were drawing near 600W, the compute units were being well-fed; if far less, the micro-benchmark story of underutilization was confirmed.

Assumption 2: The per-expert batch size calculation is correct. The assistant computed average per-expert tokens as 512 * 8 / 256 = 16. This assumes perfect load balancing across experts—that each of the 256 experts receives exactly 1/256 of the total routed tokens. In practice, MoE routing is rarely perfectly balanced; some experts may receive more tokens and others fewer. The actual distribution depends on the router's softmax gating and the input distribution. The assistant acknowledged this by saying "at most" and "on average," but the 16-token figure became the anchor for the analysis.

Assumption 3: The micro-benchmark results generalize to the server's actual kernel invocations. The assistant had been benchmarking isolated flashinfer.mm_fp4 calls with random data. Real inference involves additional operations (quantization, scaling, activation functions, allreduce) that could change the kernel mix. The assistant implicitly assumed that the FP4 GEMM was the dominant cost and that its micro-benchmark behavior would be representative.

Assumption 4: The server configuration choices are neutral for power measurement. By disabling CUDA graphs and radix cache, the assistant was removing optimizations that could affect both throughput and power draw. The assumption was that these features primarily affect latency and memory access patterns, not the fundamental compute utilization of the FP4 GEMM kernels.

Mistakes and Incorrect Assumptions

The most notable potential mistake is the assumption that power draw would cleanly distinguish between compute-bound and memory-bound regimes. In practice, Blackwell GPUs have complex power management that can throttle clocks based on temperature, power delivery, and workload type. A low power draw could indicate either underutilization or thermal throttling. The assistant's subsequent analysis (in later messages) would need to disentangle these possibilities.

Another subtle issue: the assistant restarted the server with --tp-size 8 (tensor parallelism across all GPUs) but the micro-benchmarks were run on a single GPU (cuda:0). TP8 introduces communication overhead from allreduce operations that could change the GPU's power profile. The assistant was aware of this—the NCCL environment variables (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5, NCCL_MIN_NCHANNELS=8) were set to optimize for the PCIe-bound topology—but the power measurement would necessarily include communication costs, not just compute.

The assistant also assumed that 30-42 microseconds for small GEMMs was "pure kernel launch + memory access overhead" with negligible computation. While this is directionally correct, the statement is imprecise: even at 16 tokens, a 16×2048×6144 FP4 GEMM involves approximately 400 million operations. At Blackwell's peak throughput, that should take ~0.2 microseconds. The fact that it takes 42 microseconds means the overhead is ~200x the compute time—a ratio that suggests either extremely poor kernel implementation or fundamental architectural limitations (e.g., the FP4 tensor core paths on SM120 having high startup costs). The assistant's framing of "computation is negligible" was a useful simplification but risked conflating two different bottlenecks: kernel launch overhead and memory latency.

Input Knowledge Required

To fully understand message [msg 876], the reader needs knowledge spanning several domains:

GPU architecture. The message references Blackwell GPUs (SM120 architecture), FP4 quantization, CUTLASS kernels, and tensor parallelism. Understanding why FP4 offers theoretical 2× throughput over FP8 (and 4× over BF16) requires knowledge of NVIDIA's tensor core generations. The mention of "~30-42 microseconds regardless of problem size" as evidence of launch-overhead domination is a sophisticated diagnostic that assumes familiarity with GPU kernel execution models—specifically, that compute-bound kernels scale linearly with problem dimensions while launch-bound kernels have a fixed floor.

MoE inference geometry. The calculation 512 * 8 / 256 = 16 tokens per expert draws on knowledge of Mixture-of-Experts architectures: that a model with 256 experts activates a subset (here, 8) per token, and that concurrent requests share the expert cache. Without this context, the significance of "16 tokens" is lost.

The SGLang ecosystem. The server launch command references --nsa-decode-backend trtllm, --moe-runner-backend flashinfer_cutlass, --quantization modelopt_fp4, and other SGLang-specific flags. Understanding why these particular backends were chosen (after earlier troubleshooting in the conversation) requires following the thread from the NaN decode crash resolution (<msg id=...> in segment 2) through the FlashInfer CUTLASS autotune work (segment 6).

The hardware topology. The NCCL environment variables (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5) reflect the earlier discovery that the GPUs are in a Proxmox VM with no P2P DMA between GPUs (segments 3-5). The assistant is working around a known PCIe bottleneck.

The earlier benchmark results. The message directly references the comparison between sgl_kernel and FlashInfer that was established in messages [msg 874] and [msg 875]. Without those results, the opening sentence would appear unsupported.

Output Knowledge Created

This message generates several distinct pieces of knowledge that propagate forward:

1. FlashInfer > sgl_kernel for FP4 on Blackwell. This is a concrete, actionable recommendation: for FP4 GEMM on SM120 GPUs, use FlashInfer's CUTLASS backend, not sgl_kernel's. The sgl_kernel implementation errors on small batch sizes (the exact regime used during decode) and is slower at larger sizes. This knowledge would be critical if anyone attempted to switch MoE backends in the future.

2. The decode GEMM regime is launch-overhead-dominated. The observation that small FP4 GEMMs take ~30-42 µs regardless of size reframes the optimization problem. It suggests that reducing kernel launch overhead (via CUDA graphs, kernel fusion, or persistent kernels) could yield larger gains than improving the GEMM implementation itself. This insight directly motivated the later exploration of piecewise CUDA graphs and persistent grouped GEMM kernels documented in the glb5improvement-xx.md files (mentioned in the chunk summary).

3. A methodology for diagnosing GPU utilization. The assistant's pivot from micro-benchmarks to power measurement establishes a diagnostic pattern: when synthetic benchmarks suggest underutilization, validate with real workload power draw. This methodology was applied in subsequent rounds (not shown in this message but referenced in the chunk summary as discovering that GPUs draw only ~235W out of 600W TDP during inference).

4. A reproducible server configuration for benchmarking. The run_tp8.sh script created in this message becomes a baseline configuration for all subsequent performance testing. Its specific combination of flags (TP8, FlashInfer backends, trtllm NSA, disabled CUDA graphs) defines the "control" condition against which optimizations are measured.

The Thinking Process

The message reveals a distinctive reasoning pattern: converge, then diverge.

The first paragraph converges: it synthesizes the scattered micro-benchmark results from the previous five messages into a crisp conclusion. The assistant had been exploring two different FP4 kernel implementations across a matrix of sizes, encountering shell escaping bugs, fixing dimension conventions, and gradually building up a comparative picture. The convergence statement—"FlashInfer's CUTLASS implementation is the faster one"—closes that line of inquiry decisively.

The second paragraph diverges: it takes the synthesized result and extracts a higher-level implication. The key move is the calculation of per-expert batch size (16 tokens) and the recognition that this puts decode firmly in the launch-overhead-dominated regime. This is not a conclusion that follows directly from the benchmarks—it requires combining the benchmark results with knowledge of the model architecture (256 experts, 8 active) and the deployment configuration (512 concurrent requests). The assistant is doing systems-level reasoning, connecting kernel micro-performance to workload geometry.

The third paragraph diverges again, this time into action. The phrase "Let me check something critical" signals a new hypothesis: that power draw will confirm or refute the micro-benchmark story. The assistant doesn't just speculate—it immediately constructs the experiment: restart the server with a clean configuration, then measure power. The server launch command is reproduced in full, showing careful attention to configuration details (NCCL variables, backend selection, memory fraction).

This pattern—converge on a conclusion, extract implications, then act—is characteristic of effective troubleshooting. The assistant avoids the trap of endlessly refining micro-benchmarks and instead pivots to a qualitatively different measurement that can validate the whole chain of reasoning.

Broader Significance

Message [msg 876] sits at a inflection point in the optimization journey. The earlier segments had focused on making the model run at all—resolving NaN crashes, fixing CUDA initialization, enabling P2P DMA, tuning MoE backends. By message 876, the model is running correctly, and the question shifts to why it isn't running faster. The micro-benchmarks had suggested an answer (tiny per-expert batches → underutilized GPUs), but the assistant was wise enough to distrust synthetic measurements and seek real-world validation.

The message also exemplifies a crucial skill in ML systems optimization: knowing when to stop benchmarking kernels and start measuring systems. The assistant had spent several rounds on micro-benchmarks, and could easily have continued down that path—testing more sizes, more backends, more configurations. Instead, it recognized that the next useful signal would come from a completely different measurement modality (power draw) under a realistic workload.

This decision to pivot, more than any specific finding about kernel performance, is what makes the message noteworthy. It reflects a mature understanding that optimization is not about maximizing any single metric but about building a coherent model of where time and energy are actually being spent.