The Blackwell Inference Optimization Odyssey: From GPU Underutilization to Actionable Performance Gains

Introduction

In the high-stakes world of large language model (LLM) inference, deploying a 744-billion-parameter Mixture-of-Experts (MoE) model efficiently is a formidable challenge. When that model uses NVFP4 quantization—a cutting-edge 4-bit floating-point format—and runs on NVIDIA's latest Blackwell SM120 GPUs, the optimization landscape becomes uncharted territory. This article chronicles a deep investigative session into the performance characteristics of the GLM-5-NVFP4 model deployed on eight NVIDIA RTX PRO 6000 Blackwell GPUs, where a systematic research effort uncovered the root causes of severe GPU underutilization and charted a path toward meaningful throughput improvements.

The central puzzle was stark: the GPUs reported 100% utilization but drew only 235 watts out of a 600-watt thermal design power (TDP)—a mere 39%. This is the classic signature of streaming multiprocessor (SM) stalling, where compute units are technically "busy" but spend most of their time waiting on kernel launches, memory transfers, or synchronization rather than performing useful arithmetic. The investigation that followed spanned six research threads, multiple subagent sessions, and a deep dive into the SGLang inference framework's codebase, ultimately yielding a 28% throughput improvement and a ranked optimization roadmap.

Background: The Performance Puzzle

The GLM-5-NVFP4 model is a 744-billion-parameter MoE architecture with 256 experts, of which 8 are activated per token. It uses NVFP4 quantization—a 4-bit floating-point format introduced with the Blackwell architecture—and is served through SGLang, a high-performance inference engine. The hardware consists of eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) connected via PCIe Gen5 with no NVLink interconnect.

The deployment was initially configured with tensor parallelism across all 8 GPUs (TP8), using the flashinfer_cutlass MoE runner backend. Critically, the server was launched with --disable-cuda-graph, a flag that prevents the use of CUDA graphs—a technology that captures sequences of GPU kernel launches into a single executable graph, reducing launch overhead. The rationale for disabling CUDA graphs was that MoE layers have variable batch sizes per expert (since different experts receive different numbers of tokens), which violates the fixed-shape requirement of standard CUDA graphs.

The performance anomaly was first identified through benchmarking: despite showing 100% GPU utilization, the cards drew only 235W of their 600W budget. This immediately ruled out compute saturation as the bottleneck and pointed toward SM stalling. The FP4 CUTLASS kernels were functional but constrained to 128×128 tile sizes by the 99 KB shared memory limit on SM120, limiting arithmetic intensity. The question became: what was causing the SMs to stall, and what could be done about it?

The Research Mandate: Six Threads of Investigation

The investigation was launched through a meticulously structured research mandate ([msg 0]), which posed six specific questions targeting different layers of the optimization stack:

  1. Why were CUDA graphs disabled? Was it a correctness issue or just a performance heuristic?
  2. Does SGLang support "piecewise CUDA graphs"? Could attention layers be graphed while MoE layers run eagerly?
  3. What are the SM120 clock speeds and power management characteristics? Are the GPUs being throttled?
  4. Can CUDA Multi-Process Service (MPS) or data parallelism improve utilization?
  5. Does torch.compile integration exist for kernel fusion?
  6. Is there a grouped GEMM or expert batching mechanism to consolidate small expert computations? These six questions formed a coherent investigative framework, spanning hardware diagnostics (clock/power), software architecture (CUDA graphs, piecewise graphs), parallelism strategies (MPS, DP), and kernel-level optimizations (torch.compile, grouped GEMM). The assistant dispatched these as parallel SSH commands to the remote machine, gathering initial data across all fronts simultaneously ([msg 1]).

Thread 1: CUDA Graph Disable Reason

The first thread sought to determine whether --disable-cuda-graph was a necessity or a conservative choice. By searching SGLang's model executor code, the assistant discovered that the framework does not automatically disable CUDA graphs for MoE workloads—the only automatic disable condition is for "double sparsity" mode. This meant the flag was manually set, likely because someone encountered crashes during CUDA graph capture (since MoE routing changes tensor shapes each forward pass) or assumed it wouldn't work and never tested it.

This finding reframed the problem. Standard CUDA graphs were indeed incompatible with MoE's dynamic expert routing, but the question was never really about standard CUDA graphs—it was about whether any form of graph capture could help. This led directly to Thread 2.

Thread 2: Piecewise CUDA Graphs—The Breakthrough

The most consequential finding emerged from the investigation into piecewise CUDA graphs. The assistant discovered that SGLang fully supports piecewise CUDA graphs, with a dedicated --enable-piecewise-cuda-graph flag, and that this mechanism was designed specifically for MoE models ([msg 2], [msg 3]).

The implementation in piecewise_cuda_graph_runner.py works by inserting graph break points at MoE layers. Attention layers—which have fixed shapes during decode—are captured as CUDA graph segments, while MoE layers run eagerly outside the graph. The MoE forward functions are decorated with @register_custom_op, which acts as a graph break boundary, making them opaque to the CUDA graph capture mechanism.

The assistant traced through the code to confirm that the FlashInferFP4MoE class ([msg 7], [msg 8])—the class responsible for FP4 MoE computation—already supports piecewise CUDA graphs with a dedicated moe_forward_piecewise_cuda_graph_impl function. This was the Rosetta Stone: the optimization mechanism existed, was designed for this exact use case, and was compatible with the FP4 quantization path.

The server arguments piecewise_cuda_graph_tokens (a list of token counts for graph capture) and piecewise_cuda_graph_compiler (defaulting to "eager") provided the configuration knobs ([msg 5]). The recommendation to replace --disable-cuda-graph with --enable-piecewise-cuda-graph became the single highest-impact action item.

Thread 3: Clock Speeds and Power Diagnostics

The third thread provided critical diagnostic confirmation. By running nvidia-smi queries, the assistant established that the GPUs were in P0 performance state (the highest), running at 97% of maximum SM clock (2347 MHz out of 2430 MHz), with no throttle reasons active ([msg 3], [msg 4]). Power draw at idle was only 84 watts out of a 600-watt limit. During inference, the reported 235W/600W (39% TDP) was not due to clock throttling or power capping—it was because the SMs simply had nothing to do most of the time.

This finding ruled out an entire class of potential fixes (power limit adjustments, clock boosting, thermal management) and pointed squarely at the real bottleneck: kernel launch overhead and small problem sizes. The assistant identified three contributing factors:

Thread 4: Data Parallelism Options

The fourth thread explored data parallelism as a strategy to improve GPU utilization. SGLang supports --dp-size, --moe-dp-size, and --enable-dp-attention for various DP configurations ([msg 1]). The assistant proposed a specific configuration: --dp-size 2 --tp-size 4, which would run two independent data-parallel replicas, each using 4-GPU tensor parallelism. This could double throughput for independent requests and halve the TP communication over PCIe, at the cost of halving the memory per replica.

The assistant correctly assessed this as medium impact, noting the memory constraint. For a 744B parameter model even with FP4 quantization (roughly 372 GB at 4 bits per parameter), splitting across 4 GPUs (96 GB each) would be tight but potentially feasible.

Thread 5: torch.compile Integration

The fifth thread revealed an important constraint. SGLang has --enable-torch-compile with an inductor backend that can fuse operations in attention layers ([msg 1]). However, the assistant discovered that torch.compile and piecewise CUDA graphs are mutually exclusive in the current code—server_args.py forces the compiler to "eager" when both are enabled, and model_runner.py explicitly disables piecewise graphs when torch.compile is on.

This finding is crucial because it means the team faces a binary choice: use piecewise CUDA graphs to capture attention layers as graph segments, or use torch.compile to fuse operations in attention layers, but not both. The assistant's recommendation prioritized piecewise graphs (high impact) over torch.compile (medium impact), a defensible judgment given that piecewise graphs directly address the kernel launch overhead problem.

Thread 6: Grouped GEMM—Already Optimal

The sixth thread confirmed that the most important kernel-level optimization was already in place. The flashinfer_cutedsl_moe.py module implements masked grouped GEMM, which batches all 256 experts into a single kernel call using a masked_m tensor to indicate valid rows per expert ([msg 2]). This avoids launching 256 separate GEMMs and is the correct approach for this workload. The 128×128 tile size limitation is a hardware constraint of SM120, not a software deficiency.

This finding is valuable as a negative result: it tells the team not to waste time on expert batching optimizations because they're already optimal. The assistant also noted that expert specialization (ES) kernels are FP8-only and not applicable to the FP4 path, closing another potential avenue.

The Synthesis: From Findings to Actionable Recommendations

The most impressive aspect of the investigation is how the assistant synthesized the six threads into a coherent diagnosis and prioritized action plan ([msg 9]). The core insight is that the GPU underutilization is a scheduling and launch overhead problem, not a compute or memory bandwidth problem. The SMs are clocked high, the power limit is maxed, the grouped GEMM kernels are efficient, and there are no throttle reasons. The bottleneck is that the GPU spends most of its time transitioning between small kernel launches rather than doing useful computation.

The recommendations were ranked with clear rationale:

High Impact:

  1. Enable piecewise CUDA graphs to capture attention layers as graph segments, eliminating kernel launch overhead for ~50% of each layer's operations
  2. Try torch.compile as an alternative to piecewise graphs for kernel fusion in attention layers Medium Impact:
  3. Evaluate DP configurations to reduce PCIe communication overhead
  4. Explore compute-communication overlap via --enable-single-batch-overlap or --enable-two-batch-overlap Low Impact / Already Optimal:
  5. Grouped GEMM is already in use
  6. Power/clock limits are already at maximum
  7. Expert specialization kernels are FP8-only and not applicable

From Research to Results: The 28% Throughput Improvement

The investigation did not remain theoretical. Building on these findings, the assistant conducted systematic server parameter tuning, achieving a 28% throughput improvement at 2048 concurrency. By raising --max-running-requests to 2048 and setting --num-continuous-decode-steps 8, the server reached 2,095 output tokens per second and 4,151 total tokens per second.

Further analysis confirmed that the model is compute-bound rather than communication-bound. A benchmark of TP4+PP2 (tensor parallelism 4 + pipeline parallelism 2) was 2× slower than TP8, ruling out allreduce latency as the primary bottleneck. This validated the focus on SM-level optimizations rather than interconnect improvements.

The Optimization Roadmap: Documenting the Path Forward

The assistant synthesized all findings into a ranked optimization plan and began documenting each approach as glb5improvement-xx.md files, starting with piecewise CUDA graphs. This documentation effort ensures that the investigative work translates into actionable engineering artifacts that can be referenced, tested, and iterated upon.

The documented approaches include:

Conclusion

This investigation exemplifies the systematic methodology required for GPU optimization at the frontier of LLM inference. By starting with hardware diagnostics to rule out simple fixes (clock throttling, power capping), then tracing through the software stack to find mechanisms designed for the specific use case (piecewise CUDA graphs), verifying compatibility with the quantization path (FlashInferFP4MoE), mapping constraints (torch.compile mutual exclusivity), and confirming existing best practices (grouped GEMM), the assistant built a complete mental model of the system's behavior.

The result is not just a list of findings but a coherent understanding of why the system behaves as it does and what can be done about it. The 28% throughput improvement demonstrates that systematic investigation pays dividends, and the documented optimization roadmap provides a clear path for continued gains. For anyone working on LLM inference optimization, this session demonstrates the value of structured research: eliminate red herrings first, look for designed solutions, verify compatibility before recommending, map constraints, and always provide actionable specificity.