The Smoking Gun: Tracing a 58% GPU Utilization Gap to a Missing Architecture Check

In the high-stakes world of large-scale ML inference, performance debugging often resembles detective work — gathering clues, forming hypotheses, and eventually finding the single line of code that explains a massive discrepancy. Message 734 in this opencode session represents exactly such a moment: the point where analysis crystallizes into action, where the assistant has traced a 58% GPU utilization shortfall to its root cause and is about to apply the fix. But as with many well-intentioned patches, the story does not end there.

The Context: 8 GPUs Running at 42% Power

The session had been building toward this moment for several rounds. The assistant was running GLM-5-NVFP4 — a Mixture-of-Experts model with NVFP4 quantization — on 8x RTX PRO 6000 Blackwell GPUs connected via PCIe in a Proxmox virtualized environment. Throughput had been steadily improved from ~880 tok/s to ~3,740 tok/s through a series of optimizations: enabling FlashInfer CUTLASS MoE autotune for SM120, increasing --max-running-requests to 1024, disabling CUDA graphs and radix cache. But a nagging problem remained.

The user had pointed out in [msg 725] that GPU power draw was only ~330W per GPU despite a 600W TDP. The assistant initially dismissed this as reasonable ([msg 724]), but the user insisted: "still low GPU power numbers, ~330W/gpu not 600W TDP even at high batch, debug sglang to figure out what exactly can be at fault." The user also provided a research document (sm120-attention-fix-research.md) outlining the differences between SM120 (consumer Blackwell, 100KB shared memory) and SM100 (datacenter Blackwell, 228KB shared memory).

When the assistant finally checked the actual power limit in [msg 728], the finding was stark: 600W TDP per GPU, and the GPUs were only hitting ~250W during benchmarks — just 42% of their thermal envelope. Something was starving the GPUs of work.

The Investigation: Tracing the Compute Path

The assistant launched a subagent task in [msg 729] to analyze the GLM-5 forward pass compute. The subagent produced a comprehensive analysis identifying three bottlenecks:

  1. CRITICAL: FlashInfer allreduce fusion disabled on SM120 — The communicator.py file checked for SM90 or SM100 only, not SM120. This meant every allreduce operation (2 per layer × ~78 layers = 156 allreduces per forward pass) was serialized with compute — GPU SMs sat idle during PCIe transfers.
  2. SIGNIFICANT: Custom allreduce disabled for more than 2 PCIe-only GPUs.
  3. MODERATE: MoE runner backend not auto-selected for SM120. The assistant summarized these findings in [msg 730] and began systematically checking the codebase. In [msg 731], it confirmed that communicator.py had no reference to is_sm120_supported at all. In [msg 732], it found the apply_flashinfer_allreduce_fusion() function with its architecture gate: (_is_sm90_supported or _is_sm100_supported). In [msg 733], it checked server_args.py and found the same pattern — SM120 was absent from every architecture check.

Message 734: The Smoking Gun

Message 734 is the moment of confirmation. The assistant states:

Now I can see all the SM120 gaps clearly. Let me apply all the fixes:

And then runs a bash command to display the imports section of /root/sglang/python/sglang/srt/layers/communicator.py:

    is_cuda,
    is_flashinfer_available,
    is_gfx95_supported,
    is_hip,
    is_npu,
    is_sm90_supported,
    is_sm100_supported,
)

_is_cuda = is_cuda()
_is_flashinfer_available = is_flashinfer_available()
_is_sm90_supported = _is_cuda and is_sm90_supported()
_is_sm100_supported = _is_cuda and is_sm100_supported()

The absence of is_sm120_supported is glaring. The import list includes SM90 (Hopper datacenter), SM100 (Blackwell datacenter), and even is_gfx95_supported (AMD GPUs), but not SM120 — the architecture actually running on these RTX PRO 6000 cards. This single omission cascades through the entire inference path: allreduce fusion is never enabled, so every allreduce stalls the GPU while data transfers over PCIe, and the GPU never reaches its power ceiling because it's constantly waiting on communication.

The message is brief — just two lines of commentary and a shell command — but it represents a critical transition. The assistant has moved from diagnosis to treatment. The "smoking gun" has been found, and the next messages will apply the fix.

The Assumption and Its Consequences

The assistant's assumption was straightforward: adding SM120 to the architecture checks would enable allreduce fusion and unlock the remaining GPU throughput. The fix seemed simple — add is_sm120_supported to the import list, add it to the apply_flashinfer_allreduce_fusion() condition, and add it to the auto-enable logic in server_args.py.

But this assumption proved incorrect. As the chunk summary reveals, when the assistant patched flashinfer to add SM120 support and restarted the server, throughput dropped from ~3,740 tok/s to 236 tok/s, and power fell to 125W. The allreduce fusion kernels, which were designed for SM90/SM100 architectures with 228KB shared memory, performed poorly on SM120's 100KB shared memory constraint. The synchronization primitives and pipeline depths baked into the TRT-LLM communication kernels did not translate to the consumer Blackwell architecture.

This is a classic pitfall in systems optimization: what works on one architecture may not work on another, even within the same GPU family. The assistant's mistake was not in identifying the bottleneck — allreduce fusion was indeed the right thing to fix — but in assuming that the existing kernel implementation would be compatible with SM120. The patch was reverted, and the assistant pivoted to exploring NCCL tuning and alternative MoE backends.

Input Knowledge Required

To understand this message, one needs familiarity with several concepts:

Output Knowledge Created

This message produces several valuable insights:

  1. The specific code location of the SM120 gap: communicator.py lines 65-85, where is_sm120_supported is missing from imports and the allreduce fusion condition.
  2. The pattern of SM120 exclusion: Across server_args.py, SM120 is absent from multiple architecture checks for MoE runner backend selection, allreduce fusion auto-enable, and attention backend defaults.
  3. A methodology for debugging GPU utilization: Starting from power readings, tracing to compute utilization, identifying communication bottlenecks, and finding the specific code gates.
  4. Documentation of the fix approach: The assistant's planned four fixes (import, fusion condition, auto-enable, MoE backend) serve as a roadmap for anyone porting SGLang to SM120.

The Thinking Process

The assistant's reasoning in this message and the surrounding context reveals a systematic, hypothesis-driven approach. The chain is:

  1. Observe symptom: GPU power at 250W vs 600W TDP ([msg 728]).
  2. Form hypothesis: The GPUs are underutilized because compute is serialized with communication.
  3. Gather evidence: Launch subagent to analyze forward pass ([msg 729]), which confirms allreduce fusion is disabled.
  4. Locate root cause: Check communicator.py and server_args.py for architecture gates (<msgs id=731-733>).
  5. Confirm visually: Display the imports to see the missing is_sm120_supported ([msg 734]).
  6. Plan and execute fixes: Apply patches to add SM120 support (<msgs id=740+>). The visual confirmation in message 734 is a deliberate step — the assistant could have simply applied the fix based on the grep results, but chose to display the actual code to verify the gap exists and to document it for the user. This is a hallmark of careful debugging: never assume, always verify.

Conclusion

Message 734 captures the precise moment when a performance mystery becomes a known bug. The 58% GPU utilization gap, traced through layers of abstraction, reduces to a missing architecture check in an import statement. The fix seems trivial — add one line — but the subsequent failure of that fix reveals a deeper truth about GPU architecture portability: SM120 is not just another compute capability to add to a list, but a fundamentally different architecture with different constraints that requires its own kernel implementations. The message stands as a testament to the value of systematic debugging and the humility required when assumptions about hardware compatibility prove wrong.