Inside the Decode Performance Investigation: How One Session Cracked the Occupancy Ceiling on Blackwell GPUs

Introduction

In a single intensive session spanning dozens of messages and hundreds of tool calls, an AI assistant conducted a deep, multi-threaded investigation into the decode performance of DeepSeek-V4-Flash-NVFP4 running on 8× RTX PRO 6000 Blackwell GPUs (sm120). The session was structured around three tightly coupled tasks: analyzing the occupancy of the MMA sparse-decode attention kernel, quantifying per-step launch and latency gaps, and performing a comprehensive prior-art sweep of twelve local improvement documents. What emerged was a unified theory of the system's performance bottleneck — one that explained the puzzling gap between 97% SM utilization and only 57% power draw, and produced a ranked shortlist of actionable levers for raising the high-concurrency throughput asymptote from C60 to C90.

The Setup: A System Under the Microscope

The investigation began with a well-characterized production system. The target was a DeepSeek-V4-Flash-NVFP4 model running with TP4 on 8× RTX PRO 6000 Blackwell GPUs (188 SMs each, ~100KB shared memory per SM), with cuda-graph capture enabled up to batch size 96. The model had 43 layers (approximately 42 using sparse attention), 64 total query heads, index_topk=512, and a 512K context window. The environment variables SGLANG_SM120_MMA_FLASHMLA=1 and SGLANG_SM120_TRITON_INDEXER=1 were active.

The known baseline was stark: decode step time followed the formula 18 + 1.05·bs milliseconds, with 97% SM utilization but only 57% power draw (340W of 600W) and 27% memory controller utilization. The kernel mix at batch size 32 showed MoE at 28-35%, MMA sparse-MLA attention at 23-24%, FP8 dense at ~5%, all-reduce at ~3%, and glue operations at 3-4%, with GPU idle at ~2.6%. The central mystery was why the SMs appeared nearly fully occupied yet drew barely half their rated power — a classic signature of memory-latency-bound execution with poor instruction-level parallelism.

Task 1: The Attention Occupancy Revelation

The assistant's first task was to analyze the occupancy of the _mma_sparse_decode_split_kernel and its companion kernels. Through a combination of reading the Triton kernel source in flash_mla_sm120_triton.py and parsing real CUDA traces from /root/prof/, the assistant built a precise model of the kernel's grid dimensions.

The grid logic was straightforward: with _MMA_BLOCK_H=32 and H=64 query heads, the kernel used n_hg=2 head-groups per CTA. The number of splits was computed as nsplit = ceil(256 / (B · 2)), capped at 16. This produced grids ranging from 256 CTAs at batch 32 (1.36 waves on 188 SMs) to 384 CTAs at batch 96 (2.04 waves, near-ideal packing). The high-batch occupancy regression from the earlier single-request era was definitively fixed — the grid now comfortably exceeded 188 SMs.

But the trace data revealed a deeper constraint. The split kernel used 255 registers per thread and 80,000 bytes of shared memory per CTA, with a block size of 128 threads (4 warps). With only ~100KB of shared memory per SM, the 80KB footprint meant exactly one CTA could fit per SM. This capped warp occupancy at just 4 warps per SM — a mere 8% of the 48-warp maximum. The same pattern held for the MoE FP4 grouped-GEMM kernel, which used 89,088 bytes of shared memory and a grid of exactly 188 CTAs (one per SM), and for the FP8 dense matmul.

This was the decisive insight: the 97% SM activity metric only measures whether at least one warp is resident on each SM. With only 4-12 warps per SM, those warps were spending most of their time stalled on memory latency — scattered FP8 page-gathers for attention, weight-block loads for MoE — with too few concurrent warps to hide the stalls. The 57% power and 27% DRAM bandwidth utilization were not anomalies but consequences: the SMs were "busy" but not doing useful compute, and the memory bus was not saturated because the few resident warps couldn't issue enough concurrent requests.

The assistant identified two easy levers for attention occupancy. First, a wave-aware NSPLIT selection (approximately 5 lines of code) that picks the split count minimizing fractional-wave waste, which would recover ~15-25% of attention time at the B=64-88 graph buckets where wave quantization was worst. Second, reducing _MMA_BLOCK_H from 32 to 16 or forcing the num_warps=8 autotune config, which would halve shared memory usage and allow 2 CTAs per SM — doubling warp occupancy and dramatically improving latency hiding.

Task 2: The Per-Step Launch Structure — Separating Signal from Noise

The second task required quantifying the per-decode-step kernel composition and measuring the GPU-idle gaps between kernels within a cuda-graph replay. This proved surprisingly difficult because the available traces were contaminated by prefill phases, warmup autotuning, and inter-burst idle periods.

The assistant wrote a series of increasingly refined analysis scripts — analyze.py, scan.py, step_detail.py, onestep.py, timeline.py, steady.py, and hb.py — each iteration peeling back another layer of complexity. The initial trace scans showed 61% idle on the main stream, but this was misleading: the traces captured mixed prefill+decode workloads where chunked prefill steps with batch sizes of 97, 214, and 247 dominated the idle accounting.

By isolating a clean decode window and examining consecutive kernels on the graph stream, the assistant made the critical measurement: inter-kernel gaps within a cuda-graph replay were 0.10-0.20 microseconds — effectively zero. The split-to-combine gap was 0.19µs, elementwise-to-elementwise was 0.10µs, and combine-to-elementwise was 0.11µs. The CUDA graph was packing kernels back-to-back with no measurable launch overhead.

The large gaps that dominated the raw traces — a 24ms gap between _w8a8_block_fp8_matmul and the next cutlass kernel, repeated 109 times — were inter-burst idle from a lightly-loaded benchmark server, not intra-step gaps. The true 2.6% GPU idle in production was step-boundary CPU overhead: sampling, scheduler dispatch, and input copying between graph replays, which at batch 96 with a 119ms step time accounted for roughly 3ms per step.

This finding had profound implications for the optimization strategy. The 57% power gap was not caused by serialized small kernels or launch overhead — fusion of glue kernels would only recover ~3-4% of the profile. The real bottleneck was intra-kernel memory-latency stalls at low occupancy, and the only way to close the power gap was to raise warp occupancy in the dominant kernels.

The per-step kernel inventory was equally revealing: approximately 2,600 kernel launches per decode step across all streams, including 42 attention splits, 42 combines, ~86 MoE grouped-GEMMs, ~43-87 all-reduces, 118 FP8 matmuls, ~1,100 elementwise glue operations, and 168 indexer launches. The MoE grouped-GEMM grid was pinned at exactly 188 CTAs (one per SM), confirming that raising batch size simply adds rows per CTA at the same occupancy — the mechanism behind the 1.05ms/req marginal cost.

Task 3: The Prior-Art Sweep — Separating Dead Ends from Opportunities

The third task was a systematic review of twelve local improvement documents (glb5improvement-01 through 12, plus DSV4_SM120_REPORT.md, DSV4_DECODE_PERF_PLAN.md, eagle-fast-verify.md, and memory-bandwidth-optimization-research.md). The assistant read each document, assessed its applicability to the DSV4 sm120 decode context, and produced a definitive do-not-retread list and a ranked shortlist of promising levers.

The do-not-retread list was extensive. MSCCLPP one-shot allreduce was tried on DSV4 and showed zero gain on PCIe. Single-batch overlap was rejected due to the multi-stream corruption family that had just been fixed. Expert parallelism at EP4 performed worse than TP4 (14 vs 23 tok/s) due to PCIe all-to-all overhead at the relevant concurrency levels. Allreduce fusion catastrophically collapsed throughput on GLM5 (1867→236 tok/s) because cudaGridDependencySynchronize is broken on sm120, and the flashinfer fusion auto-disables on PCIe. torch.compile hits cudaErrorStreamCaptureIsolation under cuda-graph capture. MTP/EAGLE speculative decoding is blocked by a NextN draft MoE dispatch issue (SM100-only kernel) and even if unblocked, the measured gain at high concurrency is zero — the verifier batch-saturates. FP4 2:4 structured sparsity would be double-lossy on already-NVFP4 weights and requires kernels that may not exist on sm120.

The shortlist of untried-but-promising levers was ranked by impact on the MoE weight-load / low-occupancy bottleneck that sets the C60→C90 asymptote:

  1. Opportunistic Expert Activation (OEA) — implemented and tested on GLM5 with +5.7% at C1024 and +25% peak, with a sweet spot at C32-128 that directly overlaps DSV4's operating range. OEA reduces redundant expert loads within a batch, attacking the weight-load bottleneck at its source. The caveat is verifying DSV4's grouped routing (n_group) which may constrain piggybacking.
  2. L2 persistence for hot experts — cuts hot-expert load latency from ~877 cycles to ~358 cycles (2.4×), which helps the latency-bound regime even when bandwidth is not saturated. The caveat is that cudaDeviceSetLimit(persistingL2) returned an error on this hardware, requiring a startup probe.
  3. More warps per SM in the two 1-CTA/SM kernels — either reducing shared memory to fit 2 CTAs per SM or forcing 8-warp autotune configs. This is the most direct fix for the 57% power gap.
  4. Wave-aware NSPLIT — the nearly-free fix for attention at B=64-88.
  5. Column-major / L2-aware tile scheduling for the FP4 grouped-GEMM, targeting the weight-load bound directly.
  6. Finer cuda-graph buckets — a trivial config change to reduce padding waste at bucket edges.
  7. Hand-fused RMSNorm+RoPE+dequant epilogue — replaces ~30 tiny elementwise kernels per layer, recovering ~3-4% of profile time.

Synthesis: The Unified Theory of Decode Performance

The investigation's power came from the convergence of its three threads. The occupancy analysis (Task 1) revealed that every major decode kernel runs at 1 CTA per SM with 4-12 warps, explaining the 57% power / 97% SM paradox. The gap analysis (Task 2) confirmed that inter-kernel idle is negligible — the bottleneck is intra-kernel, not inter-kernel. The prior-art sweep (Task 3) showed that the most promising levers (OEA, L2 pinning, occupancy improvements) all target the same root cause: memory-latency stalls at low warp occupancy.

The recommended order of operations was clear: start with the free win (wave-aware NSPLIT), then verify and land OEA (the highest-impact low-risk lever), probe the L2 persistence API, invest in kernel-level occupancy work, and finally pursue the column-major scheduling, finer buckets, and hand-fused epilogue. The communication, speculative decoding, expert parallelism, and fusion approaches were all confirmed dead ends on this PCIe-sm120 platform.

Conclusion

This session exemplifies the value of systematic, evidence-driven performance analysis. By combining deep code reading, custom trace analysis, and thorough literature review, the assistant transformed a confusing performance signature (97% SM / 57% power) into a precise, actionable model of the bottleneck. The investigation did not just identify what was wrong — it produced a ranked, risk-assessed roadmap for fixing it, grounded in concrete measurements from real traces and informed by the hard-won experience of prior optimization attempts on the same hardware.