The Commit That Rewrote the Bottleneck: How a Single Git Message Captured a Paradigm Shift in GPU Kernel Optimization

In the middle of a marathon debugging and optimization session for the DeepSeek-V4-Flash NVFP4 model running on Blackwell RTX PRO 6000 GPUs, the assistant issued what appears at first glance to be a mundane git commit. Message [msg 13529] reads:

cd /home/theuser/glm-kimi-sm120-rtx6000bw && git add DSV4_DECODE_PERF_PLAN.md && git commit -q -m "docs(dsv4): #3 research — bs80 profile revises premise: ATTENTION is the slope (+0.79ms/req), MoE is flat fixed floor; root cause=1 CTA/SM occupancy (SMEM-capped); levers=wave-aware NSPLIT + double warps/SM (attention occupancy knobs); do-not-retread list" && echo committed

The output is a single word: committed. But this commit message is far more than a routine check-in. It is the compressed summary of a scientific discovery that fundamentally rewrote the team's understanding of where the DeepSeek-V4-Flash decode performance bottleneck actually lived — and it represents a critical turning point in the optimization campaign.

The Premise That Was Wrong

To understand why this message matters, we must first understand the premise it overturned. Throughout the preceding optimization work, the team had been operating under a reasonable assumption: the Mixture-of-Experts (MoE) grouped-GEMM kernel was the primary scaling bottleneck at high batch sizes. The MoE FP4 kernel dominated 28-35% of the decode step time in earlier traces, and the model's architecture — with 43 layers of expert-routed computation — made MoE the natural suspect. The assistant's performance plan (document DSV4_DECODE_PERF_PLAN.md) had been organized around this assumption, with lever #3 explicitly targeting "MoE/attention occupancy."

But the assistant did something crucial before committing to a costly kernel-engineering effort: it gathered fresh data. Rather than relying on existing profiler traces captured at batch size 32 — which predated the current optimization campaign — the assistant launched two parallel subagents ([msg 13527]) to conduct a deep investigation. One subagent was tasked with analyzing the MoE grouped-GEMM kernel's occupancy characteristics and capturing a fresh high-batch profiler trace. The other investigated the attention kernel and the per-decode-step launch structure. This dual-pronged approach, combining code inspection with empirical measurement, was the key methodological choice that enabled the discovery.

What the Fresh Trace Revealed

The subagents returned with results that upended the working model. The fresh profiler trace at batch size 80 told a dramatically different story from the old batch-32 traces. The MoE GEMM, which had appeared to be the dominant cost, turned out to be essentially flat: it went from 15.1 milliseconds at batch 32 to just 15.9 milliseconds at batch 80 — an increase of only 0.015 milliseconds per additional request. The MoE was a fixed floor, not a scaling cost.

The real culprit was the attention kernel. It jumped from 13.1 milliseconds at batch 32 to 51.2 milliseconds at batch 80, contributing 0.79 milliseconds per additional request — essentially the entire 1.05 milliseconds-per-request marginal cost that defined the decode throughput curve. This was the finding that "revises premise," as the commit message so succinctly states. The attention kernel, not the MoE, was setting the ceiling on high-concurrency decode throughput.

This discovery had immediate practical consequences. The MoE was not worth further optimization for the C60→C90 push. Any effort spent on MoE kernel engineering would have yielded negligible returns at best. The attention kernel was where the leverage lived.

Root Cause: The 1 CTA/SM Occupancy Wall

The subagents also identified the root cause of the attention bottleneck: a fundamental occupancy problem rooted in shared memory constraints. The Blackwell RTX PRO 6000 GPUs (sm120 architecture) provide approximately 100KB of shared memory per streaming multiprocessor (SM). The attention kernel consumes 80KB of that, leaving almost no room for additional thread blocks. The result is that only one cooperative thread array (CTA) can run per SM at a time, with just 4-12 warps active out of a possible 48. This is an occupancy rate of roughly 8%.

The consequence is pernicious. The SMs appear "active" in profiler metrics (97% utilization), but they are not doing useful work per cycle. They are stalled on memory latency, with only 57% power draw and 27% memory controller utilization — classic signs of a latency-bound kernel where the SMs are occupied but waiting. The single CTA per SM cannot hide memory latency through warp-level parallelism, so every memory access becomes a stall point. This is the "1 CTA/SM occupancy (SMEM-capped)" that the commit message identifies as the root cause.## The Levers: Wave-Aware NSPLIT and Doubling Warps

With the bottleneck identified, the assistant catalogued two concrete, low-risk levers that could directly attack the attention kernel's occupancy problem. Both are config-level changes gated by the kernel's built-in numerical validation (a relative error check of ≤6.7×10⁻³), meaning they carry minimal correctness risk.

The first lever is wave-aware NSPLIT selection. The attention kernel uses a split-K strategy where the KV dimension is divided across multiple CTAs, each computing a partial result that is later reduced. The number of splits (nsplit) determines the grid size — how many CTAs are launched. At batch size 64 with nsplit=2, the grid produces 256 CTAs. On a machine with 188 SMs, this means wave 1 fills all 188 SMs, but wave 2 fills only 68 — wasting roughly 32% of the SM capacity in the second wave. At batch size 96, the same nsplit=2 produces 384 CTAs, which pack into waves much more efficiently (188 + 188 + 8). The fix is trivial in concept: adjust the split count so that the grid size is always a near-multiple of the SM count, minimizing fractional-wave tails. The assistant estimated this could recover 15-25% of attention time at the problematic batch sizes (64-88), with just a few lines of code changed in the nsplit selection logic.

The second lever is doubling the warps per SM by reducing the attention kernel's shared memory staging footprint. The current configuration uses MMA_BLOCK_H=32 and 128-thread blocks (4 warps). Switching to MMA_BLOCK_H=16 or increasing num_warps to 8 would double the active warps per SM, directly improving the kernel's ability to hide memory latency. This is a larger win but carries more risk — it requires careful validation that the reduced tile size doesn't introduce numerical issues or degrade the arithmetic intensity to the point where the occupancy gain is offset by lower compute efficiency.

The Do-Not-Retread List: Saving Future Effort

A subtle but valuable output of this research is the "do-not-retread list" — a catalog of optimization approaches that have been confirmed as dead ends on this specific hardware configuration (sm120 Blackwell GPUs with PCIe interconnect). The list includes mscclpp (NVIDIA's multi-node collective library, incompatible with the single-node TP setup), SBO (static batch optimization, not applicable to dynamic decode), EP (expert parallelism, which requires multi-node and doesn't help TP-bound decode), allreduce fusion (already at the PCIe bandwidth floor), torch.compile (broken on sm120 for these kernels), persistent GEMM kernels (requires GDC support, which is broken on Blackwell), MTP speculative decoding at high concurrency (memory overhead dominates), and NCCL Tree algorithm (no benefit over the default Ring on PCIe).

This list is valuable because it prevents the team from wasting time retreading paths that others have already explored and eliminated. In a complex optimization campaign spanning weeks, the cost of rediscovering a dead end is high. Having a written record of what doesn't work is almost as important as knowing what does.

The Thinking Process: Evidence Over Intuition

The assistant's reasoning in the lead-up to this commit ([msg 13528]) reveals a disciplined, evidence-driven methodology. The initial instinct was to optimize the MoE kernel — a natural choice given its dominance in earlier traces. But rather than committing to that path, the assistant paused to gather fresh data. The key insight was that the existing profiler traces were taken at batch size 32, which predated the current optimization campaign and might not reflect the behavior at the target batch sizes of 60-96. By launching parallel subagents to capture a fresh high-batch trace and analyze both the MoE and attention kernels simultaneously, the assistant ensured that the optimization effort would be directed at the actual bottleneck, not the assumed one.

This is a textbook example of the scientific method in systems engineering: form a hypothesis, design an experiment to test it, gather data, and let the data revise the hypothesis. The commit message is the artifact that captures this revision — a permanent record that the premise has changed.## Input Knowledge Required

To fully understand this message, one needs familiarity with several domains of GPU computing. The concept of occupancy — the ratio of active warps to the maximum possible per SM — is fundamental to GPU performance analysis. Low occupancy means the SM cannot hide memory latency through warp-level parallelism, which is exactly the problem diagnosed here. The distinction between compute-bound and latency-bound kernels is also essential: a kernel can show 97% SM utilization (the SMs are never idle) while being latency-bound (each warp is stalled waiting for data), which explains the counterintuitive combination of high SM activity with low power draw.

The shared memory (SMEM) per SM constraint is the physical limit that drives the occupancy problem. On Blackwell sm120, each SM has approximately 100KB of shared memory, and the attention kernel uses 80KB of it. This means only one CTA can reside on an SM at a time, regardless of how many warps the hardware could theoretically support. Understanding the CTA/SM relationship — how thread blocks are scheduled onto SMs in waves — is necessary to grasp why wave quantization matters and why a grid of 256 CTAs on 188 SMs wastes capacity.

The split-K attention mechanism and the nsplit parameter are specific to the MLA (Multi-head Latent Attention) implementation in DeepSeek-V4-Flash. The KV dimension is split across multiple CTAs that compute partial results, then reduce them. The number of splits directly determines the grid size and thus the wave quantization efficiency. The MMA_BLOCK_H parameter controls the tile height in the matrix-multiply-accumulate operation, which directly affects shared memory usage and therefore occupancy.

Finally, the do-not-retread list references a constellation of optimization techniques — mscclpp, SBO, EP, GDC, MTP — that require deep knowledge of the SGLang/vLLM ecosystem and the specific hardware limitations of sm120 Blackwell GPUs. Without this context, the list reads as jargon; with it, it represents months of accumulated engineering wisdom.

Output Knowledge Created

This commit produces several distinct forms of knowledge. First, it creates a revised bottleneck model for DeepSeek-V4-Flash decode on Blackwell: attention is the slope driver, MoE is a fixed floor. This is the most actionable finding because it redirects optimization effort from the MoE (which would yield negligible returns) to the attention kernel (where the leverage is). Second, it establishes the root cause mechanism — 1 CTA/SM occupancy due to SMEM capping — which explains not just the attention bottleneck but also why the MoE and other kernels exhibit similar characteristics. Third, it identifies two specific, low-risk levers (wave-aware NSPLIT and doubling warps/SM) that can be implemented as environment-variable or config changes, avoiding the need for invasive kernel rewrites. Fourth, it produces the do-not-retread list, a negative-knowledge artifact that prevents wasted effort on approaches known to be infeasible on this hardware.

The commit also implicitly validates the methodology of gathering fresh profiler traces at the target batch size rather than relying on old data. This is a meta-lesson: performance characteristics can shift dramatically with batch size, and optimization decisions should always be based on measurements taken at the operating point of interest.

Mistakes and Assumptions

The primary incorrect assumption that this message corrects is that the MoE kernel was the dominant scaling bottleneck. This was a reasonable hypothesis based on earlier traces at batch size 32, where the MoE appeared to dominate the step time. The mistake was not in the hypothesis itself but in the risk of acting on it without verification. The assistant avoided this trap by gathering fresh data before committing engineering resources.

A more subtle assumption embedded in the earlier work was that the performance characteristics at batch 32 would extrapolate linearly to batch 80. This turned out to be false: the MoE kernel's cost is nearly flat across batch sizes (it is dominated by weight loading, which is amortized differently than the assistant expected), while the attention kernel's cost grows superlinearly due to the increasing KV dimension. The fresh trace revealed this nonlinearity, which the old data could not have captured.

There is also an implicit assumption in the levers themselves: that the kernel's built-in numerical validation (the rel≤6.7×10⁻³ check) is sufficient to guarantee correctness after changing block sizes or split counts. This is a reasonable assumption given that the validation is designed for this purpose, but it is still an assumption that would need to be verified in production.

Conclusion

Message [msg 13529] is a single git commit — a moment of documentation that captures a scientific discovery. In the span of a few dozen words, it records that the premise of the optimization campaign was wrong, presents the corrected understanding with hard data, identifies the root cause mechanism, proposes two concrete interventions, and catalogs the dead ends to avoid. The commit message functions as a compressed research paper, a project management artifact, and a permanent record of intellectual progress. For anyone reading the project history weeks or months later, this single message contains everything needed to understand why the optimization strategy shifted from MoE to attention — and why that shift was the right call.