The No-Free-Lunch Theorem of GPU Kernel Tuning: Wave-Aware NSPLIT on Blackwell

In the high-stakes world of production GPU kernel optimization, the most valuable insights often arrive not as breakthroughs but as tradeoffs. Message 13541 of this opencode session captures exactly such a moment: the assistant, deep in a multi-day effort to squeeze every drop of decode throughput from a DeepSeek-V4-Flash model running on NVIDIA Blackwell RTX PRO 6000 GPUs, confronts the uncomfortable truth that no single knob setting is optimal for all batch sizes. The response is not to pick the lesser evil, but to reach for a more sophisticated solution: wave-aware dynamic NSPLIT selection. This message is a masterclass in evidence-driven decision-making, combining rigorous empirical benchmarking, deep architectural understanding of GPU wave quantization, and a pragmatic willingness to implement code-level changes when environment variables reach their limits.

The Optimization Landscape

To understand the significance of this message, one must appreciate the context. The assistant has been systematically tuning the sparse Multi-head Latent Attention (MLA) decode kernel—specifically the _run_mma_sparse_decode function in flash_mla_sm120_triton.py—which is the computational heart of the DeepSeek-V4-Flash model's autoregressive generation. This kernel runs on NVIDIA's Blackwell architecture (SM120), which features 188 Streaming Multiprocessors (SMs), 99KB of shared memory per SM, and approximately 1.5 TB/s of GDDR7 memory bandwidth.

The kernel's performance is governed by several tunable parameters exposed as environment variables: SGLANG_SM120_MMA_BLOCK_H (the number of attention heads processed per thread block), SGLANG_SM120_MMA_TARGET_CTAS (the target number of cooperative thread arrays to launch), SGLANG_SM120_MMA_NSPLIT_MAX (the maximum split count), and SGLANG_SM120_MMA_MIN_CHUNK (the minimum token count per split). The assistant had already spent considerable effort exploring this parameter space. A baseline run had revealed a troubling wave-quantization anomaly: at concurrency level C=96, throughput (799.3 tok/s) was lower than at C=80 (833.0 tok/s)—a non-monotonic scaling behavior that indicated severe underutilization of the GPU's SMs due to poor wave packing.

The first attempted fix, reducing BLOCK_H from 32 to 16, had failed spectacularly. The reasoning was sound on paper: halving the block height should halve shared memory usage per CTA, allowing two CTAs per SM instead of one, doubling occupancy and improving latency hiding. But the kernel uses 255 registers per thread across 128 threads, consuming 32,640 registers per CTA. Two CTAs would need 65,280 registers, exceeding the SM's 65,536-register file by a hair. The kernel was register-capped at one CTA per SM regardless of shared memory, so the change merely shrank each CTA's work without gaining occupancy, producing a 3–9% regression across the board.

The second attempt, raising TARGET_CTAS from 256 to 512, was a clear win. It resolved the wave-quantization anomaly, producing monotonic scaling from C=48 through C=96, with gains of +12.8% at C=64 and +5.7% at C=96. The fix was validated with a 60-session corruption test showing 0% corruption, committed to git, and deployed. But the assistant, driven by a relentless pursuit of optimality, decided to probe further by testing TARGET_CTAS=768.

The Decisive Tradeoff

Message 13541 opens with the analysis of that probe. The results are unambiguous and painful: TARGET_CTAS=768 improves C=96 throughput by 3.3% (from 845 to 872 tok/s) but regresses C=64 by 5.4% (from 811 to 767 tok/s). The assistant's reasoning captures the essence of the dilemma:

"This confirms that there's no universally optimal TARGET_CTAS value; the best choice depends on the specific batch characteristics."

This is the no-free-lunch theorem of GPU kernel tuning made concrete. The TARGET_CTAS parameter controls how many CTAs the kernel aims to launch. A higher target means more splits of the attention computation, which can fill more SMs and reduce wave-quantization waste—but each split introduces combine overhead (reducing the K and V matrices into smaller pieces that must be reduced later) and can push per-split work below the MIN_CHUNK floor, where the kernel becomes compute-bound on tiny tiles rather than memory-bound on large ones.

At C=64 with TARGET_CTAS=512, the kernel selects nsplit=4, achieving a wave utilization of 0.75 (meaning 75% of SMs are active in the final wave). At TARGET_CTAS=768, it selects nsplit=6, pushing utilization to 0.83—but the additional split overhead and smaller per-split tiles actually reduce overall throughput. The combine cost outweighs the wave-fill benefit. At C=96, the calculus flips: TARGET_CTAS=512 gives nsplit=3 with 0.77 utilization, while TARGET_CTAS=768 gives nsplit=4 with 0.82 utilization, and here the improved wave packing delivers a net gain.

The Reasoning Process: From Tradeoff to Solution

What makes this message remarkable is not the discovery of the tradeoff—many engineers would have simply picked the value that worked best for their most common batch size and moved on. Instead, the assistant performs a deeper analysis, recognizing that the tradeoff is structural rather than accidental, and that the correct response is not to compromise but to eliminate the tradeoff entirely through algorithmic adaptation.

The reasoning traces through several layers of analysis. First, the assistant identifies the core mechanism: wave quantization waste. When the total number of CTAs launched does not divide evenly by the number of SMs (188), the final wave is partially empty, wasting compute capacity. The waste fraction is (ceil(CTAs / 188) * 188 - CTAs) / (ceil(CTAs / 188) * 188). By adjusting nsplit, the kernel can change the total CTA count to better align with the SM count, reducing this waste.

But the assistant recognizes that the optimization is not simply "maximize nsplit." There is a countervailing force: combine overhead. Each split of the attention computation requires a separate kernel launch or a segmented computation within the kernel, and the results must be combined. The combine operation itself has a cost, and smaller splits mean less arithmetic intensity per CTA, potentially leaving the tensor cores underutilized. The optimal nsplit is the one that minimizes the combined cost of wave waste and split overhead.

The assistant articulates this tradeoff with precision:

"As nsplit increases, the quantization waste diminishes toward a constant baseline, but larger splits introduce combine overhead and hit practical caps. Looking at the actual numbers with C=64 and n_hg=2, I can see that nsplit=4 gives better wave utilization (0.75) than nsplit=6 (0.83), which explains why 512 outperformed 768—the overhead of more splits outweighs the marginal gain in wave fill."

This is the insight that drives the solution. If the optimal nsplit varies by batch size, then the kernel should not use a fixed formula based on TARGET_CTAS. Instead, it should compute the optimal nsplit dynamically, per batch, by evaluating the tradeoff across candidate values and picking the one that minimizes the efficiency ratio.

The Decision: Wave-Aware Dynamic NSPLIT

The assistant's conclusion is decisive and bold:

"The real solution is implementing wave-aware NSPLIT logic that dynamically selects the optimal split count per batch. This would capture the best performance from both scenarios — achieving the 12.8% improvement at C=64 and the 9.1% improvement at C=96 simultaneously, rather than being forced to choose between them."

This is not a trivial change. It requires modifying the Triton kernel's _run_mma_sparse_decode function to replace the current nsplit computation—which is driven by TARGET_CTAS, NSPLIT_MAX, and MIN_CHUNK—with a minimization procedure that iterates through candidate nsplit values, computes the expected wave utilization and combine overhead for each, and selects the optimal one. The assistant notes that a subagent has confirmed this is a "minimal ~5-line change with zero correctness risk," which is a critical validation—in a production system where corruption bugs have plagued the team for days, any change to the hot kernel path must be approached with extreme caution.

The assistant then proceeds to execute the first step of implementation: reading the current nsplit logic. The bash command copies the Triton kernel source file from the remote server and greps for the relevant parameters and function definitions:

ssh -o ConnectTimeout=15 root@10.1.230.171 "cp /root/sglang-dsv4/python/sglang/srt/layers/attention/flash_mla_sm120_triton.py /tmp/opencode/fmla.py"

The grep output reveals the key constants: _MMA_TARGET_CTAS (default 256), _MMA_NSPLIT_MAX (default 16), _MMA_MIN_CHUNK (default 32), and the function _run_mma_sparse_decode at line 653, with the n_hg computation at line 684 and the split logic at line 693.

Assumptions and Correctness Considerations

The assistant makes several assumptions in this message, most of which are well-supported by prior evidence. The assumption that wave-aware NSPLIT is "zero correctness risk" is grounded in the understanding that changing the split count does not alter the mathematical computation—it only changes how the work is partitioned across CTAs. The split-K reduction with Log-Sum-Exp (LSE) combine is mathematically exact regardless of split count, so correctness should be preserved. However, this assumption depends on the combine kernel being correctly implemented for all possible split counts, which the assistant has not explicitly verified for edge cases like nsplit=1 or nsplit=16 at extreme batch sizes.

Another assumption is that the combine overhead can be modeled accurately enough for the minimization to pick the right split. The assistant acknowledges this with the caveat "with a small bias toward lower nsplit to account for combine overhead," suggesting that the overhead model is approximate and may need empirical calibration.

There is also an implicit assumption that the batch size (concurrency level C) is the primary determinant of optimal nsplit, and that other factors like context length or sequence length distribution are secondary. This is reasonable for a decode kernel where all requests generate one token at a time, but it may need revisiting if the production workload includes variable-length sequences.

Input and Output Knowledge

To understand this message, the reader needs knowledge of: GPU architecture concepts (SMs, warps, CTAs, wave quantization, register pressure, shared memory), the NVIDIA Blackwell SM120 specification (188 SMs, register file size), the Triton kernel programming model, the DeepSeek-V4-Flash model architecture (specifically its sparse MLA attention mechanism), the concept of split-K reduction and LSE combine, and the prior benchmarking results from the session (the baseline, BLOCK_H=16 failure, TARGET_CTAS=512 success, and TARGET_CTAS=768 tradeoff).

The message creates several pieces of output knowledge: the definitive empirical demonstration that no single TARGET_CTAS value is optimal across batch sizes, the analytical framework for understanding the nsplit tradeoff (wave utilization vs. combine overhead), the decision to implement wave-aware dynamic NSPLIT, and the specific code location where the change will be made. This knowledge advances the project from a trial-and-error tuning phase to a principled optimization phase, where the kernel adapts to its workload rather than requiring manual configuration per deployment scenario.

The Broader Significance

Message 13541 represents a pivotal moment in the optimization journey. It marks the transition from environment-variable tuning—fast, safe, but fundamentally limited—to code-level adaptation—more powerful, more general, but requiring deeper understanding and careful implementation. The assistant's reasoning demonstrates a mature engineering sensibility: it does not settle for the best fixed configuration, but instead identifies the reason why no fixed configuration can be optimal and designs a solution that eliminates the constraint.

This approach—identify a tradeoff, understand its structure, then build a system that navigates it dynamically—is applicable far beyond GPU kernel tuning. It is the same pattern that drives adaptive bitrate streaming, dynamic voltage and frequency scaling, and any system where the optimal operating point depends on workload characteristics that vary at runtime.

The message also exemplifies the value of rigorous empirical methodology in performance engineering. The assistant did not speculate about which TARGET_CTAS value might be better; it ran controlled benchmarks, collected data, and let the numbers guide the decision. When the data revealed a tradeoff, it did not rationalize a compromise but instead asked: "Can I have both?" The answer, in this case, was yes—but only by changing the code.

As the session continues beyond this message, the assistant will implement the wave-aware NSPLIT logic, test it, and likely discover new complexities. But in this moment, captured in message 13541, we see the crystallization of an insight that will shape the rest of the optimization effort: the best knob is not a knob at all, but a decision procedure.