The Split-K Decision: A Single Edit That Embodies GPU Kernel Engineering

"Now insert the split + combine kernels after the existing _mma_sparse_decode_kernel:"

This seven-word instruction, followed by a file path and a success confirmation, is the subject message of this analysis ([msg 12563]). On its surface, it is the most mundane of coding actions: an edit to a Python file. But in the context of the broader optimization campaign for DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell (sm_120) GPUs, this single edit represents the culmination of an extraordinarily intricate chain of reasoning — a moment where dozens of interdependent engineering constraints converged into a concrete implementation decision.

The Strategic Crossroads

To understand why this message matters, one must appreciate the state of the optimization campaign at the moment it was written. The assistant had just completed a transformative rewrite of the sparse-MLA attention kernel, replacing a per-head SIMT (single-instruction, multiple-thread) kernel with a tensor-core MMA (matrix-matrix-accumulate) implementation using Triton's tl.dot operator. The results were dramatic: attention GPU time dropped from 5,350ms to 850ms — a 6.3× improvement — and total decode GPU time halved. At the user's target concurrency of C=64, throughput doubled from 29.7 to 59.6 tokens per second ([msg 12560]).

But the MMA rewrite introduced a new problem: a severe regression at C=1 (single concurrent request), where throughput actually dropped from 11.5 to 8.1 tok/s. The root cause was occupancy collapse. The new kernel used a head-batched grid structure of (B, H/BLOCK_H) blocks, where BLOCK_H=32 and H=64 meant only 2 blocks at B=1. On a GPU with 188 streaming multiprocessors (SMs), 2 blocks meant 99% of the hardware sat idle.

This created a strategic dilemma. The profile showed that attention was no longer the bottleneck — it had fallen from 57% to 17.6% of GPU time. The new dominant cost was "glue" code: unfused elementwise operations, copy kernels, and reductions that collectively consumed ~54% of runtime. The assistant had two clear paths forward:

  1. Proceed to torch.compile on the decode path to fuse the ~6,000 elementwise/copy/reduce launches — the biggest remaining throughput lever at C=16–64.
  2. Finish the kernel work by adding split-K parallelization over the topk dimension to fix the C=1 regression and improve occupancy at all batch sizes. The assistant initially leaned toward torch.compile, reasoning that "C=1 is a latency edge case, not the throughput target" and that "the real gains now come from tackling the glue (54% of remaining time)" ([msg 12560]). But the assistant presented both options to the user as a structured question. The user's answer was decisive: "Finish the kernel: add split-K first" ([msg 12561]).

The Reasoning Behind the Edit

Message [msg 12562], which immediately precedes the subject message, contains the assistant's extensive reasoning about how to implement split-K. This reasoning reveals the extraordinary depth of engineering consideration packed into the seemingly simple edit command.

The core idea of split-K is to parallelize the attention computation over the topk dimension — the set of cached key-value tokens that each query attends to. In the existing MMA kernel, a single CUDA block processed all topk tokens for its assigned head group. With split-K, the topk tokens are partitioned across multiple blocks, each computing partial accumulators. A separate "combine" kernel then merges these partial results using the log-sum-exp (LSE) trick for numerical stability.

The assistant's reasoning traversed multiple interconnected design dimensions:

Occupancy modeling. The assistant calculated target block counts of 188–376 to saturate the GPU's 188 SMs. For B=1 with 2 head groups, a naive grid of 2 blocks was hopeless. But with NSPLIT=8 (splitting topk into 8 chunks), the grid expanded to 16 blocks — still underutilized but an 8× improvement. For B=16, NSPLIT≈6 yielded 192 blocks, nearly perfect occupancy. For B=64, NSPLIT=2 gave 256 blocks. The adaptive formula naturally balanced occupancy across batch sizes.

CUDA graph safety. This was a critical constraint. SGLang uses CUDA graphs to capture and replay GPU operations with minimal CPU launch overhead. For graph capture to work, all tensor sizes and kernel launch configurations must be deterministic given the batch size. The assistant carefully ensured that NSPLIT is computed deterministically from B, that scratch buffer sizes are fixed per graph bucket, and that the branch between the single-kernel path (NSPLIT=1) and the split-combine path is stable during capture.

Memory traffic analysis. The scratch buffers for partial accumulators, max values (m), and log-sum-exp values (l) added memory pressure. The assistant calculated that with bf16 precision for accumulators, each layer would write and read approximately 6.2 MB, totaling ~530 MB across 43 layers per decode step — "adding negligible latency" in the context of the overall operation.

Precision and numerical stability. The combine kernel uses the standard online softmax trick: finding the global maximum m across all splits, computing correction factors with exp2(m_global - m_local), and normalizing the accumulated values. The assistant chose f32 for the m and l scratch buffers to preserve precision, while using bf16 for the accumulator to halve memory traffic.

Autotuner interaction. The Triton autotuner selects tile sizes (BLOCK_T) during warmup on the first kernel invocation. Since different batch sizes produce different chunk sizes, the autotuner might pick a suboptimal configuration for some cases. The assistant acknowledged this but judged it acceptable since "the tiling logic handles variable chunks correctly."

Environment configurability. The assistant added environment variable knobs for target block count, maximum split count, and minimum chunk size, allowing future tuning without code changes.

Assumptions and Their Validity

The implementation rested on several key assumptions:

  1. That the split-combine approach would actually fix the C=1 regression. This was well-founded: increasing the grid from 2 blocks to 16+ blocks directly addressed the occupancy collapse. The combine kernel added overhead, but at B=1 the combine grid of (1, 2) blocks was negligible.
  2. That bf16 precision for partial accumulators was sufficient. The assistant noted that "the tradeoff of bf16 precision for the partial accumulation is acceptable given the tolerance requirements." This was validated later when correctness tests showed relative errors of 3.8–6.7×10⁻³, consistent with the existing kernel's tolerance ([msg 12567]).
  3. That CUDA graph capture would work with the split-combine path. The assistant carefully designed for this, but it remained unproven until the graph was actually captured and replayed. The subsequent validation confirmed it worked.
  4. That the user's preference for "finish the kernel first" was the right strategic call. This assumption was the most consequential. The profile showed glue code at 54% of GPU time versus attention at 17.6%. Improving a component that was already only 17.6% of runtime had diminishing returns. However, the C=1 regression was a qualitative problem — a regression is harder to accept than a missing optimization. The user's intuition was that a complete kernel should not have regressions, and this proved correct in the broader arc of the campaign.

Input Knowledge Required

To understand this message, one needs knowledge spanning multiple domains:

Output Knowledge Created

This message created the split-K implementation — two new Triton kernels (a split kernel and a combine kernel) plus the orchestration logic in the wrapper function. The immediate output was a file edit, but the knowledge created included:

The Thinking Process Visible

The assistant's reasoning in [msg 12562] reveals a methodical, multi-pass design process. The assistant cycles through the same design multiple times, each pass adding more detail: first the high-level split-combine architecture, then memory sizing, then CUDA graph implications, then autotuner interactions, then environment configurability. This is not linear reasoning but iterative deepening — a hallmark of expert system design.

The assistant also demonstrates a strong awareness of tradeoffs. When considering whether to use a torch-based combine or a dedicated Triton kernel, the assistant notes that "the torch combine would work but adds extra elementwise kernels and launches, which defeats the purpose of reducing glue overhead." This shows the assistant thinking holistically about the system, not just implementing a feature in isolation.

Conclusion

Message [msg 12563] is a study in compression. Seven words of instruction encode hours of reasoning about occupancy models, numerical precision, memory hierarchies, autotuner behavior, and graph capture semantics. It is the point where analysis becomes action — where the split-K design, refined through multiple passes of reasoning, finally touches disk. The edit itself is trivial; the engineering judgment behind it is anything but.