The Architecture Detective: How an AI Assistant Systematically Mapped a Blackwell Sparse Attention Kernel for Split-K Rewrite

Introduction

In the high-stakes world of large language model inference on cutting-edge hardware, the difference between a responsive service and a sluggish one often comes down to a single GPU kernel. When a team deploying DeepSeek-V4-Flash on NVIDIA's Blackwell RTX PRO 6000 GPUs hit a performance wall—a mere ~28 tokens/second constrained by sm_120 fallback kernel bottlenecks—they needed to go deeper. The bottleneck was the sparse Multi-head Latent Attention (MLA) decode kernel, a piece of custom Triton code that had become the critical path, consuming 63% of decode time while launching only 64 thread blocks on a GPU with 188 streaming multiprocessors.

This article examines a complete subagent session (19 messages) from an opencode coding session, where an AI assistant was tasked with producing a precise technical specification for rewriting that kernel with a split-K optimization. The session is a masterclass in systematic code analysis: the assistant reads source files, greps for constants, traces tensor shapes through layers of abstraction, examines memory layouts, and ultimately synthesizes a comprehensive blueprint for a kernel rewrite. It is a rare window into how deep code comprehension unfolds—not as a single flash of insight, but as a gradual, methodical process of building, testing, and refining a mental model of a complex system.

The Performance Crisis: Why Split-K Was Needed

The root session had achieved a major engineering milestone: deploying DeepSeek-V4-Flash with NVFP4 quantization on 8× RTX PRO 6000 Blackwell GPUs with prefill-decode disaggregation. But performance was catastrophic relative to expectations. GPU profiling traced 63% of decode time to a single kernel: _tiled_sparse_decode_kernel, the sm_120 Triton fallback for sparse MLA attention.

The root cause was an occupancy collapse. The kernel used a grid of (B, H)—one thread block per (batch, head) pair. At batch size 1 with 64 query heads, this launched only 64 blocks on a GPU with ~170-188 SMs. Each block then serially iterated through all 512 top-k tokens in tiles of 16 or 32. The result: approximately 124 SMs sat idle, achieving less than 34% utilization. The kernel was purely memory-bound—streaming FP8 KV cache data from GDDR7—but with only 64 active blocks, it couldn't saturate the memory bus.

The proposed solution was split-K: changing the grid from (B, H) to (B, H, NSPLIT), where NSPLIT is between 8 and 16. This would launch 512 to 1024 blocks, filling all SMs with multiple waves. Each split processes a contiguous slice of the top-k list, producing partial attention outputs and log-sum-exp (LSE) values. A second combine kernel then merges the partials using an online softmax reduction. This is a well-known technique for improving occupancy when the inner reduction dimension (top-k) is large relative to the outer grid dimensions.

The Subagent Mission: Map Before Rewrite

The user's opening message ([msg 0]) was a precise technical brief: "Read these files... and produce a precise technical spec I can use to REWRITE the sm_120 sparse-MLA decode attention kernel with split-K. Do NOT modify anything; this is read-only analysis." The user specified three files to examine and six deliverables: kernel interface details, the mathematical computation, grid parallelism analysis, output combine logic, validation hooks, and decode batch shapes.

The assistant's response was not to jump into implementation but to embark on a systematic investigation spanning 18 subsequent messages. This subagent session, operating under a read-only mandate, demonstrates a sophisticated approach to understanding a complex production codebase before attempting to modify it.

Phase 1: Reading the Kernel and Dispatch Layer

The assistant began by reading all three specified files in parallel ([msg 1]). The Triton kernel file (flash_mla_sm120_triton.py) revealed the kernel's tiled structure: a BLOCK_T loop over top-k tokens, FP8 KV cache gathering with UE8M0 dequantization, online softmax with base-2 exponentials, and NOPE/ROPE output accumulation. The dispatch file (flash_mla_sm120.py) showed how the kernel is selected via environment variable (SGLANG_SM120_TRITON_FLASHMLA) and the pure-PyTorch reference implementation used for validation.

But the assistant quickly recognized that understanding a kernel requires understanding its ecosystem. As [msg 3] reveals, the assistant paused to "trace back through the code to understand the broader context of this decode dispatch—looking at the function signature, how q and swa_page_indices are initialized, and the overall flow that leads to these lines." This metacognitive moment—recognizing that the kernel cannot be understood in isolation—drove the entire subsequent investigation.

Phase 2: Precision Probing with Grep

With the broad architecture established, the assistant deployed surgically targeted grep commands to extract exact numerical constants. [msg 4] searched for head dimension parameters (head_dim_v, num_q_heads, num_kv_heads, softmax_scale, qk_nope_head_dim, qk_rope_head_dim, kv_lora_rank) and infrastructure constants (_is_sm120, SWA_WINDOW, page_size). The results confirmed the critical geometry: head_dim = 512 (448 nope + 64 rope), 64 query heads with 1 KV head (MQA), softmax_scale = 512^-0.5, and SWA_WINDOW = 128.

[msg 6] executed another targeted grep for C4_TOPK, index_topk, kv_cache_total_dim, and the DeepSeekV4TokenToKVPool class. This revealed C4_TOPK = 512 (or 1024 for larger models), kv_cache_total_dim = 584 bytes per token, and the page sizes for the dual sparse caches (SWA with page_size=128, c4 with page_size=64). These constants are the DNA of the attention mechanism—change any one and the entire computation shifts.

Phase 3: Tracing the Memory Layout

A critical discovery was the FP8 KV cache memory layout, documented across multiple files. The assistant traced the "576 stride" invariant: each token occupies 584 bytes in the cache, organized as 448 bytes of FP8 NOPE values, 128 bytes of BF16 ROPE values (64 elements × 2 bytes), and 8 bytes of UE8M0 scale factors. The data section uses a stride of 576 bytes per token (448 + 128), while the scale section is stored separately at a fixed offset within each page.

The addressing invariant, documented in the final specification ([msg 18]), is:

page = idx // page_size
slot = idx % page_size
data_base = page * page_bytes + slot * 576
nope[i]   = cache_fp8[data_base + i]           for i in [0, 448)
rope[j]   = cache_bf16[(data_base + 448) // 2 + j]  for j in [0, 64)
scale[g]  = cache_uint8[page * page_bytes + scale_section_off + slot * 8 + g]

This invariant is the single most critical piece of knowledge for the rewrite. If the split-K kernel gets this wrong, it will read garbage data and produce incorrect results.

Phase 4: Understanding the Index Topography

The sparse attention mechanism uses two distinct caches: a main SWA (Sliding Window Attention) cache with a window of 128 tokens, and an extra c4/c128 cache with content-based sparse indices (512 or 1024 tokens). Each cache has its own page indices and top-k lengths, and the kernel is called twice per decode step—once for each cache—with results merged via LSE-weighted combination.

[msg 8] examined the DSV4AttnMetadata dataclass, revealing the fields that carry per-request indexing information: swa_page_indices, swa_topk_lengths, c4_sparse_page_indices, c4_sparse_topk_lengths, and others. [msg 9] traced the construction of these indices through the init_flashmla_related function. [msg 10] confirmed the alignment rules: PAGE_INDEX_ALIGNED_SIZE = 64, with indices padded to the nearest multiple of 64 via _pad_last_dim. [msg 11] read the padding function itself, confirming that SWA indices have shape (num_qo_tokens, 128) and c4 indices have shape (num_qo_tokens, 512), both padded to 64-element alignment.

This index topography is essential for the split-K design because the kernel must know, for each batch element, how many valid tokens to process and where to find their KV cache data. The split-K partition must respect the per-row valid counts and avoid splitting across padding entries.

Phase 5: The Merge State Discovery

[msg 14] marked a pivotal shift from data collection to design synthesis. The assistant confirmed the core architectural parameters (H=64, MQA, q_len=1 at decode) and then searched for existing split-K infrastructure. The grep for merge_state|merge_attn_states|split_k|num_splits returned 381 matches, revealing that merge_state_v2 already existed in the codebase for XPU backends.

[msg 15] read the existing merge_state.py Triton kernel, discovering that it implements a two-way LSE-weighted combination of attention outputs. While this kernel was designed for merging results from two separate sparse caches (SWA + extra), not for reducing across split-K partials, it provided a ready-made template for the combine step. The assistant recognized that the split-K combine could generalize this pattern to N-way reduction.

This discovery was architecturally significant. It meant the split-K rewrite could leverage existing infrastructure rather than building everything from scratch. The combine kernel would need to be adapted from 2-way to N-way, but the mathematical framework—online softmax reduction with LSE-weighted averaging—was already proven.

Phase 6: Verification and Cross-Validation

Throughout the session, the assistant systematically verified its assumptions against multiple sources of truth. [msg 12] examined the model configuration to confirm v_head_dim = 512 and the number of attention heads. [msg 13] read the DSV4 attention test file to cross-reference head counts and tensor shapes against the production code. [msg 16] verified the SWA cache page_size relationship by reading the KV pool class definition.

This triangulation—checking the kernel code, the dispatch logic, the model configuration, the test harness, and the memory pool—reduced the risk of propagating incorrect assumptions into the specification. The assistant's reasoning in [msg 12] explicitly questioned whether the kernel's accumulation of both nope and rope into the output was semantically correct, noting that "in standard MLA, rope only affects the attention scores, not the value accumulation." This kind of critical thinking—not just documenting the code but questioning its correctness—is the hallmark of a thorough kernel engineer.

The Final Specification: A Blueprint for Action

[msg 18] is the culmination of the entire subagent session: a comprehensive technical specification for the split-K rewrite, spanning six sections with exact file:line references. The specification documents:

  1. Kernel Interface: All 24 parameters of _tiled_sparse_decode_kernel with shapes, dtypes, strides, and the FP8 KV page layout with the 576-stride addressing invariant.
  2. The Math: A step-by-step walkthrough of the computation per (batch, head) block, including the critical clarification that there is no ReLU, no per-token weighting, and no attention sink within the kernel—these are applied externally.
  3. Grid + Parallelism: Quantification of the occupancy collapse (64 CTAs on 188 SMs, ~34% utilization) and the split-K solution: grid (B, H, NSPLIT) with NSPLIT=8..16, producing 512..1024 CTAs.
  4. Output Combine: The split-K combine must be internal to _run_triton_sparse_decode, producing out[B,1,H,512] + lse[B,1,H] compatible with the existing _merge_partial_attn and _apply_attn_sink wrappers.
  5. Validation Hook: The torch reference signature and the expected tolerance (DSV4_ATOL = DSV4_RTOL = 8e-2), with the env-var toggle as the cleanest A/B test harness.
  6. Decode Batch Shapes: The model geometry (head_dim=512, H=64, MQA), the dual cache configuration (SWA topk=128, c4 topk=512), and how MTP flattens draft tokens into the batch dimension.

The Broader Significance

This subagent session exemplifies a pattern that recurs throughout high-stakes kernel engineering: the transition from diagnosis to prescription requires a complete and verified mental model of the existing system. The assistant did not start writing code. It read files, grepped for constants, traced data flows, verified assumptions against multiple sources, and only then produced a specification.

The session also reveals the challenges of working with cutting-edge hardware. The sm_120 architecture lacks the fused flash_mla CUDA kernel available on Hopper, forcing reliance on Triton fallback paths. The split-K rewrite is an attempt to mitigate this architectural limitation through clever parallelism, but as the broader root session would later confirm, even a successful split-K implementation can only deliver incremental gains against the fundamental ceiling of sm_120 fallback kernels.

For anyone interested in GPU kernel optimization, large language model inference, or the Blackwell architecture, this session offers a rare window into the engineering process behind high-performance AI systems. It shows that the difference between a model that runs at 28 tokens per second and one that saturates the GPU's memory bandwidth is not just a matter of better algorithms—it is a matter of understanding the hardware's parallelism model, tracing through thousands of lines of code, and documenting every assumption with precision.

The assistant's methodical approach—reading broadly first, then using targeted searches to fill gaps, then verifying assumptions against multiple sources, then synthesizing into a coherent specification—is a template for how to approach any complex kernel optimization task. It is a reminder that the most valuable contributions often come not from writing code, but from the systematic process of understanding, validating, and documenting the existing system before attempting to modify it.## References

[1] "The Kernel Archaeology Mandate: Deconstructing sm_120 Sparse-MLA Decode for Split-K Rewrite" — Analysis of the user's opening message that set the subagent's mission.

[2] "The Architecture Detective: Unraveling the SM120 Sparse-MLA Decode Kernel" — Deep analysis of the assistant's architectural synthesis in message 17.

[3] "Tracing the Index Topography: Understanding Sparse Attention Metadata in Blackwell's SM120 Decode Kernel" — Examination of message 10's investigation into index alignment.

[4] "The Connecting Thread: How a Single Verification Read Unlocks Sparse-MLA Decode Kernel Understanding" — Analysis of message 16's verification of page_size relationships.

[5] "Mapping the Metadata: How One Read Unlocks the Sparse-MLA Kernel's Data Layout" — Examination of message 8's pivot to metadata structures.

[6] "The Merge State Discovery: A Pivotal Moment in Sparse-MLA Decode Kernel Analysis" — Analysis of message 15's discovery of existing merge infrastructure.

[7] "Tracing the Decode Metadata: A Deep Dive into Sparse-MLA Index Construction on Blackwell GPUs" — Examination of message 9's trace through index construction.

[8] "Reading the Test File: Validating Assumptions for a Split-K Sparse MLA Kernel" — Analysis of message 13's cross-validation against test infrastructure.

[9] "The Hunt for C4_TOPK: A Subagent's Quest to Map the Sparse-MLA Decode Kernel's Memory Layout" — Examination of message 7's pivot to the memory pool.

[10] "Peeling Back the Layers: A Methodical Deep-Dive into the DeepSeek-V4 Attention Backend Initialization" — Analysis of message 5's reading of the initialization section.

[11] "The Art of Tracing Back: How One Assistant Message Reveals the Hidden Depth of Code Analysis" — Examination of message 3's metacognitive pivot.

[12] "The Architect's Scalpel: Precision Probing in the SM120 Sparse-MLA Decode Kernel Analysis" — Analysis of message 4's targeted grep commands.

[13] "The Data-Gathering Pivot: How a Single Grep Command Unlocks a Kernel Rewrite" — Examination of message 6's transition to numerical precision.

[14] "The Pivot Point: Confirming Architecture and Seeking Split-K Infrastructure in a Blackwell Sparse-MLA Decode Kernel" — Analysis of message 14's synthesis and search for merge infrastructure.

[15] "Opening the Black Box: The First Step Toward Rewriting a Blackwell Sparse Attention Kernel" — Analysis of message 1's initial file reads.

[16] "The Anatomy of a GPU Kernel Specification: Dissecting the SM120 Sparse-MLA Decode Attention Kernel Document" — Comprehensive analysis of message 18's final specification.

[17] "The Sparse-MLA Decode Kernel Deep Dive: Tracing the Path to Split-K on Blackwell" — Examination of message 12's verification of head dimensions.

[18] "Reading the Index Alignment: How a Single Read Unlocked the Sparse-MLA Kernel Architecture" — Analysis of message 11's investigation of padding invariants.

[19] "The Methodical Read: How an AI Assistant Systematically Unravels a GPU Kernel's Dispatch Logic" — Analysis of message 2's sequential reading strategy.