The Occupancy Epiphany: How One Assistant Identified GPU Grid Starvation as the Root Cause of 2% DRAM Utilization in Speculative Decoding

Introduction

In the high-stakes world of large language model inference, a 5× performance gap is the difference between a production-ready system and an expensive paperweight. When the assistant in this opencode session observed that its custom CUDA verify kernel for speculative decoding was achieving only 5.2 tokens per second at 46k context length—with GPU DRAM utilization stuck at a baffling 2%—it embarked on a methodical debugging journey that would ultimately lead to a profound insight about GPU occupancy, tensor parallelism, and the hidden costs of kernel launch parameters. Message 12326 represents the turning point: the moment when, after systematically ruling out every other hypothesis, the assistant correctly identified the root cause as occupancy starvation in the TP8 regime and took decisive action to fix it.

This article examines that single message in depth: the reasoning chain that led to the insight, the assumptions that were tested and discarded along the way, the technical knowledge required to understand the diagnosis, and the concrete action taken. The message is a masterclass in GPU performance debugging—a case study in how to use instrumentation, eliminate hypotheses, and arrive at a counterintuitive conclusion through first-principles reasoning about GPU architecture.

The Problem: A 192ms Step with Nothing Happening

To understand the significance of message 12326, we must first understand the problem that had been consuming the previous hour of the conversation. The assistant was deploying a speculative decoding system called DDTree (Draft-and-Diff Tree) on a cluster of RTX PRO 6000 Blackwell GPUs (compute capability sm_120). The system uses a small "draft" model to propose token sequences, then a full "target" model (Kimi K2.6, 61 layers, MoE) to verify them in parallel. The verify step—where the target model evaluates all draft-proposed tokens simultaneously—had been the focus of intense optimization.

The assistant had written a custom CUDA verify attention kernel because no existing MLA (Multi-head Latent Attention) kernels supported sm_120 (the Blackwell consumer GPU architecture). The kernel used a "flash-decode" approach: splitting the KV cache across blocks, computing partial attention results, and reducing them. Initial microbenchmarks on contiguous data showed reasonable performance, but when deployed in the live SGLang service, throughput collapsed.

The numbers were stark. At 46k context length, each decode step took approximately 192ms. The profiler revealed that the CPU-side orchestration—tree building at 1.8ms, mask preparation at 0.18ms, and the entire draft model forward pass at ~7ms—accounted for only about 10ms total. The remaining ~185ms was spent in the verify forward pass itself. Yet GPU hardware metrics showed DRAM utilization at only 2% and tensor core utilization at 0%. The GPU appeared to be doing almost nothing for 185ms of every 192ms step.

This was the mystery that message 12326 would finally solve.

The Investigation Trail: A Systematic Elimination of Hypotheses

The assistant's reasoning in message 12326 is the culmination of a multi-step investigation that had already ruled out several plausible explanations. Tracing this trail is essential to appreciating the final insight.

Hypothesis 1: CPU tree-building overhead. The assistant first suspected that the Python code constructing the draft tree and preparing verify inputs was the bottleneck. Profiler instrumentation proved this wrong: _build_ddtree_verify_input took 1.8ms and prepare_for_verify took 0.18ms—negligible fractions of the 192ms step.

Hypothesis 2: CUDA graph not replaying. If the verify forward was running eagerly (re-launching kernels every step via Python ctypes), the per-layer marshaling overhead for 61 layers could dominate. The assistant checked the decode logs and found cuda graph: True—the graph was indeed being replayed. This hypothesis was ruled out.

Hypothesis 3: Scattered KV page access. The SGLang memory allocator uses paged KV storage where each token's KV data may be scattered across non-contiguous pages. The assistant hypothesized that scattered memory access was causing latency-bound stalls. However, the service had just been restarted for profiling, meaning the KV pool was fresh and the single benchmark request would receive contiguous slots. This ruled out fragmentation as the primary cause.

Hypothesis 4: Draft model forward pass. Perhaps the draft model—a 6-layer DFlash model attending the full 46k context—was the bottleneck. The assistant instrumented _prepare_for_speculative_decoding and found it averaged only 7ms total. Ruled out.

Hypothesis 5: MoE expert weight bandwidth. The target model's MoE layers read approximately 10.7GB of INT4 expert weights per verify step. At 2% DRAM utilization, this would take ~300ms—exceeding the observed 185ms. But the math didn't quite work, and the low DRAM utilization suggested something more fundamental was wrong.

After eliminating all these candidates, the assistant was left with a stark contradiction: the verify forward was taking 185ms, the GPU was nearly idle, and the KV data was contiguous. Something about the kernel itself was preventing the GPU from doing useful work.

The Occupancy Revelation

Message 12326 opens with the assistant reviewing the profiler data and performing a critical recalculation. The key realization comes in a single paragraph:

With TP8 reducing H from 64 to 8, my grid becomes 1×9×8×16 = 1152 blocks instead of the 9216 blocks the microbench had. That's 8x fewer blocks on 188 SMs, giving only about 6 waves of occupancy—too low to hide latency. Each block still has to process 2875 tokens per split, so the per-block work is large but the overall occupancy is starved. The fix isn't memory fragmentation; it's increasing NSPLIT to something like 64 instead of 16, which would give 4608 blocks and much better occupancy for the H=8 regime.

This is the moment of insight. Let us unpack what the assistant realized.

The custom verify kernel uses a grid decomposition with four dimensions: batch size (1), number of query tokens (9, from the draft tree), number of attention heads per rank (H), and number of KV splits (NSPLIT). The total number of thread blocks launched is the product of these dimensions: batch × queries × heads × NSPLIT.

The critical variable here is H, the number of attention heads per GPU rank. The Kimi K2.6 model uses tensor parallelism across 8 GPUs (TP8), meaning the attention heads are sharded across the GPUs. With 64 heads total, each rank handles only 8 heads. This is a 8× reduction compared to a single-GPU configuration where all 64 heads would be local.

The assistant's microbenchmarks had been run in a configuration with all 64 heads local, yielding a grid of 1 × 9 × 64 × 16 = 9216 blocks. On 188 SMs (the RTX PRO 6000 has 188 Streaming Multiprocessors), this gives approximately 49 waves—excellent occupancy that keeps all SMs busy and hides memory latency effectively.

But in the TP8 deployment, the grid shrinks to 1 × 9 × 8 × 16 = 1152 blocks—only 6 waves. This is catastrophically low occupancy. With only 6 waves, most SMs are idle most of the time. Each block that does launch has to process a massive chunk of 2875 tokens per split, but the overall pipeline is so thinly populated that the GPU cannot hide the latency of memory accesses. The result: 2% DRAM utilization, because the memory system is constantly stalling waiting for blocks to issue requests.

Why NSPLIT=16 Was Chosen and Why It Was Wrong

The assistant had originally lowered NSPLIT to 16 to fit within CUDA graph capture memory constraints. CUDA graphs capture the entire kernel launch sequence and require that all memory allocations be fixed at capture time. A larger NSPLIT means more partial results to store in the reduction workspace, which increases memory pressure.

However, the assistant now realized that the TP8 regime fundamentally changes the memory calculation. With H=8 instead of H=64, the per-head workspace is 8× smaller. The workspace size scales as H × NSPLIT × sizeof(float), so reducing H by 8× while increasing NSPLIT by 4× (from 16 to 64) actually reduces the total workspace by 2×. The constraint that forced NSPLIT=16 no longer applies.

The assistant recalculated: at NSPLIT=64, each block processes 719 tokens across 90 tiles, shared memory usage stays around 20.8KB (well within the 100KB limit of sm_120), and the grid becomes 1 × 9 × 8 × 64 = 4608 blocks—approximately 24 waves on 188 SMs. This is still not ideal (ideally you want 4-8× more blocks than SMs for full latency hiding), but it is a 4× improvement over the 6 waves at NSPLIT=16.

The Second Optimization: Vectorized Loads

The assistant also identified a second issue: the paged kernel used scalar bf16 loads instead of vectorized float4 (128-bit) loads. Scalar loads issue one 16-bit memory transaction per element, achieving poor bandwidth utilization even when the data is contiguous. Vectorized loads coalesce four elements into a single 128-bit transaction, quadrupling the effective bandwidth for the same number of memory requests.

The assistant's contiguous kernel (used in microbenchmarks) already used float4 vectorization, which partly explained why the microbench showed reasonable performance while the in-service paged kernel did not. The paged kernel had been written with scalar loads for simplicity, but this was costing a 4× bandwidth penalty on top of the occupancy problem.

The assistant planned to address both issues: increase NSPLIT to 64 for occupancy, and add 128-bit vectorized bf16 loads for bandwidth. The message captures this dual strategy:

But I'm realizing there's a second issue: the paged kernel uses scalar bf16 loads instead of vectorized float4 loads like the contiguous kernel, which tanks bandwidth. So I need to tackle both—increase NSPLIT for occupancy and vectorize the bf16 loads to 4-element chunks.

The Action: A Quick Test via Environment Variable

Rather than rebuilding the kernel (which would require recompilation and redeployment), the assistant chose a clever shortcut: NSPLIT was already configurable via an environment variable (KDTREE_NSPLIT). By editing the systemd service file to set KDTREE_NSPLIT=64 and disabling the profiler (KDTREE_PROFILE=0), the assistant could test the occupancy hypothesis with zero code changes.

The bash command in the message performs this edit:

P=/etc/systemd/system/sglang-k26-ddtree.service
sed -i "s/KDTREE_PROFILE=1/KDTREE_PROFILE=0/" "$P"
grep -q KDTREE_NSPLIT "$P" || sed -i "/Environment=KDTREE_VERIFY/a Environment=KDTREE_NSPLIT=64" "$P"
sed -i "s/KDTREE_NSPLIT=[0-9]*/KDTREE_NSPLIT=64/" "$P"
grep -oE "KDTREE_(VERIFY|NSPLIT|PROFILE)=[0-9a-z]+" "$P"
systemctl daemon-reload && systemctl restart sglang-k26-ddtree && echo restarted

The output confirms the changes: KDTREE_VERIFY=on, KDTREE_NSPLIT=64, KDTREE_PROFILE=0. The service restarts, and the next benchmark will tell whether the occupancy hypothesis was correct.

Assumptions and Potential Pitfalls

The assistant's reasoning in message 12326 rests on several assumptions that deserve scrutiny:

Assumption 1: The KV data is truly contiguous. The assistant concluded that because the service was freshly restarted and only one benchmark request had been issued, the KV slots must be contiguous. This is a reasonable assumption for a simple allocator that allocates from the front of the free list, but it depends on the allocator implementation. If the allocator uses a slab-based or buddy-system approach, even a single allocation might be non-contiguous. The assistant's subsequent investigation (in the next chunk) would confirm that defragmentation was not the primary issue, validating this assumption.

Assumption 2: The occupancy calculation is correct. The assistant computes 6 waves for NSPLIT=16 and 24 waves for NSPLIT=64 on 188 SMs. This assumes perfect load balancing across SMs and ignores the fact that blocks have different sizes (the last split may be smaller). In practice, the actual occupancy may be slightly different, but the order-of-magnitude improvement is sound.

Assumption 3: Increasing NSPLIT alone will unlock DRAM bandwidth. The assistant implicitly assumes that the occupancy bottleneck is the primary constraint and that fixing it will allow the memory system to approach its peak bandwidth. This is plausible but not guaranteed—other factors like memory access patterns, bank conflicts, or the reduction kernel's efficiency could still limit throughput. The assistant acknowledges this by planning to also add vectorized loads.

Assumption 4: The workspace fits within CUDA graph capture memory at NSPLIT=64. The assistant recalculates the shared memory usage at 20.8KB, which is well within the 100KB limit for sm_120. However, the total workspace includes global memory buffers for partial results, which must be pre-allocated and fixed at graph capture time. The assistant's reasoning that "with H=8 (8× smaller workspace), I can actually afford NSPLIT=64 again without exceeding memory" is sound but would need to be verified empirically.

The Thinking Process: A Window into Expert Debugging

What makes message 12326 particularly valuable as a case study is the visible thinking process. The assistant does not arrive at the occupancy hypothesis directly; instead, we see a chain of reasoning that evolves in real time:

Initial assessment: "The verify forward is clearly the bottleneck: it's running at only 2% DRAM utilization despite being memory-intensive."
Scatter hypothesis: "At 46k context, my kernel reads scattered KV slots across 61 layers... the latency-bound scattered memory access keeps utilization low."
Self-correction: "But wait—the service was just restarted for profiling, so the bench run at 5.2 tok/s with 46k context should already have contiguous slots."
Recalculation: "With TP8 reducing H from 64 to 8, my grid becomes 1×9×8×16 = 1152 blocks instead of the 9216 blocks the microbench had."
Occupancy insight: "That's 8x fewer blocks on 188 SMs, giving only about 6 waves of occupancy—too low to hide latency."
Solution formulation: "The fix isn't memory fragmentation; it's increasing NSPLIT to something like 64 instead of 16."
Memory budget check: "At NSPLIT=64, each block processes 719 tokens across 90 tiles, shared memory stays around 20.8KB."
Second-order optimization: "The paged kernel uses scalar bf16 loads instead of vectorized float4 loads like the contiguous kernel, which tanks bandwidth." This pattern—hypothesis, test, contradiction, refinement—is the essence of performance debugging. The assistant's willingness to discard its own hypotheses when the data doesn't fit is a hallmark of rigorous engineering thinking.

Input Knowledge Required

To fully understand message 12326, the reader needs knowledge in several domains:

GPU architecture: Understanding of Streaming Multiprocessors (SMs), thread blocks, grid dimensions, occupancy, and wave quantization. The concept that a GPU needs many more thread blocks than SMs to hide memory latency is fundamental.

CUDA programming: Familiarity with grid decomposition, shared memory budgets, and the trade-offs between NSPLIT (number of splits in a flash-attention kernel) and occupancy.

Tensor parallelism (TP): Knowledge of how attention heads are sharded across GPUs in TP, and how this reduces the per-rank head count from H to H/TP_size.

Speculative decoding: Understanding of the draft-verify-accept cycle, how tree-based speculative decoding works, and the role of the verify kernel in evaluating multiple candidate tokens in parallel.

Memory hierarchy: Knowledge of DRAM bandwidth, coalesced vs. scattered memory access, vectorized loads (float4 vs. scalar bf16), and how memory latency interacts with occupancy.

SGLang internals: Familiarity with the paged KV allocator, CUDA graph capture, and the systemd service configuration for inference deployments.

Output Knowledge Created

Message 12326 produces several forms of knowledge:

Diagnostic knowledge: The root cause of the 2% DRAM utilization is identified as occupancy starvation in the TP8 regime, not scattered memory access, CPU overhead, or graph replay issues.

Actionable knowledge: The fix is to increase NSPLIT from 16 to 64 and add vectorized bf16 loads. The NSPLIT change can be tested immediately via environment variable without kernel recompilation.

Architectural knowledge: The interaction between tensor parallelism and kernel grid sizing is documented. A kernel that works well at H=64 may fail catastrophically at H=8 if the grid dimensions are not adjusted.

Memory budget knowledge: The shared memory and workspace requirements for NSPLIT=64 at H=8 are calculated: 20.8KB shared memory, 719 tokens per split, 90 tiles per block.

Methodological knowledge: The message demonstrates a debugging methodology: instrument the pipeline to measure each phase, eliminate hypotheses systematically, recalculate from first principles when empirical data contradicts expectations, and test the cheapest fix first.

The Broader Significance

Message 12326 is not just about one kernel parameter. It illustrates a general principle of GPU programming: kernel design must account for the deployment configuration. A kernel that achieves excellent occupancy in a single-GPU microbench may become occupancy-starved when deployed with tensor parallelism, because the per-rank workload is reduced. The grid dimensions must be scaled proportionally to maintain the same number of thread blocks per SM.

This principle extends beyond attention kernels to any GPU workload that decomposes work across a grid. Tensor parallelism, pipeline parallelism, and data parallelism all reduce the per-device workload, and kernel launch parameters must be adjusted accordingly. A kernel that is "optimized" for one configuration may be catastrophically suboptimal for another.

The assistant's insight also highlights the danger of microbenchmarking in isolation. The custom verify kernel showed reasonable performance in microbenchmarks with contiguous data and H=64, but this bore no relation to its in-service performance with TP8. The microbenchmarks were measuring the wrong thing—they tested a configuration that would never occur in production.

What Followed

The subsequent chunk (chunk 1 of segment 66) reveals that the assistant's diagnosis was correct. After increasing NSPLIT to 64 and adding 128-bit vectorized bf16 loads, the verify kernel achieved a 3–6× end-to-end decode speedup over the Triton baseline across all context lengths from 4k to 65k. The occupancy fix unlocked the DRAM bandwidth, and the vectorized loads further improved throughput. The assistant went on to implement Tier 0 defragmentation (forcing the allocator to keep KV contiguous) and identified MoE expert imbalance as the remaining bottleneck—a structural limitation of TP8 at batch size 1 that would require batching or expert parallelism to address.

The 3–6× speedup validated every assumption in message 12326. The occupancy hypothesis was correct. The NSPLIT increase was sufficient. The vectorized loads added further gains. And the debugging methodology—measure, hypothesize, test, refine—had paid off spectacularly.

Conclusion

Message 12326 captures a pivotal moment in a complex performance debugging journey. The assistant, faced with a GPU that appeared to be doing nothing for 185ms out of every 192ms step, systematically eliminated every plausible hypothesis until only one remained: occupancy starvation caused by the interaction between tensor parallelism and kernel grid sizing. The insight was counterintuitive—the kernel wasn't slow because it was doing too much work, but because it wasn't launching enough thread blocks to keep the GPU busy.

The message is a testament to the power of first-principles reasoning in GPU performance optimization. Rather than guessing or applying heuristics, the assistant calculated the exact grid dimensions, compared them to the SM count, computed the wave occupancy, and identified the precise parameter change needed. The fix was deployed as a simple environment variable change—no kernel rebuild, no code modification—and it worked.

For anyone debugging GPU performance, message 12326 offers a template: instrument everything, trust the data, recalculate from first principles, and never assume that a kernel that works in a microbench will work in production. The GPU is a deterministic machine, and if it's running at 2% utilization, there is always a reason. Finding that reason requires patience, rigor, and the willingness to discard your own hypotheses when the data doesn't fit.