The Variance That Changed Everything: How Benchmark Noise Revealed the Limits of GPU Kernel Optimization

Introduction

In the high-stakes world of GPU kernel optimization for large language model inference, the difference between a breakthrough and a dead end can be measured in milliseconds. But what happens when the measuring stick itself is unreliable? Message [msg 13571] in this opencode session captures a pivotal moment: the assistant, after days of methodical experimentation on the DeepSeek-V4-Flash attention kernel for NVIDIA Blackwell (sm_120) GPUs, discovers that the very benchmarks used to judge performance improvements are contaminated by ±5-7% run-to-run variance. This discovery doesn't just reinterpret past results—it fundamentally reshapes the assistant's understanding of what is and isn't possible with this kernel, closing the door on an entire class of optimization strategies and forcing a reckoning with the true nature of the throughput bottleneck.

The Message: A Synthesis and a Closing

The subject message is the capstone of a multi-message investigation into what the assistant calls "#3a"—a systematic A/B test campaign exploring whether increasing GPU occupancy through Triton autotune configuration knobs can improve decode throughput on the DeepSeek-V4-Flash model. The message is brief in execution (a single edit command to update a documentation file) but dense in reasoning. It represents the moment when the assistant steps back from the details of individual experiments, applies a newly discovered critical lens (benchmark variance), and renders a final verdict on an entire line of inquiry.

Let us quote the message in full:

## Agent Reasoning The configuration is locked in correctly with TARGET_CTAS at 512, overlap disabled, and the baseline kernel in place. C96 is showing meaningful variance between runs—829.2 and 775.0—which puts the benchmark range around 775-845 with a mean hovering near 810.

>

Given this ±5-7% noise band, I'm re-evaluating the V1 and V2 regressions: V1's low-C drops of 14-18% are well beyond noise and represent genuine degradation, while the high-C deltas stay within the noise floor. V2 shows a 9.9% drop at C64 that exceeds noise, with no compensating gains elsewhere. Both rejections hold—neither configuration lever improves performance, and the baseline remains optimal. The TARGET_CTAS=512 win at C96 (845 vs 799, +5.7%) sits right at the edge of the noise band, so it's marginal but consistent with the variance pattern.

>

Now I need to document the V2 results and the overall #3a conclusion in the A/B doc, update the plan doc to note that occupancy levers are refuted and register-rewrite isn't recommended, commit the changes, and then summarize everything for the user. I'll make sure to capture the benchmark variance finding since it's crucial context for interpreting these results and any future tuning work.

>

Confirmed: deployed state is correct — TARGET_CTAS=512 + MULTI_STREAM_OVERLAP=0 + overlap-off all live, baseline kernel restored. The C96 re-runs (829, 775) reveal ~±5-7% run-to-run benchmark variance — so the "798.8" was noise, not a regression. (This variance also means V1/V2's genuine regressions are the ones beyond that band: V1 low-C −14/−18%, V2 C64 −9.9%; neither showed a real high-C gain.)

>

Let me document the V2 result + the conclusive #3a finding + the variance caveat, and commit: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/DSV4_ATTN_OCCUPANCY_AB.md

The Chain of Experiments That Led Here

To understand the weight of this message, we must trace the experimental chain that preceded it. The assistant had been investigating why decode throughput on the DeepSeek-V4-Flash model saturates at approximately 845 tok/s regardless of concurrency. The working hypothesis was that the attention kernel—implemented in Triton for NVIDIA's sm_120 architecture (Blackwell RTX PRO 6000 GPUs)—was occupancy-bound: that is, it wasn't using enough of the GPU's computational resources simultaneously, leaving performance on the table.

The assistant formulated two concrete levers to pull, both within the Triton autotune configuration space:

V1: Increase num_warps from 4 to 8. The reasoning was straightforward: more warps per thread block would increase occupancy, allowing the GPU to hide memory latency by switching between warps. However, as the assistant had discovered in earlier reasoning (msg [msg 13562]), the kernel's MMA (matrix multiply-accumulate) tiles are [16,16]—a small shape. With 8 warps, each warp gets only a tiny slice of work, leading to over-subdivision that hurts efficiency rather than helping. The V1 benchmarks confirmed this: low-concurrency throughput dropped by 14-18%, a catastrophic regression well outside any noise band.

V2: Increase num_stages from 2 to 3. This was a subtler intervention. Triton's num_stages controls the depth of the software pipeline for prefetching data into shared memory. A deeper pipeline (3 stages instead of 2) could hide more latency by keeping the memory subsystem busier. The assistant calculated that at block_h=16, the shared memory budget would increase from ~60KB to ~81KB—still within the 99KB limit per SM. The theory was sound, but the practice told a different story. At high concurrency (C=64), V2 showed a 9.9% regression. At C=96, a 2.6% regression. Only at C=48 was the result roughly flat. The deeper pipeline was hurting, not helping.

The assistant's analysis in msg [msg 13569] was incisive: the loop iterations per block are too short (4-6 at high concurrency) for a 3-stage pipeline to ever fill properly. The pipeline fill and drain overhead dominates any latency-hiding benefit. The kernel simply doesn't have enough work per invocation to justify deeper prefetching.

After reverting V2 to the baseline in msg [msg 13569], the assistant noticed something alarming in msg [msg 13570]: the restored baseline was showing C=96 at 798.8 tok/s, significantly below the ~845 tok/s that the TARGET_CTAS=512 configuration had previously achieved. Had the revert broken something? The assistant immediately checked the live process environment—TARGET_CTAS=512 was confirmed present, MULTI_STREAM_OVERLAP=0 was set, and disable-overlap-schedule was live. Everything was correct.

Then came the critical insight: the assistant ran two more C=96 benchmarks back-to-back. The results: 829.2 tok/s and 775.0 tok/s. Two runs, same configuration, same hardware, same load—a 54 tok/s difference. This was the moment the assistant realized that the benchmark itself had significant run-to-run variance.## The Discovery of Benchmark Variance: A Methodological Breakthrough

The two re-runs in msg [msg 13570]—829.2 and 775.0 tok/s at identical settings—were the key that unlocked everything. The assistant immediately recognized this as a ~±5-7% noise band, a critical piece of meta-knowledge about the measurement system itself. This discovery had immediate and profound consequences for interpreting all prior experiments.

First, the "regression" that prompted the investigation (the 798.8 tok/s baseline reading) was itself revealed to be noise. The assistant had been chasing a phantom. The baseline was healthy all along; the low reading was simply the lower end of the natural variance distribution. This is a classic pitfall in performance engineering: a single measurement point is treated as truth, when in reality it is a sample from a distribution.

Second, the V1 and V2 results needed to be re-evaluated against this new noise floor. V1's low-concurrency regressions of 14-18% were so large that they remained genuine degradations even accounting for variance. V2's C64 regression of 9.9% also exceeded the noise band. But the high-concurrency numbers for both experiments, which had appeared to show small improvements or regressions, were now understood to be indistinguishable from noise. Neither V1 nor V2 produced a measurable gain at any concurrency level.

Third, and most subtly, the TARGET_CTAS=512 improvement itself—previously celebrated as a 6-13% win—was now revealed to be marginal. At C=96, the 845 tok/s reading compared against a baseline whose mean was ~810 tok/s gave a ~5.7% improvement, which sits right at the edge of the ±5-7% noise band. The assistant's honest assessment in the subject message is that this gain is "marginal but consistent with the variance pattern." It is real, but barely.

The Reasoning Process: A Masterclass in Evidence Integration

What makes this message remarkable is not the conclusion itself (both V1 and V2 are rejected), but the reasoning process by which that conclusion is reached and, more importantly, qualified. The assistant does not simply declare "V1 and V2 failed." Instead, it:

  1. Establishes the measurement context. It begins by confirming that the deployed configuration is correct—TARGET_CTAS=512, MULTI_STREAM_OVERLAP=0, overlap-off, baseline kernel. This is a crucial sanity check. Before interpreting any numbers, the assistant verifies that the system is in the expected state.
  2. Quantifies the noise floor. The two C96 re-runs (829.2 and 775.0) are used to estimate a ~±5-7% variance band. This is not a rigorous statistical analysis (more samples would be needed for that), but it is a practical, actionable estimate that immediately improves decision-making.
  3. Re-evaluates all prior experiments through this lens. The assistant systematically checks V1's low-C and high-C results, V2's results, and the TARGET_CTAS=512 result against the newly established noise band. Each is classified as either a genuine effect (beyond the noise band) or indistinguishable from noise.
  4. Draws the overarching conclusion. Neither occupancy lever (more warps, deeper prefetch) improves performance. The baseline is already optimal among feasible Triton autotune configurations. The register-reducing rewrite approach—which was the broader motivation for the #3a investigation—is empirically refuted.
  5. Commits to documentation. The final action is not a code change but a documentation update. The assistant records the V2 result, the #3a conclusion, and—crucially—the variance caveat. This ensures that future investigators will have access to this methodological insight.

Assumptions, Knowledge, and the Boundaries of the Investigation

The message rests on several key assumptions and bodies of knowledge:

Input knowledge required. To fully understand this message, one must know that the DeepSeek-V4-Flash model uses a Multi-head Latent Attention (MLA) mechanism implemented in Triton for NVIDIA's sm_120 architecture. One must understand the Triton autotune configuration space (num_warps, num_stages, BLOCK_T) and how they map to GPU occupancy and latency hiding. One must know the history of the TARGET_CTAS=512 optimization (a prior win that increased the number of cooperative thread arrays per kernel launch). And one must understand the production context: the decode worker runs as a systemd service on a remote machine with 8 GPUs, using prefill-decode disaggregation.

Assumptions made. The assistant assumes that the benchmark variance is stationary—that the ±5-7% band observed at C=96 applies equally to other concurrency levels and to the earlier V1/V2 runs. This is a reasonable assumption but not proven. The assistant also assumes that the two re-runs are sufficient to characterize the noise distribution. In a rigorous statistical setting, one would want many more samples. In a practical engineering context, however, the assistant's judgment is sound: two runs showing a 54 tok/s spread is strong evidence of significant variance, and the qualitative conclusions (V1/V2 regressions are real, high-C gains are marginal) are robust to reasonable changes in the estimated noise band.

Mistakes and incorrect assumptions. The most significant mistake was the initial assumption that the baseline had regressed after the V2 revert. The assistant's first reaction in msg [msg 13570] was to suspect a lost environment variable or a degraded state. This was a natural and prudent response, but it was wrong. The "regression" was simply noise. The assistant corrected this within a single message by running confirmatory benchmarks—a textbook example of how to avoid being misled by a single data point.

A subtler potential mistake is the decision to treat the ±5-7% band as the sole criterion for judging effects. A 5.7% improvement that sits at the edge of a 5-7% noise band is genuinely uncertain. The assistant's characterization of the TARGET_CTAS=512 gain as "marginal" is honest and appropriate, but it also means that the assistant cannot be certain that this optimization is real. Future work might need more rigorous statistical testing to confirm it.

Output Knowledge Created by This Message

This message produces several lasting contributions to the project's knowledge base:

  1. The #3a conclusion is finalized and documented. Both V1 (num_warps=8) and V2 (num_stages=3) are definitively rejected as optimization strategies. The baseline Triton autotune configuration (BLOCK_T=16, num_warps=4, num_stages=2) is established as optimal among feasible configurations.
  2. The register-reducing rewrite premise is refuted. The broader motivation for the occupancy investigation—that the kernel could be improved by reducing register pressure and increasing occupancy—is shown to be incorrect. The kernel is already well-optimized for its workload shape.
  3. Benchmark variance is characterized and documented. The ±5-7% run-to-run variance is a critical methodological finding that will inform all future performance work on this system. It means that single-benchmark comparisons are unreliable and that multiple runs are needed for any claim of improvement.
  4. The true bottleneck is clarified. By eliminating occupancy and latency-hiding as the limiting factors, the assistant implicitly points toward a more fundamental constraint: gather bandwidth or the inherent work pattern of the MLA kernel. This reframes the optimization problem for future investigations.

Conclusion: The Value of Knowing When to Stop

The subject message is, in many ways, a message about the limits of optimization. The assistant invested significant effort—multiple experiments, subagent investigations, code edits, deployments, and benchmarks—into the #3a investigation. The result is a conclusive negative: the attention kernel cannot be improved through the Triton autotune knobs that were tried.

But this negative result is itself valuable. It saves future effort that would have been wasted pursuing similar approaches. It provides a clear methodological contribution (the variance characterization) that improves all subsequent measurement. And it reframes the problem, pointing toward deeper architectural constraints that will require more fundamental changes to address.

In the broader narrative of the opencode session, this message represents a moment of intellectual honesty and methodological rigor. The assistant resists the temptation to over-interpret noisy data, resists the temptation to keep tweaking knobs in search of a phantom gain, and instead steps back, characterizes the measurement system, and renders a clear verdict. It is a model of how to conduct empirical performance optimization in complex systems.