The Wave-Fill Lever: How TARGET_CTAS=512 Unlocked 12.8% Throughput on Blackwell Decode

Introduction

In the high-stakes world of LLM inference optimization on cutting-edge hardware, the difference between a system that scales and one that plateaus often comes down to a single environment variable. Message [msg 13536] captures one such pivotal moment: the execution of a throughput benchmark after setting SGLANG_SM120_MMA_TARGET_CTAS=512 on an 8× RTX PRO 6000 Blackwell cluster running the DeepSeek-V4-Flash model. The results were striking — a 12.8% throughput improvement at C=64 concurrency and a 5.7% gain at C=96, completely eliminating the embarrassing anomaly where C=96 had been running slower than C=80. This message is the payoff of a rigorous, hypothesis-driven optimization campaign that had just navigated a dead end and pivoted to the right lever.

Context: The Occupancy Investigation

To understand why this message matters, one must trace the reasoning that led to it. The session had been deep in the weeds of Blackwell (sm120) decode performance for the DeepSeek-V4-Flash model. Earlier investigation ([msg 13527]) had revised the fundamental premise of the optimization effort: attention was the slope driver, not MoE. Profiler traces at batch size 80 revealed that MoE grouped-GEMM was essentially flat (15.1ms at bs32 to 15.9ms at bs80, adding only +0.015 ms/req), while attention was responsible for the entire marginal cost (13.1ms to 51.2ms, +0.79 ms/req). The C60→C90 throughput ceiling was set by attention alone.

The root cause was identified as a 1 CTA/SM occupancy bottleneck. All dominant decode kernels were shared-memory-capped (attention at 80KB, MoE at 89KB of ~100KB per SM on Blackwell), leaving only 4-12 warps active out of 48 possible per SM. The SMs appeared "active" in profiler traces (97% utilization) but were actually memory-latency-stalled — 57% power-limited and 27% DRAM-bound. The attention kernel specifically used 128-thread blocks with 80KB of shared memory, forcing just 4 warps per SM at 8% occupancy.

A concrete manifestation of this occupancy problem was wave quantization: at batch 64 with nsplit=2, the grid produced 256 CTAs that packed inefficiently across the 188 SMs, with only 68 SMs filled in the second wave (wasting ~32% of compute capacity). The baseline benchmark in [msg 13532] revealed the starkest symptom: C=96 (799.3 tok/s) was slower than C=80 (833.0 tok/s) — a non-monotonic scaling curve that screamed wave-fragmentation waste.

The BLOCK_H=16 Dead End

The assistant's first attempt to address this was the most intuitive lever: halving the MMA block height from 32 to 16 (SGLANG_SM120_MMA_BLOCK_H=16). The reasoning was straightforward — smaller blocks would use less shared memory per CTA, potentially allowing 2 CTAs per SM instead of 1, doubling warp occupancy and improving latency hiding. The test was run in [msg 13533] and the results came back in [msg 13534]: worse across the board, with degradation ranging from -3% to -9%.

The assistant's post-mortem reasoning in [msg 13535] is a textbook example of learning from failure. The kernel uses 255 registers per thread × 128 threads = 32,640 registers per CTA. Two CTAs would need 65,280 registers, exceeding the SM's 65,536 register limit. The kernel was register-capped at 1 CTA/SM regardless of shared memory consumption. Halving BLOCK_H had simply shrunk per-CTA work (worse MMA tile efficiency) without gaining any occupancy benefit. The hypothesis was elegant but wrong — the real bottleneck was registers, not shared memory.

The Pivot to Wave-Fill

This failure was productive. It forced a reassessment of which lever to pull. The assistant pivoted to the wave-fill approach: instead of trying to increase CTAs per SM, increase the total number of CTAs to pack waves more efficiently. The SGLANG_SM120_MMA_TARGET_CTAS environment variable controls the target number of CTAs the split kernel aims to produce. By raising it from the default of 256 to 512, the kernel would generate more splits, producing more CTAs that could fill the SM grid more evenly.

The math was compelling. At C=96 with TARGET_CTAS=256, the grid produced 384 CTAs = 2.04 waves, leaving a nearly empty third wave (4% utilization). With TARGET_CTAS=512, the grid would target more CTAs, potentially improving wave packing. The trade-off was increased split overhead — more CTAs means more launches and more partial reductions — but the baseline was so wave-inefficient that the overhead was worth accepting.

The Subject Message: Execution and Results

Message [msg 13536] executes this A/B test. The bash command performs three distinct phases:

Phase 1: Server restart and readiness check. After modifying the serve script with sed to inject export SGLANG_SM120_MMA_TARGET_CTAS=512, the decode server is restarted via systemctl. The command then sleeps 80 seconds to allow the model to load (the GLM-5-NVFP4 model is large — loading takes significant time even on 8 GPUs). It then polls the server with curl requests every 3 seconds, sending a minimal chat completion request ({"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"max_tokens":8}) and checking for HTTP 200. This polling loop runs up to 20 iterations (60 seconds max), ensuring the server is fully ready before benchmarking.

Phase 2: Throughput benchmark. The benchmark script /root/bench_tput.py is invoked four times at concurrency levels C=48, 64, 80, 96. Each run sends n = C*4 requests (e.g., 192 requests at C=48) with max_tokens=256. The grep -E '^C=' filters the output to show only the summary line.

Phase 3: Results reporting. The output is:

=== TARGET_CTAS=512 bench ===
C=48  n=192 max_tokens=256 | agg=680.6 tok/s | per-req~692.6 tok/s | p50 lat=17.22s | toks=49152 errs=0 wall=72.2s
C=64  n=256 max_tokens=256 | agg=811.7 tok/s | per-req~834.3 tok/s | p50 lat=19.10s | toks=65536 errs=0 wall=80.7s
C=80  n=320 max_tokens=256 | agg=842.7 tok/s | per-req~867.9 tok/s | p50 lat=23.18s | toks=81920 errs=0 wall=97.2s
C=96  n=384 max_tokens=256 | agg=844.6 tok/s | per-req~872.5 tok/s | p50 lat=27.87s | toks=98304 errs=0 wall=116.4s

Interpreting the Results

The comparison against the baseline (from [msg 13532]) tells a clear story:

| Concurrency | Baseline (tok/s) | TARGET_CTAS=512 (tok/s) | Change | |-------------|------------------|------------------------|--------| | C=48 | 684.7 | 680.6 | -0.6% | | C=64 | 719.5 | 811.7 | +12.8% | | C=80 | 833.0 | 842.7 | +1.2% | | C=96 | 799.3 | 844.6 | +5.7% |

The most important result is at C=96: 844.6 tok/s vs 799.3 tok/s — a 5.7% gain that completely reverses the non-monotonic anomaly. C=96 now outperforms C=80 (844.6 vs 842.7), restoring monotonic scaling. The wave-quantization waste at high batch sizes has been mitigated.

The C=64 result is even more dramatic: 811.7 vs 719.5 tok/s, a 12.8% improvement. This suggests that at mid-concurrency, the wave-fill benefit is maximized — the split overhead is modest while the wave-packing improvement is substantial.

C=48 is essentially flat (-0.6%), which is expected — at low concurrency, the grid is small enough that wave packing is already efficient, and the additional split overhead from higher TARGET_CTAS slightly outweighs any benefit.

C=80 shows a modest 1.2% gain, which makes sense — this concurrency level was already the baseline's best performer (833 tok/s), so there was less room for improvement.

What This Message Reveals About the Thinking Process

The subject message is deceptively simple — it's just a bash command and its output. But it sits at the convergence of a sophisticated reasoning chain:

  1. Hypothesis generation based on profiler data. The wave-quantization theory was grounded in concrete trace analysis showing 2.04 waves at C=96 with a near-empty third wave.
  2. Falsification of an alternative hypothesis. The BLOCK_H=16 experiment was a clean failure that taught the team about register pressure as a binding constraint. This was not wasted effort — it eliminated a plausible but incorrect model of the bottleneck.
  3. Pivot to the correct lever. TARGET_CTAS directly addresses wave-fill without trying to overcome the register cap. It's a "work with the hardware, not against it" approach.
  4. Empirical validation. The benchmark is controlled (same script, same concurrency levels, same max_tokens, same server configuration except the one variable) and produces clean, interpretable results.
  5. Pattern recognition in the output. The monotonic scaling (C=48 < C=64 < C=80 < C=96) is itself a validation signal — it means the wave-quantization anomaly is fixed.

Assumptions and Their Validity

Several assumptions underpin this test:

Assumption: TARGET_CTAS=512 is safe for numerical correctness. The kernel has a built-in numerical validation gate (relative error ≤ 6.7e-3) that would catch any precision degradation from increased splitting. This assumption proved valid — the benchmark reports 0 errors across all runs.

Assumption: The split overhead is worth the wave-fill benefit. More CTAs means more kernel launches and more partial reduction work. The results confirm this trade-off is favorable at C=64 and C=96, neutral at C=80, and slightly negative at C=48. A more sophisticated approach might use a dynamic TARGET_CTAS that varies with batch size, but a single value of 512 is a good starting point.

Assumption: The benchmark is representative of real workload. The benchmark uses fixed max_tokens=256 with a simple prompt. Real agentic workloads have variable-length generation and multi-turn context. The assistant later validates this with stress tests, but the initial benchmark assumes the optimization generalizes.

Assumption: The register cap analysis is correct. The conclusion that 2 CTAs/SM is impossible due to 65,280 > 65,536 registers assumes the compiler doesn't spill registers or that spilling would negate any benefit. This is a reasonable assumption for a hand-tuned Triton kernel, but it's worth noting that register spilling could theoretically allow 2 CTAs at a performance cost — the BLOCK_H=16 experiment already tested this and confirmed it's worse.

Knowledge Flow

Input knowledge required to understand this message includes:

Conclusion

Message [msg 13536] appears to be a routine benchmark execution, but it represents the successful culmination of a multi-step reasoning chain: from profiler traces that revised the optimization premise, through a falsified hypothesis about block height, to a pivot that directly addressed wave quantization. The 12.8% gain at C=64 and the restoration of monotonic scaling at C=96 are not just performance improvements — they are validation of a mental model about how the Blackwell architecture interacts with the DeepSeek-V4-Flash attention kernel. In the high-dimensional optimization space of modern LLM inference, a single environment variable can unlock significant gains when guided by the right diagnosis.