The Full-Stack Optimization of DeepSeek-V4-Flash on Blackwell GPUs: From 11 to 531 Tokens Per Second

Introduction

In the history of AI inference optimization, few stories are as dramatic as the transformation of DeepSeek-V4-Flash on eight NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120 architecture). What began as a system struggling to deliver 11 tokens per second at single-request concurrency ended as a production deployment achieving 531 tokens per second at batch size 64 — a 48× improvement from the starting point, with a single fix accounting for a 17.9× leap in a single day. This article synthesizes the engineering journey captured across an intensive opencode coding session, tracing the arc from bottleneck diagnosis through custom kernel development, production deployment, monitoring infrastructure, and quality assurance.

The campaign was not a single heroic optimization but a systematic, layered effort spanning multiple phases: confirming the bottleneck was structural (CUDA-core fallback kernels rather than communication or memory bandwidth), designing and implementing a custom MMA sparse-MLA decode kernel using Triton tensor-core operations, discovering and fixing the indexer's O(max_context) bottleneck that was silently consuming 69% of GPU time, deploying prefill-decode (PD) disaggregation across all 8 GPUs, setting up a full Prometheus/Grafana monitoring stack, resolving agent-coherence and tool-calling quality issues, and documenting the entire journey in a comprehensive engineering report.

Phase 1: The Diagnostic Breakthrough

The campaign began with a stark reality: the system was delivering approximately 11 tokens per second at C=1 and ~28 tok/s at C=16 — catastrophically far from the 300–600 tok/s that roofline analysis suggested the hardware should support. The assistant's initial profiling using NVIDIA's CUDA Flight (cufall) profiler and nsys tracing revealed that the GPU was drawing only 340 watts of its 600-watt TDP, with tensor-pipe utilization near zero. The SMs were active but doing scalar SIMT work, not tensor-core matrix operations.

The kernel-level breakdown was damning. The _tiled_sparse_decode_kernel consumed 57% of all decode GPU time, and it had two structural defects: its grid was (B, H) = (batch, 64 heads), causing all 64 head-programs to independently re-gather the identical KV cache data from memory (up to 64× redundant reads), and it computed attention scores using scalar tl.sum(q*k) operations on SIMT lanes rather than tensor-core tl.dot matrix multiplies. A second bottleneck at ~28% was labeled "unfused elementwise/copy glue" — thousands of tiny kernel launches for RoPE, RMSNorm, dequantization, and residual operations. A third issue at 8.8% was forced-FP32 SIMT GEMMs in the indexer and multi-head connection pre-linear layers, caused by conservative sm_120 environment overrides.

The user's strategic intervention at [msg 12518] — "Continue perf investigation, at C=1, C=16, C=64; Should scale relatively well" — reframed the campaign from tactical kernel-tweaking to systematic bottleneck diagnosis. The assistant's response ([msg 12519]) revealed a striking linear model: decode step time followed step_time ≈ 40 + 31·N milliseconds, where N was the number of concurrent requests. Each additional request added ~31 ms of non-amortizable GPU time, and throughput asymptoted at 1000/31 ≈ 32 tok/s regardless of concurrency. This linear per-request cost was happening inside the CUDA graph, ruling out CPU-side scheduler overhead — the bottleneck was purely in the GPU kernels themselves.

Phase 2: The Custom MMA Sparse-MLA Decode Kernel

With the diagnosis confirmed, the assistant embarked on the most technically demanding phase: designing and implementing a custom MMA (matrix-multiply-accumulate) sparse-MLA decode kernel to replace the 57% bottleneck. The design process, documented in [msg 12539] and [msg 12542], is a masterclass in GPU kernel engineering under hardware constraints.

The core insight was elegant: Multi-head Latent Attention (MLA) uses a single shared KV head for all 64 query heads — a property the existing kernel completely ignored by launching 64 independent programs. The new kernel would launch one program per batch that handles all heads simultaneously, computing Q @ K^T as a single matrix multiply using Triton's tl.dot, which maps directly to tensor-core MMA instructions.

The design involved several interlocking decisions:

Block size selection. The assistant considered BLOCK_H=64 (all heads in one program) versus BLOCK_H=32 (two programs per batch). BLOCK_H=64 maximized KV reuse but created shared memory pressure: a Q tile at 64 heads consumed ~64 KB, a K tile at BLOCK_T=32 consumed ~32 KB, and transposed buffers added more — exceeding the ~99 KB shared memory limit on sm_120. The assistant settled on BLOCK_H=32 as a conservative starting point, accepting that KV would be read twice instead of once.

Nope/rope separation. The 512-dimensional latent space is split into a 448-dimension NOPE (non-position-encoded) component and a 64-dimension ROPE (rotary position encoding) component. Rather than concatenating them, the assistant kept them separate throughout the computation, requiring two dot products for attention scores and two separate output accumulations.

The power-of-two padding trap. Triton's tl.arange requires power-of-2 dimensions, but the NOPE dimension was 448 — not a power of two. The solution, detailed in [msg 12550], was to pad the NOPE dimension to 512 and use runtime masking to zero out columns 448:512. Since zero columns contribute nothing to the dot product, the result is mathematically identical to computing over just the 448 real values.

Split-K parallelization. For low batch sizes where occupancy would be poor, the assistant designed a split-K approach that partitions the topk token dimension across multiple blocks, each producing partial attention outputs and log-sum-exp (LSE) values. A separate combine kernel merges these partial results using the standard online softmax combine algorithm — a technique borrowed from prior work on the Kimi K2.6 DDTree engine.

The implementation was gated behind an environment variable (SGLANG_SM120_MMA_FLASHMLA=1), enabling A/B testing without disrupting the running service. After validation against the production SIMT kernel (relative error ≤ 6.7e-3 across batch sizes 1–32 and topk values 128–512), the kernel was deployed via a server restart at [msg 12554]. The results were immediate: attention dropped from 57% to ~10% of decode GPU time, and throughput improved by 2.2–2.9× across all concurrency levels (C=1: 11.5→33.5, C=16: 26.6→58.6, C=64: 29.7→64.4 tok/s).

Phase 3: The Indexer O(max_context) Breakthrough

With the MMA kernel deployed, the assistant turned to the remaining ~69% of GPU time consumed by "glue" operations. The user suggested trying torch.compile to fuse the elementwise kernels, but the assistant confirmed this was incompatible with the stack — torch.compile fails at CUDA graph capture even with the stock kernel, due to a fundamental conflict between Inductor's compiled forward and SGLang's CUDA graph capture mechanism.

Instead, the assistant profiled the glue operations via eager-mode tracing, revealing the dominant launching ops: aten::copy_ (35%), aten::mul (13.6%), aten::clamp_min (13.2%), aten::bmm (10%), and aten::sum (7%). But the critical insight came from examining the tensor shapes: these operations were running on [32, 262208, 64] tensors — the indexer was computing scores over the full ~1-million-token max context (262,208 c4-positions) every decode step, even though the actual context was only ~512 tokens.

This single issue was the root cause of the linear per-request cost model. The DSA (Dense Sparse Attention) indexer's torch fallback was doing O(max_context) work per decode step regardless of actual sequence length. The fix was deceptively simple: capping --context-length 8192 cut the indexer's work by approximately 128×. The results were staggering:

| Concurrency | Before (tok/s) | After (tok/s) | Improvement | |---|---|---|---| | C=1 | 33.5 | 85.1 | 2.5× | | C=16 | 58.6 | 285.1 | 4.9× | | C=64 | 29.7 | 531.7 | 17.9× |

The profile transformed from 69% glue to a healthy compute+comm bound distribution: MoE 27%, NCCL 23%, attention 18%, glue ~4%. The system was now performing in the 300–600 tok/s target range the user had identified at the start of the campaign.

Recognizing that capping context length was a temporary workaround, the assistant then built a proper capture-safe Triton indexer kernel with early-exit per page, making compute O(actual_seq) regardless of max context length. This was validated at 128K context with ~96–98% throughput retention and committed as checkpoint 598928d75.

Phase 4: PD Disaggregation and Production Deployment

With the core kernel optimizations delivering target throughput, the campaign shifted to production deployment. The assistant deployed prefill-decode (PD) disaggregation across all 8 GPUs: prefill with TP4 on GPU0–3 (NUMA0), decode with TP4 on GPU4–7 (NUMA1), with NIXL/UCX transfer and a router on 0.0.0.0:30001. Three systemd services were set up and enabled at boot, achieving ~2.7× lower decode TPOT (C=64: 88→32 ms).

KV cache capacity was maximized to 2.58M tokens at 512K context with --mem-fraction-static 0.85. The assistant also fixed the missing chat template by wiring --chat-template tool_chat_template_deepseekv32.jinja --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4, validating both thinking (reasoning_content separated) and parallel tool_calls through the router.

Phase 5: Monitoring, Quality, and Documentation

The final phase addressed operational excellence. The assistant set up a full Prometheus + Grafana monitoring stack from scratch (no Docker, binaries on Ubuntu 24.04), configured scraping of the prefill, decode, and router metrics endpoints, and provisioned a 17-panel KV-cache dashboard covering prefill throughput, TTFT/TPOT latency percentiles, PD-disagg transfer speed and queue depths, cache hit rate, and request rates — all verified populating under live load.

Quality issues were systematically resolved. The root cause of agent-coherence and tool-calling failures was a harness-side mismatch: the harness used text-based XML tools instead of native OpenAI function-calling, which DeepSeek-V4 doesn't handle reliably. Fixes included dropping the wrong --chat-template override (restoring the native encoding_dsv4 path), enabling thinking by default via SGLANG_DEFAULT_THINKING=true and a 7-line patch to serving_chat.py, setting the model name to deepseek-v4-flash so harnesses auto-detect the correct tool format, and setting temperature 0.6 as the server default to prevent greedy-decoding degeneration.

Finally, the assistant wrote a comprehensive engineering report (DSV4_SM120_REPORT.md) documenting the entire optimization journey: the ~17× throughput breakthrough, the MMA/kernel campaign, the PD-disagg deployment, the monitoring stack, and all the quality fixes — leaving the NextN-MTP and O(actual)-topk items as open follow-ups.

Themes and Engineering Lessons

Several themes emerge from this campaign that are broadly applicable to AI inference optimization:

The bottleneck is often not where you think it is. The initial diagnosis pointed to the attention kernel (57%) and generic glue operations (28%). The true root cause — the indexer's O(max_context) behavior — was hiding inside the "glue" category, masquerading as generic elementwise operations. Only operation-level profiling with tensor shape inspection revealed the truth.

Hardware counters tell the real story. The cufall reading showing 340W power draw and zero tensor-pipe utilization was the single most diagnostic piece of data. It immediately ruled out memory bandwidth or communication as the primary bottleneck and pointed squarely at CUDA-core fallback kernels.

Custom kernel development is the last resort, not the first. The assistant exhausted every configuration-level optimization (NVFP4 quantization, MoE backend swaps, FP8 autotuning, MTP) before resorting to custom kernel development. This is the correct order of operations: configuration changes are cheap and reversible; custom kernels are expensive and introduce maintenance burden.

The user's strategic instinct matters. The user's intervention to investigate scaling behavior (C=1, 16, 64) rather than optimize individual kernels was the critical pivot that led to discovering the linear per-request cost model, which in turn pointed to the indexer bug.

Phased delivery reduces risk. The assistant's decision to implement the MMA kernel without split-K first (Phase 1) and add split-K later (Phase 2) was a classic risk-mitigation strategy. It delivered value early while preserving the option to optimize further.

Conclusion

The DeepSeek-V4-Flash optimization campaign on 8× RTX PRO 6000 Blackwell GPUs is a case study in full-stack AI inference engineering. It spans the entire stack: from hardware profiling (cufall, nsys) through kernel design (Triton MMA, split-K, online softmax combine) through production deployment (PD disaggregation, systemd, Prometheus/Grafana) through quality assurance (chat templates, reasoning parsers, tool-calling formats) through documentation.

The final result — 531 tok/s at C=64, a 48× improvement from the starting point — is not the product of a single magic fix but of systematic, layered optimization. Each phase built on the previous one: the MMA kernel fixed the attention bottleneck, revealing the indexer bottleneck; the indexer fix revealed the NCCL communication floor; the PD disaggregation improved latency; the monitoring stack provided visibility; the quality fixes made the system usable.

For anyone undertaking a similar optimization campaign, the lesson is clear: measure everything, trust hardware counters over intuition, exhaust configuration options before writing custom code, and always ask what the system should be capable of before accepting what it is doing.## References

[1] "From 11 to 531 Tokens Per Second: The Full-Stack Optimization of DeepSeek-V4-Flash on Blackwell GPUs" — The overarching narrative of the optimization campaign.

[2] "The Strategic Pivot: How a Single Planning Message Reframed an ML Optimization Campaign" — Analysis of the assistant's comprehensive planning message at the campaign inflection point.

[3] "The Scaling Intervention: A User's Strategic Pivot in the DeepSeek-V4 Optimization Campaign" — The user's directive to investigate scaling behavior at C=1, C=16, C=64.

[4] "The Diagnosis That Unlocked 17× Throughput: How a Single Message Transformed DeepSeek-V4 Inference on Blackwell GPUs" — The synthesis of profiling data into a precise mechanistic diagnosis.

[5] "The Turning Point: Architecting a Custom MMA Sparse-MLA Decode Kernel for Blackwell GPUs" — The architectural blueprint for the custom attention kernel.

[6] "The MMA Sparse-MLA Decode Kernel: A Deep Dive into AI Inference Optimization on Blackwell GPUs" — Detailed design and implementation of the MMA kernel.

[7] "Deploying the MMA Sparse-MLA Decode Kernel: A Production Rollout of Custom Tensor-Core Attention on Blackwell" — The deployment of the MMA kernel into production.

[8] "The Power-of-Two Trap: Debugging a Triton MMA Kernel on Blackwell GPUs" — The solution to Triton's power-of-2 dimension constraint for the 448-dimension NOPE component.

[9] "The Moment of Validation: A Custom MMA Sparse-Decode Kernel Passes Correctness" — The correctness validation of the MMA kernel against the production SIMT kernel.

[10] "The 60-Second Vigil: Deploying a Custom MMA Kernel into Production" — The server restart and CUDA graph capture verification.

[11] "The 31-Millisecond Smoking Gun: Diagnosing Linear Per-Request Decode Cost in DeepSeek-V4 on Blackwell" — The discovery of the linear per-request cost model.

[12] "The 30-Millisecond Barrier: Diagnosing a Structural Bottleneck in DeepSeek-V4-Flash Decode" — Analysis of the throughput asymptote.

[13] "The 340-Watt Clue: How a Single GPU Metric Unlocked the DeepSeek-V4 Bottleneck" — The significance of the 340W power draw reading.

[14] "The Moment Attention Became the Wall: A Precision Diagnosis in the DeepSeek-V4-Flash Optimization Campaign" — The kernel-level profiling that identified attention at 57%.

[15] "The Pivot Point: How One Message Decided the Fate of a Kernel Optimization Campaign" — The decision to prioritize the attention kernel rewrite.

[16] "The Profiling Checkpoint: A Pivotal Moment in the DeepSeek-V4-Flash Optimization Campaign" — The profiling results that guided the optimization strategy.

[17] "The Diagnostic Breakthrough: Profiling the DeepSeek-V4-Flash Decode Bottleneck on Blackwell" — The operation-level profiling that revealed the indexer O(max_context) issue.

[18] "The Validation Gate: When a Custom CUDA Kernel Passes Its Final Test" — The standalone correctness test for the MMA kernel.

[19] "The Moment of Truth: Benchmarking a Custom MMA Kernel for DeepSeek-V4-Flash on Blackwell" — The benchmark results after MMA kernel deployment.

[20] "The Moment of Truth: Analyzing a Custom MMA Kernel's Performance on Blackwell GPUs" — Detailed performance analysis of the MMA kernel.

[21] "The Moment of Truth: Profiling the Custom MMA Kernel on Blackwell" — Profiling results showing attention dropping from 57% to ~10%.

[22] "The Infrastructure of Kernel Engineering: Setting Up the Development Loop" — The workflow for remote kernel development.

[23] "The Silent Integration: Placing the MMA Kernel in Production Code" — The integration of the MMA kernel into the SGLang dispatch path.

[24] "The Final Connection: Wiring a Custom MMA Kernel into Production" — The final wiring of the MMA kernel dispatcher.

[25] "Validating the MMA Sparse Decode Kernel: A Study in Rigorous Testing" — The validation methodology for the custom kernel.

[26] "The $25,000 Typo: What a Missing __name__ Attribute Reveals About AI Kernel Engineering" — A debugging story from the kernel development process.

[27] "The Moment of Compilation: When a Triton Kernel Meets Its Constraints" — The Triton compilation process for the MMA kernel.

[28] "The Glue That Binds: A Single Parameter Rename in a Custom CUDA Kernel Pipeline" — A parameter rename that fixed a critical issue.

[29] "The Prefill Ramp: When Benchmarking Numbers Lie" — Understanding prefill vs. decode benchmarking artifacts.

[30] "The False Positive: When Error Detection Itself Becomes the Error" — A debugging story about error detection.

[31] "The False Positive That Wasn't: Debugging Error Detection in a High-Stakes ML Deployment" — Resolution of the false positive issue.

[32] "Confirming the Asymptote: The Critical Benchmark on Blackwell" — Confirmation of the throughput asymptote.

[33] "The Parenthetical That Reframed a Bottleneck: How a Single Line About PCIe Bandwidth Reshaped the DeepSeek-V4 Optimization Campaign" — The PCIe bandwidth constraint analysis.

[34] "The 33 Tokens-Per-Second Wall: Diagnosing Memory-Latency-Bound Inference on Blackwell GPUs" — Analysis of the ~33 tok/s ceiling.

[35] "The Decisive Profile: Capturing Steady-State Kernel Timing for DeepSeek-V4-Flash on Blackwell" — The profiling methodology.

[36] "The Missing C=64 Point: Diagnosing a Benchmark Failure in the DeepSeek-V4 Optimization Campaign" — A benchmark failure diagnosis.

[37] "The CUDA Graph Ceiling: Diagnosing a Concurrency Crash in DeepSeek-V4-Flash on Blackwell" — A concurrency-related crash diagnosis.

[38] "The Scaling Sweep: Validating a Performance Hypothesis on Blackwell GPUs" — The scaling benchmark methodology.

[39] "The Scaling Sweep: Benchmarking DeepSeek-V4-Flash Throughput at the Concurrency Frontier" — Detailed scaling benchmark results.

[40] "The Critical Information-Gathering Step in a Performance Optimization Campaign" — The methodology for gathering diagnostic information.

[41] "Reading the Kernel Before the Rewrite: The Pivot Point" — The analysis of the existing kernel before rewriting.

[42] "The 60-Second Vigil: Deploying a Custom MMA Kernel into Production" — The server restart and deployment verification.

[43] "Deploying the MMA Sparse-MLA Decode Kernel: A Production Rollout of Custom Tensor-Core Attention on Blackwell" — The production deployment of the MMA kernel.