The Full-Stack Optimization of DeepSeek-V4-Flash on Blackwell GPUs: From 11 to 531 Tokens Per Second

Introduction

In the history of AI inference optimization, few stories are as dramatic as the transformation of DeepSeek-V4-Flash on eight NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120 architecture). What began as a system struggling to deliver 11 tokens per second at single-request concurrency ended as a production deployment achieving 531 tokens per second at batch size 64 — a 48× improvement from the starting point, with a single fix accounting for a 17.9× leap in a single day. This article synthesizes the engineering journey captured across Segment 68 of an intensive opencode coding session, tracing the arc from bottleneck diagnosis through custom kernel development, production deployment, monitoring infrastructure, and quality assurance.

The campaign was not a single heroic optimization but a systematic, layered effort spanning multiple phases: confirming the bottleneck was structural (CUDA-core fallback kernels rather than communication or memory bandwidth), designing and implementing a custom MMA sparse-MLA decode kernel using Triton tensor-core operations, discovering and fixing the indexer's O(max_context) bottleneck that was silently consuming 69% of GPU time, deploying prefill-decode (PD) disaggregation across all 8 GPUs, setting up a full Prometheus/Grafana monitoring stack, resolving agent-coherence and tool-calling quality issues, and documenting the entire journey in a comprehensive engineering report.

Phase 1: The Diagnostic Breakthrough

The campaign began with a stark reality: the system was delivering approximately 11 tokens per second at C=1 and ~28 tok/s at C=16 — catastrophically far from the 300–600 tok/s that roofline analysis suggested the hardware should support. The assistant's initial profiling using NVIDIA's CUDA Flight (cufall) profiler and nsys tracing revealed that the GPU was drawing only 340 watts of its 600-watt TDP, with tensor-pipe utilization near zero. The SMs were active but doing scalar SIMT work, not tensor-core matrix operations.

The kernel-level breakdown was damning. The _tiled_sparse_decode_kernel consumed 57% of all decode GPU time, and it had two structural defects: its grid was (B, H) = (batch, 64 heads), causing all 64 head-programs to independently re-gather the identical KV cache data from memory (up to 64× redundant reads), and it computed attention scores using scalar tl.sum(q*k) operations on SIMT lanes rather than tensor-core tl.dot matrix multiplies. A second bottleneck at ~28% was labeled "unfused elementwise/copy glue" — thousands of tiny kernel launches for RoPE, RMSNorm, dequantization, and residual operations. A third issue at 8.8% was forced-FP32 SIMT GEMMs in the indexer and multi-head connection pre-linear layers, caused by conservative sm_120 environment overrides.

The user's strategic intervention — "Continue perf investigation, at C=1, C=16, C=64; Should scale relatively well" — reframed the campaign from tactical kernel-tweaking to systematic bottleneck diagnosis. The assistant's response revealed a striking linear model: decode step time followed step_time ≈ 40 + 31·N milliseconds, where N was the number of concurrent requests. Each additional request added ~31 ms of non-amortizable GPU time, and throughput asymptoted at 1000/31 ≈ 32 tok/s regardless of concurrency. This linear per-request cost was happening inside the CUDA graph, ruling out CPU-side scheduler overhead — the bottleneck was purely in the GPU kernels themselves.

Phase 2: The Custom MMA Sparse-MLA Decode Kernel

With the diagnosis confirmed, the assistant embarked on the most technically demanding phase: designing and implementing a custom MMA (matrix-multiply-accumulate) sparse-MLA decode kernel to replace the 57% bottleneck. The design process is a masterclass in GPU kernel engineering under hardware constraints.

The core insight was elegant: Multi-head Latent Attention (MLA) uses a single shared KV head for all 64 query heads — a property the existing kernel completely ignored by launching 64 independent programs. The new kernel would launch one program per batch that handles all heads simultaneously, computing Q @ K^T as a single matrix multiply using Triton's tl.dot, which maps directly to tensor-core MMA instructions.

The design involved several interlocking decisions:

Block size selection. The assistant considered BLOCK_H=64 (all heads in one program) versus BLOCK_H=32 (two programs per batch). BLOCK_H=64 maximized KV reuse but created shared memory pressure: a Q tile at 64 heads consumed ~64 KB, a K tile at BLOCK_T=32 consumed ~32 KB, and transposed buffers added more — exceeding the ~99 KB shared memory limit on sm_120. The assistant settled on BLOCK_H=32 as a conservative starting point, accepting that KV would be read twice instead of once.

Nope/rope separation. The 512-dimensional latent space is split into a 448-dimension NOPE (non-position-encoded) component and a 64-dimension ROPE (rotary position encoding) component. Rather than concatenating them, the assistant kept them separate throughout the computation, requiring two dot products for attention scores and two separate output accumulations.

The power-of-two padding trap. Triton's tl.arange requires power-of-2 dimensions, but the NOPE dimension was 448 — not a power of two. The solution was to pad the NOPE dimension to 512 and use runtime masking to zero out columns 448:512. Since zero columns contribute nothing to the dot product, the result is mathematically identical to computing over just the 448 real values.

Split-K parallelization. For low batch sizes where occupancy would be poor, the assistant designed a split-K approach that partitions the topk token dimension across multiple blocks, each producing partial attention outputs and log-sum-exp (LSE) values. A separate combine kernel merges these partial results using the standard online softmax combine algorithm — a technique borrowed from prior work on the Kimi K2.6 DDTree engine.

The implementation was gated behind an environment variable (SGLANG_SM120_MMA_FLASHMLA=1), enabling A/B testing without disrupting the running service. After validation against the production SIMT kernel (relative error ≤ 6.7e-3 across batch sizes 1–32 and topk values 128–512), the kernel was deployed via a server restart. The results were immediate: attention dropped from 57% to ~10% of decode GPU time, and throughput improved by 2.2–2.9× across all concurrency levels (C=1: 11.5→33.5, C=16: 26.6→58.6, C=64: 29.7→64.4 tok/s).

Phase 3: The FP32-to-bf16 GEMM Flip

With the attention kernel optimized, the profile revealed a new target: the forced-FP32 SIMT GEMMs in the indexer's batch matrix multiply (bmm) and the MHC (Multi-Head Combining) pre-linear layer, consuming 17% of GPU time (822 ms). These operations were running in FP32 via a CUDA-core SIMT kernel (cutlass_80_simt_sgemm) because the model code explicitly cast the hidden state to FP32 before the bmm.

The assistant flipped these operations to bf16 tensor-core operations (cutlass_80_tensorop_bf16_s16816gemm_relu), achieving a 2× speedup on that specific kernel (down to 499 ms). However, the initial implementation introduced dtype cast overhead that ate most of the savings: the hidden state x was being cast from bf16 to fp32 and then back to bf16, creating redundant memory traffic. The assistant fixed this by feeding the original bf16 hidden state directly, eliminating the round-trip cast.

Phase 4: The Indexer O(max_context) Breakthrough

With the MMA kernel deployed and the FP32 GEMMs flipped, the assistant turned to the remaining ~69% of GPU time consumed by "glue" operations. The user suggested trying torch.compile to fuse the elementwise kernels, but the assistant confirmed this was incompatible with the stack — torch.compile fails at CUDA graph capture even with the stock kernel, due to a fundamental conflict between Inductor's compiled forward and SGLang's CUDA graph capture mechanism.

Instead, the assistant profiled the glue operations via eager-mode tracing, revealing the dominant launching ops: aten::copy_ (35%), aten::mul (13.6%), aten::clamp_min (13.2%), aten::bmm (10%), and aten::sum (7%). But the critical insight came from examining the tensor shapes: these operations were running on [32, 262208, 64] tensors — the indexer was computing scores over the full ~1-million-token max context (262,208 c4-positions) every decode step, even though the actual context was only ~512 tokens.

This single issue was the root cause of the linear per-request cost model. The DSA (Dense Sparse Attention) indexer's torch fallback was doing O(max_context) work per decode step regardless of actual sequence length. The fix was deceptively simple: capping --context-length 8192 cut the indexer's work by approximately 128×. The results were staggering:

| Concurrency | Before (tok/s) | After (tok/s) | Improvement | |---|---|---|---| | C=1 | 33.5 | 85.1 | 2.5× | | C=16 | 58.6 | 285.1 | 4.9× | | C=64 | 29.7 | 531.7 | 17.9× |

The profile transformed from 69% glue to a healthy compute+comm bound distribution: MoE 27%, NCCL 23%, attention 18%, glue ~4%. The system was now performing in the 300–600 tok/s target range the user had identified at the start of the campaign.

Recognizing that capping context length was a temporary workaround, the assistant then built a proper capture-safe Triton indexer kernel with early-exit per page, making compute O(actual_seq) regardless of max context length. This was validated at 128K context with ~96–98% throughput retention and committed as checkpoint 598928d75.

Phase 5: PD Disaggregation and Production Deployment

With the core kernel optimizations delivering target throughput, the campaign shifted to production deployment. The assistant deployed prefill-decode (PD) disaggregation across all 8 GPUs: prefill with TP4 on GPU0–3 (NUMA0), decode with TP4 on GPU4–7 (NUMA1), with NIXL/UCX transfer and a router on 0.0.0.0:30001. Three systemd services were set up and enabled at boot:

Phase 6: Monitoring, Quality, and Documentation

The final phase addressed operational excellence. The assistant set up a full Prometheus + Grafana monitoring stack from scratch (no Docker, binaries on Ubuntu 24.04), configured scraping of the prefill, decode, and router metrics endpoints, and provisioned a 17-panel KV-cache dashboard covering prefill throughput, TTFT/TPOT latency percentiles, PD-disagg transfer speed and queue depths, cache hit rate, and request rates — all verified populating under live load.

The Quality Crisis

Despite the impressive throughput numbers, the deployment was plagued by a constellation of failures: the model's thinking capability wouldn't trigger by default, tool calls were truncated or malformed, agent coherence was broken, and the output quality was unacceptable. Each of these issues required its own diagnostic journey.

The thinking default bug. The environment variable SGLANG_DEFAULT_THINKING=true had been set in the systemd service files, and the assistant had verified it was present in the process environment. Yet when clients sent plain requests without an explicit reasoning parameter, the model responded with zero reasoning content. The debugging journey touched on environment variable parsing, object instance identity, subclass overrides, and renderer architecture. The critical discovery came from a debug log line added to serving_chat.py: the environment variable was being read correctly (env=True), and thinking_requested was being set to True, but chat_encoding_spec was None. The dsv4 encoding path — the renderer responsible for injecting <think> tags into the prompt — was being bypassed entirely. The fix was a four-line patch that propagated the resolved thinking default into chat_template_kwargs before the encoding step.

The temperature default crisis. The model was producing degenerate, repetitive, or overly deterministic outputs — a classic symptom of greedy decoding. The assistant traced this to the sampling temperature defaulting to an incorrect value. The model's generation_config.json was edited to set temperature to 0.6 and top_p to 0.95, and the server was configured to use these defaults when clients omitted sampling parameters.

The model name that broke tool calling. The most impactful quality issue was the tool-calling failure. The user reported that the model could not reliably write a simple HTML file — tool calls were truncated, malformed, or simply not invoked. The user's insight was sharp: the SGLang server was serving the model under its raw filesystem path (/root/models/DeepSeek-V4-Flash-NVFP4) rather than a clean model identifier like deepseek-v4-flash. This mismatch was causing the harness — the client framework managing the interaction — to fall back to a text-based XML tool format that DeepSeek-V4 wasn't trained on, instead of using native OpenAI-style function calling.

The fix required setting --served-model-name deepseek-v4-flash on both the prefill and decode servers. But this seemingly simple change triggered a cascade of discoveries: the sed-generated decode script needed careful verification to ensure the model name flag was correctly inserted; the router cached the old model ID at worker registration time and did not dynamically refresh it, requiring a router restart; and the root cause was ultimately a harness-side mismatch where the testing harness was injecting text-based XML tool descriptions into the system prompt instead of using the OpenAI-compatible tools parameter.

The Engineering Report

The final deliverable was a comprehensive engineering report (DSV4_SM120_REPORT.md) documenting the entire optimization journey. The report covered the hardware and software stack, the kernel optimization campaign (custom MMA kernels, indexer fix, bf16 conversions), the PD disaggregation deployment with systemd units and NUMA binding, the monitoring stack with Prometheus and Grafana, all the quality fixes, and open follow-up items (NextN-MTP integration and O(actual)-topk optimization). The report was written to the workspace directory as a permanent artifact alongside the codebase.

Themes and Engineering Lessons

Several themes emerge from this campaign that are broadly applicable to AI inference optimization:

The bottleneck is often not where you think it is. The initial diagnosis pointed to the attention kernel (57%) and generic glue operations (28%). The true root cause — the indexer's O(max_context) behavior — was hiding inside the "glue" category, masquerading as generic elementwise operations. Only operation-level profiling with tensor shape inspection revealed the truth.

Hardware counters tell the real story. The cufall reading showing 340W power draw and zero tensor-pipe utilization was the single most diagnostic piece of data. It immediately ruled out memory bandwidth or communication as the primary bottleneck and pointed squarely at CUDA-core fallback kernels.

Custom kernel development is the last resort, not the first. The assistant exhausted every configuration-level optimization (NVFP4 quantization, MoE backend swaps, FP8 autotuning, MTP) before resorting to custom kernel development. This is the correct order of operations: configuration changes are cheap and reversible; custom kernels are expensive and introduce maintenance burden.

The user's strategic instinct matters. The user's intervention to investigate scaling behavior (C=1, 16, 64) rather than optimize individual kernels was the critical pivot that led to discovering the linear per-request cost model, which in turn pointed to the indexer bug.

Phased delivery reduces risk. The assistant's decision to implement the MMA kernel without split-K first and add split-K later was a classic risk-mitigation strategy. It delivered value early while preserving the option to optimize further.

Production deployment is not an afterthought. The PD disaggregation, systemd services, KV cache tuning, chat template fixes, and monitoring setup were all essential to making the optimization gains usable in practice. The quality fixes — thinking default, temperature default, model name, tool-calling harness — transformed a system that was blazingly fast but unreliable into one that was both fast and correct.

Observability is not optional. The monitoring stack was not a nice-to-have; it was essential for understanding whether the system was healthy. The KV cache fullness gauge, in particular, is the canary in the coal mine for the entire inference system. Without it, the operator cannot detect when memory pressure is building, when requests are being queued, or when the prefill and decode stages are unbalanced.

Documentation is a first-class deliverable. The engineering report captured not just what was done, but why. The decision to pivot from MTP to PD disaggregation, the choice of binary-over-Docker for monitoring, the specific memory fractions for prefill vs. decode — all of these decisions were documented with their reasoning. The report also explicitly listed the remaining work, ensuring that future engineers would know exactly where to pick up.

Conclusion

The DeepSeek-V4-Flash optimization campaign on 8× RTX PRO 6000 Blackwell GPUs is a case study in full-stack AI inference engineering. It spans the entire stack: from hardware profiling (cufall, nsys) through kernel design (Triton MMA, split-K, online softmax combine) through production deployment (PD disaggregation, systemd, Prometheus/Grafana) through quality assurance (chat templates, reasoning parsers, tool-calling formats) through documentation.

The final result — 531 tok/s at C=64, a 48× improvement from the starting point — is not the product of a single magic fix but of systematic, layered optimization. Each phase built on the previous one: the MMA kernel fixed the attention bottleneck, revealing the indexer bottleneck; the indexer fix revealed the NCCL communication floor; the PD disaggregation improved latency; the monitoring stack provided visibility; the quality fixes made the system usable.

For anyone undertaking a similar optimization campaign, the lesson is clear: measure everything, trust hardware counters over intuition, exhaust configuration options before writing custom code, and always ask what the system should be capable of before accepting what it is doing. The journey from 11 to 531 tok/s is not just a story about kernel engineering — it is a story about systematic thinking, disciplined execution, and the relentless pursuit of understanding where every microsecond goes.## Deeper Dive: The Verification Instinct

One of the defining characteristics of this campaign was the assistant's relentless verification instinct — the refusal to accept positive signals without independent confirmation. This was exemplified early in the deployment when, after launching the two PD servers, the assistant polled their logs for the "fired up and ready" signal and received confirmation at the first 20-second interval. A naive operator would have accepted this and proceeded. But the assistant recognized a contradiction: the model weights alone take approximately 33 seconds to load from disk, and CUDA graph capture adds another ~30 seconds. A 20-second ready signal was physically impossible unless the weights were cached in the OS page cache from previous runs.

Rather than dismissing the anomaly, the assistant designed a three-pronged verification strategy: an HTTP health check to /health_generate on each server's port, a log tail inspection for key initialization markers, and a GPU memory check via nvidia-smi to confirm the model was actually loaded on the correct GPUs. All three checks passed, confirming that the servers were genuinely operational — the page cache had accelerated weight loading to ~15–20 seconds.

This verification instinct reappeared throughout the campaign. When the assistant deployed the 17-panel Grafana dashboard, it didn't just check that the file was written — it queried Prometheus for live metric data, confirmed that the prefill throughput and decode generation rates were populating, and interpreted the values diagnostically (explaining why the prefill showed zero, why TTFT p95 was elevated, and why KV transfer speed was 0.1 GB/s). When the model name fix was applied, the assistant verified the sed-generated script before restarting, discovered the router's caching behavior, and established a baseline with a controlled diagnostic test before declaring success.

The Router That Crashed But Still Worked

Among the operational challenges documented in this segment is a revealing episode involving the PD router. When the assistant first deployed the router as part of the PD disaggregation architecture, it encountered a situation where the router process crashed but the system continued to function. This counterintuitive behavior — a crashed router that still routed requests — was traced to a stale port conflict. The router's port (30001) was still held by the previous single-server instance that had been killed but not fully released. The new router process failed to bind to the port, but the old process's socket remained in a TIME_WAIT or CLOSE_WAIT state, continuing to accept connections and forward them to the backend servers.

This discovery led to a systematic cleanup of stale processes. The assistant used fuser and lsof to identify processes holding ports, kill -9 to terminate stubborn remnants, and systemctl reset-failed to clean up systemd state. The episode reinforced a critical operational principle: in distributed systems, a component can appear to be working (the router is "running") while actually being non-functional (it's a different process than the one you think is running). The assistant's response — verifying process identity via PID, checking port ownership, and explicitly cleaning up stale state — became a standard part of the deployment workflow.

The Systemd Transition: From Manual Processes to Production Services

A significant portion of this segment is devoted to the transition from manually launched background processes to systemd-managed production services. This transition is a hallmark of engineering maturity: manual processes are fine for development, but production deployments require automatic restart, dependency ordering, logging to journald, and boot-time enablement.

The assistant created three systemd unit files: sglang-dsv4-prefill.service, sglang-dsv4-decode.service, and sglang-dsv4-router.service. Each unit specified the exact command line, environment variables, and resource constraints. The prefill and decode units used ExecStartPre to bind to specific NUMA nodes via numactl, ensuring optimal memory access patterns. The router unit specified After=sglang-dsv4-prefill.service sglang-dsv4-decode.service to ensure the router only started after both backend servers were ready.

The deployment encountered several systemd-specific challenges. The first was a stalled deployment where the system went silent — the services appeared to be starting but never completed initialization. This was traced to a missing Type=simple declaration combined with a process that forked, causing systemd to lose track of the main PID. The fix was to ensure the services ran in the foreground (no nohup, no &) so systemd could monitor them directly.

The second challenge was the startup timeout. Model loading on Blackwell GPUs can take several minutes, especially with CUDA graph capture. The default systemd startup timeout of 90 seconds was insufficient. The assistant increased it to 900 seconds via TimeoutStartSec=900, giving the servers ample time to initialize.

The third challenge was dependency ordering across the PD services. The router depends on both prefill and decode being ready before it can accept requests. The assistant implemented this via After= directives combined with ExecStartPre scripts that polled the backend servers' health endpoints before starting the router.

The Empty Message and Engineering Closure

The segment concludes with a fascinating interaction: after the assistant delivered a polished summary of both completed deliverables — the 17-panel Grafana dashboard and the comprehensive engineering report — the user responded with an empty message. No text, no command, no question, no acknowledgment. Just silence.

This empty message, in the architecture of the conversation, carries multiple possible interpretations. It could have been an accidental send, an implicit "continue" or status prompt, a system artifact or UI glitch, or a deliberate test of the assistant's ability to infer intent from context. The assistant's response — a massive, structured project summary covering the goal, constraints and preferences, progress broken into Done/In Progress/Blocked, key decisions, next steps, critical context, and relevant files — suggests it interpreted the silence as an implicit request for a comprehensive status update.

This moment reveals something important about the final mile of engineering projects. The verification, documentation, and delivery are not just checkboxes to tick — they are the artifacts that transform raw technical achievement into sustainable, understandable infrastructure. The empty message, whether intentional or accidental, prompted the assistant to consolidate weeks of work into a single reference document, ensuring that the knowledge generated by the campaign would survive beyond the current session.

The Broader Significance

This optimization campaign is a case study in systematic performance engineering at the frontier of GPU computing. The Blackwell sm_120 architecture is new and poorly documented. The DeepSeek-V4-Flash model is complex, with custom attention mechanisms, FP4 quantization, and MoE routing. The SGLang inference framework has its own constraints around CUDA graph capture. The combination of these three systems — model, framework, hardware — creates a unique optimization landscape where standard tools (torch.compile, flash-attn, MSCCL++) may not work, and custom solutions are required.

The key lessons from this campaign are:

  1. Profile before you optimize. Every significant decision was driven by profiling data, not intuition. The indexer bottleneck was invisible until the assistant profiled with tensor shapes enabled.
  2. Build the right diagnostic tools. When the standard profiler couldn't correlate GPU kernels to CPU operations, the assistant built a custom parser. When that failed due to CUDA graph replay, the assistant pivoted to eager profiling.
  3. Know when to stop optimizing. The NCCL all-reduce was confirmed at the PCIe floor — no further optimization possible. The MTP/EAGLE integration was blocked by a fundamental architecture limitation. Recognizing these dead ends saved time that could be spent on productive work.
  4. Document everything. The PROFILE_FINDINGS.md file, the git commits, the todo list updates — all created a permanent record of the optimization journey that would be invaluable for future engineers working on similar systems.
  5. Production deployment is not an afterthought. The PD disaggregation, systemd services, KV cache tuning, chat template fixes, and monitoring setup were all essential to making the optimization gains usable in practice.
  6. Configuration fidelity matters enormously. The --chat-template override that broke tool calling is a perfect example of how a seemingly innocuous configuration change can have catastrophic effects. The assistant added the flag to fix a warning message, but in doing so, it overrode SGLang's architecture-aware encoding logic.
  7. Verification before trust. The assistant repeatedly refused to accept positive signals without independent confirmation. The 20-second ready signal was verified through HTTP health checks, log inspection, and GPU memory queries. The parser fix was validated with a dedicated test script before being deployed to production.