The Methodical Pursuit of Performance: Optimizing DeepSeek-V4-Flash on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference, the gap between a deployment that works and one that performs can span orders of magnitude. This article chronicles a methodical, measurement-driven optimization campaign for DeepSeek-V4-Flash — a 284-billion-parameter Mixture-of-Experts model with 13 billion active parameters — running on eight NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120 architecture) using SGLang. What begins as a successful but catastrophically slow deployment (~25 tok/s at concurrency 16, against a target of ~1,000 tok/s) evolves through a systematic investigation of every configuration lever: NCCL communication tuning, CUDA graph capture, MTP speculative decoding, FP8 GEMM autotuning, and a pivotal switch from stock MXFP4 to NVIDIA's NVFP4 quantization format.

The story that emerges is one of incremental gains against a structural ceiling. Each optimization delivers measurable improvement — 6% here, 24% there, 47% for single requests — but the aggregate still falls far short of the hardware's roofline potential. The decisive bottleneck, revealed through GPU profiling and confirmed through cross-validation, is the sm_120 fallback kernel path: the sparse-MLA attention and MoE slot-GEMV kernels that should run on tensor cores are instead executing on CUDA cores via Triton fallbacks, consuming 39% and 38% of decode time respectively. The NVFP4 checkpoint fixes the MoE path, routing it through tensor cores, but attention remains the primary limiter — and the MTP verifier's memory consumption silently halves the effective batch size.

This is a case study in the limits of configuration optimization, the importance of measurement-driven iteration, and the moment when an engineering team must recognize that the remaining gap requires custom kernel development rather than flag tuning.

Part I: The Deployment That Worked — But Wasn't Fast Enough

The campaign began with a significant engineering achievement: deploying DeepSeek-V4-Flash with prefill-decode (PD) disaggregation across two NUMA domains. The prefill server ran on GPUs 0–3 (NUMA0) with tensor parallelism 4, processing input prompts and building KV caches. The decode server ran on GPUs 4–7 (NUMA1), generating tokens autoregressively. KV cache transfer between them was orchestrated by NVIDIA's NIXL library over UCX transport. A Rust-based router on port 8000 coordinated traffic, presenting a unified OpenAI-compatible endpoint to clients ([msg 12385], [msg 12388]).

The architecture was sound. The model loaded correctly — 146 GB of FP4-quantized weights across 46 shards. The servers reported "fired up and ready to roll" within 60 seconds, benefiting from a warm JIT cache ([msg 12386]). An end-to-end test produced a coherent answer about prefill-decode disaggregation itself — a self-referential validation that the entire pipeline functioned correctly ([msg 12389]).

But the benchmark results, when they arrived after several minutes of polling, were sobering ([msg 12390], [msg 12391], [msg 12392]):

| Configuration | Throughput (tok/s) | TPOT (ms) | TTFT (ms) | |---|---|---|---| | Single-node TP4, bs=1 | 10.2 | 94 | 696 | | PD disaggregation, C=1 | 9.95 | 94 | 893 | | PD disaggregation, C=8 | 24.5 | 282 | 2,000 | | PD disaggregation, C=16 | 25.0 | 510 | 8,800 |

The pattern was unmistakable: throughput plateaued at ~25 tok/s regardless of concurrency, and time-to-first-token degraded catastrophically as queue buildup overwhelmed the decode side. The assistant's diagnosis was precise: "PD doesn't raise aggregate tok/s here — decode is the bottleneck, not prefill interference or KV transfer" ([msg 12392], [msg 12394]). Prefill-decode disaggregation, for all its architectural elegance, cannot help when the decode phase itself is the binding constraint.

Part II: The User's Challenge and the Diagnostic Pivot

The assistant's initial conclusion — that the sm_120 fallback kernels represented a hard hardware ceiling — was met with a firm rejection from the user. In [msg 12395], the user wrote: "We expect much much faster than 25T/s on this model, at C=16 should be at/above 1k tps." This was not a suggestion; it was a statement of fact about what the hardware should be capable of. The user pointed to two specific levers: NCCL LL (low-latency protocol) and alternatives to the Marlin MoE backend, hinting that optimizations previously applied in the repository (from work on the Kimi K2.6 model) were missing from the current deployment.

This intervention fundamentally reshaped the campaign ([msg 12396]). The assistant pivoted from acceptance to investigation, launching parallel searches of the local repository for prior optimization work and querying the remote server's environment configuration. The research was decisive: the K2.6 service had achieved 1,291 tok/s with TP8 on the same hardware using a specific NCCL PCIe tuning block — and the current DeepSeek-V4-Flash deployment had none of those settings. The NCCL environment variables that should have been set — NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=5, NCCL_MIN_NCHANNELS=8, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512 — were all absent.

The user reinforced this direction with a four-word message in [msg 12397]: "cuda graph too, especially." This terse correction elevated CUDA graphs — which had delivered a documented 3.8× speedup on K2.6 (from 26 to 98 tok/s) — above other optimizations in priority. The assistant recognized that "CUDA graphs enabled" and "CUDA graphs working effectively" are not the same thing, and that the current deployment might be falling back to eager execution for uncaptured batch sizes.

Part III: Applying the Proven Playbook

Message [msg 12398] captures the synthesis of this research into a concrete action plan. The assistant created a shared NCCL environment file (dsv4_nccl_env.sh) encoding the proven PCIe tuning parameters, and an updated server launch script (serve_dsv4_tp4.sh) with --cuda-graph-max-bs 32, --num-continuous-decode-steps 4, and --mem-fraction-static 0.85. The strategy was to validate on single-node TP4 first — the simplest configuration — before applying the optimizations to the disaggregated deployment.

But the execution hit a snag. The pkill command intended to stop the old PD servers was followed by a nohup relaunch, but the new server never started. The log file showed stale content from the previously killed process, and the assistant fell into a monitoring loop, checking every 30 seconds for a server that would never arrive ([msg 12399]). The user's single-word intervention — "proceed" ([msg 12400]) — broke the loop, redirecting effort from observation to diagnosis.

The subsequent investigation ([msg 12401]) revealed the root cause: the old prefill process (PID 73216) had survived the pkill command, still holding port 30000. The new server, unable to bind to the port, had failed silently — a classic race condition where asynchronous process termination and background shell semantics conspired to produce a silent failure. The assistant's five-dimensional state check — processes, scripts, logs, GPU memory, and health endpoint — provided a reusable debugging methodology for future deployment attempts.

The debugging continued through several more rounds: a failed SSH command due to quoting issues ([msg 12402]), a server that started but was immediately killed by its own pkill command matching itself ([msg 12403], [msg 12404]), and an OOM crash caused by insufficient memory for the model weights plus NCCL buffers ([msg 12407], [msg 12410]). Each failure was diagnosed and addressed: the self-kill was fixed by using kill with a specific PID instead of pkill -f; the OOM was resolved by reducing --mem-fraction-static from 0.85 to 0.70 and ensuring the NCCL environment was properly sourced.

Part IV: The Optimization Campaign — Incremental Gains Against a Ceiling

Once the server was stable, the assistant executed a systematic, measurement-driven optimization campaign. Each lever was benchmarked individually, with results documented and analyzed before proceeding to the next.

FP8 GEMM autotuning. The assistant pulled the latest upstream SGLang and generated SM120 FP8 GEMM autotune configurations. The result: a ~6% improvement in throughput. This was consistent with profiling data showing that FP8 GEMM accounts for only ~6% of decode time — optimizing a small fraction of the workload yields proportionally small gains.

NCCL low-latency tuning. The NCCL LL protocol and Ring algorithm, which had been transformative for K2.6, produced negligible improvement for DeepSeek-V4-Flash. Profiling revealed why: communication accounts for only ~2% of decode time on this configuration. The NCCL tuning, while essential on systems where all-reduce dominates, was attacking a non-bottleneck.

MTP/EAGLE speculative decoding. This was the most dramatic single-request optimization: MTP boosted throughput by 47% for individual requests, from ~10 tok/s to ~15 tok/s. However, at concurrency, the gain vanished. The MTP verifier — the draft model that validates speculative tokens — became saturated, consuming enough GPU memory to effectively halve the batch size from 16 to 8. The verifier's memory footprint meant that at C=16, only 8 slots were actually doing productive decode work, while the other 8 were consumed by verifier state.

NVFP4 quantization. The assistant pivoted to the highest-leverage fix: switching from the stock MXFP4 checkpoint to NVIDIA's official NVFP4 quantization format ([msg 25820]). This was not a trivial change — it required fetching a PR for correct NVFP4 auto-detection and downloading a 149 GB checkpoint. But the payoff was significant: both NVFP4 backends (Marlin W4A16 and native cutlass FP4 grouped GEMM) delivered identical throughput of ~28 tok/s at C=16, a ~24% improvement over the baseline. Crucially, this confirmed that the MoE path was now executing on tensor cores rather than CUDA cores.

Part V: The Decisive Profile and the Structural Ceiling

Despite these gains — a cumulative ~30% improvement from baseline — the system was still operating at roughly 1/40th of the user's target. The assistant returned to profiling to understand why.

A concurrency-16 GPU profile revealed the decisive bottleneck ([msg 12418], [msg 12442], [msg 12443]). Two kernels dominated decode time:

Part VI: The Path Forward — Custom Kernel Development

The campaign reached a natural inflection point. Configuration tuning — NCCL protocols, CUDA graphs, memory fractions, continuous decode steps — had been exhausted. Checkpoint format changes (MXFP4 to NVFP4) had delivered their ~24% gain. MTP speculative decoding had shown promise for single requests but was neutralized at concurrency by verifier memory overhead.

The remaining bottleneck was structural: the sm_120 fallback kernels for sparse-MLA attention. The fast fused paths (DeepGEMM, trtllm-gen, FP4 indexer) are architecture-gated to SM100 and do not run on sm_120. The only path forward is custom kernel development — the same playbook that delivered 3–6× gains for K2.6 through a custom verify-attention kernel.

The assistant documented this finding in a comprehensive SUMMARY.md file ([msg 12393]), capturing the deployment configuration, benchmark results, and bottleneck diagnosis. The summary served as a boundary object — marking the transition from "what configuration can achieve" to "what requires kernel engineering." Two concrete paths emerged:

  1. Disable MTP to restore the full batch capacity of 16, then re-measure to establish a clean baseline without verifier overhead.
  2. Build a split-K tensor-core sparse-attention kernel for sm_120, attacking the 38% attention bottleneck directly — the same approach that had proven transformative for K2.6.

Conclusion: The Discipline of Measurement-Driven Optimization

This chunk of the DeepSeek-V4-Flash deployment campaign is a masterclass in systematic performance engineering. The assistant demonstrated: