The Pivot to DeepSeek-V4-Flash: A Strategic Experiment in Prefill-Decode Disaggregation

Introduction

In the middle of an intensive optimization campaign on the Kimi-K2.6 model—where the assistant had just delivered a 3–6× decode speedup through custom CUDA kernels and identified MoE imbalance as the remaining bottleneck—the user issued a message that would fundamentally redirect the entire session. Message [msg 12349] reads:

New experiment, now focusing on nvfp4; Deploy and benchmark https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP TP4, with the caveat - make it work on SGlang, and deploy https://docs.sglang.io/docs/advanced_features/pd_disaggregation - Prefill TP4, Decode TP4, each using 4 gpus (comms stay on 1 cpu for each that way); Stop current sglang instance, download model, pull fresh sglang and port needed changes (if any, e.g. set up GEMMs for PRO 6000 if that's still needed)

This single message represents a dramatic shift in scope, architecture, and technical risk. It is not merely a request to try a different model—it is a strategic pivot from a deep, months-long custom-kernel effort on one architecture (Kimi-K2.6 with DDTree speculative decoding) to an entirely new model family (DeepSeek-V4-Flash) with a radically different deployment topology (prefill-decode disaggregation across NUMA domains). Understanding why this message was written, what assumptions it encodes, and what it demanded of the assistant reveals the dynamics of high-stakes ML infrastructure experimentation.

The Motivation: Why Pivot Now?

The user's motivation must be understood against the backdrop of the preceding session. The assistant had just completed a heroic effort on Kimi-K2.6: building a native C/C++/CUDA DDTree inference engine from scratch, writing custom sm_120 verify-attention kernels, achieving CUDA graph capture-safety, and delivering a verified 3–6× decode speedup over the Triton baseline. Yet despite these gains, the fundamental bottleneck had shifted to MoE expert imbalance at batch size 1—a structural limitation of tensor parallelism with small batches that no amount of kernel optimization could fix. The assistant's final report (msg [msg 12346]) candidly stated: "single-request decode is now near the MoE/comm floor."

The user's pivot to DeepSeek-V4-Flash signals a recognition that the K2.6 path had reached diminishing returns on single-request latency. Rather than pursuing the remaining levers (batching, expert parallelism), the user chose to explore a completely different model and deployment paradigm. The mention of "nvfp4" (NVIDIA's NVFP4 quantization format) indicates a hypothesis that the Blackwell tensor cores' native FP4 support could unlock throughput that the BF16/INT4 K2.6 deployment could not achieve. The user was not abandoning the K2.6 work—they were hedging their bets by evaluating whether a different model, quantized to exploit Blackwell's native capabilities, could deliver the throughput that the custom-kernel approach could not.

The inclusion of prefill-decode (PD) disaggregation is equally strategic. PD disaggregation separates the compute-heavy prefill phase from the memory-bound decode phase, running each on dedicated GPU groups. This is an advanced deployment topology that SGLang supports but few practitioners have deployed on non-NVLink hardware. The user's specification—"Prefill TP4, Decode TP4, each using 4 gpus (comms stay on 1 cpu for each that way)"—shows a sophisticated understanding of NUMA topology. The 8-GPU machine has two NUMA nodes (GPUs 0–3 on NUMA0, GPUs 4–7 on NUMA1), and the user wanted each TP4 group to stay within its NUMA domain so that intra-group communication stays on a single CPU socket, avoiding cross-socket PCIe penalties. This is a topology-aware deployment design, not a generic request.

Assumptions Embedded in the Message

The message carries several critical assumptions, some of which would prove incorrect:

First assumption: that the model can run on SGLang. The HuggingFace model card for canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP is exclusively focused on vLLM support—it references a forked vLLM with multiple experimental patches. The model uses the DeepseekV4ForCausalLM architecture, which at the time of this message was a brand-new architecture with sparse attention mechanisms, a lightning indexer, compressor, and hash-based MoE routing. The assumption that SGLang—even a fresh pull—would support this architecture was optimistic. As the assistant would soon discover, SGLang had only a roadmap issue tracking DeepSeek V4 support, and the installed version (0.5.11) almost certainly lacked the model definition entirely.

Second assumption: that the model's quantization format is NVFP4. The model name contains "W4A16-FP8-MTP," which describes INT4 weights, FP8 activations for attention, and BF16 Multi-Token Prediction heads. The user's framing as "nvfp4" conflates NVIDIA's native NVFP4 format (which routes MoE through tensor-core paths like Marlin W4A16 or cutlass FP4 grouped GEMM) with the model's actual W4A16 INT4 quantization. These are different quantization schemes with different hardware execution paths. This distinction would become critical later in the session when the assistant would need to switch to an actual NVFP4 checkpoint to get MoE onto tensor cores.

Third assumption: that the existing environment can be reused. The message says "pull fresh sglang and port needed changes (if any)." This assumes that porting the model architecture to SGLang is a matter of minor changes—perhaps adding a model file or configuring GEMM kernels. In reality, supporting a new architecture like DeepSeek-V4 would require implementing the full model forward pass, including sparse attention, the indexer, compressor, and hash-based MoE routing. This is a multi-week engineering effort, not a quick port.

Fourth assumption: that disk space is available. The message says "download model" without checking disk capacity. The model is ~159 GB, but the root filesystem on CT200 is 98% full with only 22 GB free, consumed primarily by the 548 GB Kimi-K2.6 checkpoint. The assumption that there is room to download another large model would require deleting existing weights—a non-trivial decision.

Input Knowledge Required to Understand This Message

To fully grasp what this message is asking, one needs substantial context from the preceding session:

  1. The hardware topology: 8× RTX PRO 6000 Blackwell GPUs (sm_120, 96 GB each) across two NUMA nodes, connected via PCIe with no NVLink. The user's TP4-per-NUMA design exploits this topology.
  2. The current service state: A running SGLang K2.6 DDTree service consuming all 8 GPUs, with custom CUDA kernels injected via monkeypatch. Stopping it is non-trivial because it represents weeks of work.
  3. The PD disaggregation concept: Prefill-decode disaggregation separates the two inference phases onto different GPU groups, with KV cache transferred between them via a transfer engine (Mooncake or NIXL). On PCIe-only hardware, this transfer bandwidth is limited.
  4. The GEMM compilation requirement: Blackwell sm_120 GPUs need custom GEMM kernel configurations for FP8/FP4 matrix operations. The user's parenthetical "e.g. set up GEMMs for PRO 6000 if that's still needed" references earlier struggles where the assistant had to generate sm_120 FP8 autotune configs.
  5. The quantization landscape: Understanding the difference between W4A16 INT4, FP8 block quantization, and NVFP4—and which hardware paths each activates on Blackwell—is essential to evaluating the user's "nvfp4" focus.

Output Knowledge Created by This Message

This message generates a cascade of investigation and discovery:

  1. Model architecture verification: The assistant discovers that DeepSeek-V4-Flash uses the DeepseekV4ForCausalLM architecture with 284B total parameters, 13B active, 256 routed experts (top-6), sparse attention with a lightning indexer, and hash-based MoE routing. This is a fundamentally different architecture from the K2.6 that SGLang currently supports.
  2. SGLang compatibility assessment: Research reveals that SGLang has no production support for DeepSeek-V4. A GitHub issue (#23602) tracks it as a roadmap item, with comments showing users attempting to load the model and hitting unsupported architecture errors. The installed SGLang 0.5.11 has no deepseek_v4 model file.
  3. Disk space crisis: The CT200 root filesystem has only 22 GB free. Downloading the 159 GB model requires deleting the 548 GB Kimi-K2.6 checkpoint—a decision with significant implications for reproducibility and rollback.
  4. NUMA topology confirmation: The GPU topology confirms the user's intuition: GPUs 0–3 are on NUMA node 0 (CPU affinity 0-31), GPUs 4–7 on NUMA node 1 (CPU affinity 32-63). The TP4-per-NUMA design is optimal for PCIe communication.
  5. The NVFP4 vs W4A16 distinction: The assistant would later learn that the linked model uses INT4 weights, not NVFP4. A separate NVFP4 checkpoint exists and would need to be fetched to get MoE onto tensor cores.

The Thinking Process and Decision Architecture

The assistant's response to this message (msg [msg 12350]) reveals a methodical, research-first approach. Rather than immediately executing the instructions, the assistant pauses to gather facts:

"I'm setting up a new experiment around nvfp4 that involves deploying a quantized DeepSeek model with tensor parallelism on SGLang, then configuring prefill and decode disaggregation where each stage runs across 4 GPUs with isolated communication channels. I need to gather some key information before diving in."

The assistant identifies four parallel research tracks: the model card, the PD disaggregation documentation, the current CT200 state (GPU topology, disk space, SGLang version, running service), and the GEMM compilation requirements. This parallel research design is efficient—none of these investigations depend on each other, so they can proceed simultaneously.

The most critical discovery is the SGLang compatibility gap. The model card is "100% vLLM—forks + many patches, zero SGLang mention." The assistant's web search finds a SGLang GitHub issue where users report: "INFO model_config.py:448: Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']" followed by unsupported architecture errors. This is a potential showstopper.

The disk space discovery is equally consequential. The assistant finds that the root filesystem has only 22 GB free, with the Kimi-K2.6 checkpoint consuming 548 GB. The model requires 159 GB. The assistant must either find an alternative storage location, delete existing models, or report the constraint to the user.

Mistakes and Incorrect Assumptions

The user's message contains one significant incorrect assumption: that the model's quantization is NVFP4. The model name "W4A16-FP8-MTP" describes INT4 weights with FP8 attention—a different quantization scheme from NVIDIA's NVFP4 format. NVFP4 uses 4-bit floating point for both weights and activations, with a specific hardware path through Blackwell's tensor cores. The W4A16 model uses INT4 weights with FP16/BF16 activations, which may not trigger the same tensor-core paths. This distinction would become crucial later in the session when the assistant benchmarks the W4A16 model at only ~28 tok/s and discovers that MoE is still running on CUDA cores, not tensor cores. Only by switching to an actual NVFP4 checkpoint (and applying SGLang PR #25820 for auto-detection) would the MoE path move to tensor cores.

The user also underestimated the engineering effort required to port DeepSeek-V4 to SGLang. The message's casual "port needed changes (if any)" suggests an expectation that the model might work with minimal modifications. In reality, supporting a new architecture with sparse attention, hash-based MoE, and a lightning indexer requires implementing the entire model forward pass—a multi-week effort comparable to the K2.6 kernel work.

Conclusion

Message [msg 12349] is a pivotal moment in the session—a strategic pivot from deep kernel optimization on one architecture to deployment experimentation on another. It demonstrates the user's sophisticated understanding of hardware topology (NUMA-aware PD disaggregation) and their willingness to explore radically different approaches when diminishing returns set in. But it also reveals the gap between deployment ambition and infrastructure reality: the model's architecture may not be supported by SGLang, the disk has no space, and the quantization format is not what the user thinks it is. The assistant's methodical fact-gathering response—researching before executing—would prove essential to navigating these challenges, setting the stage for the optimization campaign that follows.