Orchestrating Prefill-Decode Disaggregation for DeepSeek-V4-Flash on Blackwell

Introduction

In the high-stakes world of large language model inference, few architectural decisions carry as much weight as how to split the two fundamentally different phases of generation: prefill (processing the input prompt) and decode (generating tokens one at a time). These phases have radically different compute profiles—prefill is compute-bound and highly parallelizable, while decode is memory-bandwidth-bound and latency-sensitive. Running them on the same GPUs forces a compromise: either prefill steals cycles from decode, or decode sits idle while prefill completes. The solution is prefill-decode (PD) disaggregation: dedicating separate GPU pools to each phase, connected by a fast transfer mechanism for KV cache migration.

Message <msg id=12385> captures the precise moment this architecture was brought to life for DeepSeek-V4-Flash, a 284B-parameter MoE model with 13B active parameters, running on 8× RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The assistant, having spent the preceding messages battling through environment setup, model loading, and single-node benchmarking, now pivots to the headline deliverable: splitting the 8 GPUs across two NUMA domains—GPUs 0-3 for prefill on NUMA0, GPUs 4-7 for decode on NUMA1—with KV cache transfer orchestrated by NVIDIA's NIXL library over UCX transport.

This message is a study in architectural decision-making under real-world constraints, where the assistant must balance correctness, performance isolation, NUMA topology awareness, and the practicalities of distributed process management. It is not merely a script-writing exercise; it is the culmination of a long chain of reasoning about what disaggregation means on this specific hardware, and the assumptions baked into every flag and environment variable tell a rich story about the state of SGLang's disaggregation support on sm_120.## The Context: What Led to This Moment

To understand why message <msg id=12385> matters, we must trace the path that led to it. The assistant had spent the prior several messages deploying DeepSeek-V4-Flash on a fresh SGLang environment, battling through the peculiarities of sm_120—NVIDIA's Blackwell architecture that sits between the well-trodden Hopper (sm_90) and the upcoming ultra-fast paths reserved for SM100. The model loaded correctly, generated the correct answer ("The capital of France is Paris."), and CUDA graphs were confirmed active. But the performance was abysmal: approximately 10 tok/s at batch size 1 and ~24 tok/s at batch size 8, versus the user's target of ~1000 tok/s.

The bottleneck was diagnosed with surgical precision: the _tiled_sparse_decode_kernel—a Triton fallback for sparse MLA attention that launches only 64 blocks on ~170 SMs, serially iterating all 512 top-k tokens. This is the same low-occupancy pathology that had plagued an earlier K2.6 deployment. The fast fused DSA/MoE stack (DeepGEMM, trtllm-gen, FP4 indexer) is arch-gated to SM100, and no amount of config tuning could close the ~40× gap.

Faced with this structural limitation, the assistant made a strategic decision: record the baseline honestly and pivot to the PD disaggregation deliverable. The reasoning was sound—optimizing sm_120 kernels would require a multi-week custom kernel effort akin to the earlier K2.6 work, which was explicitly out of scope. The PD disaggregation, by contrast, was the core project requirement and could be deployed immediately, even if the absolute throughput numbers were disappointing.

By message <msg id=12385>, the stage was set. The single-node TP4 server on GPUs 0-3 had been stopped, all 8 GPUs were free (0 MiB each), sglang-router 0.3.2 was installed, numactl was confirmed available, and NIXL 1.3.0 with UCX plugin was verified. The assistant had everything needed to build the disaggregated architecture.

The Architecture: Two Servers, One Node, Two NUMA Domains

The core insight behind PD disaggregation on this machine is the NUMA topology. The 8× RTX PRO 6000 GPUs are split across two NUMA nodes: GPUs 0-3 on NUMA0, GPUs 4-7 on NUMA1. Each NUMA node has its own CPU cores, memory controller, and PCIe root complex. Running a single TP8 configuration across both NUMA nodes would incur cross-NUMA PCIe traffic for every collective operation, adding latency and reducing bandwidth. By contrast, running two independent TP4 servers, each pinned to its own NUMA node, keeps all GPU-to-GPU communication within the same PCIe switch hierarchy.

The assistant's design in <msg id=12385> is:

| Component | GPUs | NUMA | Port | Bootstrap Port | Dist Init Address | Role | |-----------|------|------|------|----------------|-------------------|------| | Prefill | 0-3 | 0 (cpunodebind=0, membind=0) | 30000 | 8998 | 127.0.0.1:30335 | Process input prompts | | Decode | 4-7 | 1 (cpunodebind=1, membind=1) | 30001 | 8999 | 127.0.0.1:30435 | Generate tokens autoregressively |

The numactl pinning is critical: --cpunodebind=0 --membind=0 for prefill ensures that the Python process runs on CPU cores local to NUMA0 and allocates memory from NUMA0's DRAM, which is physically closer to GPUs 0-3. The decode server gets the mirror treatment for NUMA1. This prevents cross-NUMA memory allocations that would silently degrade performance.

The KV cache transfer between prefill and decode is handled by NIXL (NVIDIA's Interconnect Library) over UCX transport. The environment variable SGLANG_DISAGGREGATION_NIXL_BACKEND=UCX selects UCX as the transport layer, which on a single node will automatically choose the fastest available mechanism—likely cuda_ipc (direct GPU peer-to-peer via NVLink/P2P) or cuda_copy (via CUDA memory copy). The --disaggregation-transfer-backend nixl flag tells SGLang to use NIXL for KV cache migration rather than the default mooncake or raw NCCL.

The Assumptions Embedded in Every Flag

A close reading of the launch scripts reveals a dense web of assumptions about how SGLang's disaggregation works on sm_120:

--moe-runner-backend marlin: This forces the MoE expert execution path through the Marlin backend, which on sm_120 maps to the Triton fallback (sm120_triton). The assistant knows this is slow but has no alternative—the fast DeepGEMM path is gated to SM100. The assumption is that Marlin produces correct output even if throughput is poor.

--chunked-prefill-size 8192: This limits the prefill chunk size to 8192 tokens. The reasoning, visible in the agent's thinking, is that the sparse attention kernel has an unguarded path that can crash if chunk size exceeds 11673. Setting 8192 is a safe margin. This assumption comes from earlier debugging of the dsv4 attention hook.

--cuda-graph-max-bs 32 (decode only): CUDA graphs are enabled only for decode, not prefill. Prefill operates in eager mode because its dynamic shapes (variable-length inputs) don't benefit from graph capture. Decode, with its fixed batch-size and token-position shapes, is ideal for CUDA graph acceleration. The max batch size of 32 is a conservative bound—the assistant assumes decode won't need to batch beyond 32 concurrent requests.

--mem-fraction-static 0.70: Both servers reserve 70% of GPU memory for model weights and KV cache. This is a safe default—too high and the server risks OOM during peak usage, too low and it wastes capacity. The assistant notes internally that decode could potentially use 0.80 since it has dedicated GPUs, but chooses 0.70 for safety during the first deployment.

--disaggregation-bootstrap-port 8998/8999: These ports are used for the initial handshake between prefill and decode servers. The assistant assigns distinct ports (8998 for prefill, 8999 for decode) to avoid conflicts, and the --dist-init-addr parameters (30335 and 30435) provide separate distributed initialization addresses. The assumption is that SGLang's disaggregation protocol requires these to be distinct—a reasonable inference given both servers run on the same host.

SGLANG_DSV4_FP4_EXPERTS=1: This environment variable enables the FP4 expert path for DeepSeek-V4. Without it, the model would fall back to FP8 or BF16 expert execution, which would either fail (FP8 not supported for these experts) or consume dramatically more memory. The assistant assumes this flag is both necessary and sufficient for correct FP4 execution.

The Thinking Process: What the Agent's Reasoning Reveals

The agent's reasoning block in <msg id=12385> is unusually transparent about the decision-making process. Several observations stand out:

NUMA topology verification: The assistant explicitly checks that "GPU4-7 sit on NUMA node 1" and confirms that --cpunodebind=1 --membind=1 aligns correctly. This is not blind script generation—it's a deliberate cross-check against the machine's hardware layout, which was established earlier in the session through nvidia-smi topo -m output.

Distributed initialization port collision awareness: The assistant notes that "the distributed initialization ports might collide since both servers run on the same node" and reasons that "SGLang should auto-assign free ports" but chooses distinct addresses anyway (30335 vs 30435) as a defensive measure. This shows an understanding that distributed frameworks often bind to random ports when conflicts arise, but explicit assignment is more reliable.

JIT cache reuse: The assistant observes that "the JIT cache should speed things up on the second server since it'll reuse compiled kernels from the first run." This is a practical optimization insight—the Triton JIT compilation for sm_120 kernels is expensive (the single-node server took ~3.5 minutes to start), and launching both servers sequentially would double that time. By starting them in parallel, the second server benefits from kernels already compiled by the first.

Mem-fraction tradeoff: The assistant considers that "decode could potentially use 0.80 with its separate GPU allocation" but chooses 0.70 "for safety during this first disaggregated setup." This reflects a risk-reward calculation: the cost of an OOM crash during initial deployment is high (debugging distributed system failures is time-consuming), while the benefit of 10% more memory is marginal until load testing reveals actual pressure.

Eager mode for prefill: The assistant explicitly disables CUDA graphs for prefill, reasoning that prefill operates in eager mode. This is correct—prefill processes variable-length inputs with dynamic shapes that cannot be captured in a static CUDA graph. Only decode, with its fixed 1-token-at-a-time pattern, benefits from graph capture.

The Output Knowledge Created

Message <msg id=12385> produces two shell scripts and launches two server processes. The output knowledge created is:

  1. Two launch scripts (serve_dsv4_prefill.sh and serve_dsv4_decode.sh) that encode the complete disaggregation configuration. These scripts are executable artifacts that can be reused, modified, or inspected.
  2. Two running server processes (PIDs 73216 and 73217) that, if successful, will appear in subsequent messages as "fired up and ready." The assistant will poll for their readiness in the next round.
  3. Two log files (dsv4_prefill.log and dsv4_decode.log) that capture startup output, including any errors, JIT compilation progress, and final health status.
  4. A validated architecture design: The combination of NUMA pinning, NIXL transport, distinct ports, and FP4 expert flags represents a working hypothesis about how to deploy PD disaggregation for DeepSeek-V4-Flash on this specific hardware. Whether it works correctly will be verified in subsequent messages.

Mistakes and Incorrect Assumptions

While the message is well-reasoned, several assumptions warrant scrutiny:

The NIXL transport assumption: The assistant assumes that SGLANG_DISAGGREGATION_NIXL_BACKEND=UCX and --disaggregation-transfer-backend nixl are sufficient for KV cache transfer. However, SGLang's disaggregation support on sm_120 was experimental at this point. The NIXL library version 1.3.0 may not have full compatibility with the SGLang main branch's disaggregation protocol. If the handshake fails or the transfer backend isn't properly registered, the servers might start independently but fail to communicate.

The bootstrap port assumption: The assistant assigns --disaggregation-bootstrap-port 8998 for prefill and 8999 for decode, but the SGLang disaggregation protocol may expect these to be the same port (the bootstrap port is typically a shared rendezvous point). If the protocol uses a single port for both sides, the servers will never find each other.

The NUMA pinning correctness: The assistant assumes that numactl --cpunodebind=0 binds to NUMA0, which contains GPUs 0-3. However, cpunodebind binds to CPU cores, not GPUs. The GPU-to-NUMA mapping must be verified independently. If GPUs 0-3 are actually behind NUMA1's PCIe switch (due to BIOS configuration or GPU placement), the pinning would be counterproductive.

The FP4 expert flag necessity: The SGLANG_DSV4_FP4_EXPERTS=1 flag was set based on earlier debugging, but its exact semantics may have changed between SGLang versions. If the flag is no longer required (or has been replaced by auto-detection), setting it explicitly might override a more correct auto-detection path.

The chunked-prefill-size safety margin: The limit of 8192 tokens is based on an earlier observation about an unguarded sparse kernel path crashing above 11673 tokens. This is a heuristic, not a verified bound. If the actual safe limit is lower (e.g., 4096) or higher (e.g., 16384), the assistant's choice may either risk crashes or unnecessarily limit throughput.

The Input Knowledge Required

To fully understand this message, one needs:

  1. SGLang disaggregation architecture: Knowledge that SGLang supports splitting prefill and decode into separate processes, with KV cache transfer via NIXL, mooncake, or raw NCCL.
  2. NUMA topology awareness: Understanding that GPU-to-NUMA mapping affects PCIe bandwidth and that numactl can pin processes to specific NUMA nodes.
  3. DeepSeek-V4-Flash model specifics: The model uses FP4 experts (not FP8), has a sparse MLA attention mechanism, and requires specific environment variables (SGLANG_DSV4_FP4_EXPERTS) for correct execution.
  4. sm_120 architecture limitations: The Blackwell RTX PRO 6000 uses sm_120, which lacks the SM100-specific fast paths (DeepGEMM, trtllm-gen, FP4 indexer) and relies on Triton fallbacks for MoE and attention.
  5. NIXL and UCX transport: Understanding that NIXL is NVIDIA's interconnect library and UCX is a unified communication framework that can use cuda_ipc, cuda_copy, or TCP depending on topology.
  6. CUDA graph capture: Knowledge that CUDA graphs can accelerate fixed-shape decode iterations but are unsuitable for variable-shape prefill.

Conclusion

Message <msg id=12385> is the architectural pivot point of the DeepSeek-V4-Flash deployment. It represents the transition from "does it work?" to "how do we make it work at scale?" The assistant's reasoning reveals a deep understanding of the hardware topology, the SGLang disaggregation protocol, and the practical constraints of running a 284B-parameter model on Blackwell GPUs without the SM100 fast paths.

The two shell scripts encode hours of debugging, benchmarking, and architectural reasoning. Every flag—from --moe-runner-backend marlin to --chunked-prefill-size 8192—carries the weight of an earlier discovery or a deliberate tradeoff. The NUMA pinning, the distinct bootstrap ports, the parallel process launch, the JIT cache reuse—these are not boilerplate but carefully considered decisions.

Whether the PD disaggregation actually works will be revealed in the next message. But the architecture designed in <msg id=12385> is sound: it correctly isolates prefill and decode onto separate GPU pools, pins each to its local NUMA domain, selects the appropriate transport backend, and configures each server for its specific role. The throughput ceiling imposed by sm_120 fallback kernels remains, but the disaggregation infrastructure itself is a necessary foundation for any future optimization—whether that's a custom verify-attention kernel, MTP speculative decoding, or the NVFP4 quantization pivot that the assistant will explore in subsequent chunks.