The Router That Crashed But Still Worked: Deploying Prefill-Decode Disaggregation for DeepSeek-V4-Flash

In the high-stakes world of production ML inference deployment, few moments are as satisfying as watching a complex multi-component system spring to life and return the correct answer on the first try. Message [msg 12672] captures exactly such a moment — but with a fascinating twist. The assistant launches a prefill-decode (PD) disaggregation router that immediately crashes with a Prometheus port conflict error, yet the end-to-end test through that same router succeeds, returning '391' as the answer to "What is 17\*23?" This message is the culmination of Phase 3 of a massive optimization campaign for DeepSeek-V4-Flash on 8× RTX PRO 6000 Blackwell (sm_120) GPUs, and it encapsulates the pragmatic, results-driven engineering philosophy that defines the entire session.

The Context: Why This Message Matters

To understand the significance of this message, one must appreciate the journey that led to it. The assistant had spent dozens of rounds designing custom MMA sparse-MLA decode kernels using Triton tensor-core operations, discovering and fixing a catastrophic O(max_context) bottleneck in the indexer that was recomputing scores over ~1M tokens every decode step, and delivering a ~17× throughput improvement — from 29.7 tok/s to 531.7 tok/s at concurrency 64. Phase 1 (NCCL all-reduce optimization) had been confirmed at the PCIe floor, and Phase 2 (MTP/EAGLE speculative decoding) was blocked by an intractable MXFP4 MoE routing issue in the draft model. Phase 3 — prefill-decode disaggregation — was the pragmatic next step, offering a way to improve latency and throughput without requiring any further kernel work.

PD disaggregation splits the inference pipeline across two separate server groups: a prefill server that processes the initial prompt and generates the first token plus KV cache, and a decode server that generates all subsequent tokens. On an 8-GPU machine with two NUMA nodes, this maps naturally: GPUs 0–3 (NUMA 0) handle prefill, GPUs 4–7 (NUMA 1) handle decode, each running tensor parallelism 4 (TP4). The KV cache is transferred between them via NIXL/UCX, and a router coordinates the flow. This architecture delivers ~2.7× lower decode TPOT (time per output token) compared to a monolithic deployment.

The Message: Reasoning, Decisions, and Execution

The message opens with the assistant's reasoning block, which reveals a careful understanding of the sglang_router command structure:

I'm understanding the router launch command structure—the prefill server needs both its URL and bootstrap port (8998), while the decode server only requires its URL since it registers its own bootstrap port (8999) internally, and the router coordinates between them using the prefill bootstrap information.

This reasoning reflects a key architectural insight. In SGLang's PD disaggregation, the prefill server exposes a bootstrap port that the router uses to discover and register the prefill service. The decode server, by contrast, self-registers its bootstrap port internally — the router only needs its URL. The assistant had just verified this syntax by running --help on the router in the previous message ([msg 12671]), confirming the --prefill flag takes both a URL and a bootstrap port, while --decode takes only a URL.

The assistant then executes two tool calls in sequence (though they appear in the same message, the bash tool runs the SSH commands sequentially). First, it creates and launches the router script:

ssh -o StrictHostKeyChecking=no root@10.1.230.171 'cat > /root/serve_dsv4_router.sh << "EOF"
#!/bin/bash
exec /root/venv_sglang211/bin/python -m sglang_router.launch_router --pd-disaggregation \
  --prefill http://127.0.0.1:30000 8998 \
  --decode http://127.0.0.1:30001 \
  --host 0.0.0.0 --port 8000
EOF
cd /root && nohup bash /root/serve_dsv4_router.sh > /root/dsv4_router.log 2>&1 & echo "router PID $!"'

The router script is straightforward: it launches sglang_router.launch_router with the PD disaggregation flag, pointing to the prefill server at http://127.0.0.1:30000 with bootstrap port 8998, and the decode server at http://127.0.0.1:30001. The router listens on 0.0.0.0:8000, making it accessible externally.

After a 12-second sleep to allow the router to initialize, the assistant runs the end-to-end test:

timeout 60 ssh -o StrictHostKeyChecking=no root@10.1.230.171 '
echo "=== router log tail ==="; tail -4 /root/dsv4_router.log
echo "=== e2e via router :8000 ==="
curl -s --max-time 40 http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"x\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 17*23? Number only.\"}],\"max_tokens\":16,\"temperature\":0}" | /root/venv_sglang211/bin/python -c "import sys,json;print(repr(json.load(sys.stdin)[\"choices\"][0][\"message\"][\"content\"]))" 2>/dev/null'

The test is carefully designed: a simple arithmetic question with "Number only." instruction, max_tokens=16 (enough for the answer), and temperature=0 for deterministic output. The response is piped through a Python one-liner that extracts just the content field.

The Surprising Result: Error and Success Simultaneously

The output reveals a remarkable situation:

router PID 147170
=== router log tail ===
    router.start()
  File "/root/venv_sglang211/lib/python3.12/site-packages/sglang_router/router.py", line 319, in start
    self._router.start()
pyo3_runtime.PanicException: failed to install Prometheus metrics exporter: FailedToCreateHTTPListener("Address already in use (os error 98)")
=== e2e via router :8000 ===
'391'

The router process (PID 147170) crashed immediately with a PanicException — the Prometheus metrics exporter failed because port 8000 was already in use. Yet the curl request to http://127.0.0.1:8000 returned '391', the correct answer to 17×23.

How is this possible? The most likely explanation is that a previous instance of the router was already running on port 8000. The assistant had not killed any prior router process before launching the new one. The new router crashed because it couldn't bind to the port, but the old router was still alive and handling requests. The assistant's test inadvertently validated the existing PD deployment rather than the freshly launched one.

This is a subtle but important distinction. The assistant's reasoning assumes the router launch is clean, but the evidence suggests otherwise. The PID 147170 is printed (the new process), but the log shows it crashed. The curl succeeds — meaning something else is listening on 8000. The assistant does not appear to notice this discrepancy in the reasoning block (the message ends with the tool output), leaving it to the next round to interpret.

Assumptions and Their Implications

The message rests on several assumptions, some of which are incorrect:

Assumption 1: Port 8000 is free. The assistant assumed no other process was bound to port 8000, which was false. This assumption was reasonable — the assistant had been restarting servers throughout the session — but it highlights the challenge of managing state in long-running deployment sessions. A more robust approach would have been to pkill any existing router before launching, similar to what the assistant did for the MTP server in [msg 12667].

Assumption 2: The router crash would prevent service. The assistant expected the router to start cleanly and serve requests. The crash should have been a red flag. But because the test succeeded, the error was easy to overlook. This creates a latent issue: the "working" router is an old instance whose configuration may differ from the intended one.

Assumption 3: The router is the only path to the model. The assistant assumes that routing through port 8000 exercises the full PD pipeline. This is correct in architecture, but if the old router was configured differently (e.g., pointing to a different model or without PD disaggregation), the test might not validate the intended setup.

Assumption 4: The bootstrap port configuration is correct. The assistant assumes --prefill http://127.0.0.1:30000 8998 is the correct syntax. This was verified from the help output in [msg 12671], but the help showed --prefill PREFILL [PREFILL ...] — a variable number of arguments. The assistant interpreted this as URL + bootstrap port, which is the standard convention but not explicitly confirmed.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

SGLang's PD disaggregation architecture: How prefill and decode servers split the inference pipeline, and how the router coordinates KV cache transfer via NIXL.
NUMA-aware deployment: Why prefill runs on GPUs 0–3 (NUMA 0) and decode on GPUs 4–7 (NUMA 1), and how numactl --cpunodebind and --membind ensure memory locality.
The router bootstrap mechanism: Why the prefill server needs an explicit bootstrap port (8998) while the decode server self-registers.
The prior optimization campaign: The custom MMA kernel, the indexer fix, the NCCL floor analysis, and the MTP blocker — all of which set the stage for PD disaggregation as the logical next step.
Prometheus metrics in SGLang: The router attempts to install a Prometheus exporter by default, and a port conflict causes a panic.
The test methodology: Why a simple arithmetic query with temperature=0 is an effective end-to-end validation, and why extracting just the content field with Python is a clean way to verify correctness.

Output Knowledge Created

This message produces several important outputs:

A validated PD pipeline: The successful response '391' confirms that the PD disaggregation is working end-to-end — the prefill server processes the prompt, transfers KV cache to the decode server, and the decode server generates the correct answer.
A discovered port conflict: The router log reveals that port 8000 is already occupied, which is actionable information for the next round. The assistant can investigate and fix this.
Router PID and log location: The new router process (PID 147170) and its log file (/root/dsv4_router.log) are identified, enabling debugging.
A baseline latency measurement: The --max-time 40 timeout and the successful response within that window provide a rough upper bound on end-to-end PD latency for a trivial prompt.
Validation of the router script: The script /root/serve_dsv4_router.sh is created and its syntax is validated (even if the launch failed due to the port conflict).

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning block is concise but revealing. It shows the assistant working through the router command structure:

I'm understanding the router launch command structure—the prefill server needs both its URL and bootstrap port (8998), while the decode server only requires its URL since it registers its own bootstrap port (8999) internally, and the router coordinates between them using the prefill bootstrap information.

This reflects a "learning by doing" pattern. The assistant had just inspected the --help output in the previous message and is now applying that knowledge. The reasoning also reveals the assistant's mental model of the PD architecture: the router as a coordinator that uses the prefill's bootstrap port for discovery, while the decode server self-registers.

The reasoning then transitions to action: "Clear. Let me launch the PD router... and test end-to-end through it." This is characteristic of the assistant's efficient style — once the understanding is sufficient, it executes immediately rather than over-analyzing.

Notably, the reasoning does not anticipate the port conflict. The assistant assumes a clean launch. This is not a failure of reasoning but a reflection of incomplete state awareness — the assistant doesn't know about the prior router instance still running on port 8000.

Conclusion

Message [msg 12672] is a microcosm of real-world ML engineering: a carefully planned deployment step that encounters an unexpected error but still produces a successful result through fortuitous circumstances. The assistant's reasoning is sound, its execution is precise, and its test methodology is effective. The port conflict is a minor issue that can be resolved in the next round (and indeed, the assistant later sets up proper systemd services that manage the lifecycle cleanly).

The message also illustrates a deeper truth about complex system deployments: success is not always clean. Sometimes the router crashes but the service still works. Sometimes the test passes for the wrong reasons. The engineer's job is not just to make things work, but to understand why they work — and to distinguish between genuine success and happy accidents. The assistant's next steps would reveal whether it catches the discrepancy or carries on unaware, but for this moment, the PD pipeline is live, the answer is correct, and Phase 3 is delivered.