The Systemd Transition: From Validated Parsers to Production-Grade Disaggregated Serving

Introduction

In the long arc of deploying DeepSeek-V4-Flash on 8× RTX PRO 6000 Blackwell GPUs, message [msg 12717] represents a critical inflection point: the moment when a validated but fragile single-server setup was recast into a robust, production-grade distributed architecture managed by systemd. This single message, consisting of a concise bash command that writes three systemd unit files and reloads the daemon, is deceptively brief. Behind it lies a dense web of architectural reasoning, validated assumptions, and careful trade-off analysis that spans the preceding dozen messages. Understanding this message requires tracing the thread of parser validation, disaggregation architecture, and operational reliability concerns that converged to produce these three unit files.

The Context: Why This Message Was Written

To grasp the motivation behind [msg 12717], one must understand the journey that led to it. The assistant had spent the previous messages ([msg 12711] through [msg 12716]) diagnosing and fixing a critical quality-of-service problem: the DeepSeek-V4-Flash server was running without a proper chat template, defaulting to a naive string-concatenation format that rendered thinking/reasoning output and tool-calling functionality completely broken. The server logs confirmed this with the telltale message: "No chat template found, defaulting to 'string' content format."

The assistant methodically investigated the tokenizer, discovering that the DeepSeek-V4 tokenizer contained all the necessary special tokens—<think> (ID 128821), </think> (ID 128822), and the full suite of DeepSeek tool-call markers. It identified the correct configuration flags: --chat-template tool_chat_template_deepseekv32.jinja (since the V4 detector extends the V32 detector), --reasoning-parser deepseek-v4, and --tool-call-parser deepseekv4. It validated these on a single-server setup, demonstrating that thinking output was cleanly separated into reasoning_content and content fields, and that parallel tool calls (e.g., get_weather for Paris and Tokyo) were correctly parsed.

But the user had previously expressed surprise that the prefill-decode (PD) disaggregation had been dropped. The single-server setup, while functionally correct for parsing, was a regression from the distributed architecture that had been built earlier. The user wanted PD restored, with the router exposed on port 30001. This created a tension: the validated single-server was occupying port 30001, but the PD architecture needed that port for the router. The assistant had to dismantle the working single-server setup and reconstruct the three-service PD architecture—prefill, decode, and router—while preserving the parser configuration that had just been validated.

This is the immediate motivation for [msg 12717]. The assistant had already written the three server scripts (serve_dsv4_prefill.sh, serve_dsv4_decode.sh, serve_dsv4_router.sh) in the preceding message. Now it needed to wrap those scripts in systemd units to create a reliable, boot-enabled, self-healing deployment.

The Architecture: How Decisions Were Made

The systemd unit files encode several architectural decisions that deserve close examination.

Service Separation and NUMA Binding. The prefill server is pinned to NUMA node 0 (GPU0–3) via numactl --cpunodebind=0 --membind=0, while the decode server is pinned to NUMA node 1 (GPU4–7). This reflects a deep understanding of the hardware topology: the 8 GPUs span two NUMA domains, and binding each server to its local NUMA node minimizes cross-socket memory access latency. The prefill server uses --mem-fraction-static 0.80, while decode uses 0.85—a deliberate asymmetry reflecting that decode must hold the full KV cache for active sequences, while prefill can be slightly more memory-conservative.

Port Allocation and Internal vs. External Interfaces. The prefill server binds to 127.0.0.1:30000 (localhost only, internal), the decode server to 127.0.0.1:30002 (also internal), and the router to 0.0.0.0:30001 (external, all interfaces). This creates a clean security boundary: the model servers are not directly exposed to the network; all client traffic flows through the router, which handles request routing between prefill and decode. The router also exposes a Prometheus metrics endpoint on port 29001, enabling the monitoring stack that would be built later.

Parser Flag Placement. A critical architectural question was whether the --chat-template, --reasoning-parser, and --tool-call-parser flags should go on the individual servers, on the router, or both. The assistant's reasoning (visible in [msg 12716]) shows careful consideration: the safest approach is to put them on both prefill and decode servers, since that's where the OpenAI-compatible processing happens, and the router can simply proxy requests. The response flows from decode back through the router to the client, so parsing on the servers ensures the output is already correctly formatted. The router script (serve_dsv4_router.sh) contains no parser flags—it is a pure proxy.

Systemd Dependency Ordering. The router unit declares After=network-online.target sglang-dsv4-prefill.service sglang-dsv4-decode.service and Wants=sglang-dsv4-prefill.service sglang-dsv4-decode.service. This means systemd will attempt to start the prefill and decode servers before the router, and if those services fail to start, the router will not be started. However, After= does not guarantee that the services are ready to accept connections—only that they have been started. This is a known limitation of systemd dependency modeling for network services, and the assistant implicitly acknowledges this by setting TimeoutStartSec=900 for the servers (ample time for model loading) and RestartSec=15 for automatic retries.

Restart Policy and Resilience. All three services use Restart=always with RestartSec=15, meaning they will be automatically restarted if they crash, with a 15-second delay between attempts. The KillMode=control-group on the server units ensures that if the main process dies, all child processes are cleaned up. The LimitNOFILE=1048576 raises the file descriptor limit to over a million, anticipating high connection volumes through the router.

Assumptions Embedded in the Design

Every architectural decision carries assumptions, and this message is rich with them.

Assumption 1: Parser validation on single-server transfers to PD mode. The assistant validated that the chat template and parsers work correctly on a single server handling both prefill and decode. It is now assuming that the same flags will work correctly when split across two servers communicating via NIXL/UCX. This is a reasonable assumption—the parsing logic operates at the token/text level, independent of the disaggregation mechanism—but it is untested at this point. The assistant plans to validate this after deployment.

Assumption 2: The router does not need parser awareness. The router is configured as a pure proxy, forwarding requests to prefill and responses from decode. This assumes that the OpenAI-compatible API semantics (including streaming, reasoning content, and tool calls) are preserved across the disaggregation boundary without router-side intervention. If the prefill and decode servers each apply the chat template independently, the router should see correctly formatted requests and responses. However, if the disaggregation protocol modifies the request/response format in ways that affect parsing, this assumption could break.

Assumption 3: Systemd dependency ordering is sufficient. The After= directive ensures that the prefill and decode services are started before the router, but it does not ensure they are ready. Model loading can take 5–15 minutes on these large models. The router could start, attempt to connect to the prefill or decode servers, and fail because they are still loading. The Restart=always policy on the router mitigates this—it will crash, wait 15 seconds, and retry—but this creates a window of unavailability after boot.

Assumption 4: The disaggregation bootstrap will succeed. The prefill and decode servers use --dist-init-addr with different ports (30335 and 30435) and different bootstrap ports (8998 and 8999). This assumes that the two servers can discover each other and establish the NIXL/UCX transfer channel. If the bootstrap fails (e.g., due to firewall rules, address conflicts, or version mismatches in the NIXL backend), both servers will start but disaggregation will not function.

Input Knowledge Required

To fully understand [msg 12717], a reader needs knowledge spanning several domains:

Output Knowledge Created

This message produces three systemd unit files that collectively define the production serving infrastructure:

  1. sglang-dsv4-prefill.service: Manages the prefill server on GPU0–3 (NUMA0), internal port 30000, with parser flags, NIXL/UCX transfer backend, and 512K context length.
  2. sglang-dsv4-decode.service: Manages the decode server on GPU4–7 (NUMA1), internal port 30002, with the same parser flags, CUDA graph support (--cuda-graph-max-bs 32), and higher memory fraction (0.85).
  3. sglang-dsv4-router.service: Manages the router on external port 30001, with Prometheus metrics on port 29001, ordered after both servers. These units transform the deployment from a manually managed, single-process setup into a robust, automatically recovering service infrastructure. They enable the server to survive crashes, reboot gracefully, and start automatically on system boot. This is the foundation upon which the Prometheus/Grafana monitoring stack (built in the subsequent chunk, [msg 12617]–[msg 12668]) and the final quality-of-service fixes (chat template, thinking defaults, model name) are layered.

The Thinking Process: A Window into Engineering Judgment

The assistant's reasoning, visible in the "Agent Reasoning" blocks of preceding messages, reveals a methodical, risk-aware engineering mindset. The key thought process leading to [msg 12717] can be reconstructed as follows:

Step 1: Diagnose the parsing failure. The assistant noticed that the server logs showed "No chat template found." It didn't just add a chat template blindly; it first verified that the DeepSeek-V4 tokenizer contained the expected special tokens, then confirmed that the V32 tool template was the correct base (since the V4 detector extends V32).

Step 2: Validate on single server first. Rather than deploying the parsers directly into the complex PD architecture, the assistant deliberately chose to validate on the simpler single-server setup. This is a textbook debugging strategy: isolate the variable, test in the simplest possible environment, then propagate the validated configuration to the complex system.

Step 3: Consider the architecture question. The assistant explicitly weighed whether parsers belong on the servers or the router. The reasoning shows an understanding of the data flow: requests come in through the router, get forwarded to prefill (which applies the chat template to format the prompt), then the KV cache is transferred to decode (which generates tokens and applies the reasoning/tool parsers to the output). The router is a thin proxy; the parsers belong on the servers where the actual tokenization and generation happen.

Step 4: Plan for the transition. The assistant recognized that the single-server was occupying port 30001, which the router needs. This required stopping the single-server unit, starting the prefill and decode servers (which take 5–15 minutes to load the model), and only then starting the router. The systemd dependency ordering encodes this sequence.

Step 5: Build for resilience. The assistant chose Restart=always with 15-second delays, anticipating that the services might crash during initial loading or under memory pressure. The long TimeoutStartSec=900 (15 minutes) accommodates the model loading time. The LimitNOFILE=1048576 prevents file descriptor exhaustion under load.

What is notably absent from the reasoning is any consideration of readiness checks. The assistant does not add ExecStartPost= scripts to verify that the servers are accepting connections before declaring the units "active." This is a pragmatic trade-off: proper readiness checks would require custom health-check scripts and would complicate the systemd configuration. The Restart=always policy is a simpler, if less elegant, solution.

Conclusion

Message [msg 12717] is a masterclass in operational engineering: it takes a validated but fragile configuration and wraps it in a resilient, production-grade deployment framework. The three systemd unit files encode dozens of architectural decisions about NUMA binding, port allocation, dependency ordering, restart policies, and parser placement. Each decision is grounded in the preceding analysis—the tokenizer validation, the parser testing, the hardware topology discovery, and the understanding of SGLang's disaggregation architecture.

The message is also a testament to the value of incremental validation. The assistant did not attempt to build the PD deployment and test the parsers simultaneously. Instead, it validated the parsers on the simplest possible setup, then carried that validated configuration forward into the complex distributed architecture. This pattern—isolate, validate, propagate—is visible throughout the session and is a key reason for its success.

In the broader narrative of the DeepSeek-V4-Flash deployment, [msg 12717] is the moment when the system transitions from "works on my machine" to "runs in production." The systemd units are the infrastructure that enables the subsequent monitoring setup, the quality-of-service fixes, and the comprehensive engineering report. Without this message, the deployment would remain a collection of manual commands and fragile scripts. With it, it becomes a service.