The Final Cleanup: Systemd Service Conflicts and Production Hygiene in a PD-Disaggregated LLM Deployment

Introduction

In any complex production deployment, the most dangerous bugs are often not the ones that crash your system, but the ones that silently corrupt it. A stopped service with Restart=always configured in systemd is exactly such a trap: it looks dead, it behaves dead, but at any moment it can spring back to life and seize resources from the live deployment. This is the scenario that the assistant confronts in message [msg 12955], a seemingly mundane cleanup step that reveals deep understanding of systemd behavior, GPU topology management, and the fragility of disaggregated serving architectures.

The message is the capstone of a long debugging session spanning multiple segments (65–70 of the conversation), in which the assistant has been optimizing and debugging a DeepSeek-V4-Flash NVFP4 model deployed on 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang with prefill-decode (PD) disaggregation. After diagnosing a DSA sparse attention recall failure, reverting a failed index_topk=1024 experiment, and restoring the known-good 512 configuration, the assistant faces one final housekeeping task: ensuring that a previously crashed combined (single-server) service cannot interfere with the now-running PD deployment.

What makes this message remarkable is not the complexity of the bash commands, but the reasoning process behind them. The assistant demonstrates a nuanced understanding of systemd's restart policies, the difference between an explicit stop and an unexpected crash, the resource conflict surface between two serving topologies sharing the same GPUs and ports, and the importance of verifiable state. This article examines the message in depth: the reasoning that drove it, the decisions it encodes, the knowledge it assumes and produces, and the subtle mistakes it avoids.

The Message in Full

The subject message contains two parts: an Agent Reasoning block (the assistant's internal deliberation) and a tool call executing a bash script on the remote server. The reasoning section reads:

Everything's in a clean state — git has no uncommitted changes with our patches intact, the index_topk override is back to the default 512 across all serve scripts, and the PD services are running properly as the active topology. The combined service shows as failed, but that's expected since I stopped it and it's not the current setup, though I should verify it won't try to auto-restart.

>

The combined service has Restart=always configured, which means if it's in a failed state, systemd could attempt to restart it after 15 seconds — and that would be a serious problem since it would try to claim GPUs 0-3 and port 30001, directly conflicting with the PD prefill and router services. I need to explicitly stop the combined service and ensure it stays down. When I ran systemctl stop sglang-dsv4 before starting PD, that was a clean stop, so systemd won't auto-restart it (Restart=always only kicks in on unexpected exits, not explicit stops). The "failed" status is just a leftover from the earlier OOM crash, so the service should remain stopped as long as I don't manually start it again.

>

Let me verify that the combined service is truly inactive and not in a restart loop, and confirm that PD is stable with all GPUs allocated correctly.

The assistant then executes a bash command that: (1) explicitly stops the combined service, (2) resets its failed state in systemd, (3) verifies its new status as inactive / disabled, (4) checks that all 8 GPUs are allocated to PD workers with the expected memory usage (~83–84 GB each), and (5) runs a final router smoke test to confirm the PD deployment is serving requests correctly.

The Reasoning: Why This Message Was Written

To understand why this message exists, one must trace the chain of events that led to it. The assistant had been operating in a PD-disaggregated topology where one server handles prefill (processing incoming prompts and generating KV caches) and another handles decode (generating tokens autoregressively), with a router distributing requests. Earlier in the session, the assistant had also experimented with a combined single-server topology (both prefill and decode in one process) as a comparison point for diagnosing the DSA recall bug.

During those experiments, the combined server had crashed with an OOM (out-of-memory) error. The assistant had then stopped it (systemctl stop sglang-dsv4) and started the PD services. But systemd's Restart=always policy means that if a service exits unexpectedly (with a non-zero exit code or due to a signal), systemd will automatically restart it after a configurable delay, typically 15 seconds. An explicit systemctl stop, however, is treated differently: systemd records that the stop was intentional and will not auto-restart.

The critical insight in the assistant's reasoning is the distinction between a stopped service and a failed service. The systemctl is-active command returns failed for a service that crashed and hasn't been restarted yet. The assistant correctly identifies that this failed status is a leftover from the OOM crash, not from the explicit stop. But the assistant also recognizes a subtle danger: if any systemd event triggers a re-evaluation of the service's state—for example, if the service unit is reloaded or if the system administrator runs systemctl start on a dependent unit—systemd might see the failed state and decide to restart it.

This is not a hypothetical risk. In production environments, it is common to have monitoring scripts or deployment tools that automatically restart failed services. If such a tool saw sglang-dsv4 in a failed state, it might attempt to restart it, causing a collision with the PD deployment. The assistant's decision to run both systemctl stop and systemctl reset-failed is a defense-in-depth measure: stop ensures the service is not running, and reset-failed clears the failed state so that no external tool sees a reason to restart it.

Decisions Made in This Message

The message encodes several decisions, each with its own rationale:

Decision 1: Explicitly stop and reset the combined service. The assistant chooses to run systemctl stop sglang-dsv4 (even though the service is already stopped) and systemctl reset-failed sglang-dsv4. The stop is redundant but harmless; the reset-failed is the critical action that prevents future auto-restarts. This decision shows an understanding that systemd's state machine has memory: a service that crashed and was never explicitly reset is still considered "failed" and may be restarted by various triggers.

Decision 2: Verify the service is now inactive / disabled. The assistant checks both the active state (inactive) and the enabled state (disabled). The enabled state matters because a service that is enabled will start automatically on boot. If the combined service were still enabled, a server reboot would bring it back, potentially causing a conflict. By confirming it is disabled, the assistant ensures that only the PD services will start after a reboot.

Decision 3: Verify GPU allocation. The assistant runs nvidia-smi --query-gpu=index,memory.used to confirm that all 8 GPUs are in use by PD workers and that no GPU is idle or allocated to a zombie process from the combined service. The memory usage values (~83–84 GB out of what is presumably 96 GB per GPU) are consistent with the model's footprint, confirming that the PD deployment is healthy.

Decision 4: Run a final router smoke test. The assistant sends a chat completion request to the PD router and parses the response. The response shows content: '' and a truncated reasoning field, which is actually correct behavior: the model is generating a reasoning response that begins with "We need to reply with exactly 'HEALTHY'." This confirms that the router is accepting requests, the prefill server is processing them, the decode server is generating tokens, and the response is flowing back through the router.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message, most of which are well-founded but worth examining:

Assumption 1: The combined service will not auto-restart after an explicit stop. This is correct for systemd's default behavior. When systemctl stop is used, systemd sets the service's state to inactive (not failed), and Restart=always only triggers on unexpected exits. However, there is a subtle edge case: if the service unit file has RemainAfterExit=yes or if the stop command itself fails, the state could be different. The assistant's verification step (systemctl is-active) catches this.

Assumption 2: The combined service would conflict with PD on GPUs 0-3 and port 30001. This is based on the service configuration. The assistant assumes that the combined service was configured to use GPUs 0-3 (half of the 8-GPU cluster) and port 30001, while the PD prefill and router use overlapping resources. This is a reasonable assumption given that the PD deployment was set up to use all 8 GPUs, and the combined service was an earlier topology. The verification step confirms that no conflict is occurring.

Assumption 3: The "failed" status is harmless as long as it's not acted upon. The assistant correctly identifies that failed alone does not cause a restart; it is the combination of failed + Restart=always + a triggering event that causes the problem. By resetting the failed state, the assistant eliminates the trigger.

Potential mistake: Not disabling the service entirely. The assistant stops the service and resets its failed state, but does not run systemctl disable sglang-dsv4 to prevent it from starting on boot. However, the verification output shows the service is already disabled, so this was done in a previous step. The assistant's verification confirms this without re-disabling.

Potential mistake: Not checking for port conflicts. The assistant checks GPU memory but does not explicitly verify that port 30001 is not in use by a zombie process. However, the smoke test successfully reaches the PD router on port 30001, which implicitly confirms that no other process is holding that port.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Systemd service management: Understanding the difference between systemctl stop, systemctl start, systemctl restart, systemctl reset-failed, systemctl is-active, and systemctl is-enabled. Knowing that Restart=always triggers on unexpected exits but not on explicit stops. Understanding that a service in failed state can be auto-restarted by various triggers.

GPU topology management: Understanding that GPU indices (0-7) are a finite resource and that two serving processes cannot share the same GPU without explicit MPS or MIG configuration. Knowing that memory usage (~84 GB out of ~96 GB) is consistent with a loaded model.

PD-disaggregated serving: Understanding that prefill-decode disaggregation splits the serving pipeline into two separate server pools, with a router distributing requests. Knowing that the combined (single-server) topology is an alternative that uses the same resources differently.

SGLang and DeepSeek-V4-Flash: Knowing that this is a large language model deployment using the SGLang inference engine, with NVFP4 quantization and a specific attention mechanism (DSA sparse attention).

The debugging history: Understanding that the assistant has been investigating a context-recall bug, tested index_topk=1024 as a fix, found it ineffective, and reverted to the default 512. Knowing that the combined service was used as a comparison point and crashed with OOM.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. Confirmation that the PD deployment is clean and stable. The smoke test, GPU memory check, and service status verification together provide strong evidence that the PD topology is healthy and serving requests correctly.

2. Confirmation that the combined service is safely disabled. The inactive / disabled status means it will not interfere with the PD deployment, even after a reboot.

3. A reproducible cleanup procedure. The sequence of commands (systemctl stop, systemctl reset-failed, nvidia-smi, smoke test) forms a reusable pattern for cleaning up after topology changes in a PD-disaggregated deployment.

4. Documentation of the resource allocation. The GPU memory usage values serve as a baseline for future monitoring. If memory usage changes significantly, it could indicate a problem.

5. A working smoke test endpoint. The successful router request confirms that the PD pipeline (router → prefill → decode → router) is functional.

The Thinking Process

The assistant's reasoning in this message is a model of systematic thinking under uncertainty. Let me trace the logic step by step:

Step 1: State assessment. The assistant begins by cataloging what is known: git is clean, index_topk is reverted to 512, PD services are active. The combined service shows as failed.

Step 2: Risk identification. The assistant immediately identifies the risk: Restart=always combined with failed state could lead to an auto-restart. This is not a theoretical concern—it is a specific, actionable risk with a clear failure mode (GPU and port conflict).

Step 3: Root cause analysis. The assistant traces the failed state to its origin: the earlier OOM crash. It correctly distinguishes between the crash (which caused the failed state) and the subsequent systemctl stop (which should have prevented auto-restart).

Step 4: Action planning. The assistant decides to explicitly stop the service and reset its failed state. This is a conservative choice: even though the service should already be safe (because it was explicitly stopped), the assistant adds an extra layer of safety by clearing the failed state.

Step 5: Verification planning. The assistant plans three verification steps: check service status, check GPU allocation, and run a smoke test. Each verification addresses a different failure mode: service restart (status), GPU conflict (nvidia-smi), and serving functionality (smoke test).

Step 6: Execution and confirmation. The assistant executes the plan and interprets the results. The service is now inactive / disabled. All 8 GPUs show ~83-84 GB usage, consistent with the model. The smoke test returns a response (even though it's truncated in the display), confirming the pipeline is working.

What is notable about this reasoning is what it does not do. The assistant does not assume that the service is safe just because it was stopped. It does not assume that the failed state is harmless. It does not skip verification because the deployment seems healthy. Instead, it treats every assumption as a hypothesis to be tested, and every verification as evidence to be collected.

Technical Depth: Systemd Restart Semantics

The core technical insight in this message is the distinction between different types of service termination in systemd. When a service exits, systemd records the exit code and the exit type (main, cgroup, or signal). If Restart=always is set, systemd will restart the service after the RestartSec delay (default 15 seconds) unless the stop was initiated by an administrator via systemctl stop.

The mechanism behind this is the service's ActiveState and Result properties. When systemctl stop is used, systemd sets ActiveState=inactive and Result=success (or exit-code if the stop failed). When a service crashes, systemd sets ActiveState=failed and Result=exit-code or Result=signal. The Restart=always policy only triggers auto-restart when the service enters the failed state unexpectedly.

However, there is a subtlety: if a service is in the failed state and the systemd daemon is reloaded (via systemctl daemon-reload), systemd may re-evaluate the service's state and decide to restart it. Similarly, if a dependency of the service is started, systemd might attempt to start the failed service as part of dependency resolution. The reset-failed command clears the failed state, preventing these scenarios.

This is why the assistant's decision to run systemctl reset-failed is not redundant. It is a defense against edge cases that could otherwise cause a production outage.

Broader Significance

While this message appears to be a simple cleanup step, it illustrates several principles of reliable production deployment:

1. Defense in depth. The assistant does not rely on a single mechanism to prevent the combined service from restarting. It uses systemctl stop (immediate), systemctl reset-failed (state cleanup), and verification (detection). Each layer provides protection if the previous layer fails.

2. Verify everything. The assistant verifies the service status, the GPU allocation, and the serving functionality. Each verification tests a different aspect of the system and provides different failure coverage.

3. Understand your tools. The assistant's understanding of systemd's restart semantics is what allows it to identify the risk in the first place. A less experienced operator might see failed and think "that service is down, no problem," missing the auto-restart risk entirely.

4. Clean state is a virtue. The assistant's insistence on a clean git state, clean service state, and clean GPU allocation reflects a philosophy that production systems should be in a known, reproducible state at all times. Ambiguous states (like failed without reset-failed) are liabilities.

5. Document through action. The smoke test at the end of the message serves as documentation that the deployment is working. If someone later asks "was the deployment healthy at this point?", the smoke test output provides an answer.

Conclusion

Message [msg 12955] is a masterclass in production hygiene. On its surface, it is a simple cleanup: stop a crashed service, verify the deployment is healthy. But beneath the surface, it reveals a deep understanding of systemd's state machine, the resource conflict surface of GPU deployments, and the importance of verifiable state in production systems.

The assistant's reasoning demonstrates that the most important skill in operating complex systems is not knowing how to run commands, but knowing which commands to run and why. The decision to reset the failed state, the verification of GPU allocation, and the smoke test are not random actions—they are targeted responses to specific risks identified through careful analysis of the system's behavior.

For anyone operating a PD-disaggregated LLM deployment, this message offers a template for safe topology changes. The pattern of stop → reset-failed → verify-GPUs → smoke-test is reusable across many scenarios, and the reasoning behind it applies to any system where multiple services compete for shared resources.

In the end, the most valuable output of this message is not the bash output or the smoke test result—it is the demonstration that careful, systematic reasoning can prevent production outages before they happen.