Breaking the Restart Loop: A Moment of Debugging Clarity in High-Stakes ML Deployment

In the midst of an intense benchmarking session, a single message from the assistant stands out as a moment of diagnostic clarity. The message reads:

It restarted again (PID changed). The 60s timeout + crash is a restart loop. The zombie processes from the previous instance are holding the GPU memory, causing the next instance to fail. Let me clean up properly: [bash] ssh root@10.1.230.174 'systemctl stop sglang-qwen.service' 2>&1

This is message 6537 in a long conversation spanning dozens of segments and thousands of messages. On the surface, it appears to be a simple observation followed by a routine systemctl command. But beneath that surface lies a rich story of inference serving, systemd lifecycle management, GPU memory contention, and the subtle ways that rapid experimentation can create pathological states in production infrastructure.

The Context: Chasing Speculation Depth

To understand why this message was written, we must understand what preceded it. The assistant had been systematically benchmarking the Qwen3.5-122B-A10B model deployed with SGLang on a 4-GPU system (NVIDIA RTX PRO 6000 Blackwell GPUs). The key variable under investigation was speculative_num_steps — the number of multi-token prediction (MTP) steps used by the EAGLE-3 speculative decoding engine.

The user had just asked, "Try 10 steps to see if we unplateu" ([msg 6522]), pushing the speculation depth far beyond the previous maximum of 5. The assistant dutifully edited the systemd service file to set --speculative-num-steps 10, deployed it to the remote server at 10.1.230.174, and waited for the model to load (<msg id=6523-6526>). But something went wrong. The user reported "Load crashed, resume your testing" ([msg 6527]), and the assistant began investigating.

What followed was a classic debugging chain: checking systemd status, examining journalctl logs, looking for OOM kills or CUDA errors. The assistant initially suspected an OOM kill due to the excessive KV cache overhead of 11 draft tokens per request ([msg 6532]). But closer inspection of the logs revealed a more nuanced story — the server had actually run successfully with steps=10, achieving 270-293 tok/s throughput with an accept rate of 0.92 ([msg 6530]). The "crash" was actually the assistant's own systemctl stop command from a previous iteration, which had sent SIGTERM to the server, and when the graceful shutdown exceeded systemd's 60-second TimeoutStopSec, systemd sent SIGKILL (<msg id=6533-6534>).

The Diagnosis: A Restart Loop

By message 6537, the assistant had been watching the service cycle through multiple PIDs. Each time it checked systemctl status, a new PID had appeared — the service was restarting, failing, and restarting again. The assistant's diagnosis was precise: "The 60s timeout + crash is a restart loop. The zombie processes from the previous instance are holding the GPU memory, causing the next instance to fail."

This diagnosis reveals several layers of understanding about the system's behavior:

First, the restart loop mechanism. The systemd service file had Restart=on-failure configured, meaning any non-zero exit code would trigger an automatic restart. When the server failed to start because GPU memory was still occupied, systemd dutifully tried again. And again. Each attempt consumed a few seconds of initialization before failing, producing the rapid PID cycling the assistant observed.

Second, the GPU memory persistence problem. When a Python process holding CUDA contexts is killed (especially with SIGKILL, which cannot be caught), the GPU memory it allocated may not be immediately released. NVIDIA's driver maintains per-process state, and if the process is terminated uncleanly, the CUDA driver can take time to detect the process death and release its resources. In some cases, zombie GPU processes — processes that have been killed but whose CUDA contexts linger — can block new processes from allocating memory. The assistant correctly identified that "zombie processes from the previous instance are holding the GPU memory."

Third, the root cause chain. The restart loop was ultimately caused by the assistant's own rapid iteration cycle. Each time the assistant wanted to change speculation parameters, it ran systemctl stop (which sent SIGTERM), waited briefly, then deployed a new service file and ran systemctl start. But the graceful shutdown was taking longer than expected (due to pending benchmark requests), so systemd's timeout killed the process with SIGKILL. The unclean shutdown left GPU memory in an inconsistent state, causing the next instance to fail on startup. systemd's restart policy then created the loop.

The Action: Breaking the Cycle

The assistant's response was simple but decisive: systemctl stop sglang-qwen.service. This single command would stop the service cleanly and, crucially, prevent systemd from automatically restarting it (since systemctl stop is a manual stop, not a failure-triggered restart). Once the service was fully stopped, the assistant could then clean up any lingering GPU processes with fuser -k /dev/nvidia* before starting fresh.

This moment represents a critical skill in infrastructure management: recognizing when your own tooling has created a pathological state, and knowing how to break out of it. The assistant had been iterating rapidly — stop, edit, scp, start, wait, benchmark — and this rapid cadence had accumulated side effects that manifested as a restart loop. The message is an acknowledgment that the current approach isn't working and a cleaner intervention is needed.

Assumptions and Knowledge

The message makes several implicit assumptions. It assumes that zombie GPU processes are the root cause of the restart failures, which is a reasonable inference given the symptom pattern (new PID every few seconds, service failing during initialization). It assumes that systemctl stop will break the restart loop by preventing auto-restart. It assumes that once the service is stopped, the GPU memory will eventually be released (either by the kernel cleaning up zombie contexts or by a subsequent fuser -k command).

The input knowledge required to understand this message is substantial. One must understand: the systemd service lifecycle (start, stop, restart-on-failure, TimeoutStopSec); the CUDA driver's per-process memory model and the phenomenon of zombie GPU contexts; the SGLang server's initialization sequence and its dependency on available GPU memory; and the specific deployment topology (a remote server at 10.1.230.174 running a systemd service for Qwen3.5-122B-A10B with tensor parallelism across 4 GPUs).

The output knowledge created by this message is the diagnosis of the restart loop and the decision to break it with a clean stop. This knowledge will be immediately actionable — the assistant will follow up by cleaning GPU processes and redeploying with a more deliberate approach.

The Thinking Process

The assistant's reasoning in this message is a model of diagnostic efficiency. It has observed a pattern (rapid PID cycling), correlated it with prior actions (the systemctl stop commands from earlier iterations), and identified the mechanism (GPU memory held by zombie processes). The causal chain is:

1. Previous systemctl stop → SIGTERM → graceful shutdown starts
2. Graceful shutdown exceeds 60s → systemd sends SIGKILL
3. SIGKILL leaves GPU memory in inconsistent state
4. New instance starts → fails to allocate GPU memory → exits with error
5. systemd sees failure → auto-restarts → same failure → loop The assistant doesn't waste time on further investigation — it has enough evidence to act. This is the hallmark of an experienced operator: knowing when to stop debugging and start fixing.

Conclusion

Message 6537 is a small but revealing moment in a long conversation about ML infrastructure. It shows the assistant not just executing commands, but reasoning about the state of the system, diagnosing a complex failure mode, and taking decisive corrective action. The restart loop was a self-inflicted wound — a consequence of rapid iteration without proper cleanup between deployments — and the assistant recognized it as such. This kind of meta-cognitive awareness — understanding how your own actions affect the system you're managing — is essential for reliable infrastructure operation, especially in the demanding context of large-scale ML inference serving.