The Final Fix: Breaking a Cascading Failure Loop in vLLM Systemd Deployment

Message Overview

In message [msg 2051], the assistant executes a single critical bash command that deploys a repaired systemd service file and starts the vLLM GLM-5 inference server after an extended debugging session. The command is:

ssh root@10.1.230.174 'systemctl daemon-reload && systemctl reset-failed vllm-glm5 2>/dev/null; systemctl start vllm-glm5 && echo "Service started" && systemctl status vllm-glm5 2>&1 | head -15'

The output confirms success:

Service started
* vllm-glm5.service - vLLM GLM-5 744B GGUF Q4_K_XL Inference Server
     Loaded: loaded (/etc/systemd/system/vllm-glm5.service; enabled; preset: enabled)
     Active: active (running) since Fri 2026-02-20 19:56:38 UTC; 4ms ago
    Process: 201200 ExecStartPre=/bin/bash -c for i in $(seq 1 30); do free=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | sort -rn | head -1); [ "$free" -lt 1000 ] && exit 0; echo "Waiting for GPU memory to free (max used: ${free} MiB...

At first glance, this is a routine systemd restart. But beneath the surface lies the resolution of a subtle and destructive race condition that had been silently corrupting the deployment for multiple restart cycles. This message represents the moment the assistant finally breaks a cascading failure loop and gets the service to launch cleanly.

Context: The Cascading Failure Loop

To understand why this message matters, we must trace the debugging journey that preceded it. The GLM-5 GGUF deployment had been a multi-hour saga spanning segments 13 through 17 of the conversation. The model itself is enormous — a 744B-parameter mixture-of-experts (MoE) model quantized to GGUF Q4_K_XL, weighing in at 402 GB across 10 split files that had to be merged using a custom-built llama-gguf-split tool. The vLLM codebase itself required extensive patching to support the novel glm_moe_dsa architecture, including fixes to gguf_loader.py, weight_utils.py, and the Triton MLA sparse attention backend for Blackwell SM120 GPUs.

By the time we reach [msg 2037], the service has been struggling to start. The user prompts the assistant to continue, and the assistant checks the service status in [msg 2039], finding it "active (running)" — but only 15 seconds in. In [msg 2041], the assistant watches the journal and sees it failing again with WorkerProc initialization failed. The restart counter is at 6, meaning systemd has already tried and failed six times.

The assistant's initial hypothesis in [msg 2042] is that "stale resources from the previous failed attempts" are causing the problem — a stale resource_tracker process, leftover shared memory files in /dev/shm/. The assistant stops the service, kills all Python processes, cleans shared memory, and verifies that GPU memory is fully freed (0 MiB across all 8 GPUs) in [msg 2044].

But in [msg 2047], the assistant digs deeper into the journal and discovers the real root cause. The error message from the failed worker process reads:

ValueError: Free memory on device cuda:6 (3.27/94.97 GiB) on startup is less than desired GPU memory utilization (0.9, 85.48 GiB).

The workers saw only 3.27 GiB free on each GPU because a previous failed attempt's model weights were still resident in GPU memory when the restart happened. The Restart=on-failure directive with RestartSec=10 was too aggressive — it restarted the service before the CUDA driver had fully released the GPU memory from the previous failed process. This created a devastating feedback loop: each restart would fail because GPU memory wasn't free, which would trigger another restart after 10 seconds, which would also fail because the memory still wasn't free, incrementing the restart counter each time.

This is a classic cascading failure pattern in distributed systems: a transient resource contention issue that, left unchecked, compounds itself into a permanent failure state. The assistant had already cleaned the GPUs manually in [msg 2044], but the next restart would encounter the same problem if the service file wasn't fixed.

The Decision: An ExecStartPre Guard

The assistant's diagnosis in [msg 2047] leads to a three-part plan:

Reset the restart counter by running systemctl reset-failed, which clears systemd's record of previous failures and prevents the service from entering a "failed" state that might block future restarts.
Add an ExecStartPre guard to the service file that waits for GPU memory to be freed before allowing the main process to start.
Start the service fresh with the repaired configuration. The ExecStartPre guard is the key innovation. It runs a bash script before the main vllm process starts:

/bin/bash -c 'for i in $(seq 1 30); do free=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | sort -rn | head -1); [ "$free" -lt 1000 ] && exit 0; echo "Waiting for GPU memory to free (max used: ${free} MiB)..." && sleep 2; done; exit 1'

This script polls nvidia-smi up to 30 times (with 2-second intervals, for a total of 60 seconds of patience) to check whether any GPU still has more than 1000 MiB of memory in use. If all GPUs are below that threshold, it exits successfully and the main process starts. If the timeout expires, it exits with code 1, causing systemd to report the start as failed — but crucially, without triggering the cascading restart loop because the failure happens in the pre-check, not in the main process.

This design choice reveals several assumptions:

Assumption 1: GPU memory release after a process crash is not instantaneous. The CUDA driver and NVIDIA kernel module may take seconds to fully reclaim memory from a terminated process. This is a well-known behavior on multi-GPU systems with large model allocations.
Assumption 2: A 60-second timeout is sufficient for memory to be freed. This is reasonable given that the model is 402 GB spread across 8 GPUs (~50 GB each), and CUDA memory cleanup typically completes within 10-30 seconds.
Assumption 3: The nvidia-smi polling approach is reliable and won't miss transient states. The script checks the maximum used memory across all GPUs, which is a conservative check — if any single GPU still has memory allocated, the service waits.
Assumption 4: The previous failure was purely a memory race condition, not a deeper issue with model loading, architecture patching, or GPU compatibility. This assumption is validated by the fact that the GPUs were successfully cleaned in [msg 2044] and the service had worked in earlier attempts.

The Two Edits: What Changed in the Service File

The assistant makes two edits to the service file in [msg 2048] and [msg 2049]. While the exact diff isn't shown, we can infer the changes from the context:

Edit 1 ([msg 2048]): Likely adds the ExecStartPre directive to the [Service] section. This is the core fix — the GPU memory wait loop.

Edit 2 ([msg 2049]): Possibly adjusts the RestartSec value or adds additional guards. It could also be adding the systemctl reset-failed call or adjusting environment variables.

The service file is stored locally at /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service and deployed to the remote machine via scp in [msg 2050]. This two-step process (edit locally, then copy) is a deliberate workflow choice: it allows the assistant to use the edit tool for precise modifications to the local file, then deploy the complete, validated file to production.

The Subject Message: Executing the Fix

In [msg 2051], the assistant executes the fix. The command chains four operations:

systemctl daemon-reload — Tells systemd to reload its configuration files. This is necessary because the service file was replaced via scp, and systemd caches unit files in memory. Without this step, systemd would use the old (broken) configuration.
systemctl reset-failed vllm-glm5 2>/dev/null — Resets the restart counter. The service had failed 6 times, and systemd tracks this count. If the count exceeds StartLimitBurst (default 5), systemd refuses to start the service at all until the counter is reset or the unit is reloaded. This step is essential because the cascading failures had likely pushed the counter past the burst limit. The 2>/dev/null suppresses any error output (e.g., if the service name doesn't exist yet in the failed list).
systemctl start vllm-glm5 — Starts the service. If the start succeeds, the && operator ensures the next command runs.
systemctl status vllm-glm5 2>&1 | head -15 — Prints the first 15 lines of the service status for verification. The output confirms success: the service is "active (running)" and the ExecStartPre process (PID 201200) is visible in the status output. The fact that the ExecStartPre process is shown means it ran and completed successfully — the GPU memory wait loop passed.

What This Message Achieves (Output Knowledge)

This message creates several important outcomes:

A clean service launch — The vLLM GLM-5 inference server is now running with the repaired configuration. The cascading failure loop is broken.
Validation of the ExecStartPre approach — The GPU memory wait loop works as designed. This pattern can be reused for future deployments on this machine.
Confirmation of the root cause — The fact that the service starts immediately after the ExecStartPre guard is added confirms that the race condition was indeed the primary failure mode. If the service had failed again with a different error, the diagnosis would have been incorrect.
A reset restart counter — Systemd's internal state is clean, meaning future failures (if any) will start from a fresh count rather than accumulating on the previous 6 failures.

What This Message Requires (Input Knowledge)

To fully understand this message, one needs:

Systemd internals — Knowledge of daemon-reload, reset-failed, ExecStartPre, restart counters, and the StartLimitBurst mechanism. Without understanding that systemd tracks failure counts and can lock out a service after too many failures, the reset-failed step seems superfluous.
CUDA GPU memory lifecycle — Understanding that GPU memory is not released instantaneously when a CUDA process crashes. The NVIDIA driver performs cleanup asynchronously, and on large multi-GPU systems this can take several seconds. This is why the ExecStartPre wait loop is necessary.
vLLM worker architecture — Knowing that vLLM spawns multiple worker processes (one per GPU) and that each worker checks GPU memory availability at startup. The gpu_memory_utilization parameter (set to 0.9 in the config) determines how much memory the worker expects to reserve, and if less is available, the worker raises ValueError.
The deployment history — Understanding that this is a 402 GB GGUF model running on 8 RTX PRO 6000 Blackwell GPUs, that the vLLM codebase was heavily patched to support the architecture, and that the service had been failing for multiple cycles.

Mistakes and Incorrect Assumptions

The assistant's initial hypothesis in [msg 2042] — that stale processes and shared memory files were the cause — was partially incorrect. While those resources were indeed stale and needed cleaning, they were a contributing factor rather than the root cause. The real issue was the timing race condition between systemd's restart mechanism and CUDA's memory cleanup. The assistant correctly pivoted after reading the full journal in [msg 2047], which revealed the ValueError about free memory.

This is a valuable lesson in debugging methodology: surface-level symptoms (stale processes, shared memory files) can mislead if the deeper causal chain isn't traced. The assistant's decision to read the full journal (200 lines) rather than just the tail was what uncovered the true error.

Another subtle assumption worth examining: the assistant assumed that cleaning GPU memory manually (via pkill and rm -f /dev/shm/*) was sufficient to fix the problem, without addressing the systemic issue that would cause the next restart to fail again. This is a common trap in operations work — fixing the immediate symptom without addressing the underlying mechanism that produces it. The assistant caught this in the same message ([msg 2047]) and formulated the three-part fix, but the initial cleanup in <msg id=2043-2045> was performed under the incorrect assumption that it alone would resolve the issue.

The Thinking Process Visible in Reasoning

The assistant's reasoning in [msg 2047] is particularly instructive:

"Now I can see the real root cause: Free memory on device cuda:6 (3.27/94.97 GiB) — the workers saw only 3.27 GiB free on each GPU because a previous failed attempt's model weights were still in GPU memory when the restart happened. The Restart=on-failure with RestartSec=10 was too aggressive — it restarted before the previous GPU memory was freed, creating a cascading failure loop."

This passage reveals the assistant's mental model: it connects three observations — (1) the error message about insufficient free memory, (2) the fact that the service had failed 6 times, and (3) the 10-second restart interval — into a coherent causal narrative. The phrase "cascading failure loop" shows that the assistant recognizes this as a systemic pattern, not an isolated bug.

The assistant then formulates a three-part plan that addresses each layer of the problem:

Reset the counter (fix the systemd state)
Add the ExecStartPre guard (prevent recurrence)
Start fresh (execute the fix) This layered approach — addressing state, mechanism, and execution — is characteristic of mature operational reasoning.

Conclusion

Message [msg 2051] is a deceptively simple command that represents the culmination of a complex debugging journey. It resolves a cascading failure loop that had plagued the GLM-5 GGUF deployment across multiple restart cycles, using a combination of systemd expertise, GPU memory lifecycle knowledge, and careful causal analysis. The ExecStartPre guard pattern introduced here is a reusable operational pattern for any large-model inference deployment where GPU memory cleanup timing is a concern. More broadly, this message illustrates the importance of tracing root causes beyond surface-level symptoms and of designing systemic fixes rather than one-shot interventions.