The Smallest Fix With the Largest Impact: A Systemd ExecStartPre That Saved a 744B-Parameter Model

The Message

The subject message is deceptively simple:

[assistant] [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service Edit applied successfully.

That is the entire message. No reasoning, no explanation, no fanfare. Just a tool call confirmation that a file was edited. Yet this single edit — the addition of an ExecStartPre guard to a systemd service file — resolved a cascading failure loop that had rendered a 744B-parameter LLM inference server inoperable for multiple restart cycles. To understand why this two-line change matters, we must examine the chain of events that led to it.

The Context: A 744B Model on Eight GPUs

The conversation leading up to this message documents the deployment of the GLM-5 744B GGUF Q4_K_XL model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. This is a massive undertaking: a 402GB model file, split across ten GGUF shards, loaded into 8 × 97GB of GPU memory over a PCIe-only interconnect. The assistant had already overcome extraordinary challenges — patching vLLM's gguf_loader.py to support the novel glm_moe_dsa architecture, implementing a custom Triton MLA sparse attention backend for the SM120 compute architecture, fixing dequantization shard ordering bugs, and tuning NCCL communication protocols to squeeze out ~57 tok/s decode throughput. A systemd service (vllm-glm5.service) had been created to manage the server as a production daemon.

The Problem: A Race Condition in the Restart Loop

In <msg id=2039>, the service appeared to be running — Active: active (running) — with 93GB of GPU memory already consumed per device, suggesting model weights were loaded. But by <msg id=2042>, the assistant discovered the service was actually failing: WorkerProc initialization failed. The restart counter was at 6, meaning the service had failed and been automatically restarted six times already.

The root cause emerged in <msg id=2046>. The assistant examined the journal logs and found:

ValueError: Free memory on device cuda:6 (3.27/94.97 GiB) on startup is less than desired GPU memory utilization (0.9, 85.48 GiB).

The problem was a race condition. When the service failed for any initial reason (perhaps a transient NCCL initialization issue or a stale shared-memory file from a previous run), systemd's Restart=on-failure policy would automatically restart it after RestartSec=10 — just 10 seconds. But GPU memory doesn't free instantly. The NVIDIA driver and CUDA runtime need time to release the tens of gigabytes of model weights allocated across eight GPUs. When the restart fired after only 10 seconds, the worker processes found only 3.27 GiB free on each GPU — the previous instance's weights were still resident. vLLM's startup validation then refused to proceed because available memory was below the 90% utilization target (85.48 GiB), causing an immediate failure. This triggered another restart, which again found the memory still occupied, creating a cascading failure loop that would repeat indefinitely.

The assistant's diagnosis in <msg id=2047> was precise: "The Restart=on-failure with RestartSec=10 was too aggressive — it restarted before the previous GPU memory was freed, creating a cascading failure loop."

The Fix: An ExecStartPre Guard

The fix, applied in the subject message, was to add an ExecStartPre directive to the systemd service file. The exact content of the edit is visible in the subsequent message <msg id=2051>, where the assistant deploys the updated file and starts the service:

ExecStartPre=/bin/bash -c 'for i in $(seq 1 30); do \
  free=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits \
    | sort -rn | head -1); \
  [ "$free" -lt 1000 ] && exit 0; \
  echo "Waiting for GPU memory to free (max used: ${free} MiB...)"; \
  sleep 5; \
done; exit 1'

This script does three things:

Checks GPU memory pressure by querying nvidia-smi for the maximum memory usage across all eight GPUs.
Waits up to 150 seconds (30 iterations × 5 seconds) for all GPUs to drop below 1000 MiB used, indicating the previous instance's weights have been fully released.
Fails cleanly with exit 1 if the timeout is exceeded, preventing the service from starting under memory pressure and avoiding another failure loop. Additionally, the edit included a second ExecStartPre to clean stale shared-memory files (/dev/shm/psm_* and /dev/shm/sem.mp-*) that accumulate from NCCL and multiprocessing operations across restarts.

Why This Matters: Operational Reliability at Scale

This edit is a masterclass in operational reasoning for large-scale ML deployments. The assistant recognized that the problem wasn't a code bug, a configuration error, or a hardware fault — it was a timing problem. Systemd, designed for general-purpose service management, has no intrinsic understanding of GPU memory lifecycle. It treats a process exit as a binary signal (running vs. stopped) and schedules restarts based on wall-clock time, not hardware state. The assistant had to bridge this gap by injecting GPU-aware logic into the service lifecycle.

The fix is elegant in its minimalism. Rather than modifying vLLM's startup code, altering systemd's restart policy, or writing a complex watchdog daemon, the assistant added a single self-contained bash script as a pre-condition. This approach preserves the existing restart mechanism (which is valuable for recovering from transient failures during operation) while preventing the specific failure mode of premature restart after GPU memory hasn't been freed.

Assumptions and Knowledge Required

To understand and implement this fix, the assistant needed:

Knowledge of systemd service lifecycle: Understanding that ExecStartPre runs before the main process and can block startup, and that Restart=on-failure triggers independently of GPU state.
Understanding of GPU memory management: Knowing that nvidia-smi reports memory usage accurately and that CUDA memory deallocation is asynchronous — the process may exit before the GPU driver fully releases memory.
Awareness of NCCL shared-memory artifacts: Recognizing that psm_* and sem.mp-* files in /dev/shm/ are leftover NCCL communication buffers that can interfere with fresh worker initialization.
Familiarity with vLLM's startup validation: Knowing that vLLM checks free_memory >= desired_gpu_memory_utilization * total_memory at worker startup and raises a ValueError if the check fails.

The Outcome

The fix was immediately effective. In <msg id=2051>, the assistant deployed the updated service file, ran systemctl daemon-reload and systemctl reset-failed to clear the restart counter, and started the service. The ExecStartPre passed instantly (GPUs were already clean from the manual cleanup in <msg id=2043-2045>), and the service began loading the model. The model loaded successfully, and subsequent testing confirmed coherent output at ~57 tok/s.

Broader Lessons

This episode illustrates a recurring pattern in large-scale ML operations: the most impactful fixes are often not about algorithm correctness or model architecture, but about the mundane infrastructure that surrounds the model. A 744B-parameter model, with all its architectural sophistication and engineering effort, is ultimately at the mercy of a systemd service file and a bash script. The assistant's ability to diagnose a race condition across multiple layers of abstraction — from GPU memory allocation to systemd's restart scheduler to NCCL's shared-memory protocol — is the kind of systems thinking that separates a working deployment from a perpetually failing one.

The subject message, for all its brevity, represents the culmination of that diagnostic chain. "Edit applied successfully" is the quiet sound of a problem solved.