The $free That Wasn't: A Systemd Variable Escaping Bug in a 1T-Parameter Model Deployment

In the middle of deploying a 540GB, 1-trillion-parameter MoE model across eight RTX PRO 6000 Blackwell GPUs, a single bash command reveals an entire chain of failure: a systemd service file with a subtle shell variable escaping bug, a killed pre-start process, and the quiet frustration of a deployment that refuses to stay up. Message [msg 2155] is a diagnostic check — a systemctl status call — but the output it returns tells a story spanning several rounds of debugging.

The Message

The assistant runs a straightforward command:

ssh root@10.1.230.174 'systemctl status vllm-kimi-k25 2>&1 | head -15'

And gets back:

x vllm-kimi-k25.service - vLLM Kimi-K2.5 NVFP4 1T MoE Inference Server
     Loaded: loaded (/etc/systemd/system/vllm-kimi-k25.service; enabled; preset: enabled)
     Active: failed (Result: signal) since Fri 2026-02-20 22:14:00 UTC; 5s ago
    Process: 213509 ExecStartPre=/bin/bash -c for i in $(seq 1 30); do free=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | sort -rn | head -1); [ "$free" -lt 1000 ] && exit 0; echo "Waiting for GPU memory to free (max used: ${free} MiB).....

The service has failed with Result: signal, meaning a process in the service's control group was terminated by a Unix signal (likely SIGTERM or SIGKILL). The truncated ExecStartPre line shows a bash one-liner that loops up to 30 times, checking if any GPU has less than 1000 MiB of used memory, and exits successfully when that condition is met. But the output is cut off — systemd's status display truncates long command lines — and the visible portion reveals a critical detail: the variable $free uses single-dollar syntax, not the $$free that systemd requires.

The Chain of Events

To understand why this message matters, we need to trace the deployment sequence. The assistant had just pivoted from a GLM-5 GGUF deployment to serving nvidia/Kimi-K2.5-NVFP4, a 1T-parameter MoE model based on DeepSeek V3 architecture, quantized by NVIDIA using NVFP4. The model was downloaded across 119 safetensor shards totaling 540GB. A critical blocker — FP8 KV cache incompatibility with the SM120 (RTX PRO 6000) architecture — was resolved by editing the model's quantization configuration to fall back to fp16 KV cache. The model loaded successfully and achieved approximately 60 tok/s single-request throughput.

The next step was to create a systemd service to manage the vLLM server as a production service. The assistant wrote the service file in [msg 2149], copied it to the server in [msg 2150], and ran systemctl start. But the start command timed out after 30 seconds because the ExecStartPre was blocking — it was waiting for GPU memory to free up, but the variable $free was evaluating to empty in the systemd context.

In [msg 2152], the assistant diagnosed the problem: "The $free variable is empty in the systemd context — the environment variable expansion is being eaten. The issue is that $free in the ExecStartPre needs different escaping for systemd. In systemd service files, $ needs to be $$."

This is a classic systemd pitfall. In systemd unit files, the $ character has special meaning — it introduces variable references like ${FOO} or $FOO that systemd resolves from the service's environment. To pass a literal $ to the shell, you must write $$. The original service file had:

ExecStartPre=/bin/bash -c 'for i in $(seq 1 30); do free=$(nvidia-smi ...); [ "$free" -lt 1000 ] && exit 0; ...'

Systemd was interpreting $free and $(seq 1 30) and $(nvidia-smi ...) as its own variable references, not shell syntax. Since $free wasn't a defined systemd variable, it expanded to empty, causing the memory check to always fail (or behave unpredictably), and the loop to hang.

The assistant fixed this in [msg 2153] by editing the service file. Then in [msg 2154], it ran a cleanup sequence:

ssh root@10.1.230.174 'systemctl stop vllm-kimi-k25 2>/dev/null; sleep 2; pkill -9 -f "python3.*vllm" 2>/dev/null; sleep 3; rm -f /dev/shm/psm_* /dev/shm/sem.mp-*' && scp ... && ssh root@10.1.230.174 'systemctl daemon-reload && systemctl start vllm-kimi-k25 && echo "Started"'

This is where the failure in [msg 2155] originates. The systemctl stop command sent SIGTERM to the service's control group, which killed the old ExecStartPre process (PID 213509) that was still running from the original (unfixed) service file. Because the process was killed by a signal rather than exiting cleanly, systemd recorded the service state as failed (Result: signal). The new service file was then copied, daemon-reload was run, and systemctl start was invoked — but the status check in [msg 2155] still shows the old failed state.

What the Output Actually Reveals

The systemctl status output is deceptive. It shows PID 213509 and the old ExecStartPre command line (with $free not $$free), suggesting the daemon-reload hadn't taken effect or the new start hadn't replaced the old state. In reality, systemctl stop had killed the old process, and the new systemctl start had likely also failed — but for a different reason: the GPUs were still full.

The assistant discovers this in the very next message ([msg 2156]), where it checks GPU memory and finds all eight GPUs still at 96,619 MiB each — essentially full. The pkill -9 in the cleanup sequence had killed the vLLM Python processes, but GPU memory hadn't been released yet (a stale multiprocessing resource tracker was holding references). The new ExecStartPre would have checked GPU memory, found it still occupied, and either timed out or failed.

Assumptions and Mistakes

Several assumptions collided in this message:

That systemctl stop would cleanly terminate the service. It did — but by sending a signal that killed the ExecStartPre, which caused a "failed" state rather than a clean "inactive" state. This required a systemctl reset-failed before the service could be restarted properly.
That daemon-reload would immediately apply the fix. The daemon-reload command reloads unit file definitions, but it doesn't retroactively change the state of previously started processes. The old ExecStartPre was already running with the buggy escaping.
That killing vLLM processes would free GPU memory immediately. GPU memory release can lag behind process termination, especially when there are orphaned resource trackers or when the NVIDIA driver's memory management doesn't immediately reclaim buffers.
That the $ escaping was the only problem. Even with correct $$ escaping, the ExecStartPre would still fail if GPU memory wasn't freed. The escaping fix was necessary but not sufficient.

Knowledge Required and Created

To understand this message, the reader needs knowledge of: systemd unit file syntax and the $ vs $$ escaping rule; how systemctl status displays service state and process information; the NVIDIA GPU memory management model and how nvidia-smi reports memory usage; the bash shell's command substitution syntax ($() and backtick); and the concept of ExecStartPre as a pre-condition check in systemd services.

The message creates knowledge about the failure mode: a service that fails with Result: signal after a systemctl stop during its ExecStartPre phase. It also reveals that the truncated command line in systemd status output can hide critical details — the full ExecStartPre line would have shown the $free vs $$free bug more clearly.

The Broader Significance

This message, standing alone, looks like a simple failure notification. But in the context of the session, it represents the friction point between two different worlds: the shell scripting world where $free is a perfectly normal variable reference, and the systemd world where $ is a reserved character. It's the kind of bug that's trivial to fix once spotted but maddening to diagnose because the error messages are opaque — systemd doesn't say "unknown variable $free", it just silently expands it to empty.

The debugging that follows ([msg 2156] through [msg 2159]) shows the assistant working through the consequences: resetting the failed state, killing stale resource trackers, waiting for GPU memory to drain, and eventually getting the service running. But the seed of all that work is visible right here in [msg 2155] — a single systemctl status output that says "failed" and a truncated command line that hints at why.