The Moment of Relief: Starting a Systemd Service After GPU Memory Liberation

Introduction

In the sprawling, multi-day journey of deploying a 1-trillion-parameter MoE model on an 8-GPU RTX PRO 6000 Blackwell server, few moments are as simultaneously mundane and consequential as the one captured in message index 2162. After hours of debugging architecture incompatibilities, patching vLLM's GGUF loader, wrestling with FP8 KV cache limitations on SM120 architecture, and fighting with zombie CUDA processes that refused to release GPU memory, the assistant finally issues the command to start the systemd service that will make the Kimi-K2.5-NVFP4 deployment persistent and production-ready.

This article examines that single message in depth: why it was written, the chain of reasoning that led to it, the assumptions baked into its simple commands, and what it reveals about the broader arc of the deployment effort.

The Message Itself

The subject message reads:

[assistant] GPUs clean now. Let me start the service: [bash] ssh root@10.1.230.174 'rm -f /dev/shm/psm_ /dev/shm/sem.mp-; systemctl start vllm-kimi-k25' & sleep 5 && ssh root@10.1.230.174 'systemctl status vllm-kimi-k25 2>&1 | head -15' * vllm-kimi-k25.service - vLLM Kimi-K2.5 NVFP4 1T MoE Inference Server Loaded: loaded (/etc/systemd/system/vllm-kimi-k25.service; enabled; preset: enabled) Active: activating (start-pre) since Fri 2026-02-20 22:15:11 UTC; 4s ago Cntrl PID: 213791 (bash) Tasks: 6 (limit: 618378) Memory: 16.7M (peak: 18.2M) CPU: 1.011s CGroup: /system.slice/vllm-kimi-k25.service |-213791 /bin/bash -c "for i in \$(seq 1 30); do free=\$(nvidia-smi --query-gpu=memory.u...

At first glance, this is a straightforward sequence: clean shared memory, start the service, wait, check status. But the context transforms this into something far more significant.

Why This Message Was Written: The Reasoning and Motivation

The message exists because of a specific problem that consumed the previous several minutes of the session: GPU memory that refused to be freed. In [msg 2156], the assistant discovered that all 8 GPUs were still holding 96.6 GiB of memory each — the full weight of the loaded model — even after the vLLM process had been ostensibly killed. The nvidia-smi --query-compute-apps command in [msg 2160] revealed the root cause: the vLLM worker processes (PIDs 212702 through 212709) were still alive, holding CUDA contexts. They hadn't been terminated by the earlier pkill -9 -f "python3.*vllm" command, likely because the process names matched differently or because the pkill ran before the processes had fully started.

This led to a frustrating multi-step cleanup: identifying the surviving PIDs via nvidia-smi --query-compute-apps and fuser /dev/nvidia*, then issuing individual kill -9 commands for each of the eight workers plus two ancillary processes (the resource tracker at PID 212437 and another process at PID 202515). Only after this manual intervention did the GPUs finally report 0 MiB used ([msg 2161]).

The subject message is the direct consequence of that cleanup. The assistant's opening line — "GPUs clean now" — is not a casual observation but a declaration of victory after a frustrating battle. The relief is palpable in its brevity.

The motivation for starting the service immediately is clear: every moment the GPUs sit idle is wasted compute time on a machine that cost tens of thousands of dollars. The model has already been downloaded (540 GB across 119 safetensor shards), tested (achieving ~60 tok/s in [msg 2147]), and proven coherent across multiple prompts. The only remaining step is to make it restart-proof via systemd.

How Decisions Were Made

Several design decisions are embedded in this message, some visible and some implicit.

The background-process pattern: The assistant uses & to background the SSH command that starts the service, then sleeps 5 seconds before checking status. This is a deliberate pattern to avoid the tool timeout that plagued earlier attempts. In [msg 2150], the systemctl start command timed out after 30 seconds because it was waiting for the ExecStartPre to complete. By backgrounding the start command and checking status separately, the assistant avoids the timeout issue entirely while still getting immediate feedback.

The shared memory cleanup: The rm -f /dev/shm/psm_* /dev/shm/sem.mp-* command appears before systemctl start, even though the service file's ExecStartPre already includes this cleanup. This redundancy is intentional: it ensures that any stale shared memory segments from the previous failed runs are removed before the service's pre-check runs, preventing the pre-check from failing on a race condition. It's a defensive programming pattern born from experience with the fragility of NCCL shared memory in LXC containers.

The status check with head -15: Rather than tailing the log or waiting for the full startup, the assistant checks systemctl status and limits output to 15 lines. This is a quick sanity check that confirms the service is in the expected state ("activating (start-pre)") without waiting for the full 8-minute model load. It's an efficient feedback loop.

The choice of systemctl start over systemctl restart: The assistant uses start rather than restart because the service was previously in a "failed" state (after [msg 2155] showed Result: signal). Using start on a failed service is valid in systemd — it resets the failure count and attempts to start. This choice reflects knowledge of systemd's state machine.

Assumptions Made

The message rests on several assumptions, most of which are justified by the preceding context but some of which are untested.

Assumption: The ExecStartPre will succeed. The pre-check script iterates up to 30 times, checking if GPU memory usage is below 1000 MiB. The assistant assumes that since the GPUs are now clean (0 MiB), the pre-check will pass on its first iteration. This is correct — the status output confirms the service is in "activating (start-pre)" state, meaning the pre-check is running.

Assumption: The service file is correct. The assistant fixed the $ → $$ escaping issue in [msg 2153] and redeployed the service file in [msg 2154]. However, the systemctl daemon-reload in that earlier attempt may not have fully taken effect before the service was killed. In [msg 2155], the status still showed the old ExecStartPre with $free (not $$free), suggesting the daemon-reload hadn't been processed. The assistant's systemctl reset-failed in [msg 2156] and the fresh systemctl start in this message should trigger a proper reload, but there's a lingering risk that the old service definition is cached.

Assumption: The model will load successfully again. The model loaded successfully in the manual test ([msg 2143]-[msg 2147]), but the systemd service uses slightly different parameters (e.g., --gpu-memory-utilization 0.95 and --max-model-len 131072). The assistant assumes these parameters won't cause issues, but the earlier failed attempt with 262k context ([msg 2140]) shows that KV cache configuration is sensitive. The 128k limit should work — it was specifically chosen because the KV cache calculation in [msg 2141] showed that ~178k tokens would fit, so 128k leaves comfortable headroom.

Assumption: No other process will interfere. The assistant has killed all known vLLM processes and the resource tracker, but there's always a risk that another CUDA process (e.g., from a previous test run or an automated job) claims GPU memory before the service starts. The pre-check script's 30-second timeout with polling is designed to handle this, but if a long-running process claims memory, the service will fail to start.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

Systemd service management: Understanding the "activating (start-pre)" state, the purpose of ExecStartPre, the $$ escaping requirement for shell variables in systemd unit files, and the difference between start, restart, and reset-failed.

NCCL and GPU memory management: Knowing that CUDA contexts persist after process death, that nvidia-smi --query-compute-apps reveals surviving processes, that fuser /dev/nvidia* shows which PIDs hold GPU file handles, and that shared memory segments under /dev/shm/ can interfere with NCCL initialization.

vLLM architecture: Understanding that vLLM spawns one worker process per GPU (Worker_TP0 through Worker_TP7 for 8-way tensor parallelism), that each worker holds ~70.8 GiB of model weights, and that KV cache memory is allocated separately during engine initialization.

The specific hardware: Knowing that RTX PRO 6000 Blackwell GPUs have 96 GiB of VRAM each, that they use SM120 architecture which lacks FP8 KV cache support in Triton MLA attention, and that NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS are necessary environment variables for this setup.

The model being deployed: Understanding that Kimi-K2.5-NVFP4 is a 1-trillion-parameter MoE model using DeepSeek V3 architecture, quantized to NVFP4 by NVIDIA, with 119 safetensor shards totaling 540 GB, and that it uses a reasoning parser and tool-call parser for its chat interface.

Output Knowledge Created

This message produces several concrete outputs:

A running systemd service (eventually): The service transitions from "activating (start-pre)" to "active (running)" once the model loads. This makes the deployment restart-proof — if the server reboots, systemd will automatically restart the service.

Verification that the ExecStartPre works: The status output confirms that the pre-check script is executing correctly, with the $$ escaping fix working as intended. The script is iterating through GPU memory checks, and since the GPUs are clean, it will pass immediately.

A pattern for future service starts: The background-and-check pattern demonstrated here — starting the service in a background SSH command, then checking status after a brief sleep — becomes the template for subsequent service management in this session.

Confirmation of the service configuration: The status output shows the service description ("vLLM Kimi-K2.5 NVFP4 1T MoE Inference Server"), the loaded unit file path, the enabled state, and the control PID. This confirms the service file was deployed correctly and parsed by systemd without syntax errors.

The Thinking Process Visible in the Reasoning

While this message doesn't contain explicit chain-of-thought reasoning (unlike some earlier messages with <thinking> blocks), the reasoning is visible in the structure of the commands and the sequence of actions.

The assistant is thinking: "GPUs are finally clean after that painful manual kill process. I need to start the service, but I can't block on it because systemctl start will wait for the ExecStartPre (30 seconds max) and then the model load (8+ minutes). Instead, I'll background the start command, wait just 5 seconds to confirm the service is in the expected state, and then let it run. The shared memory cleanup should happen before the service starts to avoid any race conditions with stale segments from the previous failed run."

The choice to check status after only 5 seconds (rather than 30 or 60) is telling: the assistant expects the pre-check to pass almost immediately because the GPUs were just verified clean. A longer wait would be unnecessary. The head -15 limit on the status output shows the assistant only needs a quick confirmation — the full log can be checked later.

The message also reveals a shift in strategy. Earlier attempts to start the service used blocking SSH commands that timed out ([msg 2150]). The assistant learned from that failure and adapted: background the long-running command, check status asynchronously. This is a practical lesson in tool management within the opencode environment.

Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is the assumption that the service file is fully correct. The $$ escaping fix was applied, but the service file also contains other parameters that haven't been tested in the systemd context:

1. The --gpu-memory-utilization 0.95 flag wasn't present in the manual test that succeeded ([msg 2143]-[msg 2147]). The manual test used the default GPU memory utilization. If 0.95 causes the KV cache allocation to exceed available memory (because the model weights take 70.8 GiB and 0.95 × 96 GiB = 91.2 GiB, leaving only 20.4 GiB for KV cache), the service might fail during engine initialization.
2. The --max-model-len 131072 was tested only in the manual run that succeeded, but the systemd service's environment variables (NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS) are set differently — they're exported in the service file's Environment directives rather than being set in the shell before launching. If there's a subtle difference in how vLLM interprets these environment variables when set via systemd vs. the shell, the behavior could differ.
3. The service file uses ExecStart=/root/ml-env/bin/python3 directly, without going through the shell wrapper script (/tmp/run_kimi.sh) that was used in manual tests. The shell wrapper set environment variables via export, while the service file uses Environment= directives. These should be equivalent, but any edge case in systemd's environment handling could cause issues. However, these are relatively minor concerns. The core assumption — that the model will load and serve correctly under systemd — is well-supported by the successful manual test. The assistant's decision to proceed with service deployment is reasonable given the evidence.

Conclusion

Message 2162 is a small pivot point in a much larger narrative. It represents the transition from debugging mode to deployment mode — from fighting with zombie processes and escaping issues to confidently starting a production service. The "GPUs clean now" declaration is earned through the preceding five messages of painful manual cleanup, and the simple systemctl start command carries the weight of hours of prior work.

The message is also a testament to the iterative nature of infrastructure engineering. The first attempt at starting the service failed (timeout from blocking command). The second attempt failed (GPU memory not freed, old service file cached). The third attempt is poised to succeed because the assistant learned from each failure: background long-running commands, verify GPU state before proceeding, and ensure systemd has fully reloaded the service definition.

In the end, this message is about resilience. It's about the patience to manually kill nine processes by PID, the knowledge to check fuser /dev/nvidia* when nvidia-smi shows stuck memory, and the judgment to know when the cleanup is complete and it's time to try again. These are the skills that separate a successful deployment from a stalled one, and they're all visible in this one deceptively simple message.