The Systemd Trap: A Moment of Discovery in GPU Memory Management

In the middle of a high-stakes model deployment session — pivoting from a successfully benchmarked MiniMax-M2.5 to the native INT4 Kimi-K2.5 — the assistant encounters a seemingly trivial but deeply instructive moment. The subject message, <msg id=2350>, is just three lines of action and a table of numbers. But within those lines lies a rich story about assumptions, system architecture, and the invisible hands that manage long-running services.

The Message

Let us first quote the message exactly as it appears:

[assistant] MiniMax is still loaded. The systemd service must be running: [bash] ssh root@10.1.230.174 "systemctl stop vllm-minimax-m25; systemctl disable vllm-minimax-m25; sleep 5; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader" Removed "/etc/systemd/system/multi-user.target.wants/vllm-minimax-m25.service". 0, 96801 MiB 1, 96801 MiB 2, 96801 MiB 3, 96801 MiB 4, 96799 MiB 5, 96799 MiB 6, 96799 MiB 7, 96799 MiB

The assistant states a conclusion — "MiniMax is still loaded. The systemd service must be running" — then executes a command to stop and disable the service, waits five seconds, and queries GPU memory. The result shows all eight NVIDIA RTX PRO 6000 Blackwell GPUs still holding approximately 96.8 GiB of memory each, despite the service having been stopped and disabled.

The Context: A Pivot Between Models

To understand why this message matters, we must understand the broader narrative. The session had been an intense exploration of deploying trillion-parameter models on an 8× Blackwell GPU machine. Earlier, the assistant had benchmarked MiniMax-M2.5 (a 230B FP8 GQA model) at TP=8 with Expert Parallelism, achieving an impressive ~4,000 tok/s at high concurrency ([msg 2335]). The user then directed the assistant to pivot to the native INT4 Kimi-K2.5 ([msg 2337]), a 547GB model from Moonshot AI.

The assistant had already taken several steps toward this pivot. It freed disk space by deleting the 540GB NVFP4 variant of Kimi-K2.5 ([msg 2342]). It downloaded the 547GB INT4 model across 64 safetensor shards (<msg id=2344-2346>). Then, in the message immediately preceding our subject ([msg 2348]), the assistant announced: "Now stop MiniMax and launch Kimi-K2.5 INT4." It ran a kill -9 command targeting any vLLM process matching "minimax" and checked GPU memory. The result ([msg 2349]) showed all eight GPUs still reporting ~96.8 GiB used — as if nothing had been killed.

This is the moment of cognitive dissonance that the subject message resolves.

The Reasoning: Why "Must Be Running"

The most interesting part of this message is the reasoning that precedes the command. The assistant writes: "MiniMax is still loaded. The systemd service must be running." This is a deduction, not a guess. The assistant had just attempted to kill the MiniMax process with kill -9 targeting any process matching "vllm.*minimax" ([msg 2348]). Yet the subsequent GPU memory check ([msg 2349]) showed all eight GPUs still saturated at ~96.8 GiB each — exactly the memory footprint of the MiniMax model. The kill command had apparently done nothing.

The assistant's reasoning chain is elegant and correct: if the process was killed but the memory is still allocated, the process must have been restarted. And the only mechanism that could restart a killed process automatically is systemd, which manages services as units with restart policies. The service vllm-minimax-m25 had been previously deployed as a systemd service (<msg id=2337 context>). When a systemd service is configured with Restart=always (common for production services), killing the underlying process merely triggers an immediate restart. The assistant correctly inferred that kill -9 alone was insufficient — the service definition itself needed to be stopped and disabled.

The Assumption and Its Correction

The message reveals an important assumption that was implicitly made and then corrected. In the earlier message ([msg 2348]), the assistant assumed that kill -9 with a process-name pattern match would be sufficient to free the GPUs. This assumption was reasonable: in most ad-hoc development scenarios, processes are launched directly from the command line, and killing them releases resources. But the assistant had previously deployed MiniMax as a production-grade systemd service, which introduced a layer of process management that outlives simple kill signals.

The correction happens in this message. The assistant recognizes the failure mode — "MiniMax is still loaded" — and correctly diagnoses the root cause: "The systemd service must be running." The command issued combines three operations: systemctl stop (terminates the current process and prevents restart), systemctl disable (removes the symlink so the service won't start on boot), and a five-second sleep before checking GPU memory. The sleep is a thoughtful touch — it gives the system time to clean up GPU context after the process terminates.

Input Knowledge Required

To fully understand this message, several pieces of domain knowledge are necessary. First, one must understand systemd's service lifecycle: that stop and disable are separate operations, that a service with Restart=always will respawn a killed process, and that systemctl disable only affects future boots, not the current runtime state. Second, one must know how NVIDIA GPU memory works in a multi-process context: GPU memory allocations persist until the allocating process exits, and even then, the NVIDIA driver may hold memory briefly during cleanup. Third, one must understand vLLM's deployment architecture: that it runs as a long-lived HTTP server with worker processes that allocate GPU memory during model loading, and that these workers are children of the main process. Fourth, one must recognize the significance of the specific memory value (~96.8 GiB per GPU) — this is the exact memory footprint of the MiniMax-M2.5 model at TP=8 with EP, confirming that the model is still resident in GPU memory rather than being a stale driver artifact.

Output Knowledge Created

This message produces several valuable pieces of output knowledge. Most concretely, it creates the knowledge that the MiniMax systemd service has been stopped and disabled, clearing the path for the Kimi-K2.5 INT4 deployment. The output of the nvidia-smi command — all eight GPUs still at 96.8 GiB — creates the knowledge that stopping the service alone does not immediately free GPU memory. This is a critical observation: systemctl stop terminates the process, but the GPU memory remains allocated. This sets up the next messages in the conversation, where the assistant discovers zombie worker processes still holding GPU memory (<msg id=2352-2354>) and must kill them directly.

More broadly, the message creates architectural knowledge about the interaction between systemd service management and GPU resource allocation. It demonstrates that GPU memory management has two layers: the process layer (which systemd controls) and the driver layer (which holds memory until the process fully exits). When a systemd service is stopped, the main process is killed, but child worker processes may persist as zombies, continuing to hold GPU memory. This is exactly what happens next in the conversation — the assistant discovers that processes 215004, 233458, and others are still holding GPU memory even after the service was stopped ([msg 2353]).## The Thinking Process Visible in the Reasoning

Although the assistant's reasoning is presented concisely — just two sentences — it reveals a sophisticated diagnostic process. The assistant observes a symptom (GPUs still occupied after kill), hypothesizes a mechanism (systemd restart), tests the hypothesis (systemctl stop + disable), and evaluates the result (memory still allocated, but now the service is disabled). The thinking is notable for what it does not do: it does not re-run the kill command, does not check process lists first, does not try a different signal. The assistant jumps directly to the most likely root cause based on knowledge of the system's architecture.

This is a pattern of reasoning that comes from experience with production ML deployments. A less experienced operator might have tried kill -9 again, or used pkill -f minimax, or rebooted the GPUs with nvidia-smi --gpu-reset. The assistant instead recognizes that the systemd service layer is the relevant abstraction. The thinking is also notable for what it reveals about the assistant's mental model: it understands that systemd services are not just processes but are managed entities with lifecycle policies, and that the correct way to terminate a service-managed process is through systemd's own tooling, not through raw signal delivery.

The Mistake That Wasn't

One might ask: was there a mistake in this message? The assistant stopped and disabled the service, but the GPUs remained occupied. From one perspective, the command "failed" to free the GPUs. But this is not a mistake — it is a discovery. The assistant learned that systemctl stop terminates the main process but does not guarantee that child worker processes (which hold the GPU memory) are immediately reaped. This is a subtle point about vLLM's process architecture: when vLLM runs with tensor parallelism, it spawns worker subprocesses (one per GPU) that are children of the main process. When systemd kills the main process, these workers may become orphaned or zombie processes that continue to hold GPU memory until they are explicitly killed.

The real mistake, if any, was in the earlier message ([msg 2348]) where the assistant assumed kill -9 would be sufficient. But even that assumption was reasonable in context — the assistant had not yet internalized that the MiniMax deployment was managed by systemd. The subject message represents the correction of that assumption.

Why This Message Matters

In a session filled with dramatic moments — patching vLLM source code to support new architectures, debugging Triton kernel output buffers, achieving 4,000 tok/s throughput — this message stands out for its quietness. It is a maintenance operation, a cleanup step, a piece of housekeeping. But it is precisely this kind of message that reveals the true nature of production ML engineering. The glamorous work is designing architectures and tuning kernels. The real work is managing state: freeing disk space, stopping services, killing zombie processes, ensuring that one deployment does not interfere with the next.

This message also illustrates a broader principle: in complex systems, the mechanism of process management matters. A process launched from a terminal is not the same as a process launched by systemd. A process killed with kill -9 is not the same as a service stopped with systemctl stop. The abstraction layers that make deployment reliable (service managers, container orchestrators, init systems) also create new failure modes when they are not properly understood. The assistant's ability to diagnose and correct this situation demonstrates a deep understanding of these layers.

Conclusion

The subject message at <msg id=2350> is a masterclass in diagnostic reasoning. In just two sentences and one command, the assistant identifies that a systemd service is responsible for the persistent GPU memory allocation, correctly uses systemd's own tooling to stop and disable the service, and sets the stage for the subsequent cleanup of zombie worker processes. The message reveals the importance of understanding process management abstractions in production ML deployments, the subtle interaction between systemd and GPU memory allocation, and the value of reasoning from symptoms to mechanisms rather than blindly retrying failed operations. It is a small moment in a long conversation, but it is the kind of moment that separates experienced operators from novices.