The Stubborn 96.6 GiB: A Post-Mortem of GPU Memory Cleanup

In the high-stakes world of large-scale ML inference deployment, few moments are as deflating as watching a meticulously configured systemd service fail to start, then discovering that every GPU on the machine is still pinned at 96.6 GiB of memory usage — as if the model never left. Message [msg 2157] captures exactly this moment: a brief, almost terse assistant response that belies a cascade of accumulated context, failed assumptions, and the beginning of a deeper debugging journey.

The Message

The subject message reads in full:

[assistant] GPUs are full — 96.6GB each. There's a leftover vllm process. Let me kill it properly: [bash] ssh root@10.1.230.174 'pkill -9 -f "python3.vllm"; pkill -9 -f "run_kimi"; sleep 5; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader; rm -f /dev/shm/psm_ /dev/shm/sem.mp-*'

This is a single-turn assistant message containing a bash command dispatched to a remote server. Despite its brevity — a mere two sentences of natural language followed by a shell command — it sits at the intersection of multiple converging threads: systemd service lifecycle management, GPU memory persistence, process cleanup semantics, and the accumulated weight of a multi-hour deployment session.

The Context: How We Got Here

To understand why this message exists, we must trace backward through the preceding messages. The assistant had just spent considerable effort deploying the nvidia/Kimi-K2.5-NVFP4 model — a 1-trillion-parameter MoE (Mixture of Experts) model quantized with NVIDIA's NVFP4 format — across 8 RTX PRO 6000 Blackwell GPUs. This was a pivot from the previous GLM-5 GGUF deployment ([msg 2127] onward). The model weighs 540 GB across 119 safetensor shards, and loading it took over 8 minutes, consuming 70.8 GiB of memory per GPU ([msg 2144]).

After successfully loading the model and verifying coherent output at ~60 tok/s throughput ([msg 2147]), the assistant turned to creating a systemd service for production deployment. The service file vllm-kimi-k25.service was written ([msg 2149]), copied to the server, and the service was started ([msg 2150]). But the start command timed out — the ExecStartPre script that waits for GPU memory to free was stuck.

The root cause emerged: a shell variable escaping issue. In systemd service files, the $ character needs to be escaped as $$ to prevent systemd from interpreting it. The original ExecStartPre used $free which systemd consumed as an empty variable, causing the memory-wait loop to spin indefinitely ([msg 2152]). The assistant fixed this ([msg 2153]), stopped the old service, killed lingering processes, cleaned shared memory, copied the updated file, reloaded systemd, and started the service again ([msg 2154]).

But the service still showed as failed ([msg 2155]). The assistant investigated and found that the failure was from the old ExecStartPre that had been killed by SIGTERM during systemctl stop — the daemon-reload should have fixed it. After resetting the failed state with systemctl reset-failed, the assistant checked GPU memory and found all 8 GPUs still at 96,619 MiB each ([msg 2156]).

This is the immediate precursor to our subject message. The GPUs are full. Something is holding the memory.

The Reasoning: What the Assistant Was Thinking

The subject message opens with a diagnosis: "GPUs are full — 96.6GB each. There's a leftover vllm process." This is not a guess — it's an inference drawn from the observed data. The assistant knows that:

  1. The vLLM model weights occupy ~70.8 GiB per GPU.
  2. The KV cache and other runtime allocations bring this to ~96.6 GiB.
  3. No vLLM server should be running (it was killed in [msg 2154]).
  4. Yet the memory remains allocated. The conclusion "there's a leftover vllm process" is the most parsimonious explanation. GPU memory in CUDA is tied to the lifetime of the allocating process. When a process exits (normally or via SIGKILL), the CUDA driver should release all GPU memory associated with that process. If the memory is still allocated, either: - A process is still running that holds the allocation (the "leftover" hypothesis). - The process was killed but the CUDA driver hasn't yet cleaned up (a transient condition). - Some other process (e.g., NCCL, shared memory) is pinning the allocation. The assistant opts for the first hypothesis and takes decisive action: kill all vllm and run_kimi processes with SIGKILL (-9), wait 5 seconds for cleanup, verify with nvidia-smi, and clean shared memory segments.

Assumptions Embedded in the Action

This message carries several assumptions, some explicit and some implicit:

Assumption 1: SIGKILL will free GPU memory. The pkill -9 sends an uncatchable kill signal. In normal CUDA operation, when a process is killed, the CUDA driver's cleanup handlers release all GPU memory allocations. This is generally reliable, but edge cases exist — for example, if the process was in the middle of a CUDA kernel execution or had pending asynchronous operations, cleanup can be delayed or, in rare cases, require a GPU reset.

Assumption 2: The memory is held by a vllm or run_kimi process. The assistant specifically targets these two process name patterns. This is a reasonable heuristic — these are the only processes that should have allocated GPU memory. But it's possible that some other Python process (e.g., a multiprocessing resource tracker spawned by vLLM) holds references that prevent cleanup.

Assumption 3: Five seconds is sufficient for cleanup. The sleep 5 between the kill commands and the nvidia-smi check assumes that CUDA cleanup completes within this window. For most cases this is true, but with 8 GPUs and ~773 GiB of total allocated memory (8 × 96.6 GiB), cleanup might take longer.

Assumption 4: Shared memory cleanup is relevant. The rm -f /dev/shm/psm_* /dev/shm/sem.mp-* removes NCCL shared memory segments. This is a good hygiene practice — stale shared memory can cause issues on restart — but it's unlikely to be the cause of persistent GPU memory allocation.

What Actually Happened

The follow-up message ([msg 2158]) reveals the outcome:

0, 96619 MiB 1, 96619 MiB ... 7, 96619 MiB root 212436 0.0 0.0 18412 11704 ? S 22:01 0:00 /root/ml-env/bin/python3 -c from multiprocessing.resource_tracker import main;main(93)

The GPUs are still at 96.6 GiB each. The kill commands succeeded in terminating the main vLLM processes, but a single Python process remains: a multiprocessing.resource_tracker. This is a child process spawned by Python's multiprocessing module, responsible for tracking and cleaning up shared resources. It holds 18 KB of RSS — essentially nothing in system memory — yet it appears to be the sole surviving process.

This is a critical finding: the resource tracker process, despite its tiny memory footprint, may be holding CUDA context references that prevent GPU memory deallocation. The vLLM engine uses Python multiprocessing extensively — each tensor-parallel worker runs as a separate process, and the resource tracker is part of that infrastructure. When the main processes are killed with SIGKILL, the resource tracker survives (it was likely spawned by a different parent or inherited by init), and its continued existence may keep the CUDA contexts alive.

The assistant's assumption — "there's a leftover vllm process" — was partially correct. There was a leftover process, but it wasn't a vLLM process matching the python3.*vllm pattern. It was a Python multiprocessing infrastructure process that the pkill pattern didn't catch. This is a subtle but important distinction: the memory holder wasn't a "vllm process" in the conventional sense, but a supporting process that CUDA still recognizes as a context owner.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message and its aftermath produce several pieces of knowledge:

  1. The GPU memory persistence problem: Even after killing all visible vLLM processes, GPU memory remains allocated. This is a concrete, reproducible observation.
  2. The resource tracker hypothesis: The surviving multiprocessing.resource_tracker process is the prime suspect. This becomes actionable knowledge for the next step.
  3. Process pattern limitations: The pkill -f "python3.*vllm" pattern is too narrow — it misses infrastructure processes that don't match the pattern but still hold CUDA contexts.
  4. Cleanup procedure refinement: A more aggressive cleanup is needed — possibly killing all Python processes spawned by the vLLM user, or using fuser or lsof to identify what's holding GPU contexts.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, though compressed into two sentences, reveals a clear diagnostic chain:

  1. Observe symptom: GPUs at 96.6 GiB each (from [msg 2156]).
  2. Compare to expected: The model weights are 70.8 GiB/GPU, and the service was stopped. Memory should be freed.
  3. Form hypothesis: A leftover process is holding the memory.
  4. Take action: Kill all matching processes, wait, verify, clean shared memory. The phrase "Let me kill it properly" is telling — it acknowledges that previous kill attempts (in [msg 2154]) were insufficient. The "properly" implies a more thorough approach: using SIGKILL instead of SIGTERM, targeting both vllm and run_kimi patterns, adding a verification step, and cleaning shared memory. However, the reasoning has a blind spot: it assumes that any process holding GPU memory would match the python3.*vllm or run_kimi patterns. The multiprocessing resource tracker runs as python3 -c from multiprocessing.resource_tracker import main;main(93) — it matches neither pattern. This is a classic case of a heuristic that works for the common case but fails for an edge case the designer didn't anticipate.

Broader Significance

This message, for all its brevity, exemplifies a recurring pattern in complex systems debugging: the gap between what we think is happening and what is actually happening. The assistant correctly identified that GPU memory should have been freed, correctly hypothesized that a leftover process was responsible, and took reasonable corrective action. Yet the action failed because the model of "what processes could hold GPU memory" was incomplete.

In the broader arc of the conversation, this message represents a turning point. The clean, straightforward deployment path — download model, configure vLLM, create systemd service — has hit a snag that requires deeper understanding of process management and CUDA internals. The next steps would likely involve identifying and killing the resource tracker, or using more aggressive cleanup methods like nvidia-smi GPU reset or systemd's KillMode=process to ensure all child processes are terminated.

The 96.6 GiB that refused to leave would eventually need a more creative solution — perhaps killing all Python processes belonging to the root user, or using CUDA_GRACE_PERIOD and CUDA_TIMEOUT environment variables, or even a full GPU reset via nvidia-smi --gpu-reset. But at this moment, in message [msg 2157], the assistant is still operating under the assumption that the right kill commands will suffice. The stubborn memory has not yet yielded its secrets.