The Last Kill: Resolving Stuck GPU Memory Through Precise PID Targeting

Introduction

In the high-stakes world of large-scale ML model deployment, few problems are as frustrating as a GPU that refuses to release its memory. When a 1-trillion-parameter model like Kimi-K2.5-NVFP4 occupies 96.6 GiB across each of eight RTX PRO 6000 Blackwell GPUs, and those gigabytes stubbornly persist after the serving process has supposedly been terminated, the deployment pipeline grinds to a halt. Message 2161 in this opencode session captures the precise moment when this impasse was broken — a short, technically dense message that represents the culmination of a debugging chain spanning multiple failed attempts to reclaim GPU memory.

The message itself is deceptively simple:

The worker processes (212702-212709) are still alive as zombies or didn't get killed. Let me kill them all by PID: `` ssh root@10.1.230.174 'kill -9 212702 212703 212704 212705 212706 212707 212708 212709 212437 202515 2>/dev/null; sleep 5; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader' 0, 0 MiB 1, 0 MiB 2, 0 MiB 3, 0 MiB 4, 0 MiB 5, 0 MiB 6, 0 MiB 7, 0 MiB ``

Eight lines of zeros. After hours of debugging, patching, downloading, and tuning, this is the moment the deployment can finally proceed. This article examines why this message was written, the reasoning behind its approach, and the broader lessons it offers about GPU process management in containerized ML environments.

The Context: A Deployment Hitting a Wall

To understand message 2161, we must first understand what led to it. The assistant had been working for many rounds to deploy the nvidia/Kimi-K2.5-NVFP4 model — a 1-trillion-parameter Mixture-of-Experts model based on DeepSeek V3 architecture, quantized by NVIDIA using NVFP4 format. This was a pivot from an earlier effort to deploy GLM-5, which had encountered fundamental architecture incompatibilities.

The deployment had progressed remarkably well. The 540 GB model had been downloaded across 119 safetensor shards. A critical blocker — FP8 KV cache incompatibility with the SM120 architecture of the RTX PRO 6000 Blackwell GPUs — had been resolved by removing kv_cache_quant_algo from the model configuration, falling back to fp16 KV cache. The model had loaded successfully and achieved approximately 60 tok/s single-request throughput.

The final step was to create a systemd service for production deployment. This is where things went wrong. The assistant had written a service file (vllm-kimi-k25.service), copied it to the server, and attempted to start it. But the ExecStartPre script — designed to wait for GPU memory to be freed before launching — was failing because the GPUs still showed 96.6 GiB allocated per card. The previous vLLM process, which had been killed with pkill -9 -f "python3.*vllm", had left its memory footprint intact.

The Debugging Chain: Three Failed Attempts

Message 2161 is the fourth attempt to reclaim GPU memory, and each previous attempt reveals a different facet of the problem.

Attempt 1 (Message 2157): The assistant ran pkill -9 -f "python3.*vllm" and pkill -9 -f "run_kimi", then waited 5 seconds. Result: all eight GPUs still showed 96,619 MiB used. The pkill approach, which matches process command lines against a regex pattern, had apparently succeeded — there were no vLLM or run_kimi processes visible in ps aux. Yet the GPU memory remained allocated.

Attempt 2 (Message 2159): The assistant identified a stale resource tracker process (PID 212436) — a Python multiprocessing resource tracker that often outlives its parent. Killing it with kill -9 and waiting 10 seconds. Result: still 96,619 MiB per GPU. The resource tracker was a red herring; it held no GPU memory.

Attempt 3 (Message 2160): The assistant used nvidia-smi --query-compute-apps=... to enumerate exactly which processes held GPU memory. This revealed the truth: eight VLLM worker processes (PIDs 212702 through 212709) were still alive, each holding 96,610 MiB. These were the per-GPU worker processes spawned by vLLM's tensor parallelism. They had survived the pkill because their process names appeared as VLLM::Worker_TP0 through VLLM::Worker_TP7 — names that didn't match the python3.*vllm regex pattern used by pkill. The workers had been launched as subprocesses with custom names, and the pattern-based kill had missed them entirely.

The Reasoning Behind Message 2161

The assistant's analysis in message 2161 is precise: "The worker processes (212702-212709) are still alive as zombies or didn't get killed." The phrase "zombies or didn't get killed" shows the assistant considering two possibilities. Zombies are processes that have terminated but whose exit status hasn't been collected by their parent — they consume a PID table entry but no memory. The fact that these processes still held 96,610 MiB of GPU memory each ruled out the zombie hypothesis. They were simply alive and unkilled.

The solution was straightforward but required the insight to look beyond pkill. The assistant constructed a single kill -9 command targeting all eight worker PIDs plus two additional PIDs (212437 and 202515) that had appeared in the fuser output as holding /dev/nvidia* file handles. After a 5-second sleep, the verification command showed all eight GPUs at 0 MiB.

Why GPU Memory Gets Stuck

This scenario illustrates a fundamental challenge in GPU computing: GPU memory is tied to CUDA contexts, not directly to process lifetimes. When a process creates a CUDA context (which happens implicitly when it first touches a GPU), that context persists until the process exits and the kernel driver cleans up. However, several factors can prevent cleanup:

  1. Orphaned subprocesses: If a parent process spawns worker subprocesses and the parent is killed, the workers may continue running. In vLLM's architecture, the main API server process spawns one worker per GPU for tensor parallelism. Each worker creates its own CUDA context. Killing the parent doesn't automatically kill the workers.
  2. Pattern-based killing limitations: pkill -f "python3.*vllm" matches against the full process command line as reported by /proc. If worker processes use a different executable path or a different command-line pattern (e.g., VLLM::Worker_TP0 as the process name in nvidia-smi), they can evade the pattern.
  3. Container quirks: In LXC containers, GPU memory cleanup can be delayed or fail entirely if the NVIDIA driver's context cleanup mechanism encounters issues with the container's PID namespace isolation.
  4. CUDA IPC handles: Processes that share GPU memory via CUDA IPC (inter-process communication) can leave shared handles that prevent full memory deallocation until all participating processes have exited.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with vLLM's tensor parallelism architecture (one worker per GPU), understanding of CUDA memory management and process-context relationships, knowledge of nvidia-smi --query-compute-apps for enumerating GPU-using processes, and awareness that pkill uses regex pattern matching against process command lines.

Output knowledge created by this message is both practical and diagnostic: the specific PIDs that needed killing, confirmation that targeted kill -9 works when pkill fails, and the insight that vLLM worker processes may use process names that don't match typical pkill patterns. More importantly, the message establishes that the GPU memory issue was purely a process management problem — not a driver bug, not a CUDA context leak, and not a kernel module issue. This ruled out more invasive solutions like driver reloads or GPU resets.

Assumptions and Potential Mistakes

The assistant made several assumptions during this debugging chain. The initial assumption that pkill -9 -f "python3.*vllm" would catch all vLLM processes proved incorrect — the worker processes used different naming. The assistant also initially assumed that killing the resource tracker (PID 212436) would free memory, which turned out to be wrong.

A potential mistake was not using --signal with pkill or not verifying the kill more aggressively earlier. However, the assistant's methodical approach — checking with nvidia-smi, then ps aux, then fuser, then nvidia-smi --query-compute-apps — represents a textbook escalation of diagnostic tools. Each tool provides different information, and the assistant correctly interpreted each result.

One subtle point: the assistant killed PID 202515 and 212437 alongside the workers. PID 202515 appeared in the fuser output as holding /dev/nvidia* file handles but wasn't a vLLM worker. This could have been a leftover from an earlier CUDA operation. Killing it was defensive — better to over-kill than leave a stray process holding GPU resources.

Broader Significance

Message 2161 is a small but perfect example of a common pattern in systems engineering: the difference between "the command succeeded" and "the operation succeeded." The pkill command returned successfully and appeared to have killed the vLLM processes. But the actual goal — freeing GPU memory — had not been achieved. The assistant had to look beyond the surface-level success indicator and verify the actual state of the system.

This message also demonstrates the value of using the right diagnostic tool for the job. ps aux shows processes but not their GPU memory usage. nvidia-smi shows GPU memory but not which processes hold it. nvidia-smi --query-compute-apps bridges this gap by showing the PID-to-memory mapping. And fuser /dev/nvidia* shows which processes hold GPU device file handles. Each tool contributes a piece of the puzzle.

The eight lines of "0, 0 MiB" at the end of the message are deeply satisfying. After rounds of debugging, patching, downloading, and troubleshooting, this is the moment when the deployment can finally proceed. The systemd service can start, the model can serve requests, and the 60 tok/s throughput can be delivered to users. All of that depends on this one small victory: eight GPUs, zero megabytes used, ready for the next chapter.

Conclusion

Message 2161 is a masterclass in targeted process management. It demonstrates that in complex ML deployments, the difference between success and failure often comes down to understanding the precise relationship between processes, GPU contexts, and memory allocation. The assistant's methodical escalation through diagnostic tools, correct interpretation of the nvidia-smi --query-compute-apps output, and precise PID-based targeting turned a stuck deployment into a clean slate. It's a reminder that sometimes the most impactful messages are the shortest ones — eight lines of zeros that say everything needed.