The Art of GPU Memory Reclamation: Why Killing Zombie Processes Is Never Simple

The Message

[assistant] Zombie processes again. Let me be more thorough:
[bash] ssh root@10.1.230.174 "ps aux | grep -E 'python3|VLLM|vllm' | grep -v grep | awk '{print \$2}' | xargs kill -9 2>/dev/null; sleep 5; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
0, 0 MiB
1, 0 MiB
2, 0 MiB
3, 0 MiB
4, 0 MiB
5, 0 MiB
6, 0 MiB
7, 0 MiB

At first glance, this message looks like a mundane cleanup operation — a simple shell pipeline that kills lingering Python processes and checks GPU memory. But in the high-stakes world of deploying trillion-parameter language models across eight NVIDIA Blackwell GPUs, this thirty-second interlude represents a critical inflection point. It is the moment when the assistant, having hit a wall with zombie processes that refused to release their grip on GPU memory, pivots from a targeted approach to a scorched-earth strategy. The result — all eight GPUs showing 0 MiB — unlocks the next phase of optimization. This article unpacks the reasoning, assumptions, and technical knowledge embedded in this deceptively simple message.

Context: The Optimization Pipeline

To understand why this message exists, one must understand what came before it. The session had been a marathon of model deployment and benchmarking. The assistant had already:

1. Deployed NVFP4 Kimi-K2.5 — a 540GB model using NVIDIA's FP4 quantization, which achieved ~61 tok/s single-stream but was bottlenecked by PCIe allreduce across 8 GPUs for its 61-layer MLA (Multi-head Latent Attention) architecture.
2. Pivoted to MiniMax-M2.5 FP8 — a 230B-parameter model using GQA (Grouped Query Attention), which loaded in 75 seconds and achieved up to 4,000 tok/s with Expert Parallelism, proving that GQA + smaller active parameters dramatically outperforms MLA on PCIe-bound Blackwell hardware.
3. Downloaded and deployed native INT4 Kimi-K2.5 — a 547GB model using compressed-tensors INT4 quantization, which took 36 minutes to load but delivered an impressive 81.4 tok/s single-stream, well exceeding the user's target of 40–50 tok/s. After confirming the INT4 model worked and benchmarking it, the assistant received a user instruction: "Run benchmarks and try to get to single stream >40~50; If not there try NCCL LL alg and other safe-ish tricks" ([msg 2359]). The assistant had already exceeded the target (81 tok/s), but the user wanted to push further. The natural next step was to restart the vLLM server with different NCCL (NVIDIA Collective Communications Library) settings — specifically the NCCL_PROTO=LL (Low Latency) protocol — to see if allreduce performance could be improved. But restarting vLLM requires freeing GPU memory first. And that is where the trouble began.

The Zombie Problem

In msg [msg 2367], the assistant attempted to kill the running vLLM process:

kill -9 $(pgrep -f 'vllm.*kimi') ... 

But when it checked GPU memory in msg [msg 2368], all eight GPUs still showed 96,733 MiB used — the full model allocation. The vLLM master process had been killed, but worker processes (or CUDA context holders) were still alive, clinging to the GPU memory like zombies in a horror film.

This is a well-known phenomenon in GPU computing. When a CUDA application spawns multiple processes (as vLLM does with tensor parallelism across 8 GPUs), killing the parent process does not always clean up the child processes or their CUDA contexts. The GPU memory remains allocated until every process that holds a CUDA context is terminated. The nvidia-smi output shows memory as "used" even though no visible application appears to be running.

The Decision: "Let Me Be More Thorough"

The key phrase in the message is: "Zombie processes again. Let me be more thorough." This reveals several things about the assistant's reasoning:

1. Recognition of a recurring pattern — The word "again" indicates this is not the first time zombie processes have interfered with GPU memory cleanup. Earlier in the session ([msg 2353]), a similar situation occurred when stopping the MiniMax service left GPU 5 with 96,798 MiB allocated. The assistant had to use fuser to find and kill the processes holding /dev/nvidia* file handles.
2. Learning from past failures — The assistant's previous attempt used pgrep -f 'vllm.*kimi', which only matched processes whose command line contained the string "vllm.*kimi". This was too narrow. The new approach uses ps aux | grep -E 'python3|VLLM|vllm', which catches any process containing "python3", "VLLM", or "vllm" in its command line — a much broader net.
3. Escalation of force — The assistant moves from a surgical strike (targeting only Kimi-specific vLLM processes) to a blanket sweep (killing all Python and vLLM processes). This is a pragmatic trade-off: it risks killing unrelated Python processes, but in a dedicated ML server environment, the only Python processes running are likely vLLM instances.

Technical Analysis of the Command

The shell pipeline used is worth examining in detail:

ps aux | grep -E 'python3|VLLM|vllm' | grep -v grep | awk '{print $2}' | xargs kill -9

Each stage serves a purpose:

• ps aux — lists all running processes with their command lines
• grep -E 'python3|VLLM|vllm' — filters for processes whose command line contains "python3", "VLLM", or "vllm". The case-insensitive "VLLM" catches any process that might have the string in uppercase (some tools or environment variables use this casing)
• grep -v grep — excludes the grep process itself from the kill list (a classic anti-pattern prevention)
• awk '{print $2}' — extracts the PID (process ID) from the second column
• xargs kill -9 — sends SIGKILL to each PID The 2>/dev/null suppresses errors from kill when a process has already died (race condition). After a 5-second sleep to allow the kernel to clean up, the assistant runs nvidia-smi to verify all GPUs are at 0 MiB.

Assumptions and Potential Pitfalls

This approach makes several assumptions:

1. All Python processes are vLLM-related — On a general-purpose server, killing all Python processes would be catastrophic. But the assistant assumes (correctly, in this context) that the only Python processes running are vLLM workers. This assumption is validated by the fact that the machine is a dedicated ML inference server.
2. SIGKILL is safe — Using kill -9 (SIGKILL) is the nuclear option. It does not allow processes to clean up gracefully. For CUDA applications, this can leave GPU hardware in an inconsistent state (though modern drivers handle this reasonably well). The assistant could have tried SIGTERM first, but previous experience showed that gentler approaches failed.
3. 5 seconds is enough for cleanup — After killing processes, the assistant waits 5 seconds before checking GPU memory. This is usually sufficient for the NVIDIA driver to release CUDA contexts, but under heavy load or with certain driver bugs, it might take longer.
4. No other GPU-using processes exist — The command does not check for non-Python GPU processes (e.g., NCCL benchmarks, CUDA samples). If any existed, they would survive the purge and continue holding memory.

Input and Output Knowledge

Input knowledge required to understand this message includes:

• Understanding of CUDA process management and GPU memory allocation
• Familiarity with vLLM's architecture (master process + TP worker processes)
• Knowledge of nvidia-smi memory reporting semantics
• Awareness of the session's history (previous zombie cleanup in [msg 2353])
• Understanding of the NCCL tuning goal that motivates the restart Output knowledge created by this message includes:
• Confirmation that all GPU memory is freed (0 MiB on all 8 GPUs)
• A validated cleanup procedure for this specific environment
• The insight that broad process matching (python3|VLLM|vllm) is more reliable than targeted matching (vllm.*kimi) for this machine
• The practical knowledge that zombie processes are a recurring issue in this deployment

The Broader Significance

This message, for all its brevity, encapsulates a fundamental truth about large-scale ML deployment: the hardest problems are often not the algorithms but the systems engineering. Getting a 547GB model to load and generate coherent text is impressive, but the day-to-day reality of operating such systems involves wrestling with process management, memory leaks, driver quirks, and zombie processes.

The assistant's progression from targeted kill to blanket sweep mirrors a common pattern in systems administration: start with the least destructive approach, observe the outcome, and escalate when necessary. The phrase "Let me be more thorough" signals a conscious decision to abandon precision in favor of reliability — a trade-off that makes sense when the cost of collateral damage (killing an unrelated process) is low and the cost of failure (not freeing GPU memory) is high.

Moreover, this message demonstrates the importance of verification. The assistant does not assume the kill succeeded — it checks with nvidia-smi and reports the result. This feedback loop is essential: without it, the assistant might proceed to launch a new vLLM instance only to encounter an OOM (Out of Memory) error, wasting another 36 minutes of load time.

Conclusion

Message [msg 2369] is a masterclass in practical GPU operations. It shows an AI assistant recognizing a recurring failure pattern, adapting its strategy based on experience, executing a more aggressive cleanup, and verifying the result before proceeding. The 0 MiB readings across all eight GPUs are not just numbers — they represent a cleared runway for the next phase of optimization. In the broader narrative of the session, this message is the bridge between benchmarking the INT4 Kimi-K2.5 model and tuning its NCCL parameters for maximum throughput. It is a reminder that in the world of large-scale ML deployment, victory often goes not to the most brilliant algorithm designer, but to the engineer who can reliably kill a zombie process.