The Art of GPU Cleanup: A Single Kill Command That Freed 775GB

In the middle of an intensive deep-learning profiling campaign on an 8× NVIDIA RTX PRO 6000 Blackwell system, a single message stands out not for its complexity, but for its quiet resolution of a stubborn operational problem. Message 2455 is a one-line bash command executed over SSH, yet it represents the culmination of a multi-step debugging sequence that reveals fundamental truths about GPU memory management in distributed ML inference.

The Message

[assistant] [bash] ssh root@10.1.230.174 'kill -9 244199 2>/dev/null; sleep 3; fuser /dev/nvidia* 2>&1; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'
0, 0 MiB
1, 0 MiB
2, 0 MiB
3, 0 MiB
4, 0 MiB
5, 0 MiB
6, 0 MiB
7, 0 MiB

Eight GPUs, all showing 0 MiB. The 775 gigabytes of HBM2e memory that had been consumed by the Kimi-K2.5 INT4 model (402 GB of weights plus KV cache and activation memory across 8 × 96 GB GDDR7 cards) had been released. The system was clean.

The Context: Why This Cleanup Was Necessary

To understand why this message matters, we must step back. The preceding messages document a comprehensive profiling campaign of the moonshotai/Kimi-K2.5 INT4 model running on vLLM 0.16.0rc2 with tensor parallelism 8 across eight RTX PRO 6000 Blackwell GPUs. The team had just completed a full torch.profiler capture, collecting detailed kernel-level timing data that revealed AllReduce as the dominant bottleneck at 51.5% of decode time ([msg 2447]). The profiler had generated ~300 MB of trace data across all 8 GPU ranks ([msg 2444]).

With the profiling data safely collected and analyzed, the next step was to restart the production service without profiler overhead. But this required first cleaning up the profiler-enabled vLLM instance — and that turned out to be surprisingly difficult.

The Three-Act Drama of GPU Memory Release

The cleanup unfolded across four messages in a classic debugging pattern: attempt, verify, investigate, resolve.

Act 1 — The Blunt Approach ([msg 2452]): The assistant issued a compound kill command targeting all vLLM-related processes. It used pgrep -f "vllm.entrypoints" to find the main server process, piped through kill, then xargs kill -9 for any remaining Python or VLLM processes, and finally used fuser to kill any process holding NVIDIA device files. This was a thorough attempt — it tried graceful termination first (kill without flags), then forced termination (kill -9), then device-level cleanup (fuser-based kill).

Act 2 — The Unexpected Discovery ([msg 2453]): After the kill command, the assistant checked GPU memory usage with nvidia-smi. GPUs 0 and 1 still showed 96,844 MiB each — essentially the full 96 GB GDDR7 capacity. The model was still resident in GPU memory. The kill command had not fully worked.

Act 3 — The Investigation ([msg 2454]): The assistant dug deeper, listing processes matching "python3|VLLM|vllm" (finding none) and checking fuser /dev/nvidia* to see which PID was holding the devices. The answer: PID 244199 was holding /dev/nvidia-uvm, /dev/nvidia0, /dev/nvidia1, /dev/nvidia2, and /dev/nvidia3. This was the zombie — a process that had survived the initial kill attempts, likely because it was a vLLM worker process that didn't match the pgrep pattern or had been spawned after the initial kill.

Resolution ([msg 2455]): A direct kill -9 244199 followed by confirmation that all 8 GPUs showed 0 MiB. The system was clean.

Why Kill -9 Was Necessary

The need for kill -9 (SIGKILL) rather than kill (SIGTERM) reveals something important about vLLM's process architecture. vLLM uses a multi-process model where the main API server spawns separate worker processes — one per GPU or per tensor-parallel rank. These workers communicate through NCCL and shared memory. When the main process is killed, the workers may not automatically terminate because they are independent processes. Moreover, the NVIDIA driver can hold GPU memory allocations even after the owning process has been killed, especially if there are shared memory regions or CUDA IPC handles that haven't been released.

The fuser utility is particularly useful here because it queries the Linux kernel's file descriptor tables to find which processes have opened the NVIDIA device files. The GPU device files (/dev/nvidia0 through /dev/nvidia7, plus /dev/nvidia-uvm for Unified Virtual Memory) are character devices that the CUDA driver uses for communication. Any process with an open handle to these devices can hold GPU memory allocations. The fuser command revealed that PID 244199 had opened handles to GPUs 0 through 3, which explained why those GPUs still showed allocated memory.

Assumptions and Lessons

The assistant made several assumptions during this cleanup sequence. First, it assumed that killing processes matching "vllm.entrypoints" and "python3|VLLM" would cover all relevant processes. This failed because the worker processes may use different command-line patterns or may have been spawned with names that don't match the grep patterns. Second, it assumed that checking only GPUs 0 and 1 with head -2 was sufficient to verify cleanup — a reasonable optimization but one that could miss stragglers on other GPUs. Third, it assumed that kill -9 on the fuser-identified PID would be sufficient without needing to clean up shared memory segments (/dev/shm).

The lesson here is a common one in GPU computing: process termination does not guarantee GPU memory release. The CUDA driver maintains memory allocations until the process's file descriptors are fully closed and the kernel has cleaned up the process's address space. In practice, this means that a multi-step cleanup procedure is often necessary: first try graceful termination, then forced termination, then check for remaining allocations, then use fuser to find stubborn processes, and finally clean up shared memory segments manually.

The Broader Significance

This message, for all its apparent simplicity, represents the boundary between two major phases of work. Before it, the system was in "profiling mode" — running with torch.profiler enabled, collecting detailed kernel traces, and consuming hundreds of gigabytes of GPU memory. After it, the system was a clean slate, ready to be reconfigured for production deployment. The 0 MiB readings across all 8 GPUs are the digital equivalent of a factory reset — a necessary precondition for the next phase of work.

In the context of the broader session, which spanned model deployment, debugging incoherent output, optimizing throughput, and profiling bottlenecks, this cleanup message is the quiet moment of reset between the analysis phase and the action phase. It's the deep breath before the next sprint. And it demonstrates that even in a workflow dominated by high-level concerns like model architecture, kernel optimization, and throughput tuning, the low-level realities of Linux process management and NVIDIA driver behavior remain inescapable.