The Persistent Ghost: Cleaning Up GPU Memory After a Profiling Campaign

In the middle of an intensive deep-dive profiling session on an 8x NVIDIA RTX PRO 6000 Blackwell system, the assistant encounters an unexpected operational snag. Message [msg 2454] is deceptively brief — just two lines of reasoning and a single bash command — but it represents a critical moment of friction in the workflow: the struggle to reclaim GPU memory after a torch.profiler capture session. This message, in its brevity, encapsulates the messy reality of high-performance GPU inference engineering, where the most sophisticated profiling and optimization work can be thwarted by the mundane challenge of process cleanup.

Context: What Came Before

To understand why this message exists, we must trace the sequence that led to it. The assistant had just completed a comprehensive profiling campaign of the Kimi-K2.5 INT4 model running on 8x RTX PRO 6000 Blackwell GPUs with vLLM 0.16.0rc2. The profiling had been a multi-phase effort spanning macro-benchmarks (single-stream TPOT, multi-concurrency throughput), micro-benchmarks (Marlin W4A16 GEMM operations at exact Kimi dimensions), NCCL AllReduce burst measurements, and a full torch.profiler capture that revealed the shocking finding that AllReduce accounted for 51.5% of decode time — 11.17ms per step, dwarfing the GEMM compute costs.

After gathering this definitive profiling data, the assistant needed to restart vLLM in its normal production configuration (without profiler overhead). In [msg 2452], it issued a comprehensive kill command designed to tear down the profiler-enabled vLLM instance:

ssh root@10.1.230.174 'kill $(pgrep -f "vllm.entrypoints") 2>/dev/null; 
  sleep 3; 
  ps aux | grep -E "python3|VLLM" | grep -v grep | awk "{print \$2}" | xargs kill -9 2>/dev/null; 
  sleep 5; 
  kill -9 $(fuser /dev/nvidia* 2>/dev/null | tr " " "\n" | sort -u | grep -v "^$") 2>/dev/null; 
  sleep 2; 
  nvidia-smi --query-gpu=index,memory.used --format=csv,noheader | head -2'

This was a layered kill strategy: first a gentle SIGTERM to the main vLLM process, then a more aggressive kill -9 sweep of any remaining python3 or VLLM processes, then a nuclear option — force-killing anything holding /dev/nvidia* device files. The final nvidia-smi check was meant to confirm memory was freed.

But when the assistant checked the result in [msg 2453], it found something alarming:

0, 96844 MiB
1, 96844 MiB

Both GPUs still showed 96,844 MiB of memory in use — essentially full allocation. The kill commands had failed to release the GPU memory.

The Subject Message: A Second Attempt

Message [msg 2454] is the assistant's response to this failure. The reasoning is stated plainly: "Still using memory. Let me force-kill everything." The assistant recognizes that the previous kill strategy was insufficient and escalates to a more targeted investigation.

The command issued is:

ssh root@10.1.230.174 'ps aux | grep -E "python3|VLLM|vllm" | grep -v grep; 
  echo "---"; 
  fuser /dev/nvidia* 2>&1 | head -5'

This is a diagnostic-first approach. Rather than blindly firing another kill command, the assistant first checks two things:

  1. Process listing: Are there any remaining python3, VLLM, or vllm processes? The grep -v grep filters out the grep process itself. The empty result before the "---" separator tells us: no matching processes found.
  2. Device file ownership: The fuser /dev/nvidia* command identifies which processes have the NVIDIA device files open. This reveals a single process, PID 244199, holding /dev/nvidia-uvm and /dev/nvidia0 through /dev/nvidia3. The result is puzzling: no python3 or vLLM processes are visible, yet a process (PID 244199) is still holding the GPU devices. This is a classic zombie process scenario — the process name may have changed, or it could be a child process that was orphaned and reparented to init, or a process that was started with a different name than the grep patterns expected.

Assumptions and Their Failure

The assistant made several assumptions in [msg 2452] that proved incorrect:

Assumption 1: That killing by process name would catch all GPU-holding processes. The grep patterns python3|VLLM were based on how vLLM processes typically appear in ps output. But the profiler-enabled vLLM instance may have spawned worker processes with different naming conventions, or the process may have been in a state where its command line was truncated or modified.

Assumption 2: That fuser-based kill would be sufficient. The assistant did include kill -9 $(fuser /dev/nvidia* ...) in the original command, but it appears this either failed silently or the fuser output changed between the two commands. The 2>/dev/null redirection on the fuser kill means any errors were hidden.

Assumption 3: That process cleanup would be straightforward. After a multi-hour profiling session involving CUDAGraph compilation, NCCL initialization, and torch.profiler capture, the GPU state can be left in complex conditions. CUDA contexts, NCCL communicators, and IPC shared memory handles may not be fully released by simple process termination.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. GPU memory management on Linux: Knowledge that GPU memory is tied to process lifetime — when a process that has initialized CUDA is killed, the CUDA driver should release the GPU memory. However, zombie processes or processes that have been reparented can hold memory.
  2. The fuser command: Understanding that fuser /dev/nvidia* identifies which processes have the NVIDIA device files open. On Linux, GPU devices are represented as /dev/nvidia0, /dev/nvidia1, etc., and processes must open these files to interact with the GPU via the CUDA driver.
  3. vLLM process architecture: vLLM with tensor parallelism spawns multiple worker processes (one per GPU), each running as a separate Python process. These appear as VLLM::Worker_TP0, VLLM::Worker_TP1, etc. in process listings.
  4. The profiling context: Understanding that the torch.profiler capture had just completed, and the assistant was trying to restart vLLM without profiler overhead to return to production operation.

Output Knowledge Created

This message produces several pieces of valuable diagnostic information:

  1. Confirmation that process-name-based grep is insufficient for GPU cleanup. The empty grep result combined with the fuser result proves that processes can hold GPU memory without matching expected naming patterns.
  2. PID 244199 as the target: The specific process ID holding the GPUs is identified. This can now be killed directly with kill -9 244199.
  3. Evidence of partial cleanup: The fact that only one process (PID 244199) remains, rather than the expected 8 worker processes + 1 main process, suggests that the earlier kill commands did eliminate most processes, but one survivor remained.
  4. The /dev/nvidia-uvm clue: The process also holds /dev/nvidia-uvm (the Unified Virtual Memory device), which is used for CUDA Unified Memory. This suggests the process had performed CUDA memory allocations that spanned CPU and GPU memory.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern:

Step 1 — Acknowledge failure: "Still using memory." This is an honest admission that the previous approach didn't work. The assistant doesn't try to pretend the cleanup succeeded.

Step 2 — Escalate the response: "Let me force-kill everything." The word "force" signals an escalation from the previous approach. The assistant recognizes that gentler methods have failed.

Step 3 — Diagnose before acting: Rather than immediately firing another kill command, the assistant first runs diagnostics to understand why the previous kills failed. This is a mature debugging approach — gather data before taking action.

Step 4 — Compare two data sources: By running both ps aux (process listing) and fuser (device file ownership), the assistant can cross-reference. The empty grep result tells one story; the fuser result tells another. The discrepancy between them is the key insight.

Step 5 — Identify the survivor: The fuser output reveals PID 244199 as the sole holder of GPU devices. This is the ghost process that needs to be eliminated.

Broader Significance

This message, for all its brevity, illustrates a fundamental truth about GPU inference engineering: the infrastructure around the model — process management, memory cleanup, service orchestration — is just as important as the model architecture and kernel optimizations. The assistant had just completed a world-class profiling campaign, identifying AllReduce as the dominant bottleneck at 51.5% of decode time, measuring Marlin kernel performance, and analyzing NCCL communication patterns. Yet all that sophisticated work was momentarily blocked by a single stubborn process holding GPU memory.

The message also reveals the importance of robust cleanup procedures in multi-GPU environments. A simple kill by process name is not enough; one must trace device file ownership, handle edge cases like orphaned processes, and verify that memory is actually freed (via nvidia-smi). The assistant's layered kill strategy in [msg 2452] was well-designed, but it still failed because of the gap between process naming and device ownership.

In the end, this message is a small but telling moment in a much larger story — the story of pushing a 1-trillion-parameter model to its limits on PCIe-only multi-GPU hardware, where every millisecond counts, and where even the cleanup after a profiling run can become a debugging exercise in its own right.