The Art of GPU Cleanup: A Study in Systematic Debugging of Orphaned CUDA Contexts

In the high-stakes world of large language model deployment on multi-GPU systems, few moments are as deceptively simple — yet as technically rich — as the cleanup of a stubborn GPU memory leak. The message at index 2354 of this opencode session captures one such moment: a single bash command that resolves a problem that three prior attempts could not. On its surface, it reads as a mundane administrative task. But beneath the surface lies a fascinating case study in process management, CUDA memory lifecycle, and the art of escalating debugging strategies.

The Message

The subject message is brief and direct:

GPU 5 still has memory. Let me clean up more aggressively:

>

``bash ssh root@10.1.230.174 "kill -9 \$(fuser /dev/nvidia* 2>/dev/null | tr ' ' '\n' | sort -u | grep -v '^$') 2>/dev/null; sleep 3; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader" ``

>

`` 0, 0 MiB 1, 0 MiB 2, 0 MiB 3, 0 MiB 4, 0 MiB 5, 0 MiB 6, 0 MiB 7, 0 MiB ``

Eight GPUs, all reporting zero memory used. The problem is solved. But the journey to this moment reveals far more than the output suggests.

The Context: A High-Performance Model Deployment Pipeline

To understand why this message matters, we must first understand the environment. The session involves an 8-GPU system equipped with NVIDIA RTX PRO 6000 Blackwell GPUs, each with approximately 96GB of memory. The operator has been on a whirlwind tour of deploying and benchmarking some of the largest open-weight language models available — models exceeding 500GB in size that must be sharded across all eight GPUs using tensor parallelism (TP) and expert parallelism (EP).

The immediate predecessor to this cleanup was the MiniMax-M2.5 model, a 230B-parameter FP8 model that had been running as a systemd service (vllm-minimax-m25.service). That service had achieved impressive results — up to 4,000 tokens per second at high concurrency with TP=8 and EP enabled. But the user had pivoted again, requesting deployment of the native INT4 quantized Kimi-K2.5 model from moonshotai/Kimi-K2.5 — a 547GB behemoth across 64 safetensor shards.

Before the new model could be loaded, the old one had to be fully evicted from GPU memory. This is where the trouble began.

The Problem: Orphaned GPU Contexts

When the assistant issued systemctl stop vllm-minimax-m25 and systemctl disable vllm-minimax-m25 in message 2350, the expected behavior was that all vLLM worker processes would terminate and release their CUDA contexts, freeing GPU memory. But when the assistant checked nvidia-smi afterward, all eight GPUs still showed approximately 96GB of memory allocated — as if the model were still loaded.

This is a well-known phenomenon in CUDA-based inference serving. vLLM, like many high-performance serving frameworks, spawns multiple worker processes — one per GPU, plus a controller process. When the main process is terminated via systemd, the worker processes can become orphaned if they don't receive the termination signal, or if they fail to clean up their CUDA contexts during shutdown. The GPU memory allocated via cudaMalloc is tied to the lifetime of the process that allocated it, but if that process becomes a zombie or is reparented to init, the memory can persist long after the service is "stopped."

The assistant's initial investigation in message 2352 confirmed this diagnosis: pgrep -af vllm returned no results, yet nvidia-smi still showed memory allocated. The fuser -v /dev/nvidia* command revealed the culprits — a set of zombie process IDs still holding the NVIDIA device files open. These were the orphaned vLLM workers.

The Escalation of Debugging Strategy

The assistant's approach to this problem follows a classic debugging escalation pattern — a textbook example of how a skilled operator narrows down a stubborn issue:

Attempt 1 (Message 2350): The polite approach. systemctl stop and systemctl disable were issued, which should have sent SIGTERM to the main process and allowed clean shutdown. This failed — the GPU memory remained allocated.

Attempt 2 (Message 2351): Verification. The assistant checked for remaining vLLM processes with pgrep -af vllm and pgrep -af 'python3.*api_server'. Nothing was found. This was a crucial diagnostic step: it confirmed that the main processes were indeed gone, but the GPU memory was still held by orphaned children.

Attempt 3 (Message 2352): Investigation via fuser. The assistant ran fuser -v /dev/nvidia* to identify exactly which processes held the NVIDIA device files open. The output revealed a set of PIDs (215004, 233458, 233723-233730) that were still holding the GPU resources. These were the zombie worker processes.

Attempt 4 (Message 2353): Targeted killing. Armed with the specific PIDs, the assistant issued kill -9 to each one. After waiting 3 seconds, nvidia-smi showed most GPUs freed — but GPU 5 stubbornly showed 96,798 MiB still allocated. This partial success was informative: it revealed that some processes had been missed, or that a process had spawned a child that wasn't captured by fuser.

Attempt 5 (Message 2354 — the subject): The aggressive, comprehensive approach. Rather than chasing individual PIDs, the assistant used fuser dynamically within the kill command itself: kill -9 $(fuser /dev/nvidia* 2>/dev/null | tr ' ' '\n' | sort -u | grep -v '^$'). This pipeline finds all processes holding any NVIDIA device file open, transforms the space-separated output into newline-separated PIDs, deduplicates them, filters out empty lines, and kills them all in one shot.

This fifth attempt succeeded. All eight GPUs showed 0 MiB.

Assumptions and Their Implications

Several assumptions underpin this message, and examining them reveals the operator's mental model:

Assumption 1: The problem is purely process-based. The assistant assumed that GPU memory would be released once all processes holding NVIDIA device files were killed. This is correct in the CUDA programming model — when a process terminates (especially via SIGKILL), the CUDA driver cleans up all contexts associated with that process. However, this assumption can fail if there are kernel module-level issues, persistent GPU memory allocations via cudaMallocManaged with cudaMemAttachGlobal, or if processes are stuck in an unkillable state (D-state). None of those edge cases applied here.

Assumption 2: fuser captures all relevant processes. The fuser command queries which PIDs have file descriptors open on the specified device files. This is reliable for /dev/nvidia* devices because CUDA processes open these files to interact with the GPU. However, fuser can miss processes that have closed their device files but still have CUDA contexts alive through other mechanisms. In practice, this is rare.

Assumption 3: GPU 5's persistence was due to a missed process. The assistant concluded that because GPUs 0-4 and 6-7 were freed after the targeted kill, the remaining memory on GPU 5 must be held by a process that wasn't in the original fuser list. This could happen if a process opened the device file after fuser was run, or if a child process inherited the CUDA context without explicitly opening the device file. The comprehensive fuser-based kill in message 2354 caught whatever was missed.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

CUDA memory management: Understanding that GPU memory allocations are tied to process lifetime, and that nvidia-smi reports memory allocated by all processes with active CUDA contexts on a device.

Linux process management: Familiarity with fuser, kill -9 (SIGKILL), process states (zombie, orphaned), and the /dev/nvidia* device file interface.

vLLM architecture: Knowledge that vLLM uses a distributed serving architecture with one worker process per GPU, and that these workers can become orphaned if the controller process is terminated without propagating the signal.

Systemd service management: Understanding that systemctl stop sends SIGTERM to the main process but may not reach all child processes, especially if they are in separate process groups or if the service uses KillMode=process instead of KillMode=control-group.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

  1. A working cleanup procedure: The fuser-based kill command is a reliable, repeatable method for forcefully evicting all CUDA contexts from all GPUs on a system. This is now documented implicitly in the conversation history.
  2. Confirmation of the system's health: After the cleanup, all eight GPUs reported 0 MiB used, confirming that there were no persistent memory leaks, no kernel module issues, and no hardware problems. The system was ready for the next model load.
  3. A diagnostic pattern: The escalation from systemctl stop → pgrep → fuser → targeted kill → comprehensive kill provides a reusable template for debugging similar GPU memory retention issues.

The Broader Significance

This message, for all its brevity, represents a critical transition point in the session. The assistant had just completed an extensive benchmarking campaign of the MiniMax-M2.5 model, achieving nearly 4,000 tokens per second. The user had requested a pivot to the native INT4 Kimi-K2.5 — a fundamentally different model with different quantization, different architecture support, and different performance characteristics. Before that pivot could happen, the system had to be in a clean state.

The cleanup of GPU 5's stubborn memory allocation was the last obstacle before the Kimi-K2.5 INT4 deployment could begin. In the messages that follow (not shown in this analysis), the assistant would go on to load the 547GB model, achieve 82 tok/s single-stream throughput, and deploy it as a production systemd service. But none of that would have been possible without this seemingly mundane cleanup step.

This is the hidden drama of large-scale ML infrastructure: the most critical operations are often the most prosaic. A forgotten process, a missed SIGKILL, a single GPU holding 96GB of orphaned memory — these are the gremlins that can derail an entire deployment. The assistant's methodical escalation from polite service stop to aggressive process sweep demonstrates the kind of systematic thinking that separates reliable infrastructure from fragile setups.

Conclusion

Message 2354 is a masterclass in targeted system administration. It shows how a well-constructed one-liner can solve a problem that resisted multiple previous attempts, and it illustrates the importance of understanding the full lifecycle of GPU memory — from allocation by vLLM workers to cleanup via process termination. The command kill -9 $(fuser /dev/nvidia* ...) is not just a piece of bash; it is the culmination of a debugging process that diagnosed orphaned processes, verified their identities, escalated from targeted to comprehensive force, and ultimately delivered a clean slate for the next phase of model deployment. In the high-wire act of managing 8-GPU inference servers, sometimes the most valuable tool is knowing exactly which processes to kill and how to find them.