The Unkillable Process: Diagnosing Stuck GPU Memory with `fuser`

In the middle of a grueling debugging session to deploy a 402GB GLM-5 GGUF model across 8 NVIDIA Blackwell GPUs, the assistant encountered a peculiar and frustrating problem: the GPU memory would not free up, even after sending a SIGKILL to every vLLM-related process. The subject message — message 1832 in the conversation — is a single, deceptively simple bash tool call that runs the fuser command to identify which processes are holding the NVIDIA device files open. But this message is not merely a routine diagnostic; it represents a critical pivot in the assistant's debugging strategy, a moment when conventional process management failed and a deeper, more forensic approach was required.

The Message

The assistant executed the following command on the remote machine:

ssh root@10.1.230.174 'fuser -v /dev/nvidia* 2>&1 | head -20' 2>&1

The output revealed a surprising situation:

                     USER        PID ACCESS COMMAND
/dev/nvidia-caps:    root     kernel mount /dev/nvidia-caps
/dev/nvidia-uvm:     root     kernel mount /dev/nvidia-uvm
                     root        843 F.... nvtop
                     root      42635 F.... VLLM::EngineCor
                     root      42897 F...m VLLM::Worker_TP
                     root      42898 F...m VLLM::Worker_TP
                     root      42899 F...m VLLM::Worker_TP
                     root      42900 F...m VLLM::Worker_TP
                     ...

Multiple vLLM processes — including VLLM::EngineCore (PID 42635) and several VLLM::Worker_TP processes (PIDs 42897–42900 and beyond) — were still holding file descriptors to the NVIDIA devices, despite the assistant having already attempted to kill them with pkill -9 -f "VLLM\|vllm\|python3.*vllm" in the preceding message ([msg 1831]).

Why This Message Was Written

To understand why the assistant ran fuser, we must trace the chain of events leading up to this moment. The assistant had spent the previous hour loading the GLM-5 GGUF model onto 8 RTX PRO 6000 Blackwell GPUs — a 402GB behemoth that took 25 minutes to load and consumed 51.5 GiB per GPU ([msg 1816]). The model loaded successfully, but during the CUDAGraph warmup phase, the process crashed with a set_stride error originating from DeepGEMM's fp8_paged_mqa_logits function ([msg 1817]). This was a known compatibility issue between DeepGEMM and PyTorch 2.10 on the SM120 (Blackwell) architecture.

After the crash, the assistant attempted to restart the server with --enforce-eager to bypass CUDAGraph capture ([msg 1829]). But when it checked GPU memory usage with nvidia-smi, it found that all 8 GPUs still showed 89,879 MiB allocated ([msg 1830]). The assistant then tried pkill -9 -f "VLLM\|vllm\|python3.*vllm" ([msg 1831]), a forceful kill that should have terminated every matching process. Yet when it checked again, the memory was stubbornly unchanged.

This is the moment message 1832 was born. The assistant faced a contradiction: it had sent the most aggressive possible kill signal (SIGKILL, signal 9) to all vLLM processes, but the GPU memory remained allocated. This could mean either:

The processes were somehow immune to SIGKILL (impossible for user-space processes).
The processes had already died but the GPU memory was leaked (possible with CUDA, but nvidia-smi would eventually reflect freed memory).
The process names didn't match the pkill pattern, leaving some processes alive.
Zombie processes or orphaned children were still holding GPU resources. The fuser command was the natural next step. Unlike ps or pgrep, which check process lists, fuser checks which processes have open file descriptors on specific files or devices. Since NVIDIA GPUs are exposed as device files (/dev/nvidia0, /dev/nvidia1, etc.), fuser -v /dev/nvidia* would reveal exactly which processes were holding the GPU open — regardless of their name, state, or parent process. This is a more reliable diagnostic than pattern-matching process names.## The Reasoning Behind fuser The assistant's choice of fuser over alternative tools reveals a sophisticated understanding of how GPU resources are managed on Linux. When a CUDA application initializes, it opens the NVIDIA device files (/dev/nvidia0, /dev/nvidia1, etc.) and the NVIDIA Unified Memory device (/dev/nvidia-uvm). These file descriptors are the kernel's handle on GPU resource ownership. Even if a process becomes a zombie or is in an unkillable state (e.g., stuck in an uninterruptible sleep in a kernel driver), the file descriptors remain open, and the GPU memory stays allocated. The pkill -9 command had already been tried and failed to free the memory. The assistant could have tried other approaches — checking /proc filesystem entries, using lsof, or simply rebooting the machine — but fuser was the most direct and informative choice. It would tell the assistant not just whether processes were still alive, but which specific processes were holding the devices, with their PIDs and access modes. The output confirmed the assistant's suspicion: the vLLM processes were still alive. The F.... access code in the fuser output indicates that the process has the file open (F) but doesn't have it as its current working directory, root directory, or executable — it's a pure file descriptor hold. The m suffix on some entries (e.g., F...m for Worker_TP processes) indicates that the process has the file memory-mapped, which is typical for CUDA contexts that map GPU memory into the process address space.

Assumptions and Their Validity

The assistant made several assumptions in crafting this message. First, it assumed that the pkill -9 command from the previous message had actually matched the correct processes. The fuser output showed this assumption was incorrect — the processes were still alive. This could be because pkill's pattern matching didn't account for the exact process names as they appear in /proc/pid/cmdline, or because the processes had respawned. The VLLM::EngineCore and VLLM::Worker_TP names shown by fuser are actually the process names set by vLLM using prctl(PR_SET_NAME, ...), which may differ from the command-line pattern pkill was searching for.

Second, the assistant assumed that fuser would be available on the remote system. fuser is part of the psmisc package, which is commonly installed on Linux but not guaranteed on minimal server builds. On Ubuntu 24.04, it is typically present, and the command succeeded.

Third, the assistant assumed that the GPU memory not being freed was caused by alive processes rather than a kernel-level leak or driver bug. The fuser output validated this assumption — the processes were indeed alive and holding the devices.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains. First, an understanding of Linux process management and file descriptors — specifically, that GPU devices are represented as files in /dev/ and that holding them open prevents resource reclamation. Second, familiarity with NVIDIA's CUDA driver model, where each GPU is exposed as a device file and the UVM (Unified Virtual Memory) module manages CPU-GPU memory coherence. Third, knowledge of vLLM's architecture — that it spawns separate worker processes for each GPU in a tensor-parallel configuration, and that the engine core coordinates them. Fourth, understanding of the fuser command itself: its -v (verbose) flag, its output format, and the meaning of access codes like F (file open), m (memory-mapped), and . (current working directory).

The reader also needs context from the preceding messages: the 25-minute model load, the set_stride crash during CUDAGraph warmup, the failed pkill attempts, and the assistant's growing frustration as the GPU memory refused to budge.

Output Knowledge Created

This message produced concrete, actionable knowledge. The assistant now knew:

Exactly which PIDs were holding the GPUs: PID 42635 (VLLM::EngineCore) and PIDs 42897–42900+ (Worker_TP processes). This was more precise than the pattern-matched list from ps.
The access mode: The F flag confirmed these were file descriptor holds, and the m flag confirmed memory-mapped GPU regions — consistent with a CUDA application that has allocated GPU tensors.
That the processes were truly alive: Not zombies, not orphans — they were running processes with valid PID entries in the process table.
The scope of the problem: At least 5 Worker_TP processes plus the EngineCore were still alive, suggesting the entire worker pool was intact and the pkill had missed them entirely.

The Thinking Process

The assistant's reasoning, visible in the sequence of messages, shows a methodical escalation of diagnostic techniques. First, it used nvidia-smi to check memory usage — a high-level tool that reports GPU state. When memory was stuck, it tried pkill -9 — the nuclear option for process termination. When that failed, it recognized that pattern-based killing was unreliable and switched to a file-descriptor-based approach with fuser. This progression from symptom check (nvidia-smi) to blunt intervention (pkill) to forensic investigation (fuser) is characteristic of experienced systems debugging: when the obvious solution fails, you gather more precise data before trying again.

The assistant could have immediately tried kill -9 with explicit PIDs from ps, but that would have required parsing the ps output and manually extracting PIDs — error-prone and slow. The fuser command provided a cleaner, more authoritative answer in a single invocation. It also revealed the process names as set by the application itself (VLLM::EngineCore, VLLM::Worker_TP), which explained why pkill -f "vllm" might have failed — the process name doesn't contain the string "vllm" in lowercase; it uses the uppercase VLLM:: prefix.

Mistakes and Incorrect Assumptions

The primary mistake in this message is not in the message itself but in the assumptions that led to it. The assistant had assumed that pkill -9 -f "VLLM\|vllm\|python3.*vllm" would be comprehensive. The -f flag to pkill matches against the full process command line (from /proc/pid/cmdline), but the vLLM processes might have had their command lines altered or truncated. The fuser output showed the process names as VLLM::EngineCore and VLLM::Worker_TP, which are set via prctl(PR_SET_NAME, ...) — these are the process's "name" as reported by ps -o comm, not the full command line. The pkill -f pattern should have matched the original Python command line (python3 -m vllm.entrypoints.openai.api_server ...), but it's possible that the processes had already been partially cleaned up and the remaining ones were children with different command lines.

Another subtle issue: the fuser output shows only the first 20 lines (head -20), which might truncate the full list of Worker_TP processes. With 8 GPUs and tensor parallelism, there should be 8 worker processes plus the engine core. The output shows PIDs up to 42900, but we can see at least 4 workers listed (42897, 42898, 42899, 42900) — likely more were cut off by head -20. This is a minor limitation of the command, but it doesn't undermine its diagnostic value.

Conclusion

Message 1832 is a masterclass in targeted systems debugging. Faced with a stubbornly stuck GPU memory allocation that survived a SIGKILL barrage, the assistant reached for the right tool for the job: fuser, which interrogates the kernel's file descriptor table rather than relying on process name patterns. The output revealed that the vLLM processes were still alive and holding the GPU devices open, explaining why the memory hadn't been freed. This set the stage for the next step: killing the processes by their explicit PIDs, which the assistant would need to extract from the fuser output.

In the broader context of the session, this message marks the transition from trying to work around the DeepGEMM crash (by restarting with --enforce-eager) to the more fundamental problem of process management and resource cleanup. The assistant would go on to kill these processes by PID, free the GPU memory, and eventually pivot to a different approach for the DSA indexer issue. But this single fuser command — 81 characters of SSH invocation — represents the critical diagnostic insight that made all subsequent progress possible.

The Unkillable Process: Diagnosing Stuck GPU Memory with fuser