The Art of GPU Cleanup: Why fuser -k /dev/nvidia* Matters

A Single Command That Speaks Volumes

In the middle of a high-stakes machine learning deployment session, after diagnosing a server crash caused by a missing Python attribute, the assistant issued this deceptively simple command:

[bash] ssh root@10.1.230.174 'fuser -k /dev/nvidia* 2>/dev/null; sleep 2; nvidia-smi | grep -c "No running processes"'

The output returned:

 181432 181432 181432 181432 181432 181432 181432 181432 181432 1814321

On its surface, this looks like a routine cleanup step — kill some processes, check that GPUs are free. But this message sits at a critical inflection point in a much larger story: the transition from experimental benchmarking to hardened production deployment of a 250-billion-parameter language model (Kimi-K2.5 INT4) running across eight NVIDIA RTX PRO 6000 Blackwell GPUs. Understanding why this command was written, what it accomplishes, and what it reveals about the operational realities of large-scale ML inference reveals a depth of engineering discipline that is easy to overlook.

The Context: A Server Crash and a Missing Attribute

To understand this message, we must trace back through the preceding conversation. The assistant had been running extensive benchmarks comparing EAGLE-3 speculative decoding against baseline throughput on a Kimi-K2.5 model. After finalizing an optimal configuration (EAGLE-3 with topk=1 and spec_v2 overlap scheduling), the server was launched — and promptly crashed on the first decode request.

The root cause was a subtle bug: the assistant had added a "dynamic speculation disable" feature that referenced self.spec_disable_batch_threshold in the forward_batch_generation method of EAGLEWorkerV2, but this attribute was initialized later in __init__ — after several initialization calls (init_attention_backend(), init_cuda_graphs()) that could throw exceptions. If any of those calls failed, the __init__ method would abort before reaching the attribute assignment, leaving the object in a partially initialized state. The server would start up (logging its configuration), but the first actual decode would trigger an AttributeError because the attribute simply didn't exist on the object.

The assistant spent several messages (msg 5618–5631) debugging this, examining the traceback, checking whether the right file was being imported, and eventually fixing it by moving the attribute initialization to the very top of __init__, before any code that could fail. This is a classic Python anti-pattern: relying on instance attributes that are set late in a constructor that can fail partway through.

With the fix applied, the next step was to clean up the mess left by the crashed server and restart.

Breaking Down the Command

The command executed in message 5633 is a two-part pipeline wrapped in an SSH call:

Part 1: fuser -k /dev/nvidia* 2>/dev/null

fuser is a Unix utility that identifies processes using specified files or sockets. The -k flag sends SIGKILL to those processes. The pattern /dev/nvidia* matches the NVIDIA device files — typically /dev/nvidia0, /dev/nvidia1, etc. (one per GPU), plus /dev/nvidiactl and /dev/nvidia-uvm. By killing any process holding these files open, this command forcibly terminates any leftover processes that still have GPU resources allocated.

The 2>/dev/null suppresses stderr, which would otherwise show error messages for device files that don't exist or for processes that can't be killed.

Part 2: sleep 2; nvidia-smi | grep -c "No running processes"

After a two-second pause (to allow the kernel to clean up and release GPU memory), the assistant runs nvidia-smi and pipes it through grep -c "No running processes". When no processes are using a GPU, nvidia-smi outputs the line "No running processes found" for each GPU. The -c flag counts matching lines, so if all eight GPUs are free, the output would be 8. This provides a quick sanity check that the cleanup was successful.

Interpreting the Output

The output 181432 181432 181432 181432 181432 181432 181432 181432 181432 1814321 comes from fuser -k, which prints the PIDs of killed processes. The pattern reveals that PID 181432 was using nine different NVIDIA device files (appearing nine times), while PID 1814321 (or possibly the same PID with a trailing digit artifact) was using one. This is consistent with a multi-GPU deployment: a single server process would open all GPU device files, and fuser lists the PID once per matching file.

Notably, the output does not show the result of the nvidia-smi check. In the context of the conversation, the assistant likely read this output and proceeded to the next step (restarting the server) without further comment — the fuser output itself confirmed that processes were killed, and the absence of error output from the nvidia-smi check (or its result being consumed silently) indicated success.

Why This Matters: The Hidden Complexity of GPU State Management

This command addresses a problem that is surprisingly subtle and often underestimated: GPU state is sticky. When a process crashes while holding GPU memory allocations, those allocations are not automatically released in all cases. The NVIDIA driver maintains per-process state, and if a process terminates abnormally (especially if it was in the middle of a CUDA operation), the driver can take time to clean up, or in some cases, the memory remains allocated until something explicitly closes the device files.

Simply killing the Python process (as the assistant did in msg 5632) is not always sufficient. The process might have child processes, or the CUDA runtime might have spawned helper threads that persist. The fuser -k approach is more aggressive: it targets any process holding any NVIDIA device file open, regardless of parent-child relationships.

This is particularly important in a multi-GPU setup with speculative decoding, where the server spawns multiple worker processes (one per GPU, plus draft model workers). A crash can leave a constellation of zombie processes, each holding GPU memory. If the server is restarted without proper cleanup, the new processes may fail to allocate memory, or worse, silently share memory with dead processes, leading to corruption.

Assumptions and Risks

The command makes several assumptions. First, it assumes that all processes holding NVIDIA device files should be killed. This is safe in a dedicated ML server, but would be catastrophic on a shared machine where other users or services might be running GPU workloads. The assistant is operating on a dedicated machine (root access, IP 10.1.230.174), so this assumption is valid.

Second, it assumes that fuser -k with SIGKILL is safe for GPU processes. SIGKILL cannot be caught or ignored, so processes die immediately. However, this can leave GPU hardware in an inconsistent state if a process was in the middle of a DMA transfer or kernel execution. The NVIDIA driver is designed to handle this (it resets the GPU context), but there is a small risk of requiring a GPU reset or even a machine reboot in extreme cases.

Third, the 2>/dev/null suppression of errors means that if some device files don't exist (e.g., if the nvidia-uvm module isn't loaded), the command proceeds silently without warning. This is acceptable for a routine cleanup but could mask configuration issues.

Fourth, the grep -c "No running processes" check is a heuristic, not a guarantee. It confirms that nvidia-smi reports no processes, but there can be edge cases where the driver reports processes as gone while memory is still pinned, or where nvidia-smi itself fails to enumerate all GPU state.

Input and Output Knowledge

The input knowledge required to understand this message includes:

The Thinking Process Visible in the Reasoning

The assistant's thinking, visible across the preceding messages, shows a systematic debugging approach. When the server crashed, the assistant didn't immediately restart — it examined the traceback, traced the attribute access through the code, verified the file was being imported correctly, considered edge cases like partial __init__ failures, and only then applied the fix. The fuser -k command represents the final operational step: clean up the wreckage before trying again.

The choice of fuser -k over a simple killall python3 or pkill -f sglang is telling. The assistant could have just killed all Python processes, but instead chose a more precise approach: kill only processes holding GPU resources. This reflects an understanding that not all Python processes are GPU processes, and that the crashed server might have left behind non-Python helper processes (CUDA runtime threads, NCCL watchdog processes) that wouldn't be caught by a Python-specific kill.

The two-second sleep before the nvidia-smi check also reveals operational experience. GPU memory cleanup is not instantaneous — the kernel needs time to process the SIGKILL, the driver needs to reset GPU contexts, and the memory manager needs to reclaim allocations. A brief pause avoids a false negative from the check.

Broader Significance

This message, for all its brevity, captures a moment of operational maturity. The session had moved from experimental benchmarking (where a server crash means "fix the bug and rerun") to production deployment (where a server crash means "fix the bug, clean up thoroughly, verify state, and restart cleanly"). The fuser -k command is the hallmark of someone who has been burned by GPU state leaks before — who knows that kill -9 on a Python process is not enough, that GPU memory is a persistent resource that must be explicitly reclaimed.

In the broader narrative of this coding session, message 5633 is the bridge between debugging and deployment. The attribute fix (msg 5631) was the intellectual work; the fuser -k command (msg 5633) was the operational work. Both are necessary, and the article's subject — this single command — embodies the principle that in ML engineering, the gap between "the code is fixed" and "the system is running" is filled with small, critical actions that are easy to overlook but impossible to skip.