The Fuser That Freed the GPUs: A Surgical Cleanup in the Service of Model Rotation

In the sprawling, multi-month journey of deploying and optimizing large language models on a cluster of NVIDIA RTX PRO 6000 Blackwell GPUs, most messages are about grand architectural decisions: splitting GPU topology across NUMA domains, patching SGLang for SM120 support, debugging NCCL P2P corruption under SEV-SNP IOMMU, or benchmarking throughput at high concurrency. But sometimes the most telling messages are the smallest ones — the quiet, almost invisible commands that reveal how an operator thinks about state, resources, and the boundary between software and hardware.

Message [msg 6094] is exactly such a message. It consists of a single bash command executed by the AI assistant:

[assistant] [bash] ssh root@10.1.2.6 'pct exec 129 -- bash -c "fuser -k /dev/nvidia* 2>/dev/null; echo freed"' freed

To the uninitiated, this looks like a routine cleanup. But in the context of the session — a high-stakes production environment where 8× Blackwell GPUs are being split between an LXC container running SGLang and a VM for SEV-SNP passthrough — this command represents a critical moment of state management. It is the final, definitive act of severing a running system's grip on hardware resources before those resources can be repurposed.

The Chain of Events That Led Here

Understanding why this message exists requires tracing backward through the conversation. The user had just decided to retire the /data volume on the LXC container (CT 129), which held the Qwen3.5-397B-A17B-NVFP4 model. The model was described by the user as "actually very low quality" ([msg 6091]), and the plan was to replace it with a similarly sized model on a different volume. The assistant had already:

1. Stopped and disabled the SGLang service ([msg 6092]) via systemctl stop sglang-qwen and systemctl disable sglang-qwen.
2. Killed Python processes ([msg 6093]) by running ps aux | grep python3 | awk '{print $2}' | xargs -r kill -9 inside the container via pct exec 129. But the assistant recognized that killing processes by name was not sufficient. The SGLang server, which had been serving the Qwen model across 4 GPUs with tensor parallelism, had allocated GPU memory, opened CUDA contexts, and held file handles to NVIDIA device files. Simply killing the main Python process might leave orphaned subprocesses, lingering CUDA contexts, or zombie processes that still held references to GPU resources. More critically, the NVIDIA device files (/dev/nvidia0, /dev/nvidia1, /dev/nvidia2, /dev/nvidia3, /dev/nvidiactl, /dev/nvidia-uvm, etc.) could still be held open by processes that didn't match the "python3" grep pattern.

What the Command Actually Does

The command fuser -k /dev/nvidia* is a masterclass in pragmatic Unix resource management:

• fuser identifies processes that have opened files or sockets matching the given paths.
• -k sends SIGKILL to every process identified.
• /dev/nvidia* is a glob that matches all NVIDIA device files — the compute devices (nvidia0 through nvidia3), the control device (nvidiactl), the UVM (Unified Virtual Memory) device (nvidia-uvm), and any others.
• 2>/dev/null suppresses errors from the glob if no files match or if fuser encounters issues.
• echo freed provides a clean confirmation that the command ran, regardless of whether any processes were actually killed. The result freed confirms execution. It does not tell us how many processes were killed — only that the command completed. The 2>/dev/null redaction means that if fuser found no matching processes and printed an error or a "no processes found" message, it would be hidden. But the important thing is that the operation was attempted.

Why This Matters: The Semantics of GPU Device Files

The /dev/nvidia* device files are the kernel-level interface to NVIDIA GPUs. When a CUDA program initializes, it opens these device files to establish a context. The CUDA runtime library (libcuda.so) manages these file descriptors, and they persist for the lifetime of the process — or until the process is killed. Critically, even if the main Python process is killed, child processes or threads that have their own CUDA contexts may continue holding these files open.

In the context of an LXC container, where GPU devices are passed through from the host using cgroup device permissions, the container's processes interact with these device files directly. The fuser -k approach is more thorough than process-name matching because it operates on the resource itself: instead of saying "kill anything named python3," it says "kill anything touching the GPUs." This is the difference between targeting a specific application and targeting a resource class.

Assumptions and Reasoning

The assistant made several implicit assumptions in choosing this approach:

Assumption 1: Any process holding NVIDIA device handles is unwanted. This is a reasonable assumption given the context — the SGLang server has been stopped and disabled, and the goal is to cleanly remove the model files and eventually unmount /data. There is no legitimate reason for any process in the container to still hold GPU resources.

Assumption 2: fuser is available in the container. fuser is part of the psmisc package, which is standard on virtually all Linux distributions. The assistant did not check for its presence before running the command.

Assumption 3: Killing processes holding /dev/nvidia* handles is safe. This is generally true for GPU compute workloads — the processes are either the SGLang server (which we want to kill) or system processes that shouldn't be holding GPU device files. However, there is a subtle risk: if the NVIDIA Persistence Daemon (nvidia-persistenced) or the NVIDIA UVM driver helper processes have opened these files, killing them could cause system-level instability. The assistant implicitly trusts that only application-level processes are holding the handles.

Assumption 4: The Proxmox host (10.1.2.6) has pct access to the container. The command is executed via SSH to the Proxmox host, then pct exec 129 runs inside the container. This assumes the Proxmox host is reachable and has the necessary privileges.

The Thinking Process Visible in the Sequence

The assistant's reasoning unfolds across three messages in a clear escalation of cleanup rigor:

1. [msg 6092]: "Let me stop the server and clean up /data." This is the high-level plan. The assistant stops and disables the systemd service.
2. [msg 6093]: Kill Python processes by name. This is the standard approach — find processes matching "python3" and SIGKILL them. But this is incomplete: it misses processes whose names don't contain "python3" (e.g., if the binary is renamed, or if it's a shell script that spawns Python).
3. [msg 6094]: Kill by resource handle. This is the escalation. The assistant realizes that name-based matching is insufficient and switches to resource-based matching. The fuser -k /dev/nvidia* command catches everything that the grep missed. This three-step pattern — plan, execute standard cleanup, then escalate to thorough cleanup — reveals a sophisticated understanding of process management. The assistant doesn't assume the first attempt was sufficient; it double-checks by targeting the resource itself.

Input Knowledge Required

To understand this message, one needs:

• Knowledge of Linux device files: Understanding that /dev/nvidia* represents the GPU device interface and that processes must open these files to use the GPU.
• Knowledge of fuser: Understanding that fuser identifies processes by open files, and -k sends SIGKILL.
• Knowledge of LXC and Proxmox: Understanding that pct exec 129 runs a command inside LXC container 129 from the Proxmox host, and that GPU passthrough in LXC involves mounting /dev/nvidia* devices.
• Knowledge of CUDA runtime semantics: Understanding that GPU contexts are tied to file descriptors on /dev/nvidia* devices, and that killing a process releases those contexts.
• Knowledge of the session context: Understanding that the SGLang server was using 4 GPUs with tensor parallelism, that the model was being removed, and that /data was being retired.

Output Knowledge Created

This message produces several concrete outcomes:

• All processes holding GPU device handles are killed. This ensures no process has an open CUDA context, no GPU memory is allocated, and no file handles prevent the model files from being deleted or the filesystem from being unmounted.
• The GPU devices are "freed" in the operational sense — they are no longer bound to any process in the container.
• A precondition for the next step is satisfied. In the immediately following message ([msg 6095]), the assistant deletes the Qwen model files and other deletable data from /data. This deletion would have been risky or impossible if a process still had the model loaded in GPU memory.

Potential Mistakes and Edge Cases

While the command is effective, there are considerations worth noting:

Overkill risk: If the NVIDIA Persistence Daemon or other system-level NVIDIA processes had opened /dev/nvidia* files inside the container, they would have been killed too. In practice, these daemons typically run on the host, not inside LXC containers, so this risk is minimal.

Silent failure: The 2>/dev/null redaction means that if fuser failed for any reason (e.g., the glob didn't match any files because the container's GPU device files were named differently), the assistant would see "freed" and assume success. However, the previous command ([msg 6093]) confirmed that the container had GPU access, so this is unlikely.

No verification: The assistant does not verify that the GPUs are actually freed (e.g., by running nvidia-smi to check for zero GPU memory usage). It trusts the fuser result and moves on. In a production environment, a verification step might be warranted.

Conclusion

Message [msg 6094] is a small but revealing artifact of a larger operational philosophy. It demonstrates that effective system management is not just about issuing the right high-level commands but about understanding the resource model deeply enough to know when the obvious cleanup (killing by process name) is insufficient. The assistant's decision to escalate from name-based to resource-based process termination shows an awareness that in complex GPU-accelerated environments, the connection between a process and a hardware resource is not always captured by a simple ps aux grep.

The fuser -k /dev/nvidia* command is, in essence, a declaration of intent: "I don't care what's using these GPUs — I want them back." It is the software equivalent of pulling the plug, and in the context of retiring a 223 GB model and preparing to unmount a 12 TB volume, it is exactly the right tool for the job.