The Stubborn GPU: A Case Study in Systems-Level Debugging of Multi-GPU Cleanup

In the middle of an intense optimization session for EAGLE-3 speculative decoding on an 8-GPU Kimi-K2.5 deployment, the assistant encounters a stubborn problem: a zombie process that refuses to die. The message at <msg id=4710> is deceptively brief — a single bash command and its output — but it represents a critical turning point in a debugging spiral that reveals the messy reality of GPU memory management in production ML environments.

The Message

The assistant writes:

GPU 1 still holding memory. Let me clean it up:

>

``bash ssh root@10.1.2.6 'pct exec 129 -- bash -c "fuser -k /dev/nvidia* 2>/dev/null; sleep 1; fuser -k /dev/nvidia* 2>/dev/null"' ``

>

Output: 21316 21316 21316 21316 21316 21316 21316 21316 21316 21316

The Context: An 8-Hour Zombie

To understand why this message exists, we must trace the events leading up to it. In the preceding messages, the assistant discovered that a SGLang server configured for 3-step EAGLE-3 speculation had been started approximately eight hours earlier but never completed its initialization. The server had successfully loaded all 64 checkpoint shards of the 1-trillion-parameter Kimi-K2.5 model (consuming ~76 GB per GPU), but then stalled — no "Serving" message appeared in the logs, and the health endpoint returned nothing. The process was alive but unresponsive, a classic zombie state.

The assistant's first attempt at cleanup, at <msg id=4708>, used a targeted approach: ps aux | grep python3 | grep -v grep | awk '{print $2}' | xargs -r kill -9. This killed all Python processes, which should have been sufficient since the SGLang server runs as a Python process. However, when the assistant checked GPU memory usage afterward (at <msg id=4709>), the result was alarming:

0, 51
1, 76115
2, 51
3, 3
4, 3
5, 3
6, 3
7, 3

GPU 1 was still holding 76,115 MiB of memory — essentially the full model allocation. The kill -9 on Python processes had missed something. This is the precise moment that motivates the subject message.

Why fuser -k /dev/nvidia*?

The assistant's choice of fuser -k /dev/nvidia* is a significant escalation in cleanup strategy. The fuser command identifies processes that have specific files or sockets open, and the -k flag sends SIGKILL to those processes. By targeting /dev/nvidia* — the device files for NVIDIA GPUs — the assistant is bypassing process-name-based filtering entirely and going straight to the hardware resource level.

This approach makes a key assumption: that any process still holding GPU memory must have an open file handle on one or more NVIDIA device files. This is a correct assumption for CUDA programs, which open /dev/nvidia0, /dev/nvidia1, etc. to communicate with GPUs. The command is essentially saying: "I don't care what process it is or what it's called — if it's holding a GPU, kill it."

The double invocation (fuser -k ... sleep 1 ... fuser -k) is a defensive pattern. Some processes spawn child processes or re-acquire resources after being killed. The sleep gives the first SIGKILL time to take effect, and the second invocation catches any survivors or descendants.

The Output: A Revealing Fingerprint

The output — 21316 repeated ten times — is remarkably informative. Process ID 21316 had ten file handles open on NVIDIA devices. On an 8-GPU system, each GPU typically has at least one device file (/dev/nvidia0 through /dev/nvidia7), plus the control device /dev/nvidiactl and possibly /dev/nvidia-uvm (Unified Virtual Memory). Ten handles for a single process suggests it had opened each GPU device plus the control and UVM devices — consistent with a CUDA program that initializes all available GPUs, exactly what an 8-way tensor-parallel SGLang server would do.

The fact that process 21316 survived the earlier kill -9 on Python processes is puzzling. Several explanations are possible: the process might have been a subprocess spawned with a different executable name (not containing "python3"), or it could have been a child process that was reparented. Alternatively, the initial grep python3 might have missed it if the process name was truncated or the command line didn't match the pattern. This highlights a common pitfall in process management: filtering by process name is fragile.

Assumptions Made

The assistant makes several assumptions in this message, most of which are sound:

1. That fuser -k /dev/nvidia* will find all GPU-holding processes. This is generally true for CUDA applications, but not universally — some GPU access patterns (e.g., via CUDA IPC or certain container runtimes) might not leave obvious file handles. In this case, it worked.
2. That killing processes via device files is safe. The fuser -k approach is indiscriminate — it kills anything touching NVIDIA devices, including potentially unrelated processes. In a dedicated ML server, this is acceptable, but on a shared system it could be dangerous.
3. That the double-kill pattern is necessary. The assistant assumes that a single fuser -k might not be sufficient, which is a reasonable precaution based on experience with stubborn processes.
4. That GPU memory will be freed after the process is killed. The CUDA driver and NVIDIA kernel module should release GPU memory when a process exits, but in practice, memory leaks and driver bugs can cause allocations to persist. The assistant implicitly trusts the driver to clean up, which it does in this case (as confirmed in the subsequent message at <msg id=4711> where all GPUs show 0 MiB used).

What This Message Reveals About the Debugging Process

This message is a window into the assistant's operational thinking. The assistant is operating in a multi-layered environment: it's running commands on a Proxmox hypervisor host (10.1.2.6) that manages a container with ID 129, which in turn runs the ML workload on the target machine (10.1.230.174). The nested ssh pct exec bash -c chain reflects the reality of managing virtualized GPU infrastructure.

The assistant's debugging strategy follows a clear escalation path:

1. Process-name-based kill (kill -9 on Python processes) — failed because it missed a process.
2. Resource-based kill (fuser -k on NVIDIA devices) — succeeded because it targets the resource itself rather than the process name. This is a textbook example of the "debug by resource, not by name" principle. When a process is hiding from name-based filters, attacking the resource it holds is more reliable.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

• Linux process management: How kill -9, fuser, and process signals work.
• NVIDIA GPU device files: The /dev/nvidia* device nodes and how CUDA programs interact with them.
• Proxmox container management: The pct exec command for running commands inside containers.
• GPU memory persistence: The fact that GPU memory allocations survive process death if the process isn't properly cleaned up.
• The SGLang architecture: That SGLang uses tensor parallelism across 8 GPUs, meaning a single process holds memory on all devices.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

• Confirmation that process 21316 was the culprit: The ten file handles confirm this was the GPU-holding process.
• Validation of the fuser -k approach: It successfully cleared GPU memory when name-based killing failed.
• A reusable debugging pattern: The double fuser -k pattern is now available for future GPU cleanup scenarios.

The Broader Significance

In the context of the overall session, this message is a brief but essential cleanup operation. It clears the way for the assistant to restart the 3-step EAGLE-3 server and run the benchmark that will complete the performance comparison table. Without this step, the zombie server would continue occupying GPU memory, preventing any further work.

The message also illustrates a recurring theme in the session: the gap between idealized performance measurements and real-world deployment. The assistant had previously reported 94 tok/s for EAGLE-3 speculation, but upon re-examination in this segment, that number proves non-reproducible — the stable baseline is actually 82-83 tok/s, and EAGLE-3 delivers only 59-61 tok/s (27% worse). The zombie server was an earlier attempt to run a 3-step configuration that got stuck during CUDA graph capture. This message represents the cleanup of that failed experiment, making way for the sober reassessment of EAGLE-3's viability that dominates the rest of the segment.

In the end, the fuser command succeeds where kill -9 failed, all GPUs are freed, and the assistant can proceed to the next experiment. But the deeper lesson — that GPU memory management in multi-GPU systems requires resource-level thinking, not just process-level thinking — is one that every ML engineer learns the hard way.