The Persistence of Memory: Debugging Zombie GPU Processes in an ML Inference Pipeline

In the middle of an intense optimization session for a large-scale inference pipeline, a single short message captures one of the most frustrating and recurring challenges in GPU-accelerated machine learning: zombie processes that refuse to release their grip on GPU memory. Message [msg 3896] reads:

[assistant] Zombies again. [bash] ssh root@10.1.2.6 'pct exec 129 -- bash -c "kill -9 \$(fuser /dev/nvidia* 2>/dev/null) 2>/dev/null; sleep 2"; pct exec 129 -- nvidia-smi --query-gpu=index,memory.used --format=csv,noheader' 0, 93781 MiB 1, 93789 MiB 2, 93789 MiB 3, 93789 MiB 4, 93789 MiB 5, 93789 MiB 6, 93789 MiB 7, 93701 MiB

This message, though brief, is a window into the gritty reality of production ML engineering — where the elegant mathematics of transformer architectures meets the messy world of Linux process management, GPU memory allocation, and the stubborn refusal of hardware resources to be cleanly released.

The Context: A Server Restart Gone Wrong

To understand why this message was written, we must step back into the broader narrative. The assistant had been engaged in a multi-hour effort to optimize the throughput of an SGLang inference server serving a Kimi-K2.5 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The goal was to generate synthetic training data for an EAGLE-3 speculative decoding drafter — a pipeline that required running tens of thousands of inference requests through the server.

The optimization journey had already been eventful. Earlier, the assistant had configured the server with --mem-fraction-static 0.93, a setting that allocates 93% of GPU memory to the model and KV cache. This proved too aggressive — the server crashed with an out-of-memory (OOM) error during a large prefill batch, leaving only 0.82 GB of headroom per GPU. The assistant then backed off to --mem-fraction-static 0.88 and added hierarchical cache (--enable-hierarchical-cache --hicache-size 48), which gave 159K GPU tokens and a healthy 5.47 GB headroom per GPU. The server was humming along at 930–1350 tokens per second.

But the assistant wanted more. In [msg 3893], the assistant diagnosed that the real bottleneck was GPU KV cache capacity limiting concurrent decode requests. With 159K tokens and ~4K average tokens per request, only ~40 concurrent decodes could fit. The solution identified was KV cache quantization: switching from bf16 to FP8 (--kv-cache-dtype fp8_e4m3) would roughly double the effective KV capacity to ~318K tokens.

In [msg 3894], the assistant initiated the restart: it ran pkill -f run_inference; pkill -9 -f sglang; pkill -9 python3 followed by fuser -k /dev/nvidia* to clear GPU processes. Then in [msg 3895], it checked the result — and found that all eight GPUs were still showing approximately 93,781 MiB of memory used. The kill commands had not worked. The processes were still holding the devices.

"Zombies Again": Recognizing a Recurring Pattern

The opening words of the subject message — "Zombies again." — reveal a critical piece of the assistant's reasoning process. This is not a first encounter. The assistant has dealt with this pattern before in this very session (and likely in earlier segments of the conversation). The term "zombies" refers to processes that have been killed or have terminated but whose resources — particularly GPU memory allocations — have not been released back to the system. In Linux, a true zombie process is one that has exited but still has an entry in the process table because its parent has not called wait(). But in GPU computing, the term is used more loosely to describe any situation where GPU memory remains allocated after the owning process has been terminated.

The assistant's choice to say "again" rather than investigating from scratch shows an important aspect of the reasoning: pattern recognition. Rather than spending time diagnosing why the GPU memory was still allocated, the assistant immediately recognized the symptom and reached for a known solution — escalating to a more aggressive kill strategy through the Proxmox hypervisor.

The Decision to Escalate: Going Through the Hypervisor

The bash command in the message reveals a key decision: instead of running the kill command directly on the worker machine (which had been the target of all previous operations, at IP 10.1.230.174), the assistant instead SSH'd to a different host (10.1.2.6) and used pct exec 129 to execute commands inside a Proxmox container. This is a significant escalation in the kill strategy.

Why this decision? The assistant had already tried pkill -9 -f sglang, pkill -9 python3, and fuser -k /dev/nvidia* directly on the worker machine in [msg 3894]. All had failed to release GPU memory. The likely diagnosis was that the processes had become orphaned or detached from the terminal session in a way that made them resistant to signals sent from within the same machine. By going through the Proxmox host, the assistant could potentially reach processes that were stuck in an unkillable state — for example, processes in the D (uninterruptible sleep) state, or processes whose parent had been reaped by init but which were still holding file descriptors to NVIDIA device files.

The command structure is also notable: kill -9 $(fuser /dev/nvidia* 2>/dev/null). The fuser command identifies which processes have file descriptors open to the NVIDIA device files (/dev/nvidia0, /dev/nvidia1, etc.). By killing those specific processes with SIGKILL (-9), the assistant hoped to force-release the GPU memory. This is a more targeted approach than pkill -f python3, which kills all Python processes regardless of whether they're actually holding GPU resources.

The Assumption That Failed

Despite this escalation, the command did not work. The nvidia-smi output shows that all eight GPUs still report approximately 93,781 MiB of memory used — essentially the full memory capacity of the RTX PRO 6000 Blackwell GPUs (which have 96 GB each). The kill command either failed to find the right processes, or the processes it killed were not the ones holding the GPU memory, or the GPU memory release mechanism itself was stuck.

This reveals an incorrect assumption: that killing the processes holding file descriptors to /dev/nvidia* would be sufficient to release GPU memory. In practice, GPU memory allocation in CUDA involves multiple layers of state — the CUDA driver, the GPU kernel mode driver, and the NVIDIA management library (NVML) — and a process crash or kill can sometimes leave GPU memory in a "stuck" state that requires a full GPU reset or even a system reboot to clear. The fuser approach assumes a clean one-to-one mapping between file descriptors and memory allocations, but the reality is more complex.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

Linux process management: Understanding of signals (SIGKILL), process states, orphan processes, and the fuser command. The distinction between killing a process and releasing its resources is crucial.

GPU architecture: Knowledge that NVIDIA GPUs are accessed through device files (/dev/nvidia*), that CUDA runtime allocates memory on the device, and that GPU memory persists until explicitly freed or the allocating process terminates. The fact that nvidia-smi reports memory usage even after kill attempts indicates the memory is still "pinned."

Proxmox virtualization: Understanding that pct exec 129 executes a command inside a Proxmox container (container ID 129), which provides a different process namespace and may have different visibility into running processes.

The broader session context: The assistant's previous attempts to kill processes, the history of OOM crashes, and the specific goal of restarting the server with FP8 KV cache quantization.

Output Knowledge Created

This message creates several important pieces of knowledge:

The kill attempt failed: The GPU memory remained allocated despite the escalated kill command. This is recorded in the conversation history and informs the next steps.
The pattern is recognized: The assistant now knows this is a recurring issue ("Zombies again") and can respond more quickly in future occurrences.
The Proxmox approach is insufficient: Going through the hypervisor with fuser -k did not solve the problem, ruling out one potential solution path.
GPU memory is stubbornly held: The fact that all eight GPUs show nearly identical memory usage (~93,781 MiB) suggests the memory is not being gradually freed but is held in a consistent state — possibly by a process that is itself stuck.

The Aftermath: What Followed

The subsequent messages show the continued struggle. In [msg 3897], the assistant tries yet another approach: ps aux | grep python3 | grep -v grep | awk '{print $2}' | xargs -r kill -9 combined with fuser -k /dev/nvidia*. This is a more thorough sweep — killing all Python processes by PID regardless of whether they appear in fuser output.

In [msg 3898], the results show progress: most GPUs are now free (0 or 1 MiB), but GPU 4 still shows 93,699 MiB used. One stubborn process remains.

In [msg 3899], the assistant targets GPU 4 specifically: fuser -k -9 /dev/nvidia4. This finally works — all GPUs show 0 MiB.

The entire sequence — from the first kill attempt in [msg 3894] to the final successful release in [msg 3899] — spans six messages and multiple kill strategies. The subject message represents the first escalation point, where the assistant recognized the zombie pattern and escalated through the hypervisor, but the solution was not yet complete.

The Deeper Lesson: GPU Memory Management in Practice

This message, for all its brevity, illustrates a fundamental truth about large-scale ML engineering: the software stack is not designed for clean teardown. In a perfect world, killing a process would immediately release all its resources. In practice, GPU memory management involves multiple layers of abstraction — the PyTorch allocator, the CUDA runtime allocator, the NVIDIA driver — and each layer can cache or hold memory in ways that are opaque to the user.

The "zombie" pattern is so common in GPU computing that experienced engineers develop an instinct for it. The assistant's immediate recognition ("Zombies again") and escalation through the hypervisor reflects this instinct. But even with that experience, the first escalation didn't work. It took three more attempts and a targeted strike on a single GPU to finally clear the memory.

This is the reality of production ML: the elegant mathematics of attention mechanisms and transformer architectures runs on top of a messy, stubborn, and sometimes uncooperative hardware stack. Messages like this one — a single line of frustration followed by a failed command — capture that reality more vividly than any architecture diagram ever could.