The Zombie Process Hunt: A Microcosm of GPU Memory Management

Introduction

In the middle of a high-stakes model deployment session on an 8× Blackwell GPU server, a single assistant message captures one of the most frustrating and common challenges in large-scale ML infrastructure: GPU memory that refuses to release after a process terminates. Message [msg 2353] is a brief but revealing moment where the assistant identifies zombie worker processes, kills them, and discovers that the cleanup is only partially successful. This article examines that message in detail, unpacking the reasoning, assumptions, and knowledge required to understand what might otherwise look like a trivial kill -9 command.

The Scene: Pivoting to Kimi-K2.5 INT4

The broader session had been a whirlwind tour of 1T-parameter model deployment. After benchmarking MiniMax-M2.5 FP8 at an impressive ~4,000 tok/s with TP=8 and Expert Parallelism (EP), the user gave a new directive in [msg 2337]: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably." This set off a chain of operations: checking disk space (1.7TB filesystem with 755GB used), deleting the 540GB NVFP4 variant of Kimi-K2.5, and downloading the 595GB INT4 version — a download that took approximately 28 minutes over the network.

By the time the download completed, the assistant had a 547GB model on disk and needed to transition from the running MiniMax service to the new Kimi-K2.5 INT4 deployment. This transition is where the target message lives.

The Target Message

The message reads in full:

Zombie worker processes still holding GPU memory. Kill them:

>

``bash ssh root@10.1.230.174 "kill -9 215004 233458 233723 233724 233725 233726 233727 233728 233729 233730 2>/dev/null; sleep 3; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader" ``

>

`` 0, 0 MiB 1, 0 MiB 2, 1 MiB 3, 1 MiB 4, 13 MiB 5, 96798 MiB 6, 63 MiB 7, 3 MiB ``

On its surface, this is a straightforward debugging step. But the message encodes a significant amount of reasoning and context that makes it far more interesting than it first appears.

Why This Message Was Written

The message exists because of a specific failure mode in GPU memory management. Earlier, in [msg 2350], the assistant had run systemctl stop vllm-minimax-m25 and systemctl disable vllm-minimax-m25 to stop the MiniMax service. Despite this, nvidia-smi still showed all 8 GPUs consuming approximately 96,800 MiB each — the full model footprint. In [msg 2351], the assistant checked for running vLLM processes with pgrep and found none, leading to the conclusion "Must be a driver/context leak."

This was a reasonable but incorrect hypothesis. A "driver/context leak" would imply that the NVIDIA driver or CUDA runtime had failed to release memory due to a bug or improper cleanup. However, in [msg 2352], the assistant ran fuser -v /dev/nvidia* and discovered the truth: ten process IDs (215004, 233458, 233723–233730) were still holding references to the NVIDIA devices. These were zombie worker processes — the children of the vLLM distributed runtime that had survived the parent process termination.

The target message is the assistant's response to this discovery. It represents the shift from a "driver bug" hypothesis to a "lingering process" hypothesis, and the corresponding action: force-kill the zombies.

How Decisions Were Made

Several decisions are embedded in this message, some explicit and some implicit.

The decision to use kill -9: This is the nuclear option for process termination. The assistant could have tried kill (SIGTERM) first, allowing processes to clean up gracefully, or could have attempted to trace the processes back to their parent to understand why they survived. Instead, it went straight to SIGKILL. This decision reflects the operational context: this is a deployment session on a dedicated machine, not a production system with uptime requirements. Speed matters more than graceful cleanup. The assistant needs the GPUs freed now so it can load the new model.

The decision to kill specific PIDs rather than using a pattern: The assistant had the PID list from fuser and chose to enumerate them explicitly. An alternative would have been kill -9 $(fuser /dev/nvidia* 2>/dev/null), which would have been more robust. The explicit enumeration suggests the assistant wanted to be deliberate and observable about what was being killed.

The decision to check with nvidia-smi after killing: The sleep 3; nvidia-smi pattern shows the assistant is aware that GPU memory release is not instantaneous. The 3-second sleep is a heuristic — long enough for the kernel to clean up, short enough to not waste time.

The decision to report partial results rather than immediately retry: This is the most interesting decision in the message. After killing the zombies, 7 of 8 GPUs are free (or nearly free), but GPU 5 stubbornly shows 96,798 MiB used. The assistant could have immediately tried another round of killing, or checked what was still holding GPU 5. Instead, it reported the result and waited for the next round (which came in [msg 2354], where it used fuser more aggressively to find and kill the remaining holder).

Assumptions Made

The message rests on several assumptions, some explicit and some implicit.

Assumption: The zombie processes are the only thing holding GPU memory. This turned out to be partially wrong — GPU 5 remained allocated even after all identified PIDs were killed. There was either a process the assistant missed, or the memory release was delayed.

Assumption: kill -9 is sufficient to release GPU memory. In general, SIGKILL terminates a process immediately, and the kernel should clean up the process's resources, including CUDA memory allocations. However, there are edge cases where CUDA memory can persist after process death, particularly with certain driver versions or when using CUDA IPC / shared memory. The assistant implicitly assumes the standard behavior will hold.

Assumption: The PIDs from fuser are all vLLM worker processes. The assistant labeled them "zombie worker processes" based on the context — they were children of the vLLM distributed runtime. But fuser reports any process with an open file descriptor on the NVIDIA device files. It's possible some of these PIDs were unrelated system processes. The assistant assumed they were all vLLM workers and that killing them was safe.

Assumption: The machine is dedicated and killing these processes has no side effects. This is a safe assumption given the context — this is a benchmarking/deployment session on a dedicated server. But it's still an assumption worth noting.

Mistakes and Incorrect Assumptions

The primary mistake in this message is the incomplete cleanup. The assistant killed the identified zombie processes but did not verify that all holders of GPU memory were eliminated. GPU 5 remained fully allocated, indicating that either:

  1. A process was missed by the initial fuser scan (perhaps it opened the device after the scan, or fuser doesn't catch all GPU memory consumers).
  2. The memory release was asynchronous and needed more time.
  3. There was indeed a driver-level leak on GPU 5 specifically. The assistant's response to this partial result — reporting it rather than immediately acting — is a reasonable choice. In a debugging workflow, observing the state after an action is standard practice. The next message ([msg 2354]) shows the assistant doing exactly the right thing: running fuser again to find remaining holders and killing them more aggressively. A subtler issue is the labeling. Calling these "zombie worker processes" is slightly misleading in the strict UNIX sense. A zombie process is one that has terminated but whose parent hasn't called wait() to reap its exit status. These processes were still alive (they were holding GPU memory), so they were orphaned or detached worker processes, not zombies. The term "zombie" is used colloquially here to mean "processes that should be dead but aren't."

Input Knowledge Required

To fully understand this message, a reader needs:

Knowledge of GPU memory management: Understanding that GPU memory is tied to process lifetime — when a process dies, its CUDA allocations are (normally) freed by the driver. This is why killing processes is a standard debugging step for GPU memory leaks.

Knowledge of nvidia-smi: The --query-gpu=index,memory.used --format=csv,noheader flags produce machine-readable output showing per-GPU memory usage. The reader needs to know that "0, 0 MiB" means GPU 0 has 0 MiB used (free), while "5, 96798 MiB" means GPU 5 still has ~96.8 GB allocated.

Knowledge of fuser -v /dev/nvidia*: This command (used in the previous message) lists processes with open file descriptors on NVIDIA device files. Understanding that fuser reveals which processes are holding GPU resources is essential.

Knowledge of vLLM architecture: The assistant's inference that these are "worker processes" relies on understanding that vLLM uses a distributed runtime with multiple worker processes (one per GPU) that communicate via NCCL. When the main process is killed, workers can survive if they're in a separate process group or were started with certain flags.

Knowledge of the session history: The reader needs to know that MiniMax was running with TP=8+EP, which means 8 worker processes (one per GPU). The PID list (233723–233730) has 8 consecutive PIDs, matching this expectation. The additional PIDs (215004, 233458) might be the main process or other infrastructure.

Output Knowledge Created

This message creates several pieces of knowledge:

The zombie processes have been killed: The first 7 GPUs are freed, confirming that the kill was effective for most workers.

GPU 5 has a stubborn allocation: The 96,798 MiB on GPU 5 is a problem that needs further investigation. This is new information that drives the next debugging step.

The kill approach is partially effective: The assistant learns that a single pass of kill -9 on identified PIDs is not sufficient for complete cleanup. This informs the more aggressive strategy in the next message.

The GPU memory release pattern: The non-zero values on GPUs 2 (1 MiB), 3 (1 MiB), 4 (13 MiB), 6 (63 MiB), and 7 (3 MiB) suggest residual allocations that are being cleaned up asynchronously. These small values (compared to 96,798 MiB) are likely driver overhead or deallocation in progress.

The Thinking Process

The thinking visible in this message is a classic debugging chain:

  1. Observe symptom: GPUs still show memory used after service stop.
  2. Form hypothesis: "Driver/context leak" (from previous message).
  3. Test hypothesis: Check for processes with fuser (previous message).
  4. Revise hypothesis: Found PIDs → it's zombie processes, not a driver leak.
  5. Act: Kill the identified PIDs.
  6. Observe result: Partial success — 7/8 GPUs freed, GPU 5 stuck.
  7. Report: Output the state for the next decision. The assistant doesn't panic at the partial result. It calmly reports the state, knowing that the next round of debugging will handle GPU 5. This is a mature debugging rhythm: act, observe, report, iterate. The message also shows the assistant's awareness of the operational context. It doesn't waste time investigating why GPU 5 is stuck before reporting — it just reports and moves on. In a benchmarking session where every minute counts, this efficiency is valuable.

Conclusion

Message [msg 2353] is a small but revealing moment in a complex model deployment session. It captures the transition between two hypotheses (driver leak → zombie processes), demonstrates a debugging rhythm of act-observe-report-iterate, and shows the messy reality of GPU memory management on multi-GPU systems. The incomplete cleanup of GPU 5 is a reminder that even straightforward operations like killing processes can have unexpected edge cases. For anyone who has managed large-scale GPU infrastructure, this message will feel deeply familiar — the frustration of memory that won't release, the hunt for lingering processes, and the satisfaction of finally seeing all zeros in nvidia-smi.