The Ghost in the GPUs: Debugging Zombie Memory with an Empty pgrep

In the midst of a high-stakes model deployment pivot on an 8-GPU Blackwell server, a single short message captures a moment of debugging clarity — and confusion. The message, <msg id=2351>, is deceptively brief:

Still loaded — processes are lingering: [bash] ssh root@10.1.230.174 "pgrep -af vllm; pgrep -af 'python3.api_server'" 235285 bash -c pgrep -af vllm; pgrep -af 'python3.api_server'

That's it. Two lines of commentary, one bash command, and a single line of output. On its surface, it looks like a routine check that found nothing. But this message is the fulcrum of a debugging sequence that reveals fundamental truths about GPU memory management, process lifecycle, and the assumptions we make when managing production ML infrastructure.

The Broader Context: A Model Migration

To understand why this message matters, we need to step back. The session had been a whirlwind tour of deploying and benchmarking 1-trillion-parameter models on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The team had just finished benchmarking the MiniMax-M2.5 model, a 230B FP8 GQA model that achieved an impressive ~4,000 tok/s throughput with TP=8 and Expert Parallelism. But the user had a new target: deploy the native INT4 version of Kimi-K2.5 from moonshotai/Kimi-K2.5, a 547GB behemoth that promised better single-request performance than the NVFP4 variant they'd previously tested.

The assistant had executed this pivot methodically. First, it checked disk space and found 929GB available on the 1.7TB shared volume ([msg 2339]). The NVFP4 Kimi model occupied 540GB, so it was deleted to free space ([msg 2342]). Then the 547GB INT4 model was downloaded in about 28 minutes across 64 safetensor shards (<msg id=2345-2346>). With the download complete, the assistant turned to the next step: stopping the MiniMax service that was currently running and consuming all 8 GPUs.

The Expected Path: Stop Service, Free GPUs

The assistant's mental model was straightforward. The MiniMax model was deployed as a systemd service (vllm-minimax-m25). Stopping the service should terminate the vLLM processes, which should release the CUDA contexts and free GPU memory. In <msg id=2348>, the assistant ran a kill command targeting vLLM processes associated with MiniMax. In <msg id=2350>, it escalated to systemctl stop and systemctl disable, then waited 5 seconds and checked.

The result was baffling: all 8 GPUs still showed approximately 96,800 MiB used each ([msg 2350]). The systemd service was stopped and disabled, but the GPU memory was untouched. This is the moment our subject message arrives.

Reading the Message: What the Assistant Saw

The assistant's observation — "Still loaded — processes are lingering" — is a hypothesis. The assistant assumes that the GPU memory is held by processes that survived the systemd stop. To test this, it runs two pgrep commands: one for any process containing "vllm" in its command line, and one for any Python process running api_server.

The output is a single line: 235285 bash -c pgrep -af vllm; pgrep -af 'python3.*api_server'. This is the pgrep command itself. The PID 235285 is the bash process executing the remote command. The pgrep found no vLLM processes, no api_server processes — just its own invocation.

This is a classic "pgrep finding itself" scenario. When you run pgrep -af vllm, the pgrep process itself doesn't match because it hasn't started yet. But the bash shell that wraps the command does contain the text of the command, which includes "pgrep -af vllm" — but not "vllm" as a standalone match. Wait, actually, let me reconsider. The command is pgrep -af vllm; pgrep -af 'python3.*api_server'. The -f flag matches against full command lines. The bash process's command line is bash -c pgrep -af vllm; pgrep -af 'python3.*api_server'. Does this contain "vllm"? Yes — "vllm" appears in the string "pgrep -af vllm". So pgrep matches the bash process itself. That's the output we see.

The critical insight: the pgrep output is empty of any real vLLM or api_server processes. There are no lingering vLLM workers. No orphaned Python processes. The GPU memory is being held by... nothing visible.

The Negative Result That Changed the Debugging Strategy

This message is a negative result — the assistant's hypothesis was wrong. The GPU memory wasn't held by lingering vLLM processes. The pgrep proved that. But the assistant doesn't yet know what is holding the memory. The next message ([msg 2352]) shows the assistant pivoting to a new hypothesis: "No vLLM processes. But GPUs still show 96GB used. Must be a driver/context leak."

This is a fascinating shift. The assistant moves from "find the process and kill it" to "find what's holding the device." It runs fuser -v /dev/nvidia* to find processes with open file handles on the NVIDIA devices. This reveals a list of zombie worker PIDs — processes that had been killed but whose CUDA contexts weren't released. In <msg id=2353>, the assistant kills these zombies and frees most GPUs. One GPU stubbornly holds memory until <msg id=2354>, where the assistant uses fuser output to aggressively kill everything holding the NVIDIA device files.

What This Message Reveals About GPU Memory Management

The sequence around this message illuminates several important truths about managing GPU memory in production ML systems:

1. systemd stop is not enough. When vLLM spawns worker processes (especially with tensor parallelism across multiple GPUs), stopping the parent service doesn't always cleanly terminate all children. The CUDA driver may retain memory contexts if processes are killed rather than allowed to exit gracefully.

2. GPU memory can be held by zombie processes. A process that has been killed (SIGKILL) but whose parent hasn't reaped it can still hold GPU memory. The CUDA driver tracks allocations by PID, and if the PID is still in the process table (even as a zombie), the memory isn't released.

3. pgrep has blind spots. The -f flag matches against full command lines, which means it can match its own parent shell. But more importantly, pgrep only shows processes that match the pattern — it doesn't show zombies or processes that have exited but left GPU contexts behind.

4. The debugging ladder. The assistant's debugging follows a clear escalation: kill by process name → systemd stop → pgrep investigation → fuser device inspection → aggressive PID cleanup. Each step is driven by the failure of the previous hypothesis.

Assumptions Made and Lessons Learned

The assistant made several assumptions that this message tests:

• Assumption: Stopping the systemd service would free GPU memory. Reality: The service was stopped but worker processes (or their remnants) persisted.
• Assumption: Lingering vLLM processes were the cause. Reality: No vLLM processes existed — the memory was held by zombie processes that didn't match "vllm" in their command lines.
• Assumption: pgrep would find the culprits. Reality: pgrep found nothing, forcing a change in investigative technique. The user, reading this message, would see the assistant working through the problem methodically. The empty pgrep output is informative precisely because it's empty — it rules out one class of explanations and forces consideration of others.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is compressed but visible. The phrase "Still loaded — processes are lingering" shows the assistant forming a hypothesis based on available evidence (GPUs still show memory used, systemd service is stopped). The choice of pgrep -af vllm; pgrep -af 'python3.*api_server' shows the assistant searching for two specific patterns: the generic "vllm" (covering all vLLM-related processes) and the more specific "api_server" (covering the OpenAI-compatible server entry point).

The fact that the assistant runs both commands in a single ssh call suggests it wants a comprehensive picture — not just vLLM processes but any Python API server that might be holding memory. This is a reasonable search strategy, but it misses the zombie processes that don't appear in pgrep output at all.

Input and Output Knowledge

Input knowledge required to understand this message includes: understanding of systemd service management, familiarity with pgrep and process inspection on Linux, knowledge of GPU memory allocation patterns in CUDA, and awareness of the model deployment context (MiniMax being replaced by Kimi-K2.5 INT4).

Output knowledge created by this message is the confirmed absence of identifiable vLLM or API server processes. This negative finding is valuable — it narrows the search space and forces the next debugging step. Without this message, the assistant might have continued searching for vLLM processes or tried restarting the service. Instead, the empty pgrep output redirects the investigation toward device-level inspection with fuser.

Conclusion

Message <msg id=2351> is a masterclass in debugging discipline. It's a short message that says "I checked, and what I expected to find wasn't there." In many narratives, negative results are edited out — they don't make for exciting reading. But in real engineering, negative results are the stepping stones to correct diagnosis. The empty pgrep output told the assistant something important: the problem wasn't what it seemed. The GPU memory wasn't held by visible processes. It was held by ghosts — zombie processes that had been killed but not cleaned up, their CUDA contexts lingering like unfinished business.

The message also reveals something about the assistant's character as an engineer: it doesn't assume. It checks. When the systemd stop didn't work, it didn't restart the service or reboot the machine. It investigated. It ran pgrep, got an empty result, and pivoted. That empty line of output — 235285 bash -c pgrep -af vllm; pgrep -af 'python3.*api_server' — is the sound of a hypothesis being falsified, and a better one being born.