When pkill -9 Isn't Enough: The Persistence of GPU Memory After Process Termination

In the midst of a grueling multi-hour debugging session to deploy a 402 GB GLM-5 GGUF model across 8 Blackwell GPUs, the assistant encounters a perplexing and frustrating obstacle: GPU memory that refuses to be released even after sending SIGKILL to every relevant process. Message [msg 1831] captures this moment in stark simplicity — a single bash command and its confounding result:

GPU memory not freed — zombie processes. Let me force cleanup: ``bash ssh root@10.1.230.174 'pkill -9 -f "VLLM\|vllm\|python3.*vllm"; sleep 5; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader' `` 0, 89879 MiB 1, 89879 MiB 2, 89879 MiB 3, 89879 MiB 4, 89879 MiB 5, 89879 MiB 6, 89879 MiB 7, 89879 MiB

All eight NVIDIA RTX PRO 6000 Blackwell GPUs stubbornly report 89,879 MiB of used memory — approximately 90 GB each, the exact footprint of the loaded model plus KV cache allocation. The assistant's diagnosis is concise: "GPU memory not freed — zombie processes." But the reality is more nuanced, and this message opens a window into one of the most vexing classes of problems in GPU-accelerated machine learning: the mysterious persistence of device memory after process death.

The Road to This Moment

To understand why this message matters, we must trace the path that led here. The assistant had spent the preceding hour carefully loading the GLM-5 model — a 78-layer, 402 GB GGUF-quantized behemoth — onto eight GPUs using vLLM's tensor-parallel weight loader ([msg 1809] through [msg 1815]). The loading proceeded at roughly 19 seconds per layer, taking approximately 25 minutes total, and consumed 51.51 GiB per GPU. The model loaded successfully, the KV cache was allocated, and the server entered the torch.compile warmup phase.

Then came the crash. DeepGEMM's fp8_paged_mqa_logits function, used by the DSA (Dynamic Sparse Attention) indexer, raised a RuntimeError: set_stride is not allowed on a Tensor created from .data or .detach() during CUDAGraph capture ([msg 1816]). This is a known incompatibility between DeepGEMM's CUDA kernels and PyTorch 2.10's torch.compile — the C++ extension attempts tensor metadata manipulation that the graph capture mechanism forbids.

The assistant's response was pragmatic: kill the stuck processes and relaunch with --enforce-eager to bypass CUDAGraph capture entirely ([msg 1829]). This is a standard workaround — --enforce-eager disables the graph compilation that triggers the error, at the cost of some performance. The first pkill -9 -f vllm command was issued, followed by a process count check. But when the assistant checked GPU memory in [msg 1830], all eight GPUs still showed 89,879 MiB allocated. This sets the stage for [msg 1831].

The Reasoning Behind the Second Kill Attempt

The assistant's reasoning in [msg 1831] reveals several assumptions. The phrase "GPU memory not freed — zombie processes" indicates the assistant believes that the previous pkill left behind zombie processes — processes that have terminated but whose entries remain in the process table because their parent hasn't called wait() to reap them. Zombie processes are a common Unix phenomenon, and they can sometimes hold resources if the kernel's cleanup is incomplete.

The assistant's response is to escalate: a broader pkill pattern that now includes "VLLM" (uppercase), "vllm" (lowercase), and "python3.*vllm" (to catch any Python process whose command line mentions vllm). The -9 (SIGKILL) signal is used — the most aggressive termination signal, which cannot be caught or ignored by the target process. A 5-second sleep follows to allow the kernel to complete cleanup, and then nvidia-smi checks whether memory has been freed.

The result is identical: all eight GPUs still at 89,879 MiB. The command succeeded (no error output), but the desired effect did not materialize.

The Zombie Process Assumption: A Critical Examination

The assistant's diagnosis of "zombie processes" is worth examining critically. In Unix process management, a zombie is a process that has already exited — it has released all its resources (memory, file descriptors, GPU contexts) and only retains a process table entry so its parent can read its exit status. Zombies do not hold GPU memory. The CUDA driver's context cleanup happens during the process's exit() syscall, which occurs before the process becomes a zombie. If the processes were truly zombies, the GPU memory would already be free.

The more likely explanation is one of several alternatives:

  1. The processes were not actually killed. The pkill -9 -f vllm pattern might not have matched all relevant processes. The vLLM server spawns multiple worker processes (Worker_TP0 through Worker_TP7), and these might have different command-line representations. The broader pattern in [msg 1831] addresses this, but perhaps some processes use a different executable path or have arguments that don't match.
  2. GPU memory release is asynchronous. The NVIDIA driver may not immediately release GPU memory when a CUDA context is destroyed. There can be a delay while the driver's memory manager defragments or while the NVIDIA persistence daemon (nvidia-persistenced) completes its cleanup. A 5-second sleep might not be sufficient.
  3. The CUDA context is orphaned. If the parent process that initialized CUDA is still alive (or if the GPU context was created in a way that outlives the worker processes), the driver may keep the memory allocation alive. The main vLLM process (PID 42370 from [msg 1821]) might still be in a state where its CUDA context persists.
  4. GPU memory is held by a different mechanism. Some GPU memory allocations, particularly those involving CUDA MPS (Multi-Process Service) or NVIDIA's fabric manager for multi-GPU systems, can persist independently of the creating process.

What This Message Reveals About the Debugging Process

Message [msg 1831] is a microcosm of the broader debugging methodology visible throughout this session. The assistant operates in a tight loop of observation, hypothesis, intervention, and verification:

  1. Observation: GPU memory not freed after first kill attempt ([msg 1830])
  2. Hypothesis: Zombie processes are holding the memory
  3. Intervention: Broader pkill -9 with expanded pattern matching
  4. Verification: nvidia-smi check after 5-second delay This cycle is methodical and disciplined. Each intervention is designed to isolate a specific variable. The first kill used a narrow pattern (vllm); the second broadens to VLLM and python3.*vllm. The 5-second sleep provides time for cleanup. The verification uses the same measurement tool (nvidia-smi) as the initial observation, ensuring comparability. Yet the message also reveals a limitation: the assistant is operating remotely via SSH, with limited visibility into the actual process state on the machine. There is no ps output in this message to confirm whether processes are actually zombies, no lsof to check what's holding GPU contexts, no cat /proc/*/status to inspect process states. The diagnosis of "zombie processes" is inferred from the symptom (memory not freed) rather than from direct evidence.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. The pkill -9 approach is insufficient for freeing GPU memory in this scenario, even with aggressive pattern matching.
  2. GPU memory persistence is a systemic issue on this machine, not a transient one — the memory remained allocated across two separate kill attempts.
  3. The assistant must find an alternative approach to free the GPUs, likely involving a full system-level reset (e.g., nvidia-smi --gpu-reset or nvidia-smi -r for a driver reset, or a full reboot).
  4. The debugging methodology must adapt: when process-level intervention fails, the next step is device-level or system-level intervention.

The Broader Significance

This message, despite its brevity, captures a fundamental truth about GPU-accelerated computing: GPU memory is not just another resource like RAM or disk. It is managed by a complex stack involving the CUDA runtime, the NVIDIA kernel driver, and often a persistence daemon. When things go wrong at the GPU memory level, standard Unix process management tools (kill, pkill, killall) may prove insufficient. The assistant is learning this lesson in real-time, and the frustration is palpable in the escalation from a simple pkill to a more aggressive pattern.

The message also illustrates the "last mile" problem in ML infrastructure debugging. After successfully loading a 402 GB model — a feat that required patching vLLM's weight loader, building custom tools, and debugging tensor parallelism — the deployment is blocked not by a model architecture issue or a weight loading bug, but by the mundane problem of GPU memory that won't let go. These are the problems that try ML engineers' patience: not the intellectually stimulating architecture puzzles, but the stubborn physicality of hardware that refuses to reset.

Conclusion

Message [msg 1831] is a testament to the messy reality of deploying large language models on cutting-edge hardware. It shows a skilled practitioner working methodically through a frustrating problem, forming hypotheses, testing them, and adapting when they fail. The "zombie processes" diagnosis may be incorrect, but the scientific approach is sound. And the persistence of that 89,879 MiB across all eight GPUs serves as a reminder that in the world of GPU computing, the operating system's process model is only part of the story — the hardware has a memory of its own.