The Zombie That Wouldn't Die: Debugging GPU Process Cleanup in an ML Inference Pipeline

Introduction

In the middle of an intensive machine learning deployment session, a single brief message captures a moment of operational friction that every ML engineer has faced: the server won't die. Message 3846 in this opencode conversation is deceptively short — just seven words of analysis followed by a diagnostic bash command — but it represents a critical debugging juncture where the assistant confronts the gap between what a pkill command should do and what the Linux kernel actually allows. This article unpacks that moment, exploring the reasoning, context, assumptions, and knowledge required to understand why "Still held. Zombie processes." is a meaningful diagnostic statement in a high-stakes GPU inference deployment.

The Context: Chasing Throughput with Hierarchical Cache

To understand message 3846, we must first understand what led to it. The broader session (Segment 28) was focused on optimizing the throughput of an SGLang inference server serving the Kimi-K2.5 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had identified that the server's KV cache was the bottleneck: with --mem-fraction-static 0.85, only about 116,000 total tokens could be cached, which translated to roughly 50 concurrent requests before the scheduler blocked new prefills. The effective throughput was stuck around 500–890 tokens per second, far below the user's expectation of 4–5× improvement.

The solution identified was SGLang's hierarchical cache (--enable-hierarchical-cache), which spills KV cache entries to host RAM when GPU memory is exhausted. With 408 GB of available host RAM, the assistant calculated that this could expand the effective KV cache from 116K tokens to over 410K tokens, supporting roughly 100 concurrent requests at the observed 4K average response length. This was the most promising lever for throughput improvement without sacrificing quality.

Message 3844 initiated the restart: the assistant ran a sequence of kill commands — pkill -f run_inference, pkill -9 -f sglang, pkill -9 python3, fuser -k /dev/nvidia* — each with short sleeps between them. Message 3845 then checked the result with nvidia-smi, which showed that GPU memory usage remained at 92,129 MiB per GPU — essentially unchanged from when the server was running. The kill had not freed the memory.

This is where message 3846 begins.

The Message: Diagnosis Through Process Inspection

The full text of message 3846 is:

Still held. Zombie processes. [bash] ssh root@10.1.230.174 'fuser /dev/nvidia* 2>/dev/null; echo "---"; ps aux | grep python3 | grep -v grep' 155500 156388 156531 156532 156533 156534 156535 156536 156537 156538 156539 155500 156388 156531 156532 156533 156534 156535 156536 156537 156538 156539 155500 156388 156531 156532 156533 156534 156535 156536 156537 156538 156539 155500 156388 156531 156532 156533 156534 156535 156536 156537 156538 156539 155500 156388 156531 156532 156533 156534 156535 156536 156537 156538 156539 155500 156388 156531 156532 156533 15...

The message is a two-part diagnostic. First, the assistant states its conclusion: "Still held. Zombie processes." This is the reasoning — the assistant has inferred that the GPU memory is still allocated because the killed processes have become zombie processes that haven't been reaped by their parent. Second, it runs a command to confirm this hypothesis: fuser /dev/nvidia* lists the process IDs holding the NVIDIA device files, and ps aux | grep python3 shows which Python processes are still alive.

The output (truncated in the transcript) shows a repeating pattern of process IDs — 155500, 156388, 156531, 156532, 156533, 156534, 156535, 156536, 156537, 156538, 156539 — repeated multiple times. This pattern is itself informative: the repeating PIDs suggest that fuser is reporting the same processes holding multiple NVIDIA devices (likely all 8 GPUs), and the specific PID range (156531–156539) suggests a process tree where one parent spawned multiple children, all of which are now zombies.

Why Zombie Processes Persist

The assistant's diagnosis reveals an understanding of Linux process lifecycle that is essential for GPU server management. When a process is killed with SIGKILL (signal 9, as in pkill -9), the kernel terminates it immediately, but the process entry in the process table is not removed until the parent process calls wait() or waitpid() to reap it. If the parent process is itself dead or not properly reaping its children, the terminated children remain as "zombies" — they consume no CPU or memory resources except for their entry in the process table and, critically, any resources that are tied to the process's lifetime rather than its execution state.

GPU memory is one such resource. When a process opens a CUDA context on a GPU, the NVIDIA driver associates that context with the process. Even after the process is killed, if it becomes a zombie, the CUDA context may not be fully released until the process is reaped. The NVIDIA driver holds the memory allocation until it can confirm that the process has been fully cleaned up. This is why nvidia-smi continued to show 92 GiB of used memory per GPU even after the pkill -9 commands had run.

Assumptions and Incorrect Assumptions

This message reveals several assumptions the assistant made, some of which proved incorrect:

Assumption 1: pkill -9 would immediately free GPU memory. The assistant assumed that sending SIGKILL to the SGLang server and inference processes would cause the NVIDIA driver to release the GPU memory allocations. This is generally true for normal process termination, but it fails when processes become zombies that are not reaped.

Assumption 2: fuser -k /dev/nvidia* would be sufficient. The fuser -k command sends SIGKILL to all processes accessing the specified files. However, if those processes are already zombies (having been killed by the earlier pkill), fuser -k may not help — you can't kill what's already dead.

Assumption 3: The sleep intervals (1–3 seconds) were sufficient for cleanup. The assistant inserted sleep 1, sleep 3, sleep 2 between kill commands, assuming this would give the kernel and driver time to clean up. In reality, zombie processes persist until their parent reaps them, which is not a matter of time but of parent process behavior.

Assumption 4: The processes would be cleanly killable. The assistant did not anticipate that the SGLang server, which spawns multiple worker processes for tensor parallelism across 8 GPUs, would leave behind a complex process tree where children become zombies because the parent process manager doesn't reap them promptly.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Linux process management: Understanding what zombie processes are, how they differ from running or terminated processes, and why they persist. The key insight is that a zombie is a process that has terminated but whose exit status has not been collected by its parent.

NVIDIA GPU memory lifecycle: Knowing that GPU memory allocations are tied to CUDA contexts, which are associated with process lifetimes. A zombie process can prevent CUDA context cleanup because the kernel module sees the process as still existing (even though it's terminated).

SGLang server architecture: Understanding that SGLang uses tensor parallelism across multiple GPUs, which means it spawns multiple worker processes (one per GPU) that each hold a CUDA context. Killing the main process doesn't necessarily clean up the workers if they're in a zombie state.

Process tree inspection tools: Knowing that fuser /dev/nvidia* lists PIDs accessing NVIDIA devices, and ps aux | grep python3 shows surviving Python processes. The combination of these two commands allows the assistant to cross-reference which PIDs are holding GPUs versus which are still alive.

The fuser command: Understanding that fuser without -k simply lists PIDs, and that the output format shows space-separated PIDs for each file argument. The repeating PID pattern in the output indicates that the same processes are holding multiple device files.

Output Knowledge Created

This message produces several valuable pieces of knowledge for the ongoing session:

Confirmation that the kill was incomplete. The assistant now knows that the previous kill sequence was insufficient and must be supplemented with additional cleanup steps.
Identification of specific problematic PIDs. The repeating PID pattern (156531–156539) gives the assistant specific targets for further investigation or forced cleanup.
A refined mental model of the process architecture. The assistant now understands that SGLang's multi-GPU deployment creates a process tree that requires more careful teardown than a simple pkill.
A diagnostic pattern for future use. The combination of fuser and ps becomes a standard debugging step for GPU process cleanup issues.

The Thinking Process

The reasoning visible in this message is a classic example of hypothesis-driven debugging. The chain of thought, reconstructed from the context, goes like this:

Observation: nvidia-smi shows GPU memory still fully allocated after kill commands (message 3845).
Hypothesis: The processes were killed but became zombies, preventing GPU memory release.
Prediction: fuser /dev/nvidia* will show PIDs that are also visible in ps aux as zombie entries.
Test: Run both commands and compare outputs.
Conclusion (stated): "Still held. Zombie processes." The brevity of the statement — just four words of analysis — reflects the assistant's confidence in its diagnosis. It doesn't hedge or offer alternatives. The command is chosen to confirm a specific hypothesis, not to explore unknown territory. This is the hallmark of an experienced operator who has seen this pattern before and knows exactly what to check.

Implications for the Rest of the Session

The zombie process diagnosis has immediate practical consequences. The assistant cannot simply restart the server with hierarchical cache enabled while the old CUDA contexts are still held. The next steps would need to involve either:

Killing the parent process that is preventing reaping (likely the shell or process manager that spawned the SGLang workers)
Using kill on the zombie PIDs themselves (which won't work — zombies can't be killed, only reaped)
Rebooting the machine (a nuclear option that would be disruptive but guaranteed to work)
Using fuser -k more aggressively, or finding and killing the parent PID The fact that the assistant had to investigate this at all reveals an important operational reality: GPU server management involves not just starting and stopping services, but understanding the full lifecycle of CUDA contexts, process trees, and kernel-level resource management. A simple pkill is rarely sufficient when dealing with multi-process GPU inference servers.

Conclusion

Message 3846 is a small but revealing window into the operational complexity of deploying large language models on multi-GPU hardware. In just seven words and one bash command, the assistant diagnoses why a seemingly straightforward server restart failed: zombie processes holding GPU memory. The message demonstrates that effective ML infrastructure management requires not just knowledge of model architectures and inference frameworks, but also a deep understanding of Linux process mechanics, GPU driver behavior, and the sometimes surprising ways that terminated processes can continue to consume resources. For the reader, it serves as a reminder that in the world of large-scale inference, even the simplest operations — like stopping a server — can become debugging exercises that require careful forensic investigation.