The Zombie That Wouldn't Die: A Debugging Microcosm in ML Infrastructure

The Message

[assistant] GPUs still have memory allocated. Need to be more aggressive.
[bash] ssh root@10.1.230.174 "ps aux | grep -E 'python3|sglang' | grep -v grep"
root       72309  0.0  0.0      0     0 ?        Zl   00:28 101:48 [sglang::schedul] <defunct>
root       72310  0.0  0.0      0     0 ?        Zl   00:28  95:59 [sglang::schedul] <defunct>
root       72311  0.0  0.0      0     0 ?        Zl   00:28 100:55 [sglang::schedul] <defunct>
root       72312  0.0  0.0      0     0 ?        Zl   00:28  97:55 [sglang::schedul] <defunct>
root       72313  0.0  0.0      0     0 ?        Zl   00:28  97:39 [sglang::schedul] <defunct>
root       72314  0.0  0....

This message, global index 3268 in a sprawling ML infrastructure session, captures a moment of suspended animation — a systems debugging instant where the assistant has just discovered that its aggressive process-killing cascade has failed to achieve its goal. The GPUs still show 76 GB of allocated memory each. The processes are gone, but their ghosts remain. And the assistant, facing this stubborn reality, utters a quietly determined line: "Need to be more aggressive."

Context: The Road to the Hang

To understand this message, one must appreciate the chain of events that led to it. The assistant had been engaged in a months-long effort to deploy and optimize the Kimi-K2.5 large language model across an 8-GPU machine equipped with NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The journey had already traversed driver installations, CUDA toolkit conflicts, flash-attn compilation battles, and a pivot from vLLM to SGLang after discovering that vLLM's EAGLE-3 speculative decoding integration yielded an unacceptable ~15% acceptance rate.

By segment 25 of the session, the assistant had achieved a respectable 63.6 tok/s single-stream throughput with SGLang using the triton attention backend. The goal was to push this further through NCCL environment variable tuning (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.) combined with the --num-continuous-decode-steps 4 flag. The assistant launched a new SGLang server with these optimizations — but crucially, also changed the attention backend to flashinfer, a different CUDA kernel backend for attention computation.

That decision proved fateful. The server loaded model weights successfully (all 64 safetensor shards across 8 TP workers), but then went silent. The log file stopped at weight loading. Port 8000 never opened. The scheduler processes consumed ~245 GB of system RAM each and performed a single write to an event pipe every second — the telltale sign of a busy-wait loop, likely during CUDA graph capture initialization.

The assistant spent several rounds diagnosing this hang: checking the log, verifying the process tree, running strace on the scheduler, examining dmesg for kernel errors. The conclusion was that --attention-backend flashinfer was incompatible with the SM120 architecture for this model's MLA (Multi-Head Latent Attention) implementation. The fix was to kill the server and restart with the known-working triton backend.

The Killing Spree

In message 3266, the assistant launched a four-stage kill cascade:

kill -9 72169; sleep 2; pkill -9 -f sglang; sleep 2; pkill -9 python3; sleep 3; fuser -k /dev/nvidia* 2>/dev/null; sleep 2; nvidia-smi

This is a progressively escalating sequence: first kill the specific parent Python process by PID, then kill any process matching "sglang", then kill any Python process (a nuclear option), and finally use fuser -k to kill any process holding open NVIDIA device file handles. Each stage is separated by a sleep to allow the system to process the signal and release resources.

When the assistant checked nvidia-smi after this barrage (message 3267), the result was disheartening: all 8 GPUs still showed approximately 76 GB of memory used — essentially unchanged from when the server was running. The processes were supposedly dead, but the GPU memory was not freed.

The Subject Message: Discovery and Diagnosis

Message 3268 is the assistant's response to this stubborn fact. The opening line — "GPUs still have memory allocated. Need to be more aggressive." — is both an observation and a resolution. The assistant recognizes that its kill cascade was insufficient and decides to escalate. But before reaching for a bigger hammer, it does something smart: it investigates.

The bash command ps aux | grep -E &#39;python3|sglang&#39; | grep -v grep is a diagnostic probe. It asks: are there any remaining processes matching these names? The answer is both yes and no. The grep output shows processes with PID 72309 through 72314 (and presumably more beyond the truncated output) — but these are marked as &lt;defunct&gt; and have state Zl. In Linux process terminology, Z means zombie (a terminated process whose exit status hasn't been collected by its parent), and l means multi-threaded. These are the ghost processes: the scheduler workers that were children of the now-killed parent Python process (PID 72169).

The zombie processes show 0.0% CPU, 0.0% memory, and 0 RSS (resident memory) — they are already dead from the kernel's perspective. They consume no resources except a PID table entry and a process control block waiting to be reaped. Critically, they should not be holding GPU memory. The fact that GPU memory remains allocated despite all processes being zombies (or killed) points to a deeper issue: either the CUDA driver has not yet released the memory, or some other process (perhaps a kernel module or a leftover CUDA context) is holding it.

The Thinking Process Revealed

This message is a window into the assistant's debugging methodology. Several layers of reasoning are visible:

Layer 1 — Observation: The nvidia-smi output from the previous round shows GPU memory still allocated. This is the raw data that triggers the investigation.

Layer 2 — Hypothesis: The assistant hypothesizes that some processes survived the kill cascade and are holding the GPU memory. The phrase "Need to be more aggressive" reflects this hypothesis — if the first wave of kills didn't work, a stronger wave is needed.

Layer 3 — Investigation: Rather than immediately firing off another kill command, the assistant first checks what processes actually exist. This is a crucial step — it's gathering intelligence before acting.

Layer 4 — Discovery: The zombie processes are found. But the assistant's truncated output (ending with ...) suggests the output was cut off, and the assistant likely saw the full list internally. The discovery of zombies explains why kill -9 didn't fully work — the parent was killed before it could reap its children, leaving them in zombie state. However, zombies can't hold GPU memory, so this discovery actually contradicts the hypothesis that leftover processes are holding the memory.

Layer 5 — Implicit Pivot: The assistant doesn't explicitly state it in this message, but the next message (msg 3269) shows the pivot: "Zombie processes. Their parent (72169) was killed but they're still zombie. Need to kill the parent's parent or wait for init to reap them. But the real issue is GPU memory — the zombie processes shouldn't hold GPU memory." This reveals that the assistant, upon seeing the zombie output, recognized the contradiction and adjusted its investigation strategy.

Assumptions and Their Consequences

The assistant operated under several assumptions in this sequence:

Assumption 1: That kill -9 followed by pkill -9 -f sglang and pkill -9 python3 would terminate all relevant processes. This was partially correct — the parent and any non-zombie children were killed. But the zombie children couldn't be killed because they were already dead.

Assumption 2: That fuser -k /dev/nvidia* would release GPU memory by killing processes holding NVIDIA device handles. This is a reasonable approach, but it may not work if the GPU memory is held by a CUDA driver context that hasn't been fully cleaned up, or if the zombie processes' file handles weren't properly released.

Assumption 3: That GPU memory would be freed immediately upon process termination. In practice, CUDA memory cleanup can be asynchronous. The NVIDIA driver may hold memory for a period after process death, especially if there are zombie children or if the CUDA context wasn't properly destroyed.

Assumption 4 (implicit): That the zombie processes were somehow related to the GPU memory leak. This assumption is challenged by the assistant's own knowledge that zombie processes consume no resources — and in the next message, the assistant explicitly recognizes this contradiction.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Linux process management: Understanding of zombie processes (terminated but un-reaped children), the Z state flag, and how kill -9 interacts with parent-child process relationships.
  2. CUDA memory model: Knowledge that GPU memory is allocated per CUDA context, typically tied to a process, and that process termination should trigger memory release — but that this can be delayed or fail if the CUDA driver is in an inconsistent state.
  3. SGLang architecture: Understanding that SGLang launches a parent Python process that spawns multiple scheduler worker processes (one per TP rank), and that these workers hold CUDA contexts with model weights and KV cache.
  4. The SM120/flashinfer incompatibility: The background that --attention-backend flashinfer caused a hang during CUDA graph capture on Blackwell GPUs, which is why the server was being killed in the first place.
  5. The session's history: The months of infrastructure work, the GPU upgrade to 8 cards, the previous successful SGLang deployment with triton attention at 63.6 tok/s, and the goal of pushing throughput to 90+ tok/s.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. The zombie discovery: The scheduler processes survived as zombies, explaining why the kill cascade appeared to fail. This is a concrete finding that shapes the next steps.
  2. The GPU memory persistence: The fact that GPU memory remained allocated despite all processes being zombies (or killed) is a significant data point. It suggests either a CUDA driver issue, a kernel module leak, or that the memory was never actually tied to these processes in a way that kill would release.
  3. The insufficiency of the kill cascade: The assistant learns that its multi-stage kill approach, while thorough, doesn't handle the zombie edge case. This informs future debugging sessions.
  4. The need for a different approach: The discovery implicitly rules out "kill more processes" as a solution. The next message shows the assistant pivoting to fuser -v /dev/nvidia* to find what's actually holding the GPU resources — a different diagnostic strategy.

A Microcosm of Systems Debugging

This single message, in its brevity and focus, encapsulates a fundamental pattern in systems debugging: the moment when a seemingly straightforward solution (kill the process) fails, and the debugger must confront the gap between their mental model of the system and its actual behavior.

The assistant expected that killing the SGLang parent process would terminate all children and release GPU memory. Instead, it found zombie children and persistent GPU allocation. The gap between expectation and reality is where all learning happens in systems engineering.

The line "Need to be more aggressive" is particularly telling. It reflects a natural debugging instinct — when a solution doesn't work, try harder. But the assistant quickly realizes that "more aggressive" doesn't mean "more kill signals"; it means "better understanding." The investigation in this message (the ps aux command) is the pivot from brute force to diagnosis. The next message shows the assistant reasoning about why zombies can't hold GPU memory and pivoting to fuser to find the real culprit.

This is the essence of mature debugging: not just trying harder, but understanding why the first attempt failed, and using that understanding to choose a more targeted next step. The zombie processes were a red herring — they looked like the problem but couldn't explain the GPU memory persistence. The assistant recognized this and moved on.

The Broader Lesson

For anyone who has worked with large-scale ML infrastructure, this moment is deeply familiar. GPU memory is a precious and stubborn resource. Processes crash but leave CUDA contexts behind. Kill signals are processed asynchronously. The NVIDIA driver has its own ideas about when to release memory. And zombie processes — those undead remnants of a bygone computation — serve as a reminder that in distributed systems, death is never simple.

Message 3268 is a snapshot of a debugger in the act of debugging: observing, hypothesizing, investigating, discovering, and adjusting. It's not a heroic moment of breakthrough, nor a catastrophic failure. It's the mundane, essential work of understanding why a system isn't behaving as expected — and deciding what to do next.