The Zombie Process Exorcism: A Single Bash Command That Saved an ML Pipeline

In the middle of an intense ML engineering session — tuning SGLang server throughput for the Kimi-K2.5 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs — the assistant encountered a frustratingly common yet deeply disruptive problem: zombie processes. The message at the center of this story is a single, carefully crafted bash command that represents the culmination of a debugging spiral into process management, GPU memory ownership, and the subtle differences between killing a process and truly reclaiming its resources.

The Message

ssh root@10.1.2.6 'pct exec 129 -- bash -c "ps aux | grep python3 | grep -v grep | awk \"{print \\\$2}\" | xargs -r kill -9; sleep 2; fuser /dev/nvidia* 2>&1"'

At first glance, this is a straightforward command: SSH into a Proxmox host, execute inside container 129, find all Python processes, kill them forcefully, wait, then check which processes still hold the NVIDIA GPU devices. But to understand why this particular incantation was necessary — and what it reveals about the fragility of GPU-accelerated ML workflows — we must trace the events that led to it.

The Context: A Server Optimization Spiral

The assistant had been running a large-scale inference pipeline to generate synthetic training data for the EAGLE-3 speculative decoding system. The SGLang server, hosting the 236B-parameter Kimi-K2.5 model across 8 GPUs with tensor parallelism, was the critical bottleneck. Over the preceding messages, the assistant had been iterating on server configuration to maximize throughput:

  1. Initial baseline: ~600 tok/s with mem_fraction_static=0.85
  2. Hierarchical cache experiment: Added --enable-hierarchical-cache with 48GB per rank, which briefly pushed throughput to ~1300 tok/s before settling back to ~600 tok/s as the GPU KV cache saturated
  3. OOM crash: Tried mem_fraction_static=0.93 to maximize GPU memory allocation, but the server crashed during prefill of a large batch because only 0.82GB of headroom remained
  4. Stable config: Settled on mem_fraction_static=0.88 + hicache, yielding 159K GPU tokens and ~930-1350 tok/s peak But the throughput was still limited by GPU KV cache capacity. The assistant identified KV cache quantization (--kv-cache-dtype fp8_e4m3) as the next lever — it would roughly double effective capacity from 159K to ~318K tokens, enabling more concurrent decode requests.

The Zombie Problem Emerges

In [msg 3894], the assistant issued a kill command to shut down the running SGLang server and inference client before restarting with the new FP8 configuration:

pkill -f run_inference; pkill -9 -f sglang; pkill -9 python3; sleep 3; fuser -k /dev/nvidia* 2>/dev/null; sleep 2; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader

This command chain uses multiple strategies: pattern-matched process killing (pkill -f), SIGKILL signals (-9), and device-level process termination (fuser -k). Yet when the assistant checked the result in [msg 3895], the output was empty — a worrying sign. In [msg 3896], the assistant tried a different approach, running the kill through the Proxmox host directly:

pct exec 129 -- bash -c "kill -9 \$(fuser /dev/nvidia* 2>/dev/null) 2>/dev/null; sleep 2"

But the subsequent nvidia-smi check still showed 93,781 MiB allocated on every GPU — the processes were still alive, still holding memory, still refusing to die. The assistant's diagnosis was succinct and telling: "Zombies again."

Why the Standard Kill Approaches Failed

The persistence of these processes reveals several layers of difficulty in managing GPU workloads inside containerized environments:

  1. Process ancestry and signal propagation: When SGLang launches with tensor parallelism across 8 GPUs, it spawns multiple worker processes. A simple pkill targeting the parent may not reliably cascade to all children, especially if some are stuck in CUDA kernel execution or driver-level operations that block signal delivery.
  2. Container boundary effects: The SGLang server was running inside a Proxmox container (LXC). When the assistant SSH'd directly to the container's IP (10.1.230.174), the kill commands operated within the container's PID namespace. But some processes may have been in an unkillable state — waiting on CUDA driver calls that never complete — and the container-level kernel couldn't force-terminate them.
  3. GPU device persistence: The NVIDIA driver maintains per-process state for GPU memory allocations. Even after a process receives SIGKILL, the driver may hold memory in a "zombie" state if the process's CUDA context isn't properly cleaned up. The fuser -k command attempts to address this by finding processes holding /dev/nvidia* device files open, but if the processes are already in a terminal state where they don't respond to signals, even this can fail.
  4. The -9 limitation: SIGKILL cannot be caught or ignored — it terminates the process immediately at the kernel level. However, if a process is blocked in an uninterruptible sleep state (common with CUDA driver operations), even SIGKILL may not take effect until the kernel returns from the driver call. In practice, this means the process appears alive in ps and continues holding GPU memory.## The Subject Message: A Different Strategy This is where the subject message enters. The assistant recognized that the previous kill attempts had failed because they were operating at the wrong level. The processes were inside a Proxmox container, and the container's own kernel couldn't force-kill processes stuck in CUDA driver calls. The solution was to go outside the container — to the Proxmox host itself — and use pct exec to run the kill command from the host's perspective. The command in the subject message is worth examining in detail:
ssh root@10.1.2.6 'pct exec 129 -- bash -c "ps aux | grep python3 | grep -v grep | awk \"{print \\\$2}\" | xargs -r kill -9; sleep 2; fuser /dev/nvidia* 2>&1"'

This is a triple-nested execution: SSH to the Proxmox host (10.1.2.6), then use pct exec 129 to run a command inside container 129, then use bash -c to execute a pipeline. The key difference from the previous attempt is the approach to finding processes: instead of relying on pkill pattern matching or fuser to discover processes by device ownership, it enumerates all Python processes explicitly via ps aux | grep python3 | grep -v grep, then pipes the PIDs to xargs kill -9.

The awk \"{print \\\$2}\" portion is particularly interesting — the backslash escaping reveals the nesting depth. The \$2 is escaped for the outer SSH shell, so that the inner bash -c receives $2 as a literal string, which awk then interprets as the second field (the PID column from ps aux). This kind of layered escaping is a hallmark of real-world shell scripting under duress.

After the kill, the command runs sleep 2 to give the kernel time to process the terminations, then runs fuser /dev/nvidia* to verify that no processes remain holding the GPU devices. The 2>&1 redirect ensures that both stdout and stderr are captured, so any error messages (like "No such process" or permission denials) are visible in the output.

The Result and Its Implications

The next message ([msg 3898]) shows the result of this command — not directly, but through the assistant's subsequent check. After the subject message's command ran, the assistant checked GPU memory via nvidia-smi and found:

0, 1 MiB
1, 1 MiB
2, 1 MiB
3, 99 MiB
4, 93699 MiB
5, 99 MiB
6, 3 MiB
7, 3 MiB

Most GPUs were now showing 1 MiB — essentially clean. But GPU 4 still showed 93,699 MiB, and GPUs 3 and 5 showed 99 MiB. The zombie process on GPU 4 had survived even this more aggressive kill attempt. This is a striking demonstration of how stubborn GPU-bound processes can be: even when enumerated by PID and sent SIGKILL from the host level, a process deep in a CUDA driver call can remain in an unkillable state.

What This Message Reveals About ML Infrastructure

This single message, for all its apparent simplicity, exposes several fundamental truths about operating large-scale ML inference systems:

First, containerization adds complexity to process management. The Proxmox container abstraction means that processes have a dual existence: they live inside the container's PID namespace but also appear as real processes on the host. Killing them from inside the container may fail if the container's kernel can't force-terminate processes in certain states. The pct exec approach from the host bypasses this limitation by operating from the outer kernel.

Second, GPU memory is a scarce and sticky resource. Unlike CPU memory, which is reclaimed almost instantly when a process dies, GPU memory can persist in an allocated state long after the owning process has been terminated. This is particularly problematic in multi-GPU setups where a single stuck process can hold 94GB of HBM3e memory hostage, preventing the next server from starting.

Third, the debugging process itself reveals the architecture. The assistant's progression — from simple pkill, to fuser -k, to host-level pct exec with explicit PID enumeration — mirrors the layered architecture of the system itself. Each failure mode taught something about the relationship between the container, the host kernel, and the NVIDIA driver.