The Ghost in the GPU: A Lesson in CUDA Memory Persistence

In the sprawling, multi-session effort to deploy and train speculative decoding models across a distributed GPU infrastructure, few moments are as quietly instructive as message [msg 7245]. On its surface, this message is unremarkable: a single bash command executed over SSH, designed to kill all Python processes on a remote training machine. But beneath this mundane exterior lies a rich story of debugging under pressure, the subtle failures of process management in GPU computing, and the stubborn persistence of CUDA memory even after the processes that allocated it have been violently terminated.

The Scene: A Training Run Gone Stale

To understand why this message was written, we must first understand the predicament that preceded it. The assistant had been orchestrating a complex training pipeline for DFlash speculative decoding on a remote 8× RTX 6000 Ada node (48 GB each). The pipeline required two parallel workloads: a vLLM server serving hidden states from the Qwen3.6-27B target model on GPUs 0–3, and a DFlash drafter training process on GPUs 4–7. This is a delicate dance of process coordination, GPU memory allocation, and NCCL communication.

The trouble began when the vLLM server refused to complete initialization. After multiple relaunch attempts spanning messages [msg 7221] through [msg 7244], the workers were stuck at 100% CPU utilization with GPU memory frozen at exactly 716 MiB per GPU on devices 0–3. The log file hadn't changed in over ten minutes. The assistant suspected torch.compile was silently consuming CPU cycles, but the empty compilation cache and zero bytes in the Triton inductor directory told a different story: the processes were deadlocked, not compiling.

The user had already flagged the problem in [msg 7223]: "There were old stuck vllm processes, killed the wrong one it seems." This observation proved prescient. The assistant's first cleanup attempt used pkill -9 -f python, a notoriously blunt instrument that matches against the full command line. This killed the SSH session's own shell process, leaving the remote machine in a zombie-like state where new commands appeared to execute but produced no output. The assistant learned this the hard way when a subsequent launch command produced no PID and left no log file ([msg 7229]).

The Message: A Targeted Execution

Message [msg 7245] represents the third attempt at cleanup, and it shows how the assistant's approach had evolved through failure:``` [bash] ssh -p 10978 root@217.138.104.34 '

Kill EVERYTHING

kill -9 $(ps aux | grep python | grep -v grep | awk "{print \$2}") 2>/dev/null sleep 5 nvidia-smi --query-gpu=index,memory.used --format=csv,noheader echo "---" ps aux | grep python | grep -v grep | wc -l ' 2>&1 0, 716 MiB 1, 716 MiB 2, 716 MiB 3, 716 MiB 4, 2 MiB 5, 2 MiB 6, 2 MiB 7, 2 MiB


The key difference from earlier attempts is the use of `kill -9` with a PID list constructed from `ps aux`, rather than `pkill -9 -f python`. This is a more surgical approach: instead of pattern-matching against process command lines (which can match the SSH daemon, the shell, or even the monitoring script), it explicitly enumerates all Python processes and sends SIGKILL to each one. The `2>/dev/null` suppresses errors from processes that die between enumeration and signal delivery, a common race condition in this pattern.

The result confirms that all Python processes are gone: `ps aux | grep python | grep -v grep | wc -l` returns 0. And yet, the GPU memory report shows a stubborn 716 MiB still allocated on GPUs 0–3. This is the central mystery of this message, and the reason it deserves close analysis.

## The 716 MiB Anomaly: Understanding CUDA Memory Persistence

When a CUDA process is killed with SIGKILL, the NVIDIA driver is responsible for cleaning up the GPU memory allocations associated with that process. In normal operation, this happens almost instantly. The 716 MiB figure is not random: it represents the CUDA context overhead that vLLM allocates during engine initialization — the NCCL communicators, the CUDA streams, the memory pool bookkeeping — before any model weights are loaded. This is the memory footprint of a process that has called `cudaSetDevice()` and initialized the CUDA runtime, but has not yet loaded any tensors.

The fact that this memory persists after all Python processes are confirmed dead suggests one of several possibilities:

1. **Zombie processes still holding GPU references**: A process that has been killed but not yet reaped by its parent (the `init` process in this container) can retain CUDA resources until the parent calls `waitpid()`. The `ps` check only shows processes in the process table; a zombie would appear with status `Z` but might not match the `python` filter if its command line has been cleared.

2. **A separate process tree not visible to `ps`**: The vLLM workers might have spawned subprocesses that inherited the CUDA context but have different command-line names. The `grep python` filter would miss a process named, say, `torchrun` or `ptxas`.

3. **CUDA driver-level resource leak**: In rare cases, the NVIDIA driver can fail to fully release GPU memory after a process crash, particularly if the crash occurred during a CUDA API call that left the driver in an inconsistent state. This is more common with multi-process GPU setups (MPS, NCCL communicators) where shared memory segments are involved.

4. **The processes aren't actually gone**: The `ps` check runs after `sleep 5`, but there's a race between process cleanup and the check. If the processes are in the middle of being reaped, the count could briefly be zero before the GPU memory is freed.

## The Reasoning Process Visible in This Message

The assistant's thinking is visible in the structure of the command itself. The three-part pipeline — kill, sleep, verify — reveals a debugging methodology built on first principles:

**Step 1: Exhaustive cleanup.** Rather than targeting specific process names or PIDs, the assistant uses a comprehensive enumeration of all Python processes. This is a response to the earlier failure where `pkill -9 -f python` killed the wrong process (the SSH session's shell) while leaving the actual vLLM workers alive. The lesson learned: when you don't know exactly what's running, kill everything that matches the broadest reasonable category.

**Step 2: Verification with a delay.** The `sleep 5` is critical. Process termination is not instantaneous — SIGKILL marks the process for termination, but the actual cleanup (closing file descriptors, releasing memory, notifying the parent) happens asynchronously. A 5-second delay is a heuristic: long enough for most cleanup to complete, short enough to not waste time if the kill failed entirely.

**Step 3: Double verification.** The command checks both the process count (`ps aux | grep python | wc -l`) and the GPU memory state (`nvidia-smi`). This is a defense against the "looks clean but isn't" scenario — a process might be gone from the process table while its GPU memory lingers, or vice versa.

## What This Message Reveals About the Larger Effort

This message sits at the intersection of two larger themes in the opencode session: the fragility of distributed GPU workflows and the gap between "process is dead" and "resources are free."

Throughout the session, the assistant has been fighting a war of attrition against GPU memory leaks, stuck processes, and NCCL deadlocks. The DFlash training pipeline requires precise coordination between vLLM (serving hidden states) and the training script (consuming those states). When one side crashes, the other must be killed and restarted in the correct order. But the tools for process management over SSH are primitive: `pkill` matches on process names, `kill -9` requires PID enumeration, and neither reliably handles the CUDA driver's asynchronous cleanup.

The 716 MiB ghost memory would eventually be resolved — likely by the NVIDIA driver's watchdog timer or by a subsequent GPU reset — but the assistant cannot wait for that. The training machine is remote, expensive, and needed for other work. Every minute of debugging is a minute not spent training the drafter.

## Assumptions and Their Consequences

The assistant makes several assumptions in this message that deserve scrutiny:

**Assumption 1: All relevant processes are Python processes.** This is a reasonable heuristic in a Python-centric ML workflow, but it misses auxiliary processes like `torchrun` (which wraps Python), `ptxas` (the CUDA JIT compiler), or `nvidia-cuda-mps-control` (the Multi-Process Service daemon). If any of these hold GPU references, they would survive the `kill -9` sweep.

**Assumption 2: `ps aux` shows all processes in the container.** In a Docker/LXC container with restricted `/proc` visibility, some processes (particularly those in different PID namespaces) may be invisible. The vLLM workers might have been spawned in a sub-cgroup that the `ps` invocation cannot see.

**Assumption 3: GPU memory cleanup is synchronous with process death.** The NVIDIA driver's memory cleanup is best-effort and asynchronous. A process that crashes while holding a CUDA mutex or during an NCCL collective operation can leave the driver in a state where memory cannot be freed until a driver-level timeout expires.

These assumptions are not mistakes — they are reasonable shortcuts for a debugging session under time pressure. But they explain why the 716 MiB persisted, and why the assistant would need to escalate to more aggressive measures (like `nvidia-smi --gpu-reset` or a full container restart) in subsequent messages.

## The Broader Lesson: Debugging GPU Infrastructure

Message <msg id=7245> is a microcosm of the challenges faced when operating GPU infrastructure at scale. The tools we rely on for process management — `kill`, `pkill`, `ps` — were designed for CPU-bound workloads where resource cleanup is immediate and deterministic. GPU computing introduces new failure modes: asynchronous memory release, driver-level state machines, NCCL communicators that span multiple processes and GPUs, and CUDA contexts that persist beyond process death.

The assistant's response to these challenges is methodical and evidence-based. Each failed attempt generates new information: `pkill` killed the wrong process, so switch to explicit PID enumeration. The log file stopped growing, so check CPU utilization to distinguish between compilation and deadlock. GPU memory persists after process death, so verify with both `ps` and `nvidia-smi`. This is debugging as a scientific process — forming hypotheses, testing them with targeted commands, and refining the approach based on the evidence.

In the end, the 716 MiB ghost on GPUs 0–3 would be vanquished not by a single command, but by the cumulative effect of multiple cleanup attempts and the NVIDIA driver's eventual timeout. The training pipeline would restart, the drafter would begin learning, and the assistant would move on to the next challenge. But for anyone who has ever stared at an `nvidia-smi` output showing memory allocated to no visible process, message <msg id=7245> is a familiar and deeply human moment: the quiet frustration of knowing the machine is lying to you, and the stubborn determination to prove it wrong.