The Unseen Cleanup: Why a Simple `pkill -9` Reveals the Reality of ML Engineering

The Message

[assistant] GPU memory still used from old process. Let me force kill:

[bash] ssh root@[REDACTED] 'pkill -9 -f vllm; pkill -9 -f python3; sleep 5; nvidia-smi --query-gpu=memory.used --format=csv,noheader' 2>&1

(no output)

This short message, message 6969 in a sprawling coding session, appears unremarkable at first glance. It is a single bash command executed over SSH on a remote machine running an NVIDIA RTX A6000 GPU: forcefully terminate all vLLM and Python processes, wait five seconds, then check GPU memory usage. The output is empty — no errors, no warnings, just silence. But this silence is precisely the point. This message is a critical juncture where the assistant recognized that a previous restart attempt had failed silently, diagnosed the root cause (stale GPU memory allocations from a zombie process), and executed a hard cleanup before proceeding. To understand why this message exists, we must trace the chain of events that led to it, the assumptions that were broken, and the practical realities of GPU-accelerated ML serving that make such brutal interventions necessary.

The Context: A Chain of Broken Assumptions

The story begins several messages earlier, when the assistant was attempting to deploy DFlash speculative decoding — a technique that uses a small "drafter" model to predict multiple future tokens in parallel, accelerating inference — for the Qwen3.6-27B language model on a pair of NVIDIA RTX A6000 GPUs. The initial deployment in message 6954 appeared successful: the model produced coherent output with proper reasoning. But when the assistant ran throughput benchmarks in messages 6955–6958, the results were catastrophic. The DFlash deployment achieved only ~18 tokens per second, compared to 73.5 tok/s with the earlier SGLang-based MTP (Multi-Token Prediction) deployment — a fourfold regression.

The investigation in messages 6960–6961 revealed the culprit: the DFlash drafter's acceptance rate was a mere 0.8%. The drafter was generating garbage tokens that were almost never accepted by the target model's verification step. Every speculative round, the system drafted 15 tokens, nearly all were rejected, and only the single "bonus" token from verification was kept — wasting compute and actually slowing down inference compared to no speculation at all.

The root cause, uncovered through messages 6961–6965, was a configuration mismatch. The assistant had guessed the target_layer_ids parameter — which controls which layers of the target model feed hidden states into the drafter — based on patterns observed in other DFlash models. The guess was [1, 17, 33, 49, 63], derived from a formula that spaced layer IDs evenly across the target model's 64 layers. But the actual configuration, provided by the user in message 6965 after retrieving it from the HuggingFace model repository, was substantially different: target_layer_ids: [1, 16, 31, 46, 61], mask_token_id: 248070 (not the guessed 248064), and critically, the drafter used four sliding-window attention layers followed by one full-attention layer — not all full-attention layers as the assistant had assumed.

The Failed Restart

In message 6966, the assistant acted on this new information. It wrote the corrected config.json to the remote machine, then immediately killed the running vLLM process and attempted to restart it. Message 6967 verified the config file was correctly written. Then in message 6968, the assistant launched a new vLLM server with the corrected configuration:

nohup /root/ml-env/bin/vllm serve /root/models/Qwen3.6-27B \
  --port 30000 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --language-model-only \
  --trust-remote-code \
  --speculative-config "{\"method\": \"dflash\", \"model\": \"/root/models/Qwen3.6-27B-DFlash\", \"num_speculative_tokens\": 15}" \
  > /root/vllm-serve.log 2>&1 &

But this launch was premature. The nvidia-smi output at the end of message 6968 showed 42759 MiB of GPU memory still in use — the old vLLM process had not been fully cleaned up. The pkill -9 -f vllm in message 6966 had apparently not succeeded, or the process had been restarted by some other mechanism, or the GPU memory was held by a different Python process. The assistant proceeded with the restart anyway, but the new vLLM server would likely fail to allocate memory or crash.

The Target Message: Recognizing and Fixing the Problem

This brings us to message 6969. The assistant, upon seeing that GPU memory was still occupied, recognized the problem immediately. The old process had not been properly terminated. The solution was not subtle — it was a nuclear option: pkill -9 -f vllm; pkill -9 -f python3. The -9 signal (SIGKILL) cannot be caught or ignored by processes; it forces immediate termination. The -f flag matches against the full command line, so any process whose command line contains "vllm" or "python3" would be killed — including possibly the SSH session itself if it were running Python, though the remote shell execution context prevents that.

The five-second sleep (sleep 5) is a pragmatic touch. GPU memory release after process termination is not instantaneous — the NVIDIA driver's kernel module must clean up the GPU address space, free CUDA contexts, and release pinned memory. Five seconds is a reasonable heuristic: long enough for the driver to complete cleanup, short enough to not be annoying.

The final command — nvidia-smi --query-gpu=memory.used --format=csv,noheader — serves as a verification step. If the memory usage drops to near zero (or the base amount used by the display server, if any), the assistant knows the cleanup succeeded. The empty output in this case likely means the SSH command produced no stdout (perhaps stderr was captured separately, or the output was empty), but the absence of error messages indicates the commands executed successfully.

Why This Matters: The Hidden Complexity of GPU Memory Management

This message illuminates a fundamental truth about ML engineering that is invisible in high-level tutorials and research papers: GPU memory is a finite, non-virtualized resource that must be explicitly managed. Unlike CPU memory, where an operating system's virtual memory subsystem handles allocation and deallocation transparently, GPU memory is allocated through CUDA driver APIs and persists until explicitly freed by the allocating process — or until that process is killed. If a process crashes without properly releasing its CUDA contexts, the GPU memory remains allocated until the operating system's driver cleans it up, which may not happen immediately.

This creates a class of bugs unique to GPU programming: the "zombie GPU allocation." A process exits, but its GPU memory is not freed because the CUDA driver's cleanup is asynchronous, or because the process was killed in a way that prevented proper cleanup (e.g., SIGKILL vs SIGTERM), or because a child process or thread is still holding references. The symptom is that nvidia-smi shows memory in use even though no visible process is running. The only reliable fix is to wait — or to use pkill -9 and wait.

The assistant's decision to use pkill -9 -f python3 is particularly aggressive. Python processes are the lifeblood of the ML stack: the vLLM server, the data preprocessing scripts, the monitoring tools, even the shell's Python interpreter. Killing all Python processes on the remote machine could disrupt other operations. But in this context — a dedicated GPU serving machine with a single purpose — it is the correct call. The assistant prioritizes a clean state over process isolation.

Assumptions and Knowledge Required

To understand this message, a reader needs several pieces of contextual knowledge:

GPU memory persistence: That GPU memory is not automatically freed when a process exits, and that nvidia-smi is the standard tool for inspecting GPU memory usage.
The SIGKILL signal: That kill -9 (or pkill -9) forces immediate process termination and is the nuclear option for unresponsive processes.
The vLLM architecture: That vLLM loads large language models into GPU memory at startup, and that a failed startup due to insufficient memory is a common failure mode.
The SSH execution model: That commands run via SSH execute in a non-interactive shell on the remote machine, and that the 2>&1 redirect captures stderr along with stdout.
The DFlash speculative decoding pipeline: That the drafter model and target model share GPU memory, and that restarting with a new configuration requires a clean memory state. The message also reveals the assistant's assumptions: that the old vLLM process was still holding GPU memory (confirmed by the 42759 MiB reading in message 6968), that pkill -9 would successfully terminate it, that five seconds is sufficient for GPU memory cleanup, and that the subsequent vLLM restart would succeed with the freed memory.

The Output Knowledge Created

This message produces several pieces of actionable knowledge:

Confirmation that the old process was indeed still alive: The fact that the assistant needed to issue this command confirms that the earlier pkill in message 6966 did not fully succeed, or that a new process had started in the interim.
A clean GPU memory state: After this command, the GPUs are ready for a fresh vLLM startup with the corrected DFlash configuration.
A diagnostic pattern: The sequence of "check GPU memory → if still occupied, force kill → wait → verify" becomes a reusable pattern for future deployment attempts.
Evidence of the fragility of GPU process management: The need for such a forceful cleanup highlights that graceful shutdown (SIGTERM, proper CUDA context cleanup) is often insufficient in practice.

The Thinking Process

The assistant's reasoning in this message is concise but reveals a clear diagnostic chain:

Observation: In message 6968, the nvidia-smi output showed 42759 MiB of GPU memory in use after the attempted restart.
Hypothesis: The old vLLM process (or another Python process) was not properly terminated and is still holding GPU memory.
Action: Force-kill all vLLM and Python processes with SIGKILL, which cannot be ignored.
Verification: Wait 5 seconds for driver cleanup, then check GPU memory usage.
Result: No output (no errors, no memory readings) — likely indicating the memory dropped to zero or the command produced no stdout. The absence of output is itself meaningful. In the context of shell scripting, "(no output)" after an nvidia-smi query is unusual — the command should print memory values. This could mean the SSH connection failed, the command timed out, or the output was captured differently. But the assistant does not treat this as an error; it proceeds to the next step in the following message, suggesting that the cleanup was successful enough to continue.

Conclusion

Message 6969 is a masterclass in the practical realities of ML infrastructure. It is not glamorous — there is no model training, no novel architecture, no benchmark result. It is a simple process cleanup. But it represents the invisible work that makes all the visible work possible. The assistant could have ignored the lingering GPU memory and tried to start vLLM again, only to have it crash with a CUDA out-of-memory error. Instead, it recognized the problem, applied the correct fix, and verified the result. This is the kind of engineering judgment that separates production deployments from research prototypes, and it is precisely the kind of detail that is worth examining in depth.

The Unseen Cleanup: Why a Simple pkill -9 Reveals the Reality of ML Engineering