The Brutal Art of GPU Cleanup: A Single Command That Made Production Deployment Possible

In the life of any machine learning engineer deploying a large language model to production, there comes a moment when all the careful architecture, the meticulous benchmarking, and the painstaking configuration work culminate in a single, decisive action. For the deployment of the Kimi-K2.5 INT4 model with EAGLE-3 speculative decoding across eight NVIDIA RTX PRO 6000 Blackwell GPUs, that moment arrived in the form of a single bash command executed remotely via SSH:

[assistant] [bash] ssh root@10.1.230.174 'fuser -k /dev/nvidia* 2>/dev/null; sleep 3; fuser /dev/nvidia0 2>/dev/null && echo "still busy" || echo "GPUs free"'
 189181 189181 189181 189181 189181 189181 189181 189181 189181 189181GPUs free

This message ([msg 5669]) is the 5,669th exchange in a marathon coding session that spanned environment setup, CUDA toolkit installation, flash-attn compilation, EAGLE-3 speculation tuning, and finally, production hardening. On its surface, it is unremarkable — a routine cleanup command. But in the context of the session's arc, it represents a critical inflection point: the transition from experimental, manually-managed inference serving to a hardened, systemd-controlled production deployment that would auto-start on boot and survive reboots.

The Context: A Long Road to Production

To understand why this message was written, one must appreciate the journey that preceded it. The session had spent dozens of rounds optimizing EAGLE-3 speculative decoding for the Kimi-K2.5 INT4 model. The team had tested four configurations — baseline (no speculation), topk=4 with the v1 non-overlap scheduler, topk=1 with the v2 overlap scheduler, and an older CUDA 12.8 setup — and benchmarked them across concurrency levels from 1 to 250 concurrent requests. The winner was clear: topk=1 with spec_v2 overlap scheduling, which matched or beat baseline throughput at high concurrency while retaining the latency benefits of speculation.

With the configuration finalized, the user's instruction in [msg 5659] was unambiguous: "Save findings, on the machine - save /root/production_v2.md with details + update prod deployment (systemd and all) to run this exact setup, start on boot etc." This directive set in motion a chain of events that would culminate in message 5669.

The Problem: Zombie Processes and Stubborn GPU Handles

The assistant's first attempt at transitioning to systemd management was straightforward. In [msg 5666], it killed the existing nohup-managed server process (PID 184364) using a simple kill command:

ssh root@10.1.230.174 'kill 184364 2>/dev/null; sleep 3; ps aux | grep "launch_server" | grep -v grep | wc -l'

The output confirmed zero launch_server processes remained. But this was deceptive. In [msg 5668], the assistant checked whether the GPUs were actually free using fuser /dev/nvidia0:

ssh root@10.1.230.174 'fuser /dev/nvidia0 2>/dev/null && echo "GPUs still busy" || echo "GPUs free"'
 189181GPUs still busy

PID 189181 — a process that was not caught by the kill command — was still holding /dev/nvidia0 open. This is a common pitfall in GPU-accelerated environments. The main Python process may spawn child processes (for NCCL communication, CUDA graph capture threads, or multiprocessing workers) that survive the parent's death. Alternatively, the process may have been a leftover from an earlier server launch that wasn't part of the main process tree. Regardless, the GPU devices were locked, and starting a new server would fail with a "CUDA error: device already in use" or similar error.

The Solution: Fuser's Nuclear Option

Message 5669 deployed the solution: fuser -k /dev/nvidia*. The fuser command identifies processes that have opened a given file or socket. The -k flag sends SIGKILL to those processes. By targeting /dev/nvidia* (a glob that matches /dev/nvidia0, /dev/nvidia1, etc., up to /dev/nvidia7 for eight GPUs, plus /dev/nvidiactl and /dev/nvidia-uvm), the command ensures that any process holding any NVIDIA device file is terminated.

The output reveals the effectiveness: 189181 is printed ten times, corresponding to the ten NVIDIA device files that this process had open (eight GPU devices, plus the control device and UVM device). After a three-second sleep to allow the kernel to clean up, the verification command fuser /dev/nvidia0 returns no PIDs, and the echo reports "GPUs free."

Why This Matters: The Assumptions and Their Failure

This sequence reveals a critical assumption that the assistant made — and that was proven wrong. The assumption was that killing the main Python process (the launch_server process) would release all GPU resources. This is a reasonable assumption: in a well-behaved application, child processes should exit when their parent dies, and CUDA contexts should be cleaned up when the owning process terminates.

However, several factors can violate this assumption:

  1. Orphaned child processes: If the parent process spawned children with subprocess.Popen without proper cleanup, those children may continue running after the parent is killed. In SGLang's architecture, the launch_server process may spawn NCCL communication threads, CUDA graph compilation workers, or multiprocessing pool workers that don't automatically terminate with the parent.
  2. CUDA context persistence: NVIDIA's CUDA driver maintains per-process GPU state. If a process crashes or is killed without proper cleanup, the CUDA driver may not immediately release all resources. However, in this case, the process was still alive (PID 189181 was running), so the issue was simpler: a process that simply hadn't been killed yet.
  3. The kill command's scope: The initial kill 184364 targeted a specific PID. But the server may have had multiple PIDs associated with it — especially if it used multiprocessing or if there were leftover processes from previous runs. The fuser -k approach is more comprehensive because it identifies processes by the resources they hold, not by their PID or parentage.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains:

Output Knowledge Created

This message produced several important pieces of knowledge:

  1. The GPUs are now free: The immediate output confirms that no process is holding any NVIDIA device file. This is the precondition for starting the new systemd service.
  2. PID 189181 was the culprit: The repeated output of 189181 identifies the specific process that was blocking GPU access. This is useful for forensic analysis — if this pattern recurs, the operator knows to look for processes with similar characteristics.
  3. A validated cleanup procedure: The sequence of kill → verify → fuser-kill → verify establishes a reliable cleanup pattern for this environment. Future deployments can follow the same steps.
  4. Confidence in the systemd transition: With the GPUs confirmed free, the assistant can proceed to start the systemd service without fear of conflicts. The next step would be systemctl start sglang-kimi and systemctl enable sglang-kimi.

The Thinking Process: A Study in Incremental Debugging

The reasoning visible in the sequence of messages from 5665 to 5669 reveals a classic debugging pattern: try the simplest solution first, verify, and escalate if unsuccessful.

Step 1 (msg 5665): Create the systemd service file. This is the infrastructure preparation — writing the unit file that will manage the server process.

Step 2 (msg 5666): Kill the existing nohup process. The assistant uses kill 184364 — the PID of the launch_server process identified earlier. This is the gentle approach: send SIGTERM to the main process and trust that it will clean up its children.

Step 3 (msg 5667): Verify that no Python processes remain. The assistant checks ps aux | grep python3 | grep -v grep | wc -l and gets 0, confirming no Python processes are running. But this check is too narrow — it only looks for processes named "python3," not for any process using GPU resources.

Step 4 (msg 5668): Verify that GPUs are actually free. The assistant uses fuser /dev/nvidia0 — a more targeted check that directly tests whether the GPU device file is available. The result is "GPUs still busy" with PID 189181 identified.

Step 5 (msg 5669): Escalate to fuser -k. The assistant now uses the nuclear option: kill any process holding any NVIDIA device file. After a three-second wait, the verification confirms "GPUs free."

This incremental approach — from gentle to aggressive, from process-based to resource-based — is characteristic of experienced system administrators. Each step builds on the information from the previous step, and the escalation only happens when the simpler approach is proven insufficient.

The Broader Significance

Message 5669, for all its apparent simplicity, represents the culmination of hundreds of rounds of work: CUDA toolkit installation, flash-attn compilation, NCCL tuning, EAGLE-3 speculation optimization, and systemd service configuration. All of that work would be for nothing if the server couldn't start because a zombie process was holding the GPUs.

The command also highlights a fundamental truth about GPU-accelerated computing: GPUs are shared, exclusive resources. Unlike CPU processes, which can be freely started and stopped, GPU contexts leave traces that can block subsequent users. Proper cleanup is not a nicety — it is a necessity for reliable production operation.

In the end, the message is a testament to the importance of operational discipline. The assistant didn't just write the systemd service and assume it would work. It verified each step, checked for hidden state, and cleaned up thoroughly before proceeding. This attention to operational detail is what separates a fragile experimental setup from a robust production deployment.