The Stubborn GPU: A Case Study in Process Management During ML Inference Optimization

In the course of optimizing a large-scale inference pipeline for the Kimi-K2.5 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single bash command reveals the messy reality of GPU memory management in production ML environments. Message [msg 3847] is a deceptively simple command — a forceful attempt to kill lingering processes and free GPU memory — but its output tells a story of incomplete cleanup, stubborn allocations, and the friction inherent in managing multi-GPU systems.

The Context: A KV Cache Bottleneck

The message sits at a critical inflection point in a broader optimization effort. Earlier in the conversation, the user had observed that SGLang inference throughput was only around 500 tokens per second and asked what was limiting performance ([msg 3831]). The assistant investigated and diagnosed the bottleneck: the KV cache was saturated. With --mem-fraction-static 0.85, the server could hold only about 116,000 tokens in its KV cache. Given that the B2_opencodeinstruct dataset had an average response length of ~4,100 tokens, this meant only about 28 concurrent requests could fit at steady state — far below the configured concurrency of 150 ([msg 3834]).

The solution identified was to enable SGLang's hierarchical cache (--enable-hierarchical-cache), which spills KV cache entries to host RAM when GPU memory is exhausted. With 408 GB of available host memory, the assistant calculated that 300 GB of hierarchical cache could increase effective KV capacity from 116K tokens to roughly 410K tokens, supporting about 100 concurrent requests at 4K average length ([msg 3842]).

But before the new server configuration could be launched, the old server and inference processes needed to be killed and GPU memory freed.

The Message: A Forceful Cleanup Attempt

The message itself is a bash command executed via SSH on the remote machine:

ssh root@10.1.230.174 'kill -9 155500 156388 156530 156531 156532 156533 156534 156535 156536 156537 156538 156539 2>/dev/null; sleep 3; fuser -k /dev/nvidia* 2>/dev/null; sleep 2; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'

This command chains three operations. First, it sends SIGKILL to a list of twelve specific process IDs — these were the zombie processes identified in the previous message ([msg 3846]) that were still holding references to NVIDIA devices despite earlier pkill attempts. The 2>/dev/null suppresses errors for any PIDs that may have already died. After a three-second pause to allow the kernel to clean up, the command runs fuser -k /dev/nvidia*, which identifies and kills any remaining processes with open file handles on NVIDIA device files. Another two-second pause follows, and finally nvidia-smi queries the memory usage of all eight GPUs to verify the cleanup.

The Unexpected Result: GPU 3 Holds On

The output reveals a surprising partial failure:

 194909 194909 194909 194909 194909 194909 194909 194909 1949090, 0 MiB
1, 1 MiB
2, 97 MiB
3, 92134 MiB
4, 99 MiB
5, 3 MiB
6, 3 MiB
7, 3 MiB

The first line is a concatenation of two outputs: the fuser -k command printed the PID it killed (194909, repeated because it matched multiple device files), and then the nvidia-smi header for GPU 0 got appended without a newline. The actual GPU memory states are: GPU 0 at 0 MiB, GPU 1 at 1 MiB, GPUs 2 and 4 with negligible residual allocations (97 MiB and 99 MiB respectively), GPUs 5-7 essentially free at 3 MiB each. But GPU 3 shows 92,134 MiB used — the full model weight allocation is still resident on that GPU.

This is the critical finding. Six of the seven GPUs were successfully freed, but GPU 3 stubbornly retained its full 92 GB allocation. The kill -9 on the specific PIDs and the fuser -k sweep had both failed to release GPU 3's memory.## Why GPU 3 Wouldn't Let Go

Understanding why GPU 3 remained allocated requires examining the process hierarchy and how NVIDIA's CUDA driver manages GPU memory. The kill -9 command targeted twelve specific PIDs that were identified as zombie processes in the previous message ([msg 3846]). These were the processes that ps aux had shown still running. However, the output from that earlier command was truncated in the conversation — it showed process IDs like 155500 and 156388 and many more, but the full list was cut off. It's possible that the process actually holding GPU 3's memory (PID 194909, as revealed by fuser -k) was not among the twelve explicitly killed.

The fuser -k /dev/nvidia* command is a more thorough approach — it doesn't rely on knowing specific PIDs. Instead, it queries the kernel for all processes that have open file descriptors on the NVIDIA device files (/dev/nvidia0 through /dev/nvidia7, plus /dev/nvidiactl and /dev/nvidia-modeset). This should catch any process that has called cudaSetDevice() or otherwise established a CUDA context on any GPU. The fact that it killed PID 194909 but GPU 3 still showed 92 GB allocated suggests one of two scenarios.

The first possibility is that a child process or subprocess had inherited the CUDA context and was still alive, but didn't have an open file handle on /dev/nvidia* — perhaps it was a thread within a parent process that was killed but the CUDA driver's memory cleanup hadn't completed within the two-second sleep. The second, more likely possibility is that the CUDA driver itself was holding the memory due to a stuck or unresponsive GPU kernel. When a CUDA process crashes or is killed, the driver asynchronously cleans up its memory allocations. But if a GPU kernel is in an unrecoverable state — for instance, if a CUDA graph capture was interrupted mid-execution — the driver may refuse to release the memory until the GPU is reset.

The asymmetry is telling: six GPUs freed cleanly, one did not. This pattern is consistent with a GPU that was executing a CUDA graph when the process was killed. SGLang uses CUDA graph capture for optimization (as seen in the logs: "Capture cuda graph end. Time elapsed: 13.81 s"). If the capture or execution of a CUDA graph was interrupted on GPU 3 specifically — perhaps because it was the last GPU to finish a collective operation — the driver might hold the memory allocation as a safety measure.

The Broader Pattern: GPU Memory Management Is Hard

This message exemplifies a recurring challenge in the entire session: GPU memory management on multi-GPU systems is fragile. Earlier in the conversation ([msg 3845]), the assistant had tried pkill -9 -f sglang, pkill -9 python3, and fuser -k /dev/nvidia* but the GPUs still showed 92 GB allocated. The processes were "zombie" — dead from the kernel's perspective but their CUDA contexts hadn't been released. It took multiple rounds of increasingly aggressive cleanup to finally free the GPUs.

The user's response to this message ([msg 3848]) was simply "Use all levers" — acknowledging that the cleanup was incomplete and authorizing even more aggressive measures. The assistant then had to kill PID 194909 again ([msg 3849]), and only after that second attempt did all eight GPUs finally show 0 MiB used.

Input Knowledge Required

To fully understand this message, the reader needs to know several things. One must understand that CUDA GPU memory is allocated by the CUDA driver and persists even after the allocating process dies — it is not automatically freed by the kernel's normal process cleanup. One must know that fuser -k kills processes by their file handles, which is a more reliable way to catch CUDA-using processes than PID-based killing. One must understand the SGLang architecture: that it uses tensor parallelism across 8 GPUs, meaning a single server process spawns 8 worker processes (one per GPU), and all must be killed for memory to be freed. And one must know the context of the optimization effort — that the assistant was trying to restart the server with hierarchical cache enabled to solve a KV cache bottleneck.

Output Knowledge Created

This message produces concrete knowledge about the state of the system. It confirms that most GPUs can be freed with aggressive process killing, but it also reveals that GPU 3 has a persistent allocation that requires further intervention. This output directly drives the next action: the user says "Use all levers," and the assistant proceeds to kill PID 194909 again, finally succeeding in freeing all GPUs. The message also implicitly documents a failure mode of GPU process management — that even kill -9 combined with fuser -k may not release all GPU memory on the first attempt, especially in multi-GPU configurations with CUDA graph capture.

Assumptions and Potential Mistakes

The assistant made several assumptions in this message. It assumed that a three-second sleep after kill -9 was sufficient for the kernel to clean up process entries and release file handles. It assumed that fuser -k would catch all CUDA-related processes. It assumed that the nvidia-smi output would accurately reflect memory availability after cleanup. The first two assumptions proved partially incorrect — GPU 3 remained allocated. The third assumption was correct: nvidia-smi reliably reports GPU memory state, and its output was the diagnostic that revealed the incomplete cleanup.

A potential mistake was not checking whether the twelve PIDs being killed actually included the process holding GPU 3. The earlier ps aux output was truncated, so the assistant may have been working with an incomplete list. A more robust approach would have been to use fuser -k first (without the explicit PID list) and only fall back to PID-based killing if needed. The assistant did eventually use fuser -k, but the explicit PID list was redundant and potentially misleading.

Conclusion

Message [msg 3847] is a small but revealing moment in a larger optimization narrative. It shows that even a seemingly simple operation — "kill the old server and free GPU memory" — can fail in interesting ways on multi-GPU systems. The persistent allocation on GPU 3 forced the assistant and user to iterate, ultimately requiring a second cleanup pass. This kind of friction is the daily reality of ML engineering at scale: the elegant theory of pipeline optimization constantly bumps against the messy practice of GPU resource management. The message is a reminder that in production ML systems, the hardest problems are often not the algorithms but the infrastructure.