When GPUs Refuse to Let Go: A Case Study in Container-Level CUDA Context Staleness

The Message

[assistant] Kernel-level mount — likely CUDA context stuck. Need to reset the GPUs:
[bash] ssh root@10.1.230.174 'nvidia-smi -r 2>&1 || echo "GPU reset not available in container"; nvidia-smi --gpu-reset -i 0 2>&1; nvidia-smi --gpu-reset -i 2 2>&1'
Resetting GPU 00000000:01:00.0 is not supported.
Resetting GPU 00000000:11:00.0 is not supported.
Resetting GPU 00000000:61:00.0 is not supported.
Resetting GPU 00000000:71:00.0 is not supported.
Resetting GPU 00000000:81:00.0 is not supported.
Resetting GPU 00000000:91:00.0 is not supported.
Resetting GPU 00000000:E1:00.0 is not supported.
Resetting GPU 00000000:F1:00.0 is not supported.
GPU reset not available in container

This single message, at index 3859 in a long and complex coding session, captures a moment of crisis and discovery. It is the culmination of a cascade of failures that began with an ambitious optimization attempt and ended with two GPUs locked in an unreleasable state, their 96 GiB of VRAM each held hostage by a kernel-level CUDA context that no amount of process-killing could evict. The message is brief — a single bash command and its output — but it tells a story about the boundary between user-space software and kernel-level GPU state, about the assumptions we make when managing accelerators inside containers, and about the moment when the standard toolkit of pkill, fuser, and kill -9 meets its match.

The Road to the Crash

To understand why this message was written, we must trace the events that led to it. The session was deep into optimizing throughput for a large-scale inference pipeline. The assistant had been running the Kimi-K2.5 model (a massive 671B-parameter Mixture-of-Experts architecture) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang, a high-performance inference engine. The throughput bottleneck had been identified as KV cache capacity: with --mem-fraction-static 0.85, the server could only hold about 116,000 tokens in its GPU-side KV cache, limiting concurrent requests to roughly 50 at the observed average sequence length of 4,000 tokens.

The user had given a clear directive at [msg 3848]: "Use all levers." The assistant interpreted this as a mandate to push every available optimization to its maximum. Among the levers available was SGLang's hierarchical cache (--enable-hierarchical-cache), a feature that spills KV cache entries to host RAM when GPU memory is full, dramatically increasing the effective cache capacity. The idea was sound: the machine had 449 GiB of host RAM, and the KV cache per token was small enough (roughly 137 KB per token per GPU) that spilling to CPU memory could support hundreds of thousands of additional tokens.

The mistake was in the parameter value. The assistant launched the server with --hicache-size 300, assuming this specified the total host memory to allocate for the hierarchical cache across all ranks. In reality, as discovered moments later at [msg 3854], --hicache-size is interpreted per TP rank. With tensor parallelism set to 8, this meant each of the 8 Python processes attempted to allocate 300 GiB of host RAM, for a total demand of 2.4 TiB — far exceeding the 449 GiB physically available. The result was a catastrophic OOM: 439 GiB of RAM consumed, 9 GiB free, processes hanging, and the server failing to start.## The Aftermath: Cleaning Up the Mess

The assistant's response to the OOM was methodical. First, it killed the SGLang processes with pkill -9 -f sglang and pkill -9 python3, then used fuser -k /dev/nvidia* to release any file handles on the NVIDIA devices. These are the standard incantations for cleaning up GPU state after a crash. But when the assistant checked nvidia-smi at [msg 3856], the results were alarming: GPU 0 showed 96,413 MiB used (98.5% of its 97,887 MiB VRAM), GPU 2 showed 96,417 MiB used, and the host RAM was still showing 318 GiB consumed — leaked allocations from the failed hicache initialization.

A second round of fuser -k -9 /dev/nvidia* at [msg 3857] produced no change. The GPUs were still held. The assistant then ran fuser -v to identify exactly which processes held the devices, and the output at [msg 3858] revealed the root cause:

                     USER        PID ACCESS COMMAND
/dev/nvidia0:        root     kernel mount /dev/nvidia0
/dev/nvidia2:        root     kernel mount /dev/nvidia2

The critical detail is the "kernel mount" access type. Normal user-space processes show a PID and a command name. A "kernel mount" entry means the device file is being held by a kernel-level CUDA context — a state that persists even after all user-space processes have been killed. This can happen when a CUDA application crashes without properly cleaning up its GPU context, or when the CUDA driver's reference counting on the device file gets stuck. In this case, the OOM crash during hicache initialization likely triggered an abnormal CUDA driver state where the context was never released.

The Failed Reset Attempt

The subject message at index 3859 is the assistant's attempt to escalate from process-level cleanup to hardware-level reset. The command nvidia-smi -r requests a GPU reset operation, which would forcibly tear down all CUDA contexts and return the GPU to a clean state. The output is unambiguous: "Resetting GPU ... is not supported" for all eight GPUs, followed by "GPU reset not available in container."

This is the key insight of the message. The nvidia-smi -r (or nvidia-smi --gpu-reset) command requires direct hardware access that is typically not available inside a container or virtual machine. Even though the container has full access to the GPUs via NVIDIA's container toolkit (nvidia-container-toolkit), the GPU reset capability is gated behind the NVIDIA_RESET_DEVICES environment variable or specific driver-level permissions that are not enabled by default. The Proxmox LXC container in this setup, despite having GPU passthrough, did not expose the reset capability.

The message thus represents a boundary condition: the assistant had exhausted every tool available within its operating environment. Process killing, file handle release, and even GPU reset were all attempted and failed. The GPUs were stuck in a state that could only be resolved from outside the container — either by the Proxmox host directly manipulating the GPU state, or by a full system reboot.

Assumptions Made and Lessons Learned

Several assumptions are visible in this message and the events leading up to it. First, the assistant assumed that --hicache-size 300 was a total allocation rather than a per-rank allocation. This is a reasonable assumption — many distributed computing parameters specify total resources to be divided among ranks — but it proved incorrect. The SGLang documentation or codebase apparently does not make this distinction clear, and the assistant did not verify by reading the source code before launching. (It did check the source code later at [msg 3862], after the crash.)

Second, the assistant assumed that the standard cleanup sequence (pkill, fuser -k, nvidia-smi -r) would be sufficient to recover from any server crash. This assumption had held true throughout the many previous server restarts in this session. But the OOM crash during hicache initialization created a qualitatively different failure mode — a kernel-level CUDA context staleness that normal cleanup could not address.

Third, there was an implicit assumption that the container environment would support GPU reset operations. Many container setups do, especially when using --privileged mode or specific NVIDIA driver configurations. This particular Proxmox LXC container did not.

The Resolution

The assistant's next move, at [msg 3860], was to escalate to the Proxmox host. By SSHing into the host at 10.1.2.6 and using pct exec 129 (Proxmox Container Toolkit's exec command for container ID 129), the assistant was able to run commands inside the container from the host level. This worked: after killing all Python processes from the host side, the GPUs showed 0 MiB used across all eight devices. The host RAM also returned to normal (3 GiB used, 444 GiB free) at [msg 3861].

This resolution reveals an important architectural detail: the kernel-level CUDA context was tied to the container's PID namespace. When processes were killed from inside the container, the kernel context persisted because the container's PID namespace didn't fully propagate the cleanup to the kernel driver layer. But when the same kill commands were issued from the host (outside the container's PID namespace), the cleanup was effective. This is a subtle interaction between Linux container PID namespaces and the NVIDIA driver's reference counting mechanism.

Broader Significance

This message, while brief, encapsulates a class of failure that is increasingly common in modern ML infrastructure. As practitioners push hardware to its limits — running massive models across multiple GPUs with aggressive memory optimization — they inevitably encounter the boundary conditions where software assumptions break down. The hierarchical cache feature is designed to gracefully handle KV cache overflow, but its per-rank parameter semantics created a resource allocation 8× larger than intended. The cleanup tools that work for normal crashes fail for kernel-level context staleness. The GPU reset that works on bare metal is unavailable inside containers.

The message also demonstrates a valuable debugging methodology: when standard tools fail, use progressively more specific diagnostics. The assistant moved from nvidia-smi (showing memory usage), to fuser -k (killing file handles), to fuser -v (identifying exact access types), to nvidia-smi -r (attempting hardware reset). Each step revealed more about the nature of the failure, culminating in the understanding that this was a kernel-level mount issue requiring host-level intervention.

For anyone managing GPU-accelerated containers, the lesson is clear: know your container's GPU reset capabilities before you need them, and always have a path to the host for emergencies. A simple nvidia-smi -r test at container setup time could save hours of debugging later. And when pushing the boundaries of memory allocation — especially with features like hierarchical cache — verify parameter semantics against the source code before launching, not after.