Debugging CUDA at the Container Boundary: A Case Study in LXC GPU Passthrough Recovery

Introduction

In the course of deploying and benchmarking large language models on high-end GPU hardware, infrastructure failures are inevitable. When they strike, the boundary between the application layer and the system layer becomes the critical frontier for debugging. Message [msg 11305] captures a pivotal moment in one such debugging session: an AI assistant working within an LXC container on a Proxmox VE host, trying to diagnose why CUDA initialization fails after a host reboot. The message is a masterclass in systematic diagnostic reasoning, revealing the assistant's mental model of the GPU software stack, its understanding of container-virtualized hardware access, and its methodical narrowing of a "CUDA unknown error" from a generic symptom to a specific root cause.

The Message

The message begins with the assistant's internal reasoning, then executes a diagnostic bash command, and finally presents the results:

Agent Reasoning The container doesn't have kernel modules in its filesystem (it's using the host's kernel 6.14.11-9-bpo12-pve). The nvidia_uvm is in use so we can't unload it, and we can't modprobe because the kernel modules aren't in the container.

This is a PVE (Proxmox) LXC container with GPU passthrough. The GPU driver is managed by the host. After a host reboot, the GPU state visible to the container might be stale or the device nodes might need to be recreated by the host.

This needs to be fixed from the host side. The user needs to: 1. Restart the LXC container from the Proxmox host 2. Or run nvidia-smi on the host to reinitialize the GPUs 3. Or check if the host's GPU passthrough is properly configured

But wait—nvidia-smi -L actually works and lists all 8 GPUs, so the driver interface is functional. The real issue is that the CUDA API fails at cuInit, which suggests either a version mismatch between the host driver and what the container expects, the GPU's compute mode being misconfigured, or a known incompatibility with specific driver versions in containers.

Let me check the GPU's compute mode and see if I can adjust it.

The assistant then runs a bash command that checks GPU compute mode, attempts to set it explicitly, queries power state and clock information, and performs a raw CUDA driver API call to get the driver version and test cuInit directly. The output reveals that all 8 GPUs are in "Default" compute mode with ECC enabled, and the raw CUDA driver call succeeds in getting the driver version (13020, indicating CUDA 13.2) but still fails at cuInit with error code 999.

Context: How We Got Here

To understand the significance of this message, we must trace the events that led to it. The broader session (Segment 63) involves benchmarking speculative decoding techniques—specifically DFlash and DDTree—on a machine called CT200 equipped with 8× RTX PRO 6000 Blackwell GPUs. The assistant had been running a comprehensive benchmark plan for the Qwen3.6-27B model, comparing autoregressive baselines against various speculative decoding configurations.

A host reboot during networking maintenance shattered this workflow. When the assistant attempted to resume benchmarking, it encountered a cryptic failure: "CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start." This error is notorious in the CUDA ecosystem—it provides almost no actionable information, and its suggestion about CUDA_VISIBLE_DEVICES is often a red herring.

The assistant spent several preceding messages ([msg 11298] through [msg 11304]) methodically ruling out possibilities:

Driver loaded? Yes—nvidia-smi works and lists all 8 GPUs.
Device nodes present? Yes—/dev/nvidia0 through /dev/nvidia7 exist, as does /dev/nvidia-uvm.
Persistence mode? Was off, now enabled.
GPU reset? Not supported on these GPUs.
CUDA runtime version mismatch? The venv uses CUDA 13.0, the driver reports CUDA 13.2—this should be backward compatible.
Different PyTorch venv? Both CUDA 12.8 and CUDA 13.0 venvs fail identically.
Kernel module reload? nvidia_uvm is in use and can't be unloaded; the container lacks kernel modules anyway. Each negative result tightened the diagnostic noose. By message [msg 11305], the assistant has converged on the critical insight: this is an LXC container with GPU passthrough, and the fix must come from the host.

The Reasoning Process: A Window into Systematic Debugging

The assistant's reasoning in this message is notable for its structure. It begins by stating what it has learned from the previous failed attempts: the container cannot reload kernel modules because they live in the host's filesystem, not the container's. This is a fundamental property of LXC containers—they share the host kernel but have their own userspace. The nvidia_uvm module is "in use" because the container's processes hold references to it through the device nodes.

The reasoning then makes a critical conceptual leap: "The GPU driver is managed by the host. After a host reboot, the GPU state visible to the container might be stale." This is the key insight. In an LXC GPU passthrough setup, the host's NVIDIA driver initializes the GPU hardware during boot. The container inherits access to the device nodes, but if the driver's initialization sequence didn't complete properly—or if the container started before the GPU was fully ready—the CUDA runtime inside the container will see a broken state.

The assistant then enumerates three possible host-side fixes: restart the container, run nvidia-smi on the host, or check GPU passthrough configuration. This list is pragmatic and prioritized—restarting the container is the most likely to work and the simplest to execute.

But then the assistant catches itself with "But wait—nvidia-smi -L actually works." This self-correction is a hallmark of good debugging. If nvidia-smi works, the driver interface is functional at some level. The assistant reframes the problem: the issue is specifically with cuInit, the CUDA driver API entry point that initializes the driver state. This suggests a more subtle problem—perhaps a compute mode issue, a version mismatch, or a known container-specific incompatibility.

The decision to check compute mode is logical. If a GPU is in "Exclusive Process" or "Prohibited" mode, CUDA initialization would fail. But the output shows "Default" for all GPUs, ruling this out. The raw CUDA driver API test is clever—by calling cuDriverGetVersion and cuInit directly via ctypes, the assistant bypasses PyTorch's error handling to get the raw error codes. The result—driver version 13020 (CUDA 13.2) but cuInit returning 999—confirms the driver is loaded but its initialization fails.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

Container technology: Understanding that LXC containers share the host kernel but have isolated userspaces. The container cannot load kernel modules—it relies on the host for that. The nvidia_uvm module being "in use" means processes in the container hold file descriptors to the device node.

CUDA software stack: The distinction between the NVIDIA driver (kernel module + userspace library libcuda.so), the CUDA runtime (libcudart.so), and PyTorch's CUDA integration. The cuInit function is the entry point to the driver API—if it fails, nothing CUDA-related will work.

GPU passthrough in virtualization: Proxmox VE uses LXC containers with device node passthrough for GPU access. The container sees the GPU devices but doesn't control the hardware initialization—that happens on the host during boot.

Error code semantics: CUDA error 999 is CUDA_ERROR_UNKNOWN, a catch-all that typically indicates a driver-level failure rather than a programming error. It's different from error 35 (CUDA driver version mismatch) or error 100 (no device).

The specific hardware: RTX PRO 6000 Blackwell GPUs are professional-grade cards with ECC memory enabled by default, which can affect initialization sequences.

Output Knowledge Created

This message creates several valuable outputs:

A confirmed diagnosis: The problem is specifically a cuInit failure in an LXC container post-reboot, not a runtime version mismatch, not a missing device node, and not a compute mode issue.
A prioritized action plan: Three host-side fixes ranked by likelihood of success.
A negative result that eliminates possibilities: Compute mode is correct, the driver version is compatible, and the raw driver API is reachable.
A reproducible diagnostic procedure: The raw CUDA driver API test via ctypes is a reusable technique for distinguishing between driver-level and runtime-level failures.
Documentation of a specific failure mode: CUDA error 999 in LXC containers after host reboot, where nvidia-smi works but cuInit fails, is a distinctive symptom pattern.

Assumptions and Potential Mistakes

The assistant makes several assumptions worth examining:

Assumption that the container was restarted after the host reboot: This is likely correct—LXC containers typically auto-start after a host reboot if configured to do so. But the assistant doesn't verify this explicitly.

Assumption that nvidia-smi working implies the driver is fully functional: This is partially correct. nvidia-smi uses a different communication path with the driver than CUDA does. It's possible for nvidia-smi to work while cuInit fails, as this case demonstrates. The assistant correctly identifies this nuance.

Assumption that the fix must come from the host: This is the correct conclusion given the constraints, but the assistant doesn't explore one more option: could the container's CUDA libraries be replaced or patched to work around the driver state? This would be a fragile solution but might be faster than involving the host admin.

Potential mistake: Not checking the host's nvidia-smi output: The assistant is working from within the container and can't directly access the host. But the error pattern—nvidia-smi works, cuInit fails—is actually a known issue with certain NVIDIA driver versions in virtualized environments, sometimes caused by the GPU being in a "frozen" state after a VM/LXC migration or reboot.

Assumption about ECC memory: The assistant notes that ECC is enabled but doesn't explore whether ECC initialization might be the culprit. On some GPU models, ECC memory initialization can take significant time after power-on, and CUDA calls made before initialization completes will fail with error 999.

The Broader Significance

This message is more than a debugging log—it's a case study in how AI systems reason about complex infrastructure problems. The assistant demonstrates several hallmarks of expert troubleshooting:

Progressive elimination: Each hypothesis is tested with a concrete experiment, and the results narrow the search space.
Layered abstraction: The assistant moves from the application layer (PyTorch error) to the runtime layer (CUDA version) to the driver layer (cuInit) to the kernel layer (module loading), understanding which layer is responsible for each failure.
Constraint awareness: The assistant recognizes that it's operating within a container and that certain actions (module reloading, GPU reset) are impossible from its position.
Self-correction: The "But wait" moment shows the assistant re-evaluating its conclusion when new evidence contradicts it.
Actionable output: The message doesn't just diagnose—it produces a prioritized list of next steps for the user. The message also illustrates a fundamental tension in AI-assisted system administration: the AI can diagnose problems with remarkable sophistication, but when the fix requires actions at a lower privilege level (the host), it must hand off to the human operator. The assistant's ability to clearly communicate what needs to happen and why is as important as its diagnostic skill.

Conclusion

Message [msg 11305] captures a turning point in a complex debugging session. After eliminating a cascade of possible causes—runtime version mismatch, missing device nodes, GPU compute mode, kernel module state—the assistant converges on the root cause: a CUDA driver initialization failure in an LXC container after host reboot, where the container's GPU passthrough state has become stale. The message demonstrates systematic reasoning, domain knowledge spanning virtualization, GPU architecture, and the CUDA software stack, and the ability to produce actionable diagnostics from ambiguous error signals. It's a small but illuminating window into how AI systems reason about the messy reality of production infrastructure.