When CUDA Goes Silent: Debugging GPU Initialization in LXC Containers After a Host Reboot

Introduction

In the high-stakes world of machine learning infrastructure, few moments are as frustrating as watching a perfectly configured environment suddenly refuse to initialize its GPUs. Message [msg 11304] captures one such moment: the assistant, deep in a multi-day benchmarking campaign across eight NVIDIA RTX PRO 6000 Blackwell GPUs, hits a wall when CUDA initialization fails with the cryptic error code 999 — CUDA_ERROR_UNKNOWN. This message is a masterclass in systematic debugging at the system-software boundary, where the assistant must distinguish between application-level issues, runtime version mismatches, and genuine driver-level failures, all while operating inside the constrained environment of an LXC container.

The message is brief in its visible output but dense with reasoning. It represents the culmination of a diagnostic chain that began several messages earlier, when a host reboot (triggered by networking maintenance) broke the GPU compute environment. By the time we reach [msg 11304], the assistant has already ruled out PyTorch version differences, CUDA runtime compatibility, and device node permissions. What remains is a stubborn, low-level driver initialization failure that cannot be fixed from inside the container.

The Context: A Benchmarking Campaign Interrupted

To understand the significance of this message, one must appreciate the broader context. The assistant had been running an extensive benchmarking suite comparing speculative decoding strategies — DFlash, DDTree, and EAGLE-3 — across multiple model architectures (Qwen3.6-27B and Kimi K2.6) on a machine with eight RTX PRO 6000 Blackwell GPUs. The work was methodical: each benchmark phase (TP1, TP4, TP8) swept across budgets, concurrency levels, and context lengths, producing detailed JSON results that fed into a LaTeX report generator.

Then the host rebooted. And everything stopped.

The preceding messages ([msg 11297] through [msg 11303]) show the assistant's escalating diagnostic efforts. First, the benchmark runner itself fails with "Failed to get device capability: CUDA unknown error." Then nvidia-smi works fine — the driver is loaded, all eight GPUs are visible, memory usage is zero. But cudaGetDeviceCount returns 999. The cuInit(0) call returns 999. Two different PyTorch venvs (one with CUDA 13.0 runtime, one with CUDA 12.8) both fail identically. The device nodes at /dev/nvidia* exist and have correct permissions. The nvidia_uvm module is loaded. Everything looks correct, yet CUDA refuses to initialize.

The Reasoning: Narrowing Down to a Container-Boundary Problem

The assistant's reasoning in [msg 11304] is worth quoting in full, as it demonstrates a mature understanding of GPU system architecture:

Both venvs fail: - venv_sglang211: torch 2.11.0+cu130 - CUDA fails - venv: torch 2.11.0+cu128 - CUDA fails

>

cuInit returns 999 (CUDA_ERROR_UNKNOWN). This is a system-level issue, not a PyTorch or CUDA runtime version issue.

>

In a container, this usually means the host GPU driver/firmware needs to be re-initialized, or the container needs to be restarted. Since we can't restart the container or the host from within the container, we need external help.

This reasoning is significant for several reasons. First, the assistant correctly identifies that the problem is system-level rather than application-level. By testing two different PyTorch venvs with different CUDA runtime versions (12.8 and 13.0), the assistant eliminates the possibility of a runtime-driver version mismatch — which would typically produce error code 35 (CUDA_ERROR_INSUFFICIENT_DRIVER), not 999. Error 999 is CUDA_ERROR_UNKNOWN, a catch-all that in practice often indicates that the GPU firmware or driver state is corrupted or incomplete.

Second, the assistant correctly identifies the LXC container as the critical factor. In a bare-metal Linux installation, a CUDA_ERROR_UNKNOWN after reboot might be fixable by resetting the GPUs via nvidia-smi --gpu-reset or reloading the NVIDIA kernel modules. But inside an LXC container, the GPU devices are passed through from the host via cgroup device permissions. The container cannot reset the GPU hardware, cannot reload kernel modules (the modules live in the host kernel, not the container's filesystem), and cannot restart the NVIDIA persistence daemon. The container is, in effect, a passenger — it can see the GPUs through the device nodes, but it cannot control their initialization state.

The Attempted Fix: Reloading nvidia_uvm

The assistant's chosen intervention is to reload the nvidia_uvm kernel module — the module responsible for the Unified Virtual Memory (UVM) subsystem that enables CUDA Unified Memory and certain GPU memory management features. The command sequence is:

rmmod nvidia_uvm 2>&1 && echo 'nvidia_uvm removed' || echo 'cannot remove nvidia_uvm'
modprobe nvidia_uvm 2>&1 && echo 'nvidia_uvm loaded' || echo 'cannot load nvidia_uvm'

The results are instructive:

rmmod: ERROR: Module nvidia_uvm is in use
cannot remove nvidia_uvm
modprobe: FATAL: Module nvidia_uvm not found in directory /lib/modules/6.14.11-9-bpo12-pve
cannot load nvidia_uvm

Both commands fail, but for different reasons that reveal important constraints of the container environment. The rmmod fails because the module is "in use" — some process in the container (likely the NVIDIA userspace driver library or a lingering CUDA context) holds a reference to the module. The modprobe fails because the module file itself doesn't exist in the container's kernel module directory — the container runs on a Proxmox host (kernel 6.14.11-9-bpo12-pve), and the NVIDIA driver modules are installed on the host, not inside the container. The container can use the modules that are already loaded, but it cannot load or reload them itself.

This is a critical insight about LXC GPU passthrough: the container inherits the host's loaded kernel modules, but it cannot manage them. If the GPU initialization state becomes corrupted — as apparently happened during the host reboot — the container has no way to reset it from within. The only options are host-level intervention (restart the container, reload modules on the host, or reboot the host) or external help from someone with host access.

Assumptions and Their Consequences

The assistant makes several assumptions in this message, most of which are reasonable but turn out to be incorrect or incomplete:

  1. That reloading nvidia_uvm might fix the issue. This assumption is plausible — the UVM subsystem is involved in GPU memory management, and a corrupted UVM state could certainly prevent CUDA initialization. However, the error 999 from cuInit suggests a deeper problem, possibly at the GPU firmware or PCIe initialization level, that reloading a single module would not fix.
  2. That the module can be unloaded. The assistant assumes that no process holds a permanent reference to nvidia_uvm. In practice, the NVIDIA driver stack creates a tight coupling between nvidia_uvm and the main nvidia module — even if no CUDA program is running, the module may remain "in use" due to internal driver references.
  3. That modprobe would find the module. This assumption fails because the container doesn't have the NVIDIA kernel module files in its filesystem. The assistant didn't check whether the module file existed before attempting the reload — a step that would have saved time and provided earlier insight into the container's limitations.
  4. That the problem is fixable from within the container. This is the most consequential assumption. The assistant's reasoning correctly identifies that "the host GPU driver/firmware needs to be re-initialized, or the container needs to be restarted," but then attempts an in-container fix anyway. This is understandable — trying something is better than giving up — but the failure mode provides valuable information about the boundaries of what a container can control.

Knowledge Inputs and Outputs

To fully understand this message, the reader needs several pieces of input knowledge:

The Thinking Process: A Window into Systematic Debugging

The assistant's reasoning in [msg 11304] is the culmination of a multi-step diagnostic process visible across the preceding messages. Let me trace the full chain:

In [msg 11297], the benchmark runner fails with "CUDA unknown error." The assistant's first reaction is to check the GPU state via nvidia-smi and run a simple CUDA test ([msg 11298]). This reveals that nvidia-smi works but PyTorch CUDA fails.

In [msg 11299], the assistant tries GPU reset (nvidia-smi --gpu-reset) and a raw CUDA C API call via ctypes. The reset is "not supported" (a common limitation on newer GPUs), and cudaGetDeviceCount returns 999. The assistant also checks for the NVIDIA fabric manager and finds it's not running — a potential clue.

In [msg 11300], the assistant checks whether the system is in a container (it is — LXC), examines kernel logs for GPU errors (none found), and attempts torch.cuda.init() directly (fails). The assistant also checks loaded kernel modules and finds the NVIDIA modules are present.

In [msg 11301], the assistant checks device node permissions, GPU UUIDs, and library paths. All device nodes exist. The assistant notes that nvidia-uvm-tools has no permissions (mode 000) — a potential issue.

In [msg 11302], the assistant fixes the nvidia-uvm-tools permissions, checks CUDA runtime library paths, and attempts cuInit(0) directly. cuInit returns 999 — confirming the problem is at the driver level, not the PyTorch or CUDA runtime level.

In [msg 11303], the assistant tries nvidia-modprobe (not available in the container) and tests the other venv (also fails). This confirms both venvs are affected identically.

By [msg 11304], the assistant has synthesized all this information into a clear diagnosis: system-level driver issue in a container, requiring host-level intervention. The attempted nvidia_uvm reload is the last in-container fix the assistant can try before concluding that external help is needed.

Broader Significance

This message is valuable beyond its immediate context because it illustrates a fundamental principle of debugging in containerized environments: containers provide isolation but also impose constraints. The same GPU debugging techniques that work on bare metal — module reloading, driver reinitialization, GPU reset — may be unavailable inside a container. The assistant must recognize when it has exhausted the in-container solution space and needs to escalate.

The message also demonstrates the importance of understanding the full software stack, from application code (PyTorch) through runtime libraries (CUDA) to kernel modules (nvidia, nvidia_uvm) and hardware (GPU firmware). A less experienced engineer might have spent hours chasing PyTorch version issues or CUDA installation problems, never realizing that the root cause was at the driver level and required host intervention.

In the broader narrative of the session, this message marks a turning point. The assistant has identified the problem but cannot fix it alone. The next step — which occurs in subsequent messages — is to request host-level intervention (likely a container restart or host GPU reinitialization) from someone with access to the Proxmox host. The debugging effort was not wasted: by ruling out every in-container possibility, the assistant ensures that when the host intervention happens, it will actually solve the problem.

Conclusion

Message [msg 11304] is a snapshot of a debugging process at a critical juncture. The assistant has systematically eliminated every plausible in-container cause of CUDA initialization failure and correctly identified the root cause as a driver/firmware initialization issue at the host level. The attempted nvidia_uvm reload fails, but the failure itself is informative — it confirms the container's inability to manage kernel modules and reinforces the need for external intervention.

For anyone working with GPU-accelerated workloads in containers, this message offers a valuable lesson: know the boundaries of your environment. Understand what you can control from inside the container and what requires host access. And when cuInit returns 999, don't waste time reinstalling PyTorch — the problem is almost certainly deeper than that.