Debugging CUDA Initialization Failure in an LXC Container: A Diagnostic Deep Dive

Introduction

In the complex world of high-performance GPU computing, few things are more frustrating than a setup that almost works. The NVIDIA driver loads. nvidia-smi cheerfully reports all eight Blackwell RTX PRO 6000 GPUs. The device files are present. And yet, when any CUDA runtime application tries to initialize, it fails with the cryptic error code 3 — CUDA_ERROR_NOT_INITIALIZED. This is precisely the situation facing the team in message [msg 512] of an opencode session, where the assistant takes a critical diagnostic step in a months-long effort to deploy the GLM-5-NVFP4 large language model across eight GPUs with peer-to-peer (P2P) DMA.

This message represents a pivotal moment of hypothesis testing in a debugging chain. It is not a triumphant breakthrough, nor a catastrophic failure — it is the quiet, methodical work of eliminating possibilities, one by one.

The Broader Context: Why This Message Exists

To understand why this message was written, one must understand the journey that led to it. The team had been wrestling with GPU P2P communication for days. Their initial deployment used a KVM virtual machine on Proxmox, passing through eight RTX PRO 6000 Blackwell GPUs via VFIO. While the model ran, nvidia-smi topo -m revealed a devastating topology: every GPU showed as PHB (PCIe Host Bridge) relative to every other GPU, meaning all inter-GPU communication had to traverse the host bridge rather than using direct P2P DMA. This crushed performance for tensor-parallel inference workloads.

The team's solution was elegant in theory: bypass the VFIO/IOMMU layer entirely by running the ML workload in an LXC container on the Proxmox host itself. Inside a container, the GPUs would appear with their true bare-metal topology — NODE within the same PCIe socket, SYS across sockets — enabling P2P DMA. The assistant guided the user through installing NVIDIA driver 590.48.01 on the Proxmox host, converting the container to privileged mode, bind-mounting all eight GPU device nodes, and copying the 405GB model cache to shared storage.

The topology check inside the container confirmed success: the GPUs showed the correct bare-metal topology. But then came the gut punch: cuInit returned error code 3. The CUDA runtime could not initialize, even though nvidia-smi worked perfectly. This is the moment captured in [msg 512].

The Message Itself: Hypothesis and Test

The message opens with the assistant's reasoning:

/proc/driver/nvidia is accessible. The driver module is loaded. But cuInit returns 3. This might be a cgroup permission issue. Let me check:

This single sentence reveals the assistant's diagnostic process. It has already verified two key facts: (1) the /proc/driver/nvidia filesystem is populated, confirming the kernel module is loaded and communicating with the GPUs, and (2) cuInit fails despite this. The assistant then forms a hypothesis: perhaps the issue is a cgroup permission problem. This is a reasonable guess — LXC containers use Linux cgroups to restrict device access, and if the cgroup controller is blocking CUDA's device node operations, cuInit would fail even though simple file existence checks pass.

The assistant then executes a bash command that performs two diagnostic checks:

  1. Device file accessibility: It attempts to open /dev/nvidia0, /dev/nvidiactl, and /dev/nvidia-uvm with read-write permissions using Python's os.open(). These are the three critical device files for NVIDIA GPU operation: one per GPU (nvidia0), the control device (nvidiactl), and the UVM (Unified Virtual Memory) device (nvidia-uvm).
  2. Cgroup device rules: It checks /sys/fs/cgroup/devices.list (for cgroup v1 device whitelists) and reads /proc/self/cgroup to see which cgroup the process belongs to. The results come back clean:
/dev/nvidia0: OK
/dev/nvidiactl: OK
/dev/nvidia-uvm: OK
===
no cgroup1 devices.list
0::/user.slice/user-0.slice/session-13.scope

All three device files open successfully. There is no cgroup v1 device list restricting access. The process runs in a standard user.slice scope, which is normal for an LXC container.

The Thinking Process: What the Assistant Was Reasoning

The assistant's reasoning in this message follows a classic debugging pattern: observe → hypothesize → test → evaluate. Let me reconstruct the full chain:

Observation: cuInit() returns error code 3 (CUDA_ERROR_NOT_INITIALIZED). This is distinct from "device not found" (error code 100 / CUDA_ERROR_NO_DEVICE) or "driver not loaded" (error code 135 / CUDA_ERROR_UNKNOWN). Error code 3 specifically means the CUDA driver API initialization failed — the runtime tried to set up its state and something went wrong.

Prior knowledge: The assistant knows that nvidia-smi works, meaning the kernel module and the NVML (NVIDIA Management Library) layer are functional. But cuInit goes through a different path — it initializes the CUDA driver API, which involves opening device files, setting up context state, and potentially loading GSP (GPU System Processor) firmware. The fact that nvidia-smi works but cuInit doesn't suggests the problem is in the CUDA-specific initialization path, not the general kernel module layer.

Hypothesis formation: Given the LXC container context, cgroup device permissions are a natural suspect. LXC uses cgroup v2 device controllers to restrict which devices a container can access. Even though the device files are bind-mounted, the cgroup controller might block certain ioctl operations or mmap calls that CUDA needs during initialization. The assistant hypothesizes that a cgroup device controller is blocking CUDA's deeper access to the GPU.

Test design: The assistant designs a two-part test. First, it checks if the device files can actually be opened with read-write access — a necessary but not sufficient condition for CUDA operation. Second, it checks the cgroup device rules to see if any restrictions are in place.

Evaluation: The test disproves the hypothesis. All device files open successfully, and no cgroup device restrictions exist. The assistant has eliminated one possible cause, narrowing the search space. The problem must lie elsewhere — likely in driver-kernel compatibility, GSP firmware, or CUDA version mismatch.

Assumptions Made and Their Validity

The assistant made several assumptions in this message, some explicit and some implicit:

Assumption 1: The problem is at the cgroup/permission layer. This was reasonable given the LXC context but turned out to be incorrect. The real culprit, as later investigation revealed, was a deeper driver compatibility issue: the Proxmox VE kernel (6.8.12-9-pve) lacked proper support for Blackwell GPU GSP firmware, and the NVIDIA driver 590.48.01 did not include the required gsp_* firmware files for Blackwell architecture. This is a fundamentally different class of problem — not "access denied" but "initialization sequence incomplete."

Assumption 2: Device file accessibility is a meaningful diagnostic signal. This assumption was valid. If the device files couldn't be opened, that would have been a clear smoking gun pointing to cgroup or permission issues. The fact that they can be opened is also informative — it rules out a whole class of problems.

Assumption 3: The cgroup v1 device list is the relevant mechanism. The assistant checked /sys/fs/cgroup/devices.list, which is the cgroup v1 device whitelist. However, modern Proxmox systems use cgroup v2, where device access control works differently (via BPF programs rather than a flat file). The assistant's check was still useful — the absence of a devices.list file confirms the system isn't using cgroup v1 device controllers — but it doesn't fully rule out cgroup v2 restrictions. This is a subtle but important limitation of the diagnostic.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. CUDA architecture knowledge: Understanding the difference between nvidia-smi (which uses NVML, a separate library that talks to the kernel module) and cuInit (which initializes the CUDA driver API, a deeper and more complex initialization path). The fact that one works and the other doesn't is a critical diagnostic signal.
  2. Linux device model: Understanding that /dev/nvidia0, /dev/nvidiactl, and /dev/nvidia-uvm are the three device files that NVIDIA uses, and what each one does. The control device manages API versions and permissions; the UVM device handles unified memory management.
  3. LXC and cgroup mechanics: Understanding that LXC containers use Linux cgroups (v2 by default on modern systems) to restrict resource access, including device access. The assistant's check of /sys/fs/cgroup/devices.list and /proc/self/cgroup is specific to this containerization context.
  4. Error code semantics: Knowing that CUDA error code 3 means CUDA_ERROR_NOT_INITIALIZED, which is distinct from "no device" or "driver not found."
  5. The broader project context: Understanding that this is part of an effort to deploy GLM-5-NVFP4 across 8 Blackwell GPUs with P2P DMA, and that the LXC approach was chosen specifically to bypass VFIO/IOMMU topology limitations.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Negative result: The cgroup permission hypothesis is ruled out. This is valuable — it prevents wasted effort on permission-related fixes and redirects attention to the real cause.
  2. Device file accessibility confirmed: All three NVIDIA device files are accessible with read-write permissions from within the container. This confirms the bind-mount configuration is correct.
  3. No cgroup v1 restrictions: The absence of a devices.list file confirms the system isn't using legacy cgroup v1 device controllers. This is consistent with a modern Proxmox setup using cgroup v2.
  4. Process cgroup location: The process runs in user.slice/user-0.slice/session-13.scope, which is the standard location for a root shell session in an LXC container. This is normal and not indicative of any problem.
  5. Narrowed search space: By eliminating permissions as the cause, the message implicitly directs future investigation toward other possibilities: driver-kernel compatibility, GSP firmware, CUDA toolkit version mismatch, or kernel module parameters.

The Broader Significance

This message exemplifies a crucial but often invisible aspect of engineering work: the disciplined elimination of hypotheses. In a debugging session spanning days, most messages are not breakthroughs — they are tests that disprove one theory after another, gradually converging on the truth. [msg 512] is a clean example of this process: a clear hypothesis, a well-designed test, and an unambiguous result that moves the investigation forward even though it doesn't solve the problem.

The message also reveals the assistant's systematic approach to debugging. Rather than guessing randomly or trying brute-force fixes, it reasons from first principles: "The driver module is loaded, but cuInit fails. In an LXC context, cgroup restrictions could cause this. Let me test that specific hypothesis." This methodology — observe, hypothesize, test, evaluate — is the hallmark of effective troubleshooting.

In the end, the cgroup hypothesis was wrong. The real cause lay deeper, in the NVIDIA driver's GSP firmware compatibility with the Proxmox kernel. But this message was an essential step on the path to that discovery. Without ruling out permissions, the team might have spent hours chasing cgroup configuration changes, adding device rules, or tweaking LXC settings — all futile efforts. By cleanly eliminating this possibility in a single, well-crafted diagnostic command, the assistant saved time and kept the investigation moving toward the true root cause.

This is the quiet heroism of good debugging: not the dramatic fix, but the methodical elimination of what isn't wrong, until only what is wrong remains.