The Diagnostic Deep Dive: Tracing a CUDA Initialization Failure on Blackwell GPUs in an LXC Container
Introduction
In the course of deploying the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, a team encountered a stubborn and perplexing problem. After successfully setting up an LXC container on a Proxmox VE host to bypass the VFIO/IOMMU PCIe P2P bottleneck that had crippled GPU-to-GPU communication in their KVM virtual machine, they hit a new wall: CUDA runtime initialization failed with error code 3 (CUDA_ERROR_NOT_INITIALIZED), even though nvidia-smi cheerfully reported all 8 GPUs as healthy and operational. Message 521 in this conversation captures a pivotal moment in the debugging journey — a systematic diagnostic probe into the container's GPU device infrastructure, designed to isolate whether the failure stems from a missing device node, a driver binding issue, or a deeper incompatibility between the NVIDIA driver and the Proxmox kernel.
The Message
The assistant writes:
No errors. Let me check if the container's /proc/driver/nvidia/gpus has the right content and test with nvidia-cuda-mps-control:
>
[bash] ssh root@10.1.230.174 bash <<'EOF' # Check /proc/driver/nvidia/gpus ls /proc/driver/nvidia/gpus/ 2>&1 echo "===" # Check one GPU's info cat /proc/driver/nvidia/gpus/0000:01:00.0/information 2>&1 echo "===" # Check if the problem is /sys access ls /sys/bus/pci/devices/0000:01:00.0/driver 2>/dev/null readlink /sys/bus/pci/devices/0000:01:00.0/driver 2>&1 echo "===" # Check if there are missing /dev entries for compute ls -la /dev/nvidia-caps/ 2>&1 EOF 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0 === Model: NVIDIA RTX PRO 6000 Blackwell Server Edition IRQ: 812 GPU UUID: GPU-6793b112-91ac-b02e-b2b2-a7e4143a878b Video BIOS: 98.02.67.00.0a Bus Type: PCIe DMA Size: 52 bits DMA Mask: 0xfffffffffffff Bus Location: 0000:01:00.0 Device Minor: 1 GPU Firmware: 590.48.01 GPU Excluded: No === 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0 0000:81:00.0 0000:91:00.0 000...
Context and Motivation
To understand why this message was written, one must appreciate the debugging trajectory that preceded it. The team had spent the prior several messages (510–520) systematically eliminating possible causes of the cuInit failure. They had confirmed that the device files /dev/nvidia0 through /dev/nvidia7, /dev/nvidiactl, and /dev/nvidia-uvm were all accessible with proper read-write permissions. They had verified that the NVIDIA kernel module was loaded and that /proc/driver/nvidia/ was populated. They had tested with both PyTorch 2.9.1 and 2.10.0, finding that neither could initialize CUDA. They had even compiled and run a raw CUDA C program that called cudaGetDeviceCount and cudaSetDevice directly, which also failed with "initialization error." They had checked the host's kernel log (dmesg) for NVIDIA errors and found only benign messages about module signature verification and a deprecation warning about persistence mode.
Each of these checks had narrowed the problem space but had not identified the root cause. The failure pattern was deeply puzzling: the NVIDIA driver's user-space components (like nvidia-smi) could enumerate and report GPU details, but the CUDA runtime library (libcuda.so) could not initialize. This dissociation between the driver's administrative interface and its compute runtime pointed toward a problem in the driver's initialization sequence — something that happens after the kernel module loads but before the CUDA runtime can take control.
Message 521 represents the next logical step in this diagnostic narrowing. The assistant pivots from checking the obvious (device files, driver module presence) to inspecting the internal plumbing of the NVIDIA driver's device management infrastructure. The /proc/driver/nvidia/gpus/ directory is a kernel-level interface that exposes per-GPU state maintained by the NVIDIA kernel module. Each GPU gets a subdirectory named after its PCIe bus address, containing an information file with detailed hardware and driver state. By reading this file, the assistant can verify that the kernel module has fully enumerated each GPU, assigned it a device minor number, loaded its firmware, and — critically — that the GPU is not marked as "Excluded."
The Diagnostic Methodology
The message executes four distinct checks, each targeting a different layer of the GPU software stack:
Check 1: GPU enumeration in /proc/driver/nvidia/gpus/. The ls output confirms all 8 GPUs are present, each identified by its PCIe bus address (e.g., 0000:01:00.0, 0000:11:00.0). This tells us the kernel module has discovered and registered all GPUs during its initialization — a prerequisite for any CUDA runtime operation.
Check 2: Per-GPU state via the information file. The assistant reads the file for GPU at bus 0000:01:00.0. The output is rich with diagnostic value: the model name confirms these are Blackwell-generation RTX PRO 6000 cards; the firmware version matches the driver (590.48.01); the DMA size of 52 bits is correct for modern GPUs; and crucially, GPU Excluded: No indicates the driver has not blacklisted this GPU. The Device Minor: 1 field is particularly important — it tells us which /dev/nvidia* device node corresponds to this GPU (minor number 1 maps to /dev/nvidia1).
Check 3: Driver binding via /sys/bus/pci/devices/. The assistant checks whether the PCI device at bus 0000:01:00.0 has a driver bound to it by looking at the driver symlink in the sysfs PCI device directory. The output shows the same list of 8 bus addresses, confirming each GPU has an active driver binding. This rules out a scenario where the NVIDIA driver failed to attach to some GPUs.
Check 4: The /dev/nvidia-caps/ directory. This is a less commonly known NVIDIA device interface that exposes GPU capabilities (like video encode/decode, compute preemption, etc.) as separate device files. Its absence (the ls command fails with an error, though the output is truncated in the message) is not necessarily a problem — this directory is only created when the nvidia-caps kernel module is loaded or when certain driver features are enabled. However, the assistant is being thorough, checking even this obscure corner.
Assumptions and Blind Spots
The assistant's diagnostic approach carries several implicit assumptions. First, it assumes that the problem is environmental — that something in the container or host configuration is preventing CUDA from initializing — rather than a fundamental incompatibility between the NVIDIA driver version 590.48.01 and the Blackwell GPU architecture. This assumption is reasonable given that nvidia-smi works, but it may be incorrect: the open-source kernel module (nvidia-open) might have different initialization paths for its administrative interface versus its compute runtime.
Second, the assistant assumes that the container's bind-mounts of /dev and /proc are complete and correct. The earlier messages (514–515) had already identified that /dev/nvidia-modeset was missing on the host, but the assistant correctly judged this irrelevant for compute workloads. However, the assistant has not yet checked whether the container has access to the NVIDIA UVM (Unified Virtual Memory) driver's control interface, which is sometimes required for CUDA context creation.
Third, there is an implicit assumption that the Proxmox host kernel (6.8.12-9-pve) is compatible with the NVIDIA driver 590.48.01. The driver loaded without errors, and nvidia-smi works, but there may be subtle API mismatches between the kernel's memory management subsystem and what the NVIDIA driver expects — particularly for Blackwell GPUs, which use a new GSP (GPU System Processor) firmware interface that may require specific kernel support.
Input Knowledge Required
To fully understand this message, the reader needs several layers of context. One must know that NVIDIA GPUs on Linux are accessed through a layered stack: the kernel module (nvidia.ko or nvidia-open.ko), the CUDA runtime library (libcuda.so), and user-space tools like nvidia-smi. The /proc/driver/nvidia/ filesystem is a diagnostic interface exposed by the kernel module, while /sys/bus/pci/ is the Linux kernel's standard PCI device topology interface. One must also understand the LXC container model — that containers share the host kernel but have their own mount namespace, and that GPU access requires bind-mounting device nodes from the host.
The reader also needs to know the history: that this is an 8-GPU Blackwell system running on Proxmox VE, that the team previously tried a KVM VM approach but hit P2P limitations, and that the LXC container was set up specifically to give the GPUs bare-metal topology visibility. The nvidia-smi topo -m output from earlier in the chunk showed NODE and SYS connections instead of the PHB (PCIe Host Bridge) topology seen in the VM — the very reason the LXC approach was pursued.
Output Knowledge Created
This message produces several valuable pieces of knowledge. First, it confirms that all 8 GPUs are fully enumerated and recognized by the NVIDIA kernel module at the /proc/driver/nvidia/gpus/ level — the module sees them, has assigned them minor numbers, loaded their firmware, and not excluded any. Second, it establishes that the driver binding via sysfs is correct for each GPU. Third, it reveals the specific device minor number mapping (e.g., bus 0000:01:00.0 maps to minor 1, which is /dev/nvidia1), which can be cross-referenced against the container's device nodes to verify consistency.
The most important negative finding is that the problem is not at the kernel module level — the module is loaded, functional, and has properly initialized all GPUs. This pushes the investigation upward in the stack, toward the CUDA runtime library's initialization sequence. The failure must occur somewhere between cuInit(0) being called and the CUDA driver API establishing a context with the kernel module. This points toward possible causes like GSP firmware incompatibility, missing NVML/NV-CONTROL interfaces, or a cgroup/namespace restriction that the CUDA library encounters but nvidia-smi does not.
The Thinking Process
The reasoning visible in this message reveals a methodical, hypothesis-driven debugging approach. The assistant begins with the phrase "No errors" — this refers to the dmesg output from message 520, which showed no NVIDIA-related errors beyond the benign module verification warning. Having ruled out kernel-level errors, the assistant moves to the next plausible failure domain: the driver's internal device state.
The choice to check /proc/driver/nvidia/gpus/ is strategic. This interface sits between the kernel module and the CUDA runtime — it's populated by the module during its initialization but is read by diagnostic tools and the CUDA library. If this interface shows healthy GPUs, the problem is likely in the CUDA library's own initialization logic or in some environmental constraint the library checks that nvidia-smi does not.
The assistant's decision to read the information file for a single GPU (rather than all 8) is a pragmatic optimization — the GPUs are identical, so one representative sample suffices. The choice of bus 0000:01:00.0 (the first in alphabetical order) is arbitrary but reasonable.
The check of /sys/bus/pci/devices/ driver symlinks is a clever addition. Even though the kernel module is loaded, individual PCI devices might fail to bind if there's a resource conflict or if the device's PCI configuration space is corrupted. The sysfs interface provides definitive proof of binding at the PCI layer.
The /dev/nvidia-caps/ check is the most obscure and perhaps the least likely to yield results, but its inclusion demonstrates thoroughness. When debugging a failure that has already survived multiple rounds of elimination, one must check even improbable corners.
Conclusion
Message 521 is a textbook example of systematic diagnostic reasoning in a complex systems environment. It does not solve the problem — the CUDA initialization failure persists after this message — but it narrows the search space considerably. By confirming that the NVIDIA kernel module has properly initialized all 8 Blackwell GPUs, assigned them minor numbers, loaded their firmware, and bound them to the PCI driver, the assistant eliminates an entire class of potential causes and focuses attention on the CUDA runtime library's interaction with the kernel module. The subsequent investigation (in messages 522 and beyond) would pivot toward examining the GSP firmware files, the kernel version compatibility, and the proprietary vs. open-source kernel module distinction — ultimately leading to the conclusion that the Proxmox VE kernel lacks the necessary support for Blackwell's GSP firmware requirements.
This message, though it appears as a simple diagnostic command dump, is actually a carefully reasoned probe into the GPU software stack's intermediate layer — the point where the kernel module's work meets the user-space runtime's expectations. It exemplifies the kind of methodical, hypothesis-driven thinking that complex system debugging demands.