When CUDA Won't Initialize: Diagnosing Blackwell GPU Driver Incompatibility on Proxmox

In the high-stakes world of large-scale ML inference, every millisecond counts. When deploying an 8-GPU system built around NVIDIA's latest Blackwell architecture (RTX PRO 6000), the team behind this opencode session had already navigated a labyrinth of technical challenges. They had resolved flash-attn build failures, wrestled with CUDA toolkit versions, and discovered that their KVM virtual machine's VFIO/IOMMU configuration prevented GPU peer-to-peer (P2P) DMA — a critical requirement for efficient multi-GPU tensor parallelism. Their solution was elegant: bypass the VM entirely by using an LXC container on the Proxmox host, where the GPUs would appear with their true PCIe topology. But in message 524, that elegant solution collided with a far more fundamental problem.

The LXC Experiment and a Surprising Discovery

The preceding messages in the conversation tell a story of mounting frustration. The team had successfully configured an LXC container with bind-mounts for all eight GPU device nodes, verified that nvidia-smi topo -m showed the correct bare-metal topology (with NODE and SYS connections instead of the PHB topology seen in the VM), and copied the 296GB model cache to avoid re-downloading. Everything looked promising — until CUDA itself refused to initialize.

Earlier attempts to run CUDA inside the container had failed with cuInit returning error code 3 (CUDA_ERROR_NOT_INITIALIZED). The initial assumption was a container configuration issue: perhaps missing device nodes, cgroup restrictions, or incorrect bind-mounts. The team checked /dev/nvidia* accessibility, verified minor numbers, examined cgroup device lists, and even tried downgrading PyTorch from 2.10.0 to 2.9.1. Nothing worked. The GPUs were visible to nvidia-smi, the kernel module was loaded, the device files were accessible — yet CUDA runtime initialization consistently failed.

Then came the critical insight in message 524.

A Critical Hypothesis and Investigation

The assistant's reasoning in message 524 marks a pivotal shift in diagnosis. The message opens with a succinct but crucial realization:

"cuInit fails on the host too! The issue is not container-specific — the NVIDIA driver installed but CUDA initialization fails."

This single sentence reframes the entire problem. The issue was never about LXC configuration, cgroup permissions, or bind-mount correctness. The NVIDIA driver installation on the Proxmox host itself was fundamentally broken for these Blackwell GPUs. The assistant then proposes a hypothesis:

"This is likely because the open-source kernel module (NVIDIA UNIX Open Kernel Module) doesn't support these Blackwell GPUs properly, or some other module loading issue."

This hypothesis is grounded in an important technical detail: NVIDIA provides two kernel modules for their drivers — an open-source module and a proprietary module. The open-source module, while functionally equivalent for most workloads, has historically lagged behind in support for newer architectures. The assistant suspects this might be the root cause.

To test this hypothesis, the assistant executes a bash command that performs three diagnostic steps: checking the loaded module version and type, inspecting the module's firmware manifest, and attempting to reload the nvidia-uvm (Unified Virtual Memory) module before retesting CUDA initialization.

The Smoking Gun: Missing GSP Firmware

The output from the diagnostic commands reveals the truth. The loaded module is confirmed as:

NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  590.48.01

This is the open-source kernel module, version 590.48.01 — a relatively recent driver from December 2025. But the critical detail comes from the modinfo output, which lists the firmware files bundled with the module:

firmware:       nvidia/590.48.01/gsp_tu10x.bin
firmware:       nvidia/590.48.01/gsp_ga10x.bin

Two firmware files. One for Turing architecture (tu10x — GeForce RTX 30 series, RTX A-series) and one for Ampere/Ada Lovelace (ga10x — GeForce RTX 40 series, RTX 6000 Ada). There is no firmware file for Blackwell — no gb2xx, no blackwell, nothing that would correspond to the RTX PRO 6000 Blackwell GPUs installed in this machine.

This is the root cause. Modern NVIDIA GPUs (since Turing) include a GPU System Processor (GSP) — a dedicated microcontroller that handles GPU initialization, power management, and other system-level functions. The GSP requires firmware to operate, and this firmware is loaded by the kernel module during initialization. Without the appropriate GSP firmware for Blackwell, the GPU's GSP cannot complete its initialization sequence, and CUDA's cuInit call fails with error code 3 — not because the GPU hardware is missing or inaccessible, but because the firmware handshake never completes.

The driver version 590.48.01, despite being released in December 2025, simply does not include Blackwell GSP firmware. This could mean the driver predates Blackwell production hardware, or that Blackwell support was added in a later driver branch. Either way, the driver installed on the Proxmox host cannot initialize these GPUs at the CUDA runtime level, even though the basic PCIe enumeration and nvidia-smi detection work (since those operate at a lower level in the kernel module).

Assumptions, Mistakes, and Lessons Learned

This message reveals several assumptions that had been guiding the investigation, and the moment they were overturned.

Assumption 1: The host driver works correctly. The team had been operating under the implicit assumption that the NVIDIA driver installation on the Proxmox host was functional. After all, nvidia-smi showed all eight GPUs, the kernel module loaded without errors, and the device files were created correctly. The focus had been entirely on the container configuration because the container was where CUDA applications would run. Message 524 shatters this assumption by testing CUDA directly on the host.

Assumption 2: Driver version 590.48.01 supports Blackwell. The driver was released in December 2025, well after Blackwell's announcement. The team reasonably assumed that a driver from that timeframe would include support for Blackwell GPUs. The missing GSP firmware reveals that driver support is not binary (works/doesn't work) but layered — basic PCIe enumeration and nvidia-smi detection can work even when CUDA runtime initialization fails.

Assumption 3: The open-source vs. proprietary kernel module distinction matters for Blackwell support. The assistant hypothesized that the open-source module might lack Blackwell support. In reality, the issue is not about open-source vs. proprietary — it's about the GSP firmware files bundled with either module. Both modules would need the same firmware files, and neither would work without them.

A subtle mistake in the assistant's reasoning is the suggestion to try reloading nvidia-uvm. The nvidia-uvm module handles Unified Virtual Memory management and is loaded after the main nvidia module. Reloading it would not fix a GSP firmware initialization failure, because the GSP firmware is loaded by the main nvidia module during its initialization, before nvidia-uvm even comes into play. This diagnostic step, while well-intentioned, was unlikely to yield different results — and indeed it did not.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several technical domains:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. The host driver is the root cause. The investigation can now shift from container configuration to host-level driver compatibility. This saves significant debugging effort that would otherwise be wasted on container-specific tweaks.
  2. The specific failure mode is documented. cuInit returning error code 3 (CUDA_ERROR_NOT_INITIALIZED) in the presence of a loaded kernel module and accessible device files is a known symptom of GSP firmware mismatch. This diagnostic pattern can inform future troubleshooting.
  3. Driver 590.48.01 lacks Blackwell GSP firmware. This is a concrete, actionable finding. The solution space now includes: finding a newer driver version with Blackwell GSP firmware, installing the proprietary kernel module (which might ship with different firmware), or manually extracting and placing the required firmware files.
  4. The LXC approach is validated as conceptually correct. The fact that the same failure occurs on the host confirms that the LXC container configuration was not the problem. The P2P topology shown inside the container was genuine — the GPUs just couldn't be initialized by CUDA due to the driver issue.

Implications for the Project

This discovery has significant implications. The team's strategy of using LXC to bypass VFIO/IOMMU P2P limitations was sound in principle, but it depends on a working NVIDIA driver on the Proxmox host. The host, running Proxmox VE kernel 6.8.12-9-pve, may not be able to run the latest NVIDIA drivers that include Blackwell GSP firmware — Proxmox's custom kernel can lag behind mainstream kernel support for new GPU architectures.

The team now faces a fork in the road: either find a way to get Blackwell-compatible NVIDIA drivers working on the Proxmox host (which may require kernel upgrades, DKMS builds, or manual firmware extraction), or abandon the LXC approach and return to the KVM VM with its P2P limitations. The latter would mean accepting the PCIe bandwidth bottleneck that originally motivated the LXC experiment.

Conclusion

Message 524 is a turning point in this debugging saga. It elegantly reframes a seemingly container-specific problem as a host-level driver incompatibility, saving countless hours of fruitless container debugging. The assistant's reasoning demonstrates a crucial debugging principle: when a problem appears in an isolated environment (container), always verify whether it reproduces on the host. The discovery that driver 590.48.01 lacks Blackwell GSP firmware is both a diagnosis and a roadmap — the team now knows exactly what needs to be fixed, even if the fix itself remains challenging.

In the broader narrative of this opencode session, message 524 represents the moment when a workaround (LXC) was validated as conceptually correct but blocked by a more fundamental dependency (driver support). It is a reminder that in complex systems engineering, the most elegant architectural solutions can be defeated by the lowest-level software components — in this case, a missing firmware binary for a microcontroller most developers never even know exists.