The Missing Firmware: Diagnosing CUDA Initialization Failure on Blackwell GPUs with Proxmox

In the middle of a complex effort to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant and user hit a perplexing wall. After successfully setting up an LXC container on the Proxmox host to bypass the VFIO/IOMMU PCIe bottleneck that had plagued their KVM virtual machine, they discovered that CUDA runtime initialization failed with error code 3 (CUDA_ERROR_NOT_INITIALIZED) — both on the host and inside the container. Message 537 captures the precise moment when the assistant connects the dots, identifies the likely root cause, and pivots the investigation toward a new hypothesis.

The Context: A Tale of Two Approaches

The broader session had been a saga of GPU deployment challenges. The team had initially deployed the GLM-5-NVFP4 model on 8 Blackwell GPUs using SGLang within a KVM virtual machine on Proxmox, but encountered persistent NaN crashes during decode. After iterative debugging of attention backends and quantization compatibility, they achieved a working baseline — only to discover that virtualization-induced PCIe P2P (peer-to-peer) latency was crippling performance. The GPUs in the VM showed a PHB (PCIe Host Bridge) topology instead of the bare-metal NODE topology, meaning direct GPU-to-GPU memory transfers had to traverse the host bridge rather than using direct P2P DMA.

The LXC container approach was conceived as a workaround: by running the ML workload in a privileged LXC container on the Proxmox host itself, the GPUs would be accessed directly (via bind-mounted device nodes) rather than through VFIO passthrough. The topology inside the container indeed showed the correct NODE topology, confirming the approach was theoretically sound. But a new blocker emerged: CUDA itself refused to initialize.

The Diagnostic Trail Leading to Message 537

The messages preceding 537 document a systematic debugging effort. The assistant had verified that nvidia-smi detected all 8 GPUs correctly inside the container and on the host. The device nodes (/dev/nvidia0 through /dev/nvidia7) were present with correct major/minor numbers. The /proc/driver/nvidia/gpus/ entries showed detailed GPU information including UUIDs, firmware versions, and bus locations. Yet every attempt to initialize CUDA — whether through PyTorch, raw CUDA C programs, or direct libcuda.so calls via ctypes — returned error code 3.

The assistant had already tried switching between the open and proprietary NVIDIA kernel modules. The open module (NVIDIA UNIX Open Kernel Module) could see the GPUs but CUDA couldn't initialize. The proprietary module (NVIDIA UNIX x86_64 Kernel Module) couldn't see the GPUs at all (CUDA_ERROR_NO_DEVICE). This was a critical clue: the open module was the only one that worked with Blackwell, but something deeper was wrong.

The Subject Message: A Moment of Insight

Message 537 captures the assistant's reasoning as it examines the GSP (GPU System Processor) firmware files installed by the NVIDIA driver:

The firmware files are gsp_ga10x.bin and gsp_tu10x.bin — these are for Ampere (GA10x) and Turing (TU10x). There's no Blackwell GSP firmware in this driver! The Blackwell GPUs (compute capability 12.0/SM120) need their own GSP firmware. This means the driver 590.48.01 likely doesn't fully support these GPUs on the Proxmox kernel.

This is the key insight. The driver package contains firmware for two GPU architectures: gsp_ga10x.bin for Ampere (e.g., RTX 30 series, A100) and gsp_tu10x.bin for Turing (e.g., RTX 20 series, T4). Blackwell — the architecture of the RTX PRO 6000 Blackwell Server Edition — requires its own GSP firmware binary (presumably named something like gsp_bb10x.bin or similar), which is absent.

The GSP is a dedicated microcontroller on modern NVIDIA GPUs that handles tasks like power management, clock gating, and — critically — GPU initialization. The open-source kernel module relies on the GSP firmware to properly initialize the GPU hardware. Without the correct firmware, the kernel module can detect the GPU via PCIe enumeration (hence nvidia-smi works) but cannot complete the initialization sequence required for CUDA to function.

The Reasoning Process: Connecting Contradictions

What makes this message particularly interesting is how the assistant grapples with a seeming contradiction:

Wait — in the KVM VM, this same driver version works with CUDA. The difference is the VM has its own driver directly accessing VFIO-passed GPUs. Here the open kernel module is loaded but the GSP firmware for Blackwell may be embedded in the kernel module itself rather than a separate file.

The assistant is reasoning through why the same driver version (590.48.01) works inside the KVM VM but fails on the Proxmox host. This is a crucial distinction. In the VM scenario, the NVIDIA driver is running inside the guest operating system, which accesses the GPU via VFIO (Virtual Function I/O). The VFIO subsystem presents the GPU to the guest as a PCIe device, and the guest's NVIDIA driver handles initialization independently — including loading whatever firmware it needs. The host's driver stack is irrelevant because the host never directly initializes the GPU; it merely passes through the PCIe device.

On the Proxmox host, however, the NVIDIA driver must initialize the GPUs directly. The open kernel module attempts to load GSP firmware, finds only gsp_ga10x.bin and gsp_tu10x.bin, and presumably either fails silently or loads firmware incompatible with Blackwell's GSP architecture. The result is a GPU that's visible at the PCIe level but not fully initialized for CUDA.

A Second Hypothesis: The Kernel Version

The assistant then pivots to a second potential cause:

The actual issue is likely the PVE kernel version (6.8.12-9). This is an older kernel. The open kernel module might need a newer kernel for Blackwell support. The VM worked because it has its own NVIDIA driver kernel module that handles the VFIO interface differently.

This hypothesis suggests that even with correct firmware, the Proxmox VE kernel (based on Linux 6.8) might lack necessary kernel-level support for Blackwell GPUs. The open NVIDIA kernel module may require newer kernel APIs or memory management features present only in more recent kernels. The KVM VM, by contrast, runs its own kernel (likely a standard Ubuntu or similar distribution kernel) which could be newer or configured differently.

To test this, the assistant runs a command to check the kernel version:

Let me check if the kernel is too old: [bash] ssh root@10.1.2.6 "uname -r && cat /proc/version" 6.8.12-9-pve Linux version 6.8.12-9-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-9 (2025-03-16T19:18Z)

The kernel is confirmed as 6.8.12-9-pve, a Proxmox-specific build from March 2025. While not ancient, it predates the Blackwell GPU architecture (which launched in late 2024/early 2025). The NVIDIA driver 590.48.01 itself is from December 2025, suggesting it should support Blackwell — but the driver's firmware payload and kernel module compatibility may still be tied to specific kernel versions.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message, some explicit and some implicit:

That missing firmware files are the root cause. This is a strong hypothesis but not yet proven. The GSP firmware could be embedded in the kernel module binary itself (as the assistant briefly considers), or the open module might use a different firmware loading mechanism that doesn't rely on separate .bin files. The assistant acknowledges this uncertainty: "the GSP firmware for Blackwell may be embedded in the kernel module itself rather than a separate file."
That the KVM VM's driver handles things "differently." While broadly correct, this glosses over the specifics of how VFIO passthrough interacts with GPU initialization. The VFIO mechanism doesn't just "pass through" the GPU — it also handles interrupt mapping, BAR (Base Address Register) allocation, and DMA remapping, which could affect firmware loading behavior.
That a newer kernel would fix the issue. The kernel version hypothesis is plausible but speculative. Even with a newer kernel, the missing GSP firmware would still be missing. The two hypotheses (missing firmware vs. old kernel) are somewhat contradictory, and the assistant doesn't prioritize between them.
That the driver version is the same. The assistant states "this same driver version works with CUDA" in the VM, but this isn't fully verified. The VM might be using a different build or configuration of the 590.48.01 driver, or the driver inside the VM might have been installed with different options.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

NVIDIA GPU architecture generations: Ampere (GA10x), Turing (TU10x), Blackwell (BB10x or similar), and the concept of compute capability/SM versions.
GSP firmware: The GPU System Processor is a relatively recent addition to NVIDIA GPUs (introduced with Turing) that handles low-level hardware management. Understanding that modern NVIDIA GPUs require firmware to initialize is essential.
VFIO passthrough: The mechanism by which KVM/QEMU assigns physical PCIe devices to virtual machines, including how the host and guest interact with device initialization.
Proxmox VE kernel: The Proxmox kernel is based on the Ubuntu kernel but includes additional patches and drivers for virtualization. It may lag behind upstream kernels in supporting new hardware.
CUDA initialization sequence: The distinction between PCIe-level device detection (what nvidia-smi does) and CUDA runtime initialization (what cuInit does), and why one can succeed while the other fails.
The LXC container approach: Understanding why the team was trying LXC instead of KVM, and the topology differences (NODE vs PHB) that motivated the switch.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

The driver 590.48.01 lacks Blackwell GSP firmware. This is a concrete, verifiable finding. The firmware directory contains only gsp_ga10x.bin and gsp_tu10x.bin. Anyone attempting to use this driver version with Blackwell GPUs on bare metal or LXC should be aware of this limitation.
The open kernel module is required for Blackwell. The proprietary module fails to detect the GPUs entirely (CUDA_ERROR_NO_DEVICE), while the open module at least sees them. This confirms that Blackwell requires the open-source kernel module path.
The Proxmox host kernel is 6.8.12-9-pve. This establishes the baseline for further investigation. If the kernel is indeed too old, upgrading to a newer Proxmox kernel (or a custom kernel) might resolve the issue.
The VM vs. host discrepancy is documented. The observation that the same driver works in a VFIO VM but not on the host is a valuable data point that could help others troubleshooting similar setups.

The Thinking Process: A Window into Debugging Methodology

What makes message 537 particularly instructive is the assistant's thinking process. It follows a classic debugging pattern:

Observe the symptom: CUDA initialization fails despite GPU detection.
Gather data: Examine firmware files, kernel module types, and system logs.
Formulate hypotheses: Missing GSP firmware, kernel version incompatibility, VFIO interaction differences.
Test assumptions: Check if the VM scenario is truly comparable, verify kernel version.
Acknowledge uncertainty: "The GSP firmware for Blackwell may be embedded in the kernel module itself rather than a separate file." The assistant also demonstrates intellectual honesty by correcting its own reasoning mid-stream. It starts with a strong assertion ("There's no Blackwell GSP firmware in this driver!") but immediately qualifies it by considering alternative explanations. This willingness to hold multiple competing hypotheses simultaneously is a hallmark of effective debugging.

Conclusion

Message 537 represents a critical juncture in the deployment effort. The assistant has identified a plausible root cause for the CUDA initialization failure — missing GSP firmware for Blackwell GPUs in the NVIDIA driver 590.48.01 — while also surfacing a secondary hypothesis about kernel version compatibility. The next steps would involve either finding or building a driver package that includes Blackwell GSP firmware, upgrading the Proxmox kernel, or investigating alternative approaches to GPU initialization on the host.

The message also highlights a broader lesson about GPU virtualization: what works seamlessly in a VFIO VM may fail on the host, because the host must handle GPU initialization directly rather than delegating it to the guest. For anyone deploying modern NVIDIA GPUs in virtualized environments, understanding the GSP firmware requirements and the differences between host-level and guest-level driver stacks is essential knowledge.