The Open Kernel Module Revelation: Diagnosing Blackwell GPU Initialization Failure in a Passthrough VM

In the course of deploying and optimizing large language model inference infrastructure, hardware compatibility issues often lurk beneath the surface, revealing themselves only through cryptic kernel log messages. Message [msg 5764] captures one such moment — a diagnostic breakthrough where an assistant identifies why eight NVIDIA RTX PRO 6000 Blackwell GPUs, freshly passed through to a virtual machine via PCI passthrough on Proxmox, refuse to initialize. The message is short, but it represents the culmination of a careful diagnostic chain and delivers a critical insight about NVIDIA's bifurcated driver ecosystem.

The Context: A Tangent Experiment with GPU Passthrough

The broader conversation leading up to this message had been focused on production deployment of the Kimi-K2.5 INT4 model using SGLang, running in an LXC container on a Proxmox host. The user had just finished hardening that deployment with a systemd service and hierarchical KV cache configuration when they decided to pivot — a "tangent experiment" as they called it — to pass all eight GPUs through to a VM instead. The Proxmox host (10.1.2.6) already had the PCI mappings configured from previous use, and VM 131 (ml-pipelines) was already running with the GPUs bound to vfio-pci. But the guest VM (10.1.16.221) reported that nvidia-smi found no devices, despite the GPUs being visible in lspci and /dev/nvidia* device nodes existing.

The assistant had been methodically working through the diagnostic tree: first confirming the GPUs were visible in the VM's PCI bus, then verifying the NVIDIA driver was loaded, and finally checking what nvidia-smi actually reported. The answer was "No devices were found" with exit code 6 — a clear indication that the driver loaded but failed to initialize any GPUs.

The Diagnostic Breakthrough

Message [msg 5764] opens with the assistant quoting the critical error from the kernel log:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This pair of messages tells the entire story. The first line is NVIDIA's own assertion that the installed driver package requires the open kernel modules — a requirement that is not merely a recommendation but a hard dependency. The second line confirms the consequence: the Resource Manager (RmInitAdapter) failed to initialize the GPU adapter, returning error code 0x22:0x56:1001.

The assistant immediately interprets this correctly: "The VM has the proprietary (nvidia.ko) kernel module loaded, but Blackwell GPUs (RTX PRO 6000, SM120) require the open kernel modules (nvidia-open). The proprietary driver doesn't support these GPUs."

This is the key insight. NVIDIA's driver ecosystem has split into two branches: the proprietary kernel module (nvidia.ko, packaged as nvidia-dkms-590) and the open-source kernel module (nvidia-open, packaged as nvidia-dkms-590-open). Starting with the Blackwell architecture (compute capability SM120), NVIDIA made the open kernel modules mandatory — the proprietary module simply cannot drive these GPUs. This is not a bug or a configuration issue; it is an architectural decision by NVIDIA that anyone deploying Blackwell GPUs must understand.

The Verification Step

Having identified the root cause, the assistant does not stop at the error message. It immediately runs a diagnostic command to confirm the hypothesis:

ssh theuser@10.1.16.221 'dpkg -l | grep -i "nvidia.*driver\|nvidia.*kernel\|nvidia.*open\|nvidia.*dkms" | head -10; echo "---"; ls /lib/modules/$(uname -r)/updates/dkms/nvidia* 2>/dev/null; echo "---"; uname -r'

The output confirms that the VM has nvidia-dkms-590 (the proprietary package) installed, not nvidia-dkms-590-open. The kernel module file at /lib/modules/6.8.0-101-generic/updates/dkms/nvidia.ko.zst is the proprietary variant. This is the definitive proof.

Assumptions and Knowledge Required

To fully understand this message, several pieces of background knowledge are necessary:

First, one must understand the NVIDIA driver architecture and the distinction between the proprietary and open kernel module branches. NVIDIA introduced the open kernel modules with the R515 driver release in 2022, initially as an optional alternative. By the time of the Blackwell architecture, the open modules had become mandatory.

Second, one needs to know that the RTX PRO 6000 Blackwell GPUs use the SM120 compute capability, which is only supported by the open kernel modules. This is not widely documented — it is something discovered through experience or careful reading of NVIDIA release notes.

Third, the error code RmInitAdapter failed! (0x22:0x56:1001) is NVIDIA's internal Resource Manager error. The 0x22 likely indicates a firmware or initialization state mismatch, and 0x56:1001 provides additional context about the specific failure path. Without the kernel log message about requiring open modules, this error alone would be difficult to interpret.

Fourth, understanding the Proxmox passthrough setup is important context. The GPUs are passed through via VFIO, meaning the host unbinds them from the NVIDIA driver and binds them to vfio-pci. Inside the VM, a fresh NVIDIA driver instance must initialize them. The passthrough itself was working correctly — the GPUs appeared in the VM's PCI bus — but the wrong driver variant was loaded.

The Thinking Process

The assistant's reasoning in this message follows a classic diagnostic pattern: observe the symptom, find the error, interpret the error, verify the hypothesis. The progression from the previous messages is clear:

  1. [msg 5762]: The assistant discovers nvidia-smi returns "No devices were found" with exit code 6. This narrows the problem to driver initialization rather than PCI visibility.
  2. [msg 5763]: The assistant digs deeper, checking /proc/driver/nvidia/version, lspci -ks for the kernel driver binding, and dmesg for kernel errors. The dmesg output contains the critical error messages that the assistant then quotes in the subject message.
  3. [msg 5764] (the subject): The assistant presents the error, interprets it, and runs a confirmation command to check which DKMS package is installed. The thinking process reveals a methodical approach: the assistant does not jump to conclusions or guess. It lets the kernel logs speak, then interprets what they say in the context of NVIDIA's driver architecture. The error message about requiring open kernel modules is explicit, but it requires domain knowledge to understand why — the assistant provides that context by explaining that Blackwell GPUs require the open modules.

Output Knowledge and Impact

This message creates several important outputs:

  1. A clear diagnosis: The root cause is identified — wrong kernel module variant installed in the VM.
  2. A specific fix path: The solution is to install nvidia-dkms-590-open instead of nvidia-dkms-590. The assistant explicitly notes that the userspace libraries (libnvidia-gl, libnvidia-cfg1, etc.) are compatible with either kernel module, so only the DKMS package needs to change.
  3. A reusable diagnostic pattern: The sequence of checking nvidia-smi exit code, then dmesg for NVIDIA errors, then verifying the installed package, is a pattern that can be applied to other GPU initialization failures.
  4. Documentation of a compatibility requirement: The message implicitly documents that Blackwell GPUs require the open kernel modules — knowledge that is valuable for anyone deploying these GPUs. The follow-up messages ([msg 5765] and [msg 5766]) confirm the fix is straightforward: nvidia-dkms-590-open is available in the Ubuntu repositories at the same version (590.48.01), and the assistant offers to execute the installation.

Broader Significance

This message illustrates a broader theme in modern GPU computing: the growing complexity of the NVIDIA driver ecosystem. With the introduction of open kernel modules, GSP (GPU System Processor) firmware, and architecture-specific requirements, deploying NVIDIA GPUs has become more nuanced than simply installing the latest driver. The Blackwell generation in particular represents a transition point where the open modules are no longer optional but mandatory.

The message also demonstrates effective remote debugging of a passthrough setup — a scenario that combines virtualization complexity with GPU driver complexity. The assistant successfully navigates both layers, isolating the problem to the guest VM's driver selection rather than the Proxmox passthrough configuration, the VFIO binding, or the hardware itself.

In conclusion, message [msg 5764] is a textbook example of targeted diagnostic communication: it presents the discovered error, interprets it with domain knowledge, verifies the hypothesis with a targeted command, and implicitly outlines the fix. It transforms a cryptic kernel error into actionable knowledge, saving future debugging time and contributing to a growing understanding of Blackwell GPU deployment requirements.