The Open Kernel Module Revelation: Diagnosing Blackwell GPU Passthrough Failure in a Proxmox VM

Introduction

In the course of a high-stakes machine learning deployment spanning eight NVIDIA RTX PRO 6000 Blackwell GPUs, a seemingly straightforward tangent experiment—passing these GPUs from a Proxmox host into a guest virtual machine—unraveled into a revealing diagnostic journey. Message [msg 5765] captures the critical turning point: the moment the assistant confirmed the root cause of a perplexing nvidia-smi failure and identified the precise fix. This message, though brief in its surface form, represents the culmination of a multi-step debugging process that illuminates fundamental truths about NVIDIA's GPU driver architecture, the Blackwell generation's unique requirements, and the careful detective work required when hardware and software assumptions collide.

The Context: A Tangent Experiment

The session had been focused on hardening the production deployment of the Kimi-K2.5 INT4 model, a 547 GB parameter giant running across eight GPUs in an LXC container on a Proxmox VE host (<msg id=5737-5751>). The hierarchical KV cache had just been configured with a 4× ratio, allocating 358 GB of pinned host memory as a CPU-side prefix cache. Everything was operational and performing well.

Then the user pivoted: "Doing a tangent experiment on the proxmox node" ([msg 5752]). The question was whether the GPUs could be passed through to a VM instead of the container, and how to make the switch between configurations reasonably painless. This is a common operational concern in homelab and production environments where the same hardware must serve multiple purposes—sometimes running inference directly on the host, sometimes inside a virtualized guest.

The assistant's initial investigation (<msg id=5753-5758>) revealed that the Proxmox host was already well-configured for GPU passthrough. IOMMU was enabled (iommu=pt), VFIO had been used before, and VM 131 (ml-pipelines) was already running with all eight GPUs passed through via a pro6000 PCI mapping. The Proxmox PCI passthrough mechanism handles the driver switch automatically: when a VM with hostpci entries starts, the host unbinds the devices from the nvidia driver and binds them to vfio-pci; when the VM stops, they revert. This seemed to promise a smooth workflow.

The Failure: "No devices were found"

The first sign of trouble came when the assistant attempted to verify GPU visibility inside the VM guest ([msg 5759]). The initial SSH command failed with a login error, but subsequent checks ([msg 5761]) showed that the GPUs were visible via lspci—eight NVIDIA Device 2bb5 (the RTX PRO 6000 Blackwell) entries appeared—and /dev/nvidia* device nodes existed. The kernel driver was loaded. Yet nvidia-smi returned a stark "No devices were found" with exit code 6 ([msg 5762]).

This is a particularly confusing failure mode. The PCI devices are visible to the guest operating system. The NVIDIA kernel module is loaded. The device files exist. But the management tool reports no devices. Exit code 6 in nvidia-smi specifically indicates that the NVIDIA Management Library (NVML) could not find any devices, which typically means the kernel module's initialization of the GPU hardware failed at a fundamental level.

The Diagnostic Breakthrough

The assistant's next step ([msg 5763]) was to examine the kernel logs via dmesg. This revealed the critical error messages:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This is a deeply informative error. The RmInitAdapter function is part of NVIDIA's Resource Manager (RM), the low-level firmware interface that initializes communication with the GPU hardware. A failure here means the driver could not establish the basic control channel to the GPU—the card was recognized on the PCI bus but could not be brought online.

The first line is the key: "installed in this system requires use of the NVIDIA open kernel modules." This is a Blackwell-specific constraint. Starting with the Blackwell architecture (compute capability SM120), NVIDIA made the open kernel modules (nvidia-open) mandatory. The proprietary closed-source kernel module (nvidia.ko), which had served every previous generation from Maxwell through Hopper, simply cannot initialize Blackwell GPUs. The hardware itself rejects the proprietary driver's initialization attempts.

Message 5765: The Confirmation and the Fix

With this diagnostic context established, message [msg 5765] arrives as the confirmation step. The assistant states:

Confirmed — nvidia-dkms-590 (proprietary) is installed. Needs nvidia-dkms-590-open instead.

This single sentence encapsulates the entire diagnostic chain. The VM guest had the standard nvidia-dkms-590 package installed, which builds the proprietary nvidia.ko kernel module via DKMS. This is the default NVIDIA driver package on Ubuntu 24.04 and is perfectly appropriate for every GPU generation from Maxwell through Ada Lovelace. But for Blackwell GPUs, it is entirely non-functional.

The assistant then executes a verification command to confirm that the correct package is available in the Ubuntu repositories:

ssh theuser@10.1.16.221 'apt list --all-versions nvidia-dkms-590-open 2>/dev/null; echo "---"; apt list --all-versions nvidia-open 2>/dev/null'

The output confirms that nvidia-dkms-590-open version 590.48.01-0ubuntu0.24.04.1 is available in the noble-updates and noble-security repositories. Additionally, the nvidia-open package (which provides the userspace libraries) shows multiple versions available, ranging from 580.65 through 590.48.01.

The Reasoning Process: What Made This Diagnosis Possible

The diagnostic path that led to message [msg 5765] reveals several layers of reasoning:

Layer 1: Eliminating trivial causes. Before reaching the kernel module diagnosis, the assistant had to rule out simpler explanations. Was the PCI passthrough working correctly? (Yes—lspci showed all eight devices.) Were the device nodes created? (Yes—/dev/nvidia* existed.) Was the driver loaded? (Yes—/proc/driver/nvidia/version reported version 590.48.01.) Each of these checks eliminated a potential failure point and narrowed the search space.

Layer 2: Recognizing the Blackwell-specific constraint. The assistant had prior knowledge (gained earlier in the session, see <msg id=5737-5751> context about SM120 patches) that Blackwell GPUs require the open kernel modules. This knowledge was essential—without it, the RmInitAdapter error could have been misinterpreted as a generic hardware failure or a VFIO compatibility issue.

Layer 3: Correlating the error with the installed packages. The dmesg output explicitly stated the requirement for open kernel modules. The assistant then checked which DKMS package was installed (dpkg -l | grep nvidia) and confirmed it was the proprietary nvidia-dkms-590. The correlation was immediate: the wrong kernel module package was installed for this hardware.

Layer 4: Verifying the fix is available. Before declaring the solution, the assistant checked that nvidia-dkms-590-open exists in the repositories at the same version (590.48.01). This is a critical operational step—recommending a package that doesn't exist or is at an incompatible version would waste time and potentially break the system further.

Assumptions Made and Corrected

Several assumptions were tested and corrected during this diagnostic process:

Assumption 1: Standard NVIDIA driver packages work on all GPUs. This is the most natural assumption—NVIDIA has maintained backward compatibility for decades, and the same driver package works across multiple generations. The Blackwell generation broke this pattern by requiring the open kernel modules, a change that surprised many administrators.

Assumption 2: GPU passthrough issues are usually on the host side. The initial investigation focused heavily on the Proxmox host configuration—IOMMU groups, VFIO bindings, PCI mappings. The assistant assumed the problem might be in how the GPUs were being passed through. But the host side was actually working perfectly; the failure was entirely within the guest's driver configuration.

Assumption 3: If lspci shows the devices and /dev/nvidia* exists, the driver is working. This is a reasonable heuristic for most GPU troubleshooting, but it fails in this case because the kernel module can load and create device nodes without successfully initializing any GPU hardware. The RmInitAdapter failure occurs after the module is loaded but before any GPU is brought online.

Assumption 4: The same VM configuration that worked for previous GPUs would work for Blackwell. VM 131 (ml-pipelines) was already configured with the pro6000 mapping and presumably had been working with other NVIDIA GPUs. The assistant initially assumed this proven configuration would work seamlessly with the Blackwell cards.

Input Knowledge Required

To fully understand message [msg 5765], several pieces of knowledge are required:

NVIDIA driver architecture: Understanding the distinction between the proprietary kernel module (nvidia.ko, provided by nvidia-dkms-*) and the open kernel module (nvidia-open.ko, provided by nvidia-dkms-*-open). These are separate codebases with different hardware support matrices.

Blackwell (SM120) requirements: Knowing that the Blackwell architecture specifically requires the open kernel modules, a change introduced with this generation. This is not widely documented and represents a departure from NVIDIA's historical approach.

DKMS (Dynamic Kernel Module Support): Understanding that DKMS packages automatically rebuild kernel modules when the kernel is updated, and that switching between nvidia-dkms-590 and nvidia-dkms-590-open requires rebuilding the module and potentially rebooting or reloading.

Proxmox PCI passthrough mechanics: Understanding that VFIO passthrough hands the raw PCI device to the guest, and the guest is responsible for loading the appropriate driver. The host's driver choice is irrelevant once the device is bound to vfio-pci.

nvidia-smi exit codes: Exit code 6 indicates NVML could not enumerate any devices, which differs from exit code 2 (driver not loaded) or exit code 3 (driver version mismatch).

Output Knowledge Created

Message [msg 5765] creates several valuable pieces of knowledge:

The root cause is definitively identified: The VM guest has the proprietary nvidia-dkms-590 installed instead of the required nvidia-dkms-590-open. This is a clear, actionable diagnosis.

The fix is confirmed available: The nvidia-dkms-590-open package at version 590.48.01 exists in the standard Ubuntu repositories. No third-party PPAs or manual driver downloads are needed.

The fix path is clear: Install nvidia-dkms-590-open, which will replace the proprietary kernel module with the open one, rebuild it via DKMS, and (after a reboot or module reload) allow nvidia-smi to initialize the Blackwell GPUs.

The diagnostic methodology is validated: The multi-step process of checking PCI visibility, device nodes, driver version, kernel logs, and installed packages proved effective. This methodology can be applied to similar GPU passthrough issues in the future.

A broader lesson is documented: Blackwell GPUs require the open NVIDIA kernel modules. This is a change from all previous generations and will catch many administrators off guard when upgrading hardware.

The Broader Implications

This message, while seemingly minor, touches on several important themes in modern GPU infrastructure management:

The fragmentation of NVIDIA's driver ecosystem. With the introduction of the open kernel modules as a mandatory requirement for Blackwell, NVIDIA has created a split in its driver landscape. Administrators must now track which GPU generations require which kernel module variant, adding complexity to configuration management.

The challenge of VM GPU passthrough. While Proxmox's VFIO mechanism handles the host-side driver switching automatically, the guest must still have the correct driver installed. This creates a two-layer dependency: the host must support the GPU architecture (which it does via VFIO), and the guest must have the right kernel module (which it didn't, in this case).

The importance of kernel log inspection. The dmesg output was the key diagnostic tool. Without examining the kernel logs, the RmInitAdapter error would have remained hidden, and the "No devices were found" symptom would have been much harder to trace to its root cause.

Conclusion

Message [msg 5765] represents the diagnostic climax of a GPU passthrough troubleshooting session. In a single sentence—"Confirmed — nvidia-dkms-590 (proprietary) is installed. Needs nvidia-dkms-590-open instead."—the assistant crystallizes the entire investigative chain: the initial assumption of a host-side passthrough problem, the discovery that GPUs were visible but non-functional, the critical dmesg clue about open kernel module requirements, and the final correlation with the installed package. The subsequent verification command confirms that the fix is readily available, transforming a puzzling failure into a straightforward package swap.

This message exemplifies the kind of diagnostic reasoning that separates surface-level troubleshooting from genuine root cause analysis. It also serves as a cautionary tale about the assumptions we carry forward when working with new hardware generations—sometimes the rules change, and the old solutions stop working.