The Open Kernel Module Switch: A Single Command That Unblocked Blackwell GPU Passthrough

Introduction

In the course of deploying large language models across a heterogeneous infrastructure, hardware compatibility issues often emerge at the most unexpected layers. Message 5768 of this opencode session captures one such moment: a single apt install command that resolved a critical incompatibility between NVIDIA's proprietary driver stack and the Blackwell-generation RTX PRO 6000 GPUs. While the command itself appears mundane — installing a package via SSH on a remote VM — it represents the culmination of a systematic diagnostic process and the key to unlocking GPU passthrough for a production AI workload. This article examines why this message was written, the reasoning that led to it, the assumptions embedded in its execution, and the knowledge it both required and produced.

The Message

The subject message is a bash tool invocation executed by the assistant:

[assistant] [bash] ssh theuser@10.1.16.221 'sudo apt install -y nvidia-dkms-590-open 2>&1'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  libfontenc1 libnvidia-cfg1-590 libnvidia-common-590 libnvidia-compute-590
  libnvidia-decode-590 libnvidia-egl-wayland1 libnvidia-encode-590
  libnvidia-extra-590 libnvidia-fbc1-590 libnvidia-gl-590 libpkgconf3
  libvdpau1 libwayland-server0 libxcvt0 lib...

The output is truncated — the bash tool likely timed out or the command produced more output than could be captured in a single invocation — but the critical action is clear: the assistant is installing nvidia-dkms-590-open, the open-source variant of NVIDIA's kernel module, on a VM running at 10.1.16.221. This VM is a guest on a Proxmox hypervisor (10.1.2.6) that has been configured to pass through eight Blackwell RTX PRO 6000 GPUs via VFIO PCI passthrough.

The Narrative Leading to This Command

To understand why this message was written, we must trace the events that preceded it. The session had been focused on deploying the Kimi-K2.5 INT4 model with EAGLE-3 speculative decoding on an LXC container with 8 GPUs. After extensive benchmarking and optimization — including upgrading to CUDA 13, patching SGLang for SM120 support, enabling FlashInfer allreduce fusion, and tuning hierarchical KV cache parameters — the deployment was stable and performing well.

The user then pivoted to a "tangent experiment" ([msg 5752]): passing all 8 GPUs through to a Proxmox VM. The assistant investigated the host state, discovering that the Proxmox node already had IOMMU enabled (iommu=pt) and that a VM (ID 131, ml-pipelines) was already configured with all 8 GPUs mapped via hostpci entries using a pro6000 mapping. The GPUs were correctly bound to the vfio-pci driver on the host, meaning Proxmox had successfully unbounded them from the NVIDIA driver for VM passthrough.

However, inside the VM, nvidia-smi returned "No devices were found" ([msg 5762]). The assistant systematically debugged this: the GPUs appeared in lspci, the /dev/nvidia* device files existed, the NVIDIA kernel module was loaded (version 590.48.01), but the GPUs were not enumerated. The critical clue came from dmesg ([msg 5764]):

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This error message is the key to understanding the entire situation. NVIDIA's proprietary kernel module (nvidia.ko) does not support Blackwell-generation GPUs (SM120 architecture, device ID 10de:2bb5). These GPUs require the open kernel module (nvidia-open.ko), which is distributed as a separate DKMS package: nvidia-dkms-590-open instead of nvidia-dkms-590.

The assistant confirmed the diagnosis by checking the installed packages ([msg 5765]), verifying that the VM had nvidia-dkms-590 (proprietary) installed and that nvidia-dkms-590-open was available in the Ubuntu repository at the same version (590.48.01-0ubuntu0.24.04.1). After presenting the findings, the assistant asked "Want me to run it?" and the user responded "Run it" ([msg 5767]), leading directly to message 5768.

Why This Matters: The Proprietary vs. Open Kernel Module Split

NVIDIA's driver architecture has evolved significantly in recent years. Historically, the company shipped a single proprietary kernel module that handled all GPU generations. Starting with the Turing architecture, NVIDIA began developing an open-source kernel module, initially as a parallel effort. For Blackwell (SM120) and later architectures, the open kernel module became a requirement — the proprietary module simply cannot initialize these GPUs.

This creates a deployment trap: a system administrator who installs the standard nvidia-driver-590 package (which pulls in nvidia-dkms-590) will find that their Blackwell GPUs are detected by lspci and the driver loads without errors, but nvidia-smi reports no devices. The GPUs are visible to the PCI subsystem but fail to initialize at the NVIDIA driver level. The error message — "installed in this system requires use of the NVIDIA open kernel modules" — is the only clue.

The fix is straightforward but non-obvious: install nvidia-dkms-590-open instead. This package provides the same driver version (590.48.01) and uses the same userspace libraries (libnvidia-*), but replaces the kernel module with the open-source variant. DKMS (Dynamic Kernel Module Support) handles the compilation and installation of the module against the current kernel.

The Reasoning and Decision-Making Process

The assistant's decision to install nvidia-dkms-590-open was the result of a methodical diagnostic chain:

1. Verify hardware visibility: Check that the GPUs appear in lspci inside the VM — they do.
2. Check driver presence: Verify that /dev/nvidia* device files exist and the NVIDIA kernel module is loaded — both true.
3. Check driver functionality: Run nvidia-smi — returns "No devices were found" with exit code 6.
4. Examine driver logs: Read dmesg for NVIDIA errors — reveals the open kernel module requirement.
5. Identify the specific incompatibility: The error message explicitly states that the installed driver requires the open kernel module.
6. Verify package availability: Check that nvidia-dkms-590-open exists in the repository at the same version.
7. Confirm the fix with the user: Present the diagnosis and ask for authorization before executing. This is a textbook example of layered debugging: starting from the highest level (is the GPU visible?), moving to the driver layer (is the driver loaded?), and finally to the kernel module level (is it the right module?). Each step narrows the possible causes until the root cause is identified.

Assumptions Embedded in the Fix

The command in message 5768 carries several implicit assumptions:

That the open kernel module is a drop-in replacement. The assistant assumes that nvidia-dkms-590-open at version 590.48.01 is fully compatible with the existing userspace libraries (libnvidia-gl-590, libnvidia-cuda-590, etc.) and that no other packages need to change. This is correct — the open kernel module uses the same userspace API and the same firmware interfaces.

That DKMS will successfully build the module. The VM's kernel is 6.8.0-101-generic, and the package provides pre-built modules or DKMS build scripts. The assistant assumes the build will succeed, which is reasonable given that the package is from the official Ubuntu repository and has been tested against this kernel.

That a reboot will be required. The open kernel module won't take effect until the module is loaded at boot (or manually unloaded/reloaded). The assistant implicitly assumes the VM will be rebooted after installation, which is standard practice for kernel module changes.

That the proprietary packages will be auto-removed. The output shows "The following packages were automatically installed and are no longer required" — apt's automatic dependency resolution identifies that libnvidia-cfg1-590 and other proprietary packages can be removed once the open module replaces the proprietary one. The assistant assumes this auto-removal is safe, which it is since the open module provides equivalent functionality.

Input Knowledge Required

To understand and execute this fix, the assistant needed:

• Knowledge of NVIDIA's driver bifurcation: Understanding that there are two kernel module variants (proprietary and open) and that Blackwell requires the open variant. This is not widely documented and is a relatively recent development in NVIDIA's driver history.
• DKMS familiarity: Understanding how DKMS packages work, that they compile kernel modules against the running kernel, and that swapping DKMS packages triggers a rebuild.
• Ubuntu package management: Knowing the package naming convention (nvidia-dkms-590 vs nvidia-dkms-590-open) and how to verify package availability with apt list.
• VFIO and Proxmox passthrough mechanics: Understanding that the host-side driver binding is handled by Proxmox's PCI mapping and that the VM sees the GPUs as native devices.
• Diagnostic tooling: Knowing to check dmesg for NVIDIA driver errors, how to interpret nvidia-smi exit codes, and how to verify kernel module versions with modinfo.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

• The VM's driver stack is transitioning to open kernel modules: The apt command initiated the installation, which will download, compile, and install nvidia-open.ko.
• The proprietary packages are marked for removal: The output indicates that libnvidia-cfg1-590 and other proprietary libraries are "no longer required" and will be auto-removed after the installation completes.
• The system is in an intermediate state: Until the VM is rebooted, the proprietary module remains loaded and the GPUs remain non-functional. The fix is not complete until a reboot occurs.
• The diagnostic chain is validated: The error message from dmesg was correctly interpreted, and the package swap is the confirmed solution.

Broader Implications

This message illustrates a recurring theme in modern AI infrastructure: the gap between commodity Linux distributions and cutting-edge hardware. Ubuntu 24.04's default NVIDIA driver packages target the proprietary kernel module, which works for most GPU generations. But Blackwell — like Hopper before it with certain features — introduces requirements that the standard packages don't satisfy. System administrators deploying Blackwell GPUs must know to install the -open variant, a detail that is easy to miss and costly to debug.

The fix also highlights the value of the open kernel module program. By making the kernel module source available under GPL/MIT, NVIDIA enabled distribution maintainers to package it alongside the proprietary variant, giving users a seamless upgrade path. The nvidia-dkms-590-open package is built from the same source tree as the proprietary module but with the open-source build configuration, ensuring feature parity and API compatibility.

Conclusion

Message 5768 is, on its surface, a routine package installation command. But in context, it represents the resolution of a significant infrastructure puzzle: why eight Blackwell GPUs, correctly passed through via VFIO, refused to initialize inside a VM. The answer lay in a single kernel module variant, and the fix was a single apt command. The message captures the moment when systematic diagnosis meets decisive action — the assistant identified the root cause, verified the solution, obtained user authorization, and executed the fix. It is a small but perfect example of how deep hardware knowledge, layered debugging, and careful verification combine to resolve complex deployment issues in modern AI infrastructure.