The One-Line Fix: How a Single Package Swap Brought Blackwell GPUs to Life Inside a VM

In the sprawling, multi-week journey of deploying large language models across a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs, most challenges were sprawling and multi-layered: CUDA version mismatches, flash-attn compilation failures, NCCL tuning parameters, and speculative decoding throughput analysis. But sometimes, the most critical insight arrives in the most compact form imaginable. Message [msg 5766] is exactly that — a single, five-line assistant response that identifies, diagnoses, and prescribes the fix for a problem that had stumped the user: Blackwell GPUs passed through to a Proxmox VM were invisible to nvidia-smi, returning the cryptic error "No devices were found."

The message reads in its entirety:

Package is available. Just swap the DKMS package:

>

``bash sudo apt install nvidia-dkms-590-open ``

>

This will replace nvidia-dkms-590 (proprietary) with the open kernel modules and rebuild via DKMS. Then reboot the VM. That's the only fix needed — everything else (driver version, userspace libs) stays the same at 590.48.01.

>

Want me to run it?

To the uninitiated, this looks like a trivial package installation command. To anyone who has wrestled with NVIDIA's bifurcated driver ecosystem on modern hardware, it represents the culmination of a precise diagnostic chain that cut through layers of misdirection to land on the exact root cause. This article unpacks every dimension of that message: the reasoning that motivated it, the assumptions it rests on, the knowledge it required, and the knowledge it created.

The Context: A Tangent Experiment That Hit a Wall

The broader session (Segment 38) was in the midst of transitioning the Kimi-K2.5 INT4 model from experimental benchmarking to hardened production deployment. The assistant had just finished creating a systemd service with hierarchical KV cache, verifying that 358 GB of pinned host memory was allocated as an L2 prefix cache, and confirming the model was generating correctly with tool call and reasoning parsers. Then the user pivoted to a tangent experiment: passing all eight GPUs to a Proxmox VM (guest at 10.1.16.221, host at 10.1.2.6).

The assistant had initially assumed this would be straightforward. The Proxmox host already had IOMMU enabled (iommu=pt), PCI mappings defined for the pro6000 group, and another VM (ID 131) was already running with all eight GPUs passed through successfully. The host-side driver binding was handled automatically by Proxmox: when a VM with hostpci entries starts, the kernel unbinds the devices from the nvidia driver and rebinds them to vfio-pci. When the VM stops, they revert. No blacklisting, no reboot, no manual intervention.

But the user reported a problem: inside the VM, nvidia-smi showed nothing. The assistant's diagnostic sequence in messages [msg 5761] through [msg 5765] systematically ruled out the obvious suspects. The GPUs were visible via lspci — all eight RTX PRO 6000 devices appeared. The /dev/nvidia* device nodes existed. The NVIDIA kernel module was loaded, version 590.48.01. Yet nvidia-smi returned exit code 6 with "No devices were found."

The Diagnostic Chain: Reading the Kernel's Error Messages

The breakthrough came when the assistant examined the kernel ring buffer via dmesg. Hidden among the boot-time messages were two critical lines:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This is the moment where everything clicked. The VM had the proprietary NVIDIA kernel module installed (nvidia-dkms-590), which provides the nvidia.ko binary blob. But the RTX PRO 6000 Blackwell GPUs (architecture SM120) require the open kernel modules (nvidia-dkms-590-open), which provide nvidia-open.ko. NVIDIA made this distinction starting with the 590 driver series: the proprietary module simply refuses to initialize Blackwell GPUs, returning RmInitAdapter failed with a specific error code.

The assistant confirmed this by checking the package inventory inside the VM: dpkg -l | grep nvidia showed nvidia-dkms-590 was installed, not nvidia-dkms-590-open. A quick apt list query confirmed the open variant was available in the Ubuntu repositories at the exact same version (590.48.01).

The Reasoning: Why This Fix Works and What It Preserves

The assistant's recommendation is elegant because it minimizes disruption. The key insight is that only the kernel module differs between the proprietary and open driver stacks. The userspace libraries — libnvidia-cfg1-590, libnvidia-gl-590, libnvidia-fbc1-590, and crucially libcuda.so — are identical. The CUDA runtime API, the NVIDIA Management Library (NVML) that nvidia-smi uses, and all the GPU compute libraries work the same way regardless of which kernel module sits underneath.

By swapping only the DKMS package, the assistant preserves:

  1. Driver version consistency: Both packages are 590.48.01, so there is no version mismatch between kernel module and userspace libraries.
  2. CUDA toolkit compatibility: The existing CUDA 13.0 installation remains untouched.
  3. All userspace configuration: Environment variables, NCCL tuning parameters, and SGLang server arguments all stay valid.
  4. The reboot requirement: DKMS rebuilds the kernel module against the running kernel, but the old module remains loaded until the next reboot. The assistant explicitly calls this out. The question "Want me to run it?" is not just politeness — it signals that the assistant recognizes this is a destructive action (package removal/installation) that requires the user's consent, and that the VM will need a reboot afterward.

Assumptions and Potential Pitfalls

The message makes several assumptions that are worth examining:

Assumption 1: The open kernel module supports Blackwell GPUs on this kernel version. The assistant verified that nvidia-dkms-590-open version 590.48.01 is available, but did not explicitly verify that this particular version includes SM120 support. This is a reasonable inference — the proprietary module at the same version explicitly refuses Blackwell GPUs with a message saying "requires use of the NVIDIA open kernel modules," implying the open variant at the same version supports them. However, there is a subtle risk: very early 590.xx releases may have had incomplete Blackwell support even in the open module. The assistant hedged by checking that the version matched exactly.

Assumption 2: The DKMS build will succeed. Inside a VM with 8 GPUs passed through, the kernel source headers and build dependencies must be present for DKMS to compile the module. If linux-headers-$(uname -r) is missing, the install would fail. The assistant did not verify this beforehand.

Assumption 3: No other kernel modules depend on the proprietary module. The nvidia module has aliases and dependencies (nvidia_drm, nvidia_modeset, nvidia_uvm). The open module provides its own versions of these. The assistant implicitly assumes the package manager will handle the transition cleanly.

Assumption 4: The VM's kernel version is compatible. The VM was running kernel 6.8.0-101-generic. The assistant verified this via uname -r in the prior message. DKMS would build against this kernel, and the open module source must compile cleanly against it. This is a safe assumption for an Ubuntu 24.04 LTS kernel with the official NVIDIA package.

The Knowledge Created

This message creates several important pieces of knowledge:

  1. A diagnostic pattern: "nvidia-smi shows no devices but lspci sees the GPUs" → check dmesg for RmInitAdapter failed → check if proprietary vs open kernel module is installed. This pattern is reusable for anyone deploying Blackwell GPUs in virtualized environments.
  2. A deployment rule: Blackwell GPUs (SM120) require nvidia-dkms-*-open, never the proprietary nvidia-dkms-* package. This is not obvious from NVIDIA's documentation and is easy to miss when cloning a VM template that was originally provisioned for older GPUs.
  3. A minimal-diff strategy: When fixing driver issues, change only what is necessary. The assistant resisted the temptation to reinstall the entire driver stack or upgrade to a different version. By swapping only the kernel module package, the fix is minimal, reversible, and low-risk.
  4. A workflow for GPU passthrough on Proxmox: The broader context established that Proxmox handles driver rebinding automatically via vfio-pci — no blacklisting, no kernel cmdline changes, no reboot of the host. The only per-VM requirement is the correct kernel module flavor inside the guest.

The Broader Significance

In the context of the entire coding session, this message represents a recurring theme: the difference between a working system and a broken one often comes down to a single, easily overlooked detail. Earlier in the session, the assistant spent hours debugging flash-attn compilation failures, only to discover that MAX_JOBS needed to be reduced from 128 to 20 to avoid memory exhaustion. Later, it would spend similar effort diagnosing NaN outputs from the Qwen3.5 model, traced to incompatible FP4 GEMM and MoE backends. In each case, the root cause was narrow and specific, but the diagnostic chain required broad system knowledge.

The message also illustrates the assistant's diagnostic methodology: start with the most obvious checks (is the device visible in lspci? are the device nodes present?), escalate to more specific tools (dmesg kernel messages), and cross-reference with package management state. By the time the assistant proposes the fix, it has already ruled out PCIe passthrough issues, driver loading failures, and version mismatches. The proposed command is the logical conclusion of that elimination process, not a guess.

Conclusion

Message [msg 5766] is a masterclass in concise, precise technical communication. In five lines, it encapsulates a complete diagnostic journey: the problem (Blackwell GPUs invisible to nvidia-smi), the root cause (proprietary kernel module incompatible with SM120 architecture), the fix (swap to open kernel module), the verification (package available at same version), and the execution plan (install, reboot). It makes explicit assumptions about the environment that are grounded in prior verification, creates reusable knowledge about NVIDIA's driver bifurcation, and demonstrates the value of minimal, targeted interventions over wholesale reinstallation. For anyone deploying modern NVIDIA GPUs in virtualized environments, this message is a template for how to think about driver compatibility — and how to fix it when it breaks.