The Blackwell GPU Passthrough Diagnostic: When nvidia-smi Sees Nothing

Introduction

In the course of a sprawling machine learning infrastructure session spanning multiple days, a single diagnostic message stands out as a masterclass in systematic troubleshooting. At message index 5763, the assistant executes a remote bash command on a Proxmox VM guest to investigate a puzzling failure: eight NVIDIA Blackwell RTX PRO 6000 GPUs are visible to the Linux kernel, their PCI devices are bound to the nvidia driver, the NVIDIA kernel module version matches the host's — yet nvidia-smi returns the cryptic error "No devices were found" with exit code 6. This message captures the precise moment when the assistant pivots from high-level configuration to low-level kernel debugging, and the results it uncovers set the stage for a critical insight about NVIDIA's proprietary versus open kernel modules on Blackwell hardware.

The Context: A Tangent Experiment

The broader session had been focused on hardening the production deployment of the Kimi-K2.5 INT4 model with SGLang, including setting up a systemd service, enabling hierarchical KV cache with 358 GB of pinned host memory, and finalizing the EAGLE-3 speculative decoding configuration. But the user then initiated a tangent experiment: passing all eight GPUs from the Proxmox host (10.1.2.6) to a VM guest (10.1.16.221). This required unbinding the GPUs from the host's NVIDIA driver and binding them to the vfio-pci driver for PCI passthrough — a well-established Proxmox workflow.

The initial checks ([msg 5758]) confirmed that the host had successfully switched the GPUs to vfio-pci, and the VM guest could see all eight devices via lspci and even had /dev/nvidia* device files. Yet nvidia-smi returned "No devices were found" ([msg 5762]). This was the puzzle that message 5763 set out to solve.

The Diagnostic Command

The assistant constructed a compound bash command that ran four diagnostic probes in sequence, piped through ssh to the VM guest as user theuser:

cat /proc/driver/nvidia/version 2>/dev/null
echo "---"
lspci -ks 06:10.0 2>/dev/null
echo "---"
modinfo nvidia 2>/dev/null | head -5
echo "---"
sudo dmesg | grep -i "nvidia\|nvrm\|gpu\|vfio\|error" | tail -20

Each probe targeted a different layer of the GPU software stack. The first checked the NVIDIA kernel module version via the /proc filesystem — a direct communication channel with the NVRM (NVIDIA Resource Manager). The second examined a specific PCI device (06:10.0, one of the eight GPUs) to see which kernel driver was bound to it. The third queried the module metadata to confirm the installed driver package. The fourth searched kernel ring buffer logs for any error messages related to NVIDIA, the NVRM driver, GPU initialization, VFIO, or generic errors — the last 20 lines being the most relevant.

The Results: A Contradiction Emerges

The output revealed a fascinating contradiction:

NVRM version: NVIDIA UNIX x86_64 Kernel Module  590.48.01
---
06:10.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
    Kernel driver in use: nvidia
---
filename: /lib/modules/6.8.0-101-generic/updates/dkms/nvidia.ko.zst
version: 590.48.01

On the surface, everything looked correct. The NVIDIA kernel module version 590.48.01 was loaded and responding via /proc. The PCI device at 06:10.0 showed Kernel driver in use: nvidia — meaning the nvidia driver had successfully claimed the device. The module was installed via DKMS (Dynamic Kernel Module Support) at version 590.48.01, compiled against the 6.8.0-101-generic kernel.

Yet nvidia-smi — the primary diagnostic tool for NVIDIA GPU health — reported no devices. Exit code 6 in nvidia-smi specifically means "No devices were found," which typically indicates that while the kernel module loaded, it failed to initialize any GPU hardware during its probe phase.

The Deeper Problem: Blackwell and the Proprietary Driver

This is where the assistant's diagnostic chain reveals its true purpose. The contradiction — driver loaded, driver bound, but no devices visible to nvidia-smi — is a classic symptom of the proprietary NVIDIA driver failing to initialize Blackwell (SM120 architecture) GPUs inside a virtual machine. The proprietary nvidia kernel module (590.48.01) does not support SM120 in virtualized environments. NVIDIA's Blackwell architecture requires the open kernel module (nvidia-open) for proper operation in VMs, even when the host uses the proprietary driver.

The assistant's choice to include vfio in the dmesg grep pattern was prescient — it anticipated that VFIO-related errors might appear in the kernel log. The inclusion of nvrm (the NVIDIA Resource Manager component) and gpu alongside error shows a systematic approach: check every subsystem that could have logged a failure during GPU initialization.

The modinfo output confirmed the module was the proprietary version (the filename nvidia.ko.zst rather than nvidia-open.ko.zst), which was the root cause. The VM had the proprietary driver installed and loaded, but it couldn't initialize the Blackwell GPUs in the virtualized PCI environment.

Assumptions and Reasoning

The assistant made several key assumptions in constructing this diagnostic. First, it assumed the problem was at the kernel driver level rather than the userspace CUDA toolkit level — a reasonable inference since nvidia-smi failed but the devices appeared in lspci. Second, it assumed that checking the PCI device's kernel driver binding would reveal whether the vfio-pci driver had inadvertently retained control (which would have been a different failure mode). Third, it assumed the NVIDIA module version would match the host's, which it did — ruling out a version mismatch between host and guest.

The assistant also implicitly assumed that the dmesg log would contain initialization failure messages. This was a bet on the NVIDIA driver being chatty about its failures — a bet that paid off in the subsequent resolution (installing the open kernel module).

One subtle reasoning choice: the assistant used lspci -ks 06:10.0 with the -k flag (show kernel drivers) and -s (show specific slot) rather than a broader query. This focused on a single representative GPU rather than dumping all eight, suggesting the assistant expected the answer to be uniform across devices — a reasonable assumption for identical GPUs on the same driver.

Knowledge Required to Understand This Message

To fully grasp this message, one needs knowledge of several domains:

Knowledge Created by This Message

This message produced several critical pieces of knowledge:

  1. The NVIDIA driver version inside the VM was confirmed as 590.48.01, matching the host — ruling out version mismatch as the cause.
  2. The kernel driver binding was correct — nvidia was in use, not vfio-pci, meaning the driver handoff from hypervisor to guest driver succeeded.
  3. The module was the proprietary version, not the open module — the filename was nvidia.ko.zst, not nvidia-open.ko.zst.
  4. The failure was silent at the driver level — no obvious error messages in the dmesg snippet (though the full output might have contained more).
  5. The problem was narrowed to driver initialization, not driver loading or device discovery. This diagnostic effectively eliminated three of the four possible failure modes (driver not loaded, wrong driver bound, device not visible to kernel) and isolated the problem to the fourth: driver initialization failure for Blackwell GPUs in a VM.

The Thinking Process

The assistant's reasoning is visible in the construction of the diagnostic command. It proceeds from the most fundamental check (is the kernel module loaded and what version?) to the most specific (are there error messages in the kernel log?). The echo "---" separators show the assistant expected to parse the output visually, grouping related information.

The choice to check /proc/driver/nvidia/version first is telling — this is the most direct way to confirm the NVRM is alive and communicating. If this file didn't exist or showed an error, the problem would be at the module loading stage. Since it returned a valid version string, the assistant could rule out module loading failures.

The lspci -ks check then confirmed the driver binding, and the modinfo check revealed the module identity. Together, these three checks painted a picture of a system where everything was nominally correct — the driver was loaded, bound, and responsive — yet the GPUs weren't initialized. This contradiction is the hallmark of a compatibility issue rather than a configuration issue.

The Resolution Path

The resolution, which occurred in subsequent messages, was to install the open kernel module package (nvidia-dkms-590-open) inside the VM. This swapped the proprietary nvidia.ko for the open nvidia-open.ko, which properly supports Blackwell GPUs in virtualized environments. The diagnostic in message 5763 was the crucial step that identified the proprietary module as the culprit, setting the stage for this fix.

Conclusion

Message 5763 exemplifies the systematic diagnostic approach that characterizes effective infrastructure engineering. By methodically checking each layer of the GPU software stack — kernel module version, driver binding, module identity, and kernel logs — the assistant narrowed a confusing failure from a vague "no devices found" error to a specific compatibility issue between the proprietary NVIDIA driver and Blackwell GPUs in a VM. The message is a case study in how to decompose a "works on paper but fails in practice" problem into its constituent layers, test each layer independently, and let the contradiction between layers reveal the root cause.