The Diagnostic Pivot: Tracing a GPU Passthrough Failure from PCI Bus to Kernel Module

In the middle of a high-stakes production deployment of the Kimi-K2.5 INT4 model across 8 RTX PRO 6000 Blackwell GPUs, the conversation takes an unexpected detour. The user, having just finished hardening the SGLang server with hierarchical KV cache and systemd service, asks about a "tangent experiment" — passing all 8 GPUs to a Proxmox VM instead of using them in the LXC container ([msg 5752]). This seemingly simple question triggers a rapid-fire diagnostic sequence that culminates in message 5761, a carefully crafted SSH command that reveals the root cause of a silent GPU initialization failure.

The Context: A Sudden Platform Shift

The moments before message 5761 are critical to understanding its purpose. The assistant had just confirmed that the Proxmox host (10.1.2.6) was already configured for GPU passthrough — VM 131 (ml-pipelines) was actively running with all 8 GPUs bound to the vfio-pci driver via a pro6000 PCI mapping ([msg 5758]). The host-side switching mechanism was working: Proxmox automatically unbinds GPUs from the NVIDIA driver and rebinds them to vfio-pci when a VM with hostpci entries starts, then reverses the process when the VM stops. No blacklisting or reboot required.

But then came the critical failure signal. The assistant attempted to verify the VM guest could see the GPUs by running nvidia-smi on the guest (10.1.16.221), but the SSH command failed with a login error — it had tried to connect as root when the VM only allowed the user theuser ([msg 5759]). The user then reported the alarming symptom: "Inside the vm nvidia-smi doesn't show anything" ([msg 5760]).

This is the moment message 5761 enters. The assistant cannot trust the previous failed probe. It must start fresh, with the correct credentials, and systematically determine why eight perfectly passed-through Blackwell GPUs are invisible to the NVIDIA management tool.

The Message: A Three-Pronged Diagnostic Probe

Message 5761 executes a single SSH command that runs three diagnostic checks in sequence:

ssh theuser@10.1.16.221 'lspci | grep -i nvidia; echo "---"; ls /dev/nvidia* 2>/dev/null || echo "no /dev/nvidia devices"; echo "---"; dmesg 2>/dev/null | grep -i nvidia | tail -10 || sudo dmesg | grep -i nvidia | tail -10'

This is not a random collection of commands. Each probe targets a distinct layer of the GPU software stack, and together they form a complete fault isolation chain:

  1. lspci | grep -i nvidia — Checks the PCI bus enumeration. If the GPUs were not properly passed through by the hypervisor, they would not appear here. This is the lowest-level check: is the hardware visible to the operating system at all?
  2. ls /dev/nvidia* — Checks for NVIDIA device files. The NVIDIA driver creates character devices (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, etc.) when it successfully initializes. If these files are missing, the driver either isn't loaded or failed during initialization.
  3. dmesg | grep -i nvidia — Checks the kernel ring buffer for NVIDIA driver messages. This is the most informative probe: it captures driver load events, initialization sequences, and crucially, any error messages that explain why initialization failed. The command structure also reveals careful engineering. The 2>/dev/null redirects suppress error output from ls when no NVIDIA devices exist, and the || fallback prints a clear message instead of a raw shell error. The dmesg command includes a sudo fallback — if the unprivileged user cannot read kernel logs, it tries with elevated privileges. This robustness matters when diagnosing remote systems with unknown configurations.

The Output: A Contradiction That Points to the Answer

The output from message 5761 is revealing:

06:10.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
06:11.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
06:1b.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
06:1c.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
08:0d.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
08:0e.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
08:0f.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
08:10.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)

All 8 GPUs appear in lspci. The PCI passthrough is working perfectly. The device ID 2bb5 confirms these are Blackwell-generation RTX PRO 6000 GPUs (SM120 architecture). The ls /dev/nvidia* command did not fall through to the "no /dev/nvidia devices" fallback message, meaning device files exist — the driver is loaded. But nvidia-smi reports "No devices were found." This is the contradiction: the driver loaded but cannot communicate with any GPU.

The dmesg output is truncated in the conversation data, but the assistant's follow-up in message 5763 reveals what it contained:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This is the smoking gun. The VM has the proprietary NVIDIA kernel module (nvidia.ko) installed, but Blackwell GPUs require the open kernel module (nvidia-open). The proprietary driver loads but fails during adapter initialization — RmInitAdapter failed — because it doesn't contain the necessary firmware initialization paths for the SM120 architecture.

The Reasoning Behind the Diagnostic Strategy

The assistant's thinking process in message 5761 is a textbook example of layered fault isolation. Rather than jumping to conclusions or trying random fixes, the assistant systematically eliminates possibilities:

Layer 1 — Hardware visibility: Before the user reported the issue, the assistant had confirmed on the Proxmox host that all 8 GPUs were bound to vfio-pci ([msg 5758]). But that only proves the host-side configuration. The VM-side PCI enumeration must be verified independently — a misconfigured VM BIOS or PCI topology could hide devices even with correct host-side binding.

Layer 2 — Driver presence: Even with visible PCI devices, the NVIDIA driver might not be installed in the VM at all. The VM is a fresh Ubuntu installation, and the assistant had not yet installed any NVIDIA software inside it. The ls /dev/nvidia* check confirms the driver is present and loaded.

Layer 3 — Driver correctness: This is the subtle but critical layer. A loaded driver is not necessarily a compatible driver. The NVIDIA driver ecosystem has two parallel kernel module tracks: proprietary (nvidia.ko) and open (nvidia-open.ko). For consumer and datacenter GPUs through Hopper (H100), both work. But starting with Blackwell (SM120), NVIDIA requires the open kernel modules. The proprietary driver loads but immediately fails during GPU initialization because it lacks the firmware interfaces needed for the new architecture.

The assistant's earlier assumption that "the nvidia modules are loaded but not bound to any devices" ([msg 5759]) was correct for the host but dangerously incomplete for the guest. The host had the proprietary modules loaded with zero refcount (no GPUs bound), which was harmless. But the VM had the proprietary modules loaded with all 8 GPUs bound — and failing silently.

Input Knowledge Required

To understand message 5761, the reader needs knowledge spanning multiple domains:

GPU virtualization: Understanding that PCI passthrough (vfio-pci) makes GPUs appear to the guest VM as if they were physically attached, requiring the guest to have its own GPU driver. The host's driver state is irrelevant once the device is bound to vfio-pci.

NVIDIA driver architecture: The distinction between the proprietary kernel module (nvidia.ko, built from closed-source code) and the open kernel module (nvidia-open.ko, released under GPL). Both expose the same interfaces to userspace libraries like libcuda.so, but the open module is required for Blackwell GPUs.

Linux device enumeration: Understanding that lspci reads from the PCI configuration space and does not require a functional driver — it shows devices at the bus level regardless of driver state. Conversely, /dev/nvidia* files are created by the driver during its initialization sequence.

Kernel diagnostics: Knowing that dmesg captures driver initialization messages, including error codes like RmInitAdapter failed (0x22:0x56:1001), which indicate the GPU firmware initialization step failed inside the NVIDIA kernel module.

Output Knowledge Created

Message 5761 produces a clear diagnostic picture:

  1. PCI passthrough is fully functional — all 8 GPUs are visible on the VM's PCI bus with correct device IDs.
  2. The NVIDIA driver is loaded — device files exist in /dev, indicating the kernel module initialized.
  3. The driver is incompatible — the contradiction between device file presence and nvidia-smi failure points to a driver-level initialization failure, confirmed by dmesg in the subsequent message. This output directly drives the next action: installing the open kernel module package (nvidia-dkms-590-open) to replace the proprietary module, which the assistant proposes in message 5766 and the user approves in message 5767.

The Broader Significance

Message 5761 exemplifies a critical pattern in systems engineering: the moment when a working configuration fails in a new context. The Proxmox host was perfectly configured for GPU passthrough. The VM had the NVIDIA driver installed. But the combination failed because of a subtle architectural requirement — Blackwell GPUs need the open kernel module — that was not obvious from any single error message.

The assistant's diagnostic approach — probing from hardware (PCI bus) through kernel (device files) to driver initialization (dmesg) — mirrors the OSI model of network troubleshooting, applied to GPU virtualization. Each layer's verification narrows the problem space until the root cause becomes unambiguous. This systematic methodology, captured in a single SSH command, transforms a confusing "nvidia-smi doesn't show anything" report into a precise, actionable fix: swap the DKMS package and reboot.

The message also highlights the importance of robust remote diagnostics. The SSH command handles edge cases (missing device files, unreadable kernel logs) gracefully, ensuring the probe returns useful information regardless of the VM's configuration state. This attention to detail — the 2>/dev/null redirects, the || fallback, the sudo escalation path — separates a fragile one-off check from a reusable diagnostic tool.