The Diagnostic That Uncovered Blackwell's Open Kernel Module Requirement

In the middle of a tangent experiment to pass eight NVIDIA RTX PRO 6000 Blackwell GPUs through to a Proxmox virtual machine, a single short message — message 5762 — stands as a textbook example of systematic diagnostic narrowing. The message itself is deceptively brief:

GPUs are visible — 8 devices show up in lspci and /dev/nvidia* exists. Driver is loaded. Let me check what nvidia-smi actually says:

>

``bash ssh theuser@10.1.16.221 'nvidia-smi 2>&1; echo "exit code: $?"' No devices were found exit code: 6 ``

This message, a single tool call and its output, represents the critical pivot point in a debugging session that would ultimately reveal a subtle but crucial incompatibility between NVIDIA's proprietary kernel driver and the Blackwell GPU architecture.

The Context: A Tangled GPU Passthrough

The story begins with the assistant having just finished hardening a production deployment of the Kimi-K2.5 INT4 model on an LXC container running on a Proxmox host. The user then pivoted to a "tangent experiment": passing all eight GPUs to a VM instead. The host (10.1.2.6) already had IOMMU enabled and VFIO mappings configured — VM 131 (ml-pipelines) was already running with all eight GPUs bound to the vfio-pci driver on the host side. The new VM (10.1.16.221) was the target for this experiment.

The user reported a puzzling symptom in [msg 5760]: "Inside the vm nvidia-smi doesn't show anything." The assistant's first response in [msg 5761] was to verify the basics: lspci showed all eight NVIDIA devices (device ID 2bb5 for the RTX PRO 6000 Blackwell), and /dev/nvidia* devices existed, confirming the NVIDIA driver module was loaded. But something was still wrong — the user had already said nvidia-smi showed nothing.

The Diagnostic Pivot

Message 5762 is where the assistant stops taking surface-level indicators at face value and goes directly to the source of truth: nvidia-smi itself. The command is carefully constructed:

ssh theuser@10.1.16.221 'nvidia-smi 2>&1; echo "exit code: $?"'

The 2>&1 redirect is important — it captures both standard output and standard error, ensuring no diagnostic message is lost. The echo "exit code: $?" appends the shell exit code, which is a critical piece of data that many users overlook. The exit code from nvidia-smi is a structured signal: exit code 6 specifically means "NVIDIA driver is loaded but no devices were found." This is fundamentally different from exit code 2 (driver not loaded) or exit code 0 (success).

The output — "No devices were found" with exit code 6 — tells a precise story. The NVIDIA kernel module (nvidia.ko) is loaded and operational at the module level (hence /dev/nvidia* exists). But when the driver attempts to initialize each GPU — calling RmInitAdapter in NVIDIA's internal parlance — it fails. The driver sees the PCI devices but cannot communicate with them at the hardware level.

Why This Matters: The Blackwell Open Kernel Module Requirement

This specific error pattern is the fingerprint of a known incompatibility. NVIDIA's RTX PRO 6000 Blackwell GPUs (architecture SM120) require the open kernel module (nvidia-open) rather than the proprietary nvidia.ko module. The proprietary driver, which has been NVIDIA's standard for decades, does not contain the necessary initialization paths for the Blackwell architecture's new hardware interfaces.

The follow-up messages ([msg 5763] through [msg 5766]) confirm this diagnosis. Checking /proc/driver/nvidia/version shows the proprietary NVRM version: NVIDIA UNIX x86_64 Kernel Module 590.48.01. The kernel log reveals the smoking gun:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

The VM had nvidia-dkms-590 (proprietary) installed. The fix was straightforward: install nvidia-dkms-590-open instead, which provides the open kernel module at the same driver version (590.48.01), keeping all userspace libraries compatible.

The Thinking Process: Systematic Elimination

What makes message 5762 exemplary is the systematic reasoning it represents. The assistant is working through a diagnostic tree:

1. Are the GPUs visible to the VM? → Yes, lspci shows 8 devices. (Solved in [msg 5761])
2. Is the NVIDIA driver loaded? → Yes, /dev/nvidia* exists. (Solved in [msg 5761])
3. Does nvidia-smi agree? → No, "No devices were found" with exit code 6. (Solved in [msg 5762])
4. What specific error does the driver log? → RmInitAdapter failed, requiring open kernel modules. (Solved in [msg 5764]) Each step eliminates a class of possible causes. Step 1 eliminates PCI passthrough configuration issues. Step 2 eliminates missing or failed driver loading. Step 3 narrows the problem to the driver-device initialization handshake. Step 4 identifies the exact incompatibility. This is a textbook example of the "narrowing cone" debugging methodology: start with broad checks (device visibility, driver presence), then progressively focus on more specific diagnostics until the root cause is isolated.

Input Knowledge Required

To fully appreciate this message, a reader needs to understand several pieces of context:

• GPU passthrough mechanics: The GPUs are passed from the Proxmox host to the VM via VFIO. On the host side, they're bound to vfio-pci; inside the VM, they appear as normal PCI devices that the NVIDIA driver can claim.
• The meaning of /dev/nvidia*: These device nodes (typically nvidia0, nvidia1, ..., nvidiactl, nvidia-uvm, nvidia-modeset) are created by the NVIDIA kernel module when it loads. Their existence confirms the module loaded successfully.
• nvidia-smi exit codes: Exit code 6 specifically means "driver loaded, no devices found" — a different failure mode from "driver not loaded" (exit code 2) or "driver error" (exit code 8).
• Blackwell architecture: NVIDIA's SM120 architecture (RTX PRO 6000 Blackwell) requires the open kernel module, a relatively recent change in NVIDIA's driver strategy. This is not widely known and catches many users off guard.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. A confirmed symptom signature: "No devices were found" + exit code 6 + /dev/nvidia* existing = driver loads but cannot initialize GPUs. This signature is reusable for diagnosing similar issues on other systems.
2. A narrowed problem space: The issue is not PCI passthrough, not missing drivers, not incorrect VM configuration — it's a driver-device compatibility issue at the kernel module level.
3. A clear next step: Check the kernel log (dmesg) for RmInitAdapter errors, which will point to the open kernel module requirement.

The Broader Lesson

Message 5762 illustrates a principle that applies far beyond GPU passthrough debugging: surface-level indicators can be misleading. The GPUs appeared in lspci, the driver appeared loaded via /dev/nvidia*, yet the system was non-functional. The assistant resisted the temptation to declare victory based on partial evidence and instead went to the authoritative diagnostic tool — nvidia-smi with exit code capture — to get the complete picture.

In production AI infrastructure, where hardware and software stacks are complex and rapidly evolving, this kind of disciplined diagnostic approach is invaluable. A less careful operator might have seen lspci showing eight GPUs and /dev/nvidia* existing and concluded the passthrough was working, only to discover later that inference was silently failing or using CPU fallback paths. The assistant's thoroughness in message 5762 saved what could have been hours of confused debugging.

The message also demonstrates the value of understanding tool semantics. Knowing that nvidia-smi exit code 6 means "no devices found" rather than "driver error" or "permission denied" is the kind of deep platform knowledge that separates efficient debugging from aimless tinkering. It's a small detail, but it's the detail that broke the case wide open.