A Diagnostic Pivot: Troubleshooting GPU Passthrough After a Proxmox Host Reboot

In the middle of an intense session deploying large language models across a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message appears that is deceptively simple. It is a bash command — three commands chained together over SSH — and its output. But beneath this brevity lies a critical diagnostic pivot, a moment where the assistant must re-establish the state of a complex virtualized infrastructure after an unexpected host reboot. Message [msg 5774] is not merely a status check; it is a carefully crafted probe designed to answer a single urgent question: after the Proxmox host came back up, did the GPU passthrough configuration survive?

The Message Itself

The assistant executes the following over SSH to the Proxmox host at 10.1.2.6:

ssh root@10.1.2.6 'qm list; echo "---"; pct list; echo "---"; lspci -ks 01:00.0 | head -5'

And receives back:

      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       128 llm-one              stopped    450000          1203.50 0         
       131 ml-pipelines         running    444800            53.50 7477      
---
VMID       Status     Lock         Name                
129        stopped                 llm-two             
---
01:00.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
	Subsystem: NVIDIA Corporation Device 204e
	Kernel driver in use: vfio-pci
	Kernel ...

Three commands, three pieces of data, one coherent picture. But to understand why these specific commands were chosen and what the assistant is reasoning through, we must step back and trace the full arc of events that led to this moment.

The Arc Leading to This Message

The story begins with the user running the Kimi-K2.5 INT4 model in a Proxmox LXC container, using all eight Blackwell GPUs for tensor-parallel inference with SGLang. After extensive optimization work — upgrading CUDA to version 13, patching SGLang for SM120 support, enabling FlashInfer allreduce fusion, and tuning EAGLE-3 speculative decoding — the production deployment was hardened into a systemd service with hierarchical KV cache ([msg 5742]).

Then the user pivoted. A "tangent experiment" ([msg 5752]) required passing all eight GPUs to a VM rather than the LXC container. The assistant investigated the Proxmox host and discovered that VM 131 (ml-pipelines) was already configured with all eight GPUs via the pro6000 PCI mapping ([msg 5756]). The GPUs were already bound to vfio-pci on the host — Proxmox had handled the unbinding from the NVIDIA driver automatically when the VM started ([msg 5758]).

But inside the VM guest, nvidia-smi reported "No devices were found" ([msg 5762]). The assistant dug into the kernel logs and found the critical error: the VM had the proprietary NVIDIA kernel module (nvidia-dkms-590) installed, but Blackwell GPUs (SM120 architecture) require the open kernel module (nvidia-dkms-590-open) ([msg 5764]). This is a well-known requirement for NVIDIA's latest GPU architecture — the proprietary driver branch does not support SM120.

The fix was straightforward: install nvidia-dkms-590-open, which replaces the proprietary module via DKMS while keeping the same userspace libraries at version 590.48.01 ([msg 5766]). The assistant executed the installation ([msg 5768]), rebooted the VM ([msg 5769]), and after a brief wait, confirmed that all eight GPUs were visible inside the guest ([msg 5771]). The assistant declared success: "All 8 GPUs visible. Open kernel modules working perfectly." ([msg 5772])

Then the user dropped a bombshell: "Had to reboot host, gpus not visible again" ([msg 5773]).

This is where message [msg 5774] enters.

The Reasoning Behind the Diagnostic

The user's report is ambiguous. "GPUs not visible again" could mean several things:

  1. The GPUs are not visible to the VM guest — the VM might have lost its PCI passthrough configuration after the host reboot, or the VFIO driver might not have re-bound to the devices.
  2. The GPUs are not visible on the host at all — perhaps the host reboot caused a driver loading failure, or the NVIDIA kernel modules failed to initialize.
  3. The VM itself did not survive the host reboot — if the QEMU process was killed, the VM would need to be restarted.
  4. The LXC container accidentally took the GPUs back — if the container started during boot, it might have claimed the NVIDIA driver before VFIO could bind. The assistant's response is immediate and systematic. Rather than asking clarifying questions, it runs a three-pronged diagnostic that simultaneously checks all four hypotheses: qm list — Lists all QEMU VMs. This checks whether VM 131 (ml-pipelines) survived the host reboot. The output shows it is running with PID 7477, meaning Proxmox either preserved the QEMU process across reboot (unlikely) or automatically restarted it (likely via HA or autostart configuration). This eliminates hypothesis 3. pct list — Lists all LXC containers. This checks whether CT 129 (llm-two) — the container that previously owned the GPUs — accidentally started and claimed them. The output shows it is stopped, eliminating hypothesis 4. lspci -ks 01:00.0 | head -5 — Shows the kernel driver bound to the first GPU. The critical line is "Kernel driver in use: vfio-pci". This confirms that the GPU is still correctly bound to the VFIO driver for VM passthrough, not to the NVIDIA driver. This eliminates hypothesis 2 and suggests that hypothesis 1 is the most likely explanation. The assistant's thinking is visible in the structure of this diagnostic. It proceeds from the coarsest check (are the VMs and containers in the right state?) to the finest (is the PCI driver binding correct?). Each command eliminates a class of failure, narrowing the problem space until only the most specific hypothesis remains.

What the Assistant Assumed

Several assumptions underpin this diagnostic:

That lspci -ks 01:00.0 is representative. The assistant checks only one GPU's PCI binding. If different GPUs ended up in different driver states — some on vfio-pci, some on nvidia — this single check would miss it. This is a reasonable heuristic (GPUs of the same model in the same system typically share the same binding fate), but it is not guaranteed.

That the host reboot was clean. The assistant assumes the Proxmox host rebooted normally and that all services (including QEMU autostart) functioned correctly. If the reboot was forced or interrupted, the system state might be inconsistent.

That the VM's internal state is the problem. By confirming that the host-level PCI binding is correct, the assistant implicitly concludes that the issue lies inside the VM guest. This is a logical inference — if the GPU is bound to vfio-pci on the host and the VM is running, the VM should be able to see the device. But this assumption depends on the VM's internal driver stack being correctly configured, which was only fixed in the previous boot cycle.

That the user wants to restore the VM setup. The assistant assumes the user's goal is to get the GPUs working in the VM again, rather than switching back to the LXC container. Given the user initiated the tangent experiment and only reported the problem, this is a safe assumption.

Input Knowledge Required

To fully understand this message, the reader needs:

Knowledge of Proxmox VE virtualization. The distinction between qm (QEMU/KVM VMs) and pct (LXC containers) is fundamental. Proxmox runs both types of workloads, and GPU passthrough works differently for each. VMs use VFIO for direct device assignment; containers share the host kernel and can use the NVIDIA driver directly.

Knowledge of PCI passthrough mechanics. The lspci -ks command shows the kernel driver bound to a PCI device. vfio-pci is the driver used for VM passthrough — it presents the device to the guest OS rather than driving it on the host. If the NVIDIA driver were bound instead, the device would be consumed by the host and unavailable to the VM.

Knowledge of Blackwell GPU requirements. The NVIDIA RTX PRO 6000 Blackwell (SM120 architecture) requires the open kernel module (nvidia-open or nvidia-dkms-590-open). The proprietary nvidia.ko module does not support these GPUs. This was the root cause of the earlier failure inside the VM.

Knowledge of the conversation history. The user's previous message ("Had to reboot host, gpus not visible again") is the immediate trigger for this diagnostic. Without that context, the message appears to be a routine status check.

Output Knowledge Created

This message produces three critical pieces of information:

  1. VM 131 is running. The ml-pipelines VM survived the host reboot and is active with PID 7477. This means the GPU passthrough configuration is still in place from Proxmox's perspective.
  2. CT 129 is stopped. The LXC container that previously used the GPUs did not claim them during boot. The path is clear for the VM to access the GPUs.
  3. GPU 01:00.0 is bound to vfio-pci. The host-level PCI binding is correct. The NVIDIA driver did not reclaim the device during boot. Together, these three facts narrow the problem to the VM guest itself. The host infrastructure is correctly configured; something inside the VM is preventing the GPUs from being visible. This is a significant diagnostic achievement — it saves the user from investigating the wrong layer of the stack.

A Subtle Mistake

The assistant makes one subtle error: it checks only GPU at PCI address 01:00.0. While this is likely representative, a thorough diagnostic would check all eight GPUs or at least sample a few across different PCI domains (e.g., 01:00.0, 11:00.0, 61:00.0, 71:00.0 based on the earlier lspci output from [msg 5754]). If even one GPU ended up on the NVIDIA driver, it could cause issues in a tensor-parallel setup where all GPUs must be available.

However, this is a minor omission. The primary value of this diagnostic is its speed and focus — the assistant correctly identifies that the host layer is intact and the problem is inside the VM, which is precisely where the previous fix was applied.

The Deeper Pattern

This message exemplifies a broader pattern in the conversation: the assistant consistently responds to failures with systematic, multi-axis diagnostics rather than asking the user for more information. When the VM initially failed to see GPUs ([msg 5760]), the assistant checked PCI visibility, device files, kernel logs, driver version, and module info in rapid succession ([msg 5761], [msg 5762], [msg 5763]). When the host reboot disrupted the setup, the assistant again ran a multi-point check.

This approach is particularly valuable in infrastructure debugging, where the failure could be at any layer — hypervisor, host kernel, VM kernel, userspace driver, or application. By checking multiple layers simultaneously, the assistant can quickly isolate the problem domain and avoid wasting time on red herrings.

Conclusion

Message [msg 5774] is a masterclass in targeted diagnostics. In three carefully chosen commands, the assistant establishes the state of the virtualization layer, the container layer, and the PCI device layer, conclusively ruling out host-level causes for the GPU visibility failure. The message demonstrates that effective troubleshooting is not about running every possible check, but about running the right checks — those that maximally reduce uncertainty about where the problem lies.

The assistant's thinking is visible in the structure of the diagnostic: it proceeds from the most likely failure modes (VM dead, container stole GPUs, driver binding lost) to the least likely, eliminating each in turn. By the time the output is printed, the assistant has already formed a working hypothesis: the host is fine, the VM is running, the PCI binding is correct — the problem must be inside the VM guest, likely related to the open kernel module installation that was only just completed before the unexpected host reboot intervened.