The Moment of Discovery: GPU Passthrough That Already Worked

In the middle of a high-stakes machine learning infrastructure session — where the assistant had just finished hardening a production deployment of the Kimi-K2.5 INT4 model with hierarchical KV cache and a systemd service — the user suddenly pivoted to a tangent experiment. They wanted to pass eight Blackwell RTX PRO 6000 GPUs through to a virtual machine running on a Proxmox hypervisor. The question they posed seemed straightforward but carried an underlying anxiety about complexity: "Because it's now a VM and not a container do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?" ([msg 5752]). This question reveals an assumption that GPU passthrough on a host that also uses the NVIDIA driver for LXC containers requires invasive system changes — driver unloading, module blacklisting, and a full reboot cycle. The assistant's response in message 5759 delivers a surprising and elegant resolution: none of that is necessary, because the infrastructure is already set up correctly.

The User's Question and Hidden Assumptions

The user's framing of the problem is telling. They asked about unloading the NVIDIA driver, blacklisting modules, and rebooting — all classic steps from the early days of GPU passthrough when VFIO configuration was manual and fragile. This suggests the user either had prior experience with GPU passthrough on older systems, or was extrapolating from common documentation that assumes a host where the NVIDIA driver actively owns the GPU devices. The implicit assumption was that the Proxmox host had the GPUs bound to the nvidia driver (since the LXC container used them for ML workloads), and that switching to vfio-pci for the VM would require tearing down that binding at the kernel level.

The assistant, rather than immediately answering yes or no, did something more valuable: it investigated the actual state of the host. Over several rounds ([msg 5753] through [msg 5758]), it probed the Proxmox node at 10.1.2.6, checking loaded kernel modules, PCI device bindings, kernel command-line parameters, VM configurations, and PCI mapping files. This investigative process itself is a model of how to approach infrastructure questions — never assume, always verify.

The Investigation Trail

The assistant's investigation revealed a series of important facts. First, the NVIDIA kernel modules (nvidia_drm, nvidia_uvm, nvidia_modeset, nvidia) were loaded on the host, but critically, nvidia_drm showed a refcount of 0 — meaning no device was actually using it ([msg 5754]). The lspci output showed eight NVIDIA 3D controllers (device ID 2bb5, the Blackwell RTX PRO 6000) but did not show which driver was bound to them at that point. The kernel command line included iommu=pt, confirming IOMMU was already enabled in passthrough mode — a prerequisite for VFIO that was already satisfied.

Then the assistant examined the VM configurations. VM 131 (ml-pipelines) was running with 204 GB of memory and had eight hostpci entries all using mapping=pro6000 ([msg 5756]). The PCI mapping file (/etc/pve/mapping/pci.cfg) contained the actual device-to-group mappings ([msg 5757]). The crucial check came when the assistant queried lspci -ks for individual GPU slots: the kernel driver in use was vfio-pci ([msg 5758]). This was the smoking gun — the GPUs were already bound to the VFIO driver, not the NVIDIA driver. VM 131 was already running with all eight GPUs passed through.

The Revelation in Message 5759

Message 5759 is the culmination of this investigation, and it delivers the key insight with remarkable concision:

All GPUs are already on vfio-pci — VM 131 has them. So it's already working. The nvidia modules are loaded but not bound to any devices (refcount 0 on nvidia_drm).

This single sentence reframes the entire problem. The user was worried about a complex transition that had already happened. The NVIDIA modules were loaded but idle — they existed in memory but had no devices attached to them. This is a perfectly normal state on a Proxmox host that uses VFIO passthrough: the kernel modules are available for when devices are returned to the host, but while the VM owns the GPUs, they sit unused.

The assistant then lays out the switching workflow in two simple bullet points:

To use GPUs in VM (current state): Just start the VM. Proxmox auto-unbinds nvidia, binds vfio-pci.
To use GPUs in LXC: Stop the VM → GPUs rebind to nvidia → start CT 129. The critical phrase is "auto-unbinds" — Proxmox handles the driver rebinding automatically as part of its VM lifecycle management. When a VM with hostpci mappings starts, the hypervisor unbinds the PCI devices from their current kernel driver and binds them to vfio-pci. When the VM stops, the reverse happens: devices are released from VFIO and rebind to their original kernel driver (in this case, nvidia). This is transparent to the user and requires no manual intervention.

The Switching Workflow

The workflow the assistant describes is elegantly simple. To switch from VM mode to LXC mode, the user only needs to stop the VM — the GPUs automatically rebind to the NVIDIA driver — and then start the LXC container. To switch back, start the VM. No blacklisting, no rebooting, no editing of kernel boot parameters, no modprobe -r nvidia commands. The only pain point the assistant identifies is the ~10 minute model load time when switching back to the LXC container, since the ML model weights (547 GB for Kimi-K2.5) must be reloaded into GPU memory from scratch. This is a consequence of the GPU memory being reset during the PCIe hot-unbind/rebind cycle, not something that can be avoided.

The Verification Attempt and Its Significance

The message ends with a verification attempt — the assistant tries to SSH into the VM guest at 10.1.16.221 to confirm the GPUs are visible inside the VM:

[bash] ssh root@10.1.16.221 'nvidia-smi -L 2>/dev/null || echo "nvidia-smi not available"; lspci | grep -i nvidia | head -3' Please login as the user "theuser" rather than the user "root".

This failed, but the failure itself is informative. The VM guest has SSH configured to reject root login — a security best practice — and expects the user theuser. This is a minor operational detail but it underscores an important point: even when the host-side GPU passthrough is working perfectly, the guest-side configuration (drivers, SSH access, user permissions) still needs to be validated separately. The assistant's attempt to verify was cut short, but the host-side evidence was already conclusive: lspci -ks on the Proxmox host showed vfio-pci as the kernel driver for all GPU slots, which means the devices were successfully passed through.

Knowledge Required and Created

To fully understand message 5759, the reader needs knowledge of several infrastructure concepts: Proxmox VE hypervisor architecture, VFIO (Virtual Function I/O) driver framework, PCI passthrough mechanics, the difference between kernel module loading and device binding, and the NVIDIA driver stack (nvidia, nvidia_drm, nvidia_modeset, nvidia_uvm). The message also assumes familiarity with the broader context — that the user has been running ML workloads in an LXC container on this host, that there are eight Blackwell GPUs installed, and that a VM (ID 131) named ml-pipelines exists with GPU mappings configured.

The knowledge created by this message is substantial. It documents a working GPU passthrough configuration on Proxmox with Blackwell GPUs, confirms that the NVIDIA driver modules can coexist with VFIO without conflict (as long as no devices are actively bound to both), and establishes a repeatable switching workflow between VM and LXC container usage. This is valuable operational knowledge for anyone managing GPU infrastructure on Proxmox, especially with the newer Blackwell architecture which has specific driver requirements (the open kernel module nvidia-dkms-590-open was needed earlier in the session for SM120 support).

Broader Implications

This message represents a moment of clarity that every infrastructure engineer experiences: the moment when a complex-seeming problem dissolves into a simple verification. The user's instinct was to ask "how do I make this work" — the assistant's response was to check "does it already work?" The answer was yes, and the only remaining question was about the switching workflow, which turned out to be trivially simple.

The deeper lesson is about the value of investigation over assumption. The user assumed a difficult multi-step process involving kernel configuration changes and reboots. The assistant assumed nothing and checked the actual state of the system. This investigative approach — checking loaded modules, PCI bindings, VM configurations, and kernel parameters — is a transferable skill that applies to any infrastructure debugging scenario.

The message also highlights a key architectural insight about Proxmox: the hypervisor abstracts away the complexity of PCI device rebinding. Users don't need to manually unbind devices from nvidia and bind them to vfio-pci — Proxmox handles this automatically through the hostpci device mapping mechanism. This is a significant quality-of-life improvement over raw KVM or Xen setups where such operations must be scripted manually.

Conclusion

Message 5759 is a small but perfectly formed piece of infrastructure reasoning. It delivers a surprising positive result — the GPUs are already passed through — with a clear explanation of why the user's concerns were unfounded. It establishes a simple two-step switching workflow, identifies the only real pain point (model reload time), and attempts (though fails) to verify the guest-side state. The message embodies a philosophy of infrastructure management: investigate before acting, verify before assuming, and document the workflow for future reference. For anyone managing GPU passthrough on Proxmox — especially with modern Blackwell GPUs and mixed VM/container workloads — this message provides a template for thinking about the problem and a concrete example of a working configuration.