The Tangent Experiment: GPU Passthrough Between Containers and VMs on Proxmox

The Message

"Doing a tangent experiment on the proxmox node - 10.1.2.6, 10.1.16.221 guest vm; Because it's now a VM and not a container do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?"

This single message, sent at a pivotal moment in a long optimization session, captures a fundamental infrastructure question that arises when managing GPU-accelerated workloads across virtualization boundaries. The user, deep in the process of deploying and tuning a massive 547 GB Kimi-K2.5 INT4 model across 8 RTX PRO 6000 Blackwell GPUs, pauses to ask about switching their execution environment from an LXC container to a KVM virtual machine. The question is deceptively simple, but it exposes layers of complexity about NVIDIA driver architecture, Proxmox PCI passthrough mechanics, and the operational cost of infrastructure flexibility.

Why This Message Was Written

To understand the motivation behind this message, we must look at what preceded it. The user had just successfully deployed the Kimi-K2.5 model with hierarchical KV cache (hicache) on an LXC container (CT 129) running on a Proxmox host at 10.1.2.6. The deployment was working well — the model was generating correct responses with reasoning parsing and tool calls, and the hicache was allocating 44.77 GB of pinned host memory per GPU, totaling 358 GB of L2 prefix cache across all 8 GPUs.

But the user wanted to do a "tangent experiment" — running the GPUs inside a VM instead of the container. This is a classic infrastructure exploration: the LXC container shares the host kernel and drivers, offering low overhead but less isolation. A KVM VM offers stronger isolation, the ability to run different kernel versions or driver stacks, and a more reproducible environment. The user had a VM at 10.1.16.221 already provisioned and wanted to try passing all 8 GPUs to it.

The core tension in the question is between two competing operational models. In the LXC container model, the GPUs are bound to the host's NVIDIA driver, and the container accesses them through the host's driver stack. In the KVM passthrough model, the GPUs are detached from the host driver and bound to the vfio-pci driver, which presents them to the VM as if they were physically attached. The user's instinct — and it's a reasonable one — was that this transition would require manually unloading the NVIDIA driver, blacklisting it to prevent re-binding, and rebooting the host.

Assumptions Embedded in the Question

The user makes several assumptions that reveal their mental model of the system. First, they assume that switching between container and VM usage requires manual intervention on the host — unloading the NVIDIA driver module, adding it to a blacklist, and rebooting. This is the standard approach for setting up GPU passthrough from scratch on a bare Proxmox installation, and it's a common pain point documented in countless forum posts and tutorials.

Second, the user assumes this process would be "painful" — that each switch would require significant downtime and manual reconfiguration. They're asking for a way to make it "reasonably painless," which tells us they anticipate doing this switch more than once. They want a workflow, not a one-time fix.

Third, the user implicitly assumes that the VM guest would be able to use the GPUs once passthrough was correctly configured on the host side. This assumption turns out to be the most interesting one, because it's partially wrong — but not for the reasons the user feared.

There's also an unstated assumption that the LXC container (CT 129) would need to stop while the VM runs. The user had already confirmed in a previous interaction that they wanted "all 8 GPUs to VM" and that the LXC container would stop. This is a necessary constraint — you can't share physical GPUs between a host-bound container and a VM using passthrough simultaneously.

What Actually Happened: The Discovery

The assistant's investigation revealed a much more favorable reality than the user expected. When the assistant checked the Proxmox host's configuration, it found that VM 131 (ml-pipelines) was already running with all 8 GPUs passed through. The Proxmox PCI mapping system, configured in /etc/pve/mapping/pci.cfg with a mapping called pro6000, was already set up to handle the GPUs. The VM configuration (/etc/pve/qemu-server/131.conf) had eight hostpci entries, one per GPU, all using this mapping.

The key insight is that Proxmox handles the driver switch automatically. When a VM with hostpci entries starts, Proxmox unbinds the PCI devices from the NVIDIA driver and binds them to vfio-pci. When the VM stops, the devices go back to the NVIDIA driver. No blacklisting, no rebooting, no manual module unloading. The switching mechanism was already in place and automated.

The assistant confirmed this by checking lspci -ks 01:00.0 on the host, which showed "Kernel driver in use: vfio-pci" — the GPUs were already detached from the NVIDIA driver and attached to the passthrough driver. The NVIDIA kernel modules were still loaded in memory (as shown by lsmod), but they had zero references to any devices. They were idle, consuming memory but not bound to hardware.

The Real Problem: VM Guest Driver Mismatch

But when the assistant checked the VM guest at 10.1.16.221, a different problem emerged. The GPUs were visible in lspci — all 8 devices showed up — and /dev/nvidia* devices existed. But nvidia-smi returned "No devices were found" with exit code 6. The driver was loaded but couldn't initialize the GPUs.

Checking the kernel messages (dmesg) revealed the culprit:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

The VM had the proprietary NVIDIA kernel module (nvidia-dkms-590) installed. But Blackwell GPUs (SM120 architecture, like the RTX PRO 6000) require the open kernel modules (nvidia-dkms-590-open). The proprietary driver doesn't support these GPUs at all — it can't initialize them. This is a relatively recent change in NVIDIA's driver strategy: starting with the Blackwell architecture, the open kernel module is mandatory, and the proprietary module simply refuses to work.

The fix was straightforward: install nvidia-dkms-590-open in the VM, which replaces the proprietary module with the open one via DKMS rebuild, then reboot. The userspace libraries (libnvidia-gl, libnvidia-encode, etc.) stay the same at version 590.48.01 — only the kernel module changes. The assistant confirmed the package was available in the Ubuntu 24.04 repositories and ran the installation, followed by a reboot.

Input Knowledge Required

To fully understand this message and its resolution, several pieces of domain knowledge are necessary. One must understand the Proxmox virtualization stack — the distinction between LXC containers (which share the host kernel) and KVM VMs (which run their own kernel), and the PCI passthrough mechanism using vfio-pci. One must understand the NVIDIA driver architecture, particularly the recent split between proprietary and open kernel modules, and the requirement that Blackwell GPUs use the open module. One must understand DKMS (Dynamic Kernel Module Support), which rebuilds kernel modules when the kernel or driver version changes. And one must understand the PCI device mapping system in Proxmox, which uses IOMMU groups and PCI paths to assign devices to VMs.

Output Knowledge Created

This exchange produced several important pieces of knowledge. First, it documented that the Proxmox host already had a working GPU passthrough configuration for all 8 GPUs using the pro6000 mapping. Second, it established that the switching mechanism is automatic — starting a VM with hostpci entries unbinds the NVIDIA driver and binds vfio-pci, and stopping the VM reverses the process. Third, it identified that the VM guest needed the open NVIDIA kernel module (nvidia-dkms-590-open) instead of the proprietary one for Blackwell GPU support. Fourth, it created a documented workflow for switching between LXC container and VM usage: stop the container, start the VM (automatic driver switch), or stop the VM, start the container (automatic driver switch back). The only real pain point is the ~10 minute model load time when switching back to the LXC container.

The Thinking Process Visible in the Message

The user's thinking process reveals a pragmatic, systems-oriented mindset. They're not asking "can this be done?" — they're asking "how can this be done without pain?" They've already identified the likely steps (unload driver, blacklist, reboot) and are seeking validation and optimization. The phrase "tangent experiment" is telling — they're aware this is a side quest from the main optimization work, and they want to minimize the time spent on infrastructure switching. The inclusion of specific IP addresses (10.1.2.6 for the Proxmox host, 10.1.16.221 for the VM guest) shows they're thinking concretely about the two machines involved and the network path between them.

The question also reveals a healthy skepticism about "it just works" claims. The user has clearly been burned before by infrastructure transitions that seemed simple on paper but required hours of debugging. They're proactively asking about the pain points before starting the experiment, which is a hallmark of experienced systems engineers.

Conclusion

This message, brief as it is, captures a moment of infrastructure reflection in the middle of a deep optimization session. The user's instinct that GPU passthrough switching would be painful was reasonable based on common experience, but the actual infrastructure was more sophisticated than expected — Proxmox's PCI mapping system handled the driver switch automatically. The real challenge was elsewhere: in the VM guest's driver stack, where the proprietary NVIDIA module couldn't initialize the Blackwell GPUs. The resolution — swapping to the open kernel module — was a single package installation away. The message stands as a reminder that infrastructure assumptions should always be verified, and that the pain points we anticipate are often not the ones that actually materialize.