The Discovery That Proxmox Already Solved It: A Case Study in GPU Passthrough Automation

Introduction

In the midst of a complex machine learning infrastructure deployment spanning multiple servers, GPU configurations, and virtualization layers, a seemingly simple question from the user triggered a revealing investigation. The user asked: "Do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?" The assistant's response, message 5758 in the conversation, delivers a surprising and elegant answer: the problem was already solved, and the solution was running right under their noses.

This article examines that single message — its reasoning, its investigative process, the assumptions it challenged, and the knowledge it produced. What makes this message particularly interesting is that it represents a moment of discovery rather than construction: the assistant did not build anything new, but instead uncovered an existing configuration that rendered the user's concerns moot.

Context: The Broader Infrastructure Challenge

To understand the significance of message 5758, one must appreciate the environment in which it was written. The preceding conversation (segments 33–38 of the session) documents an intensive effort to deploy and optimize large language models across a multi-GPU cluster. The team had been working with Kimi-K2.5 INT4 and Qwen3.5-397B-A17B-NVFP4 models, managing 8 NVIDIA RTX PRO 6000 Blackwell GPUs, tuning speculative decoding performance, and hardening production deployments with systemd services and hierarchical KV caches.

The infrastructure involved a Proxmox hypervisor (host 10.1.2.6) running multiple virtual machines and LXC containers. One container (CT 129, llm-two) had been the primary workhorse for the ML workloads, but the user wanted to experiment with passing all 8 GPUs to a guest VM (10.1.16.221) instead. This raised a practical concern: when a host's NVIDIA driver owns the GPUs and a VM needs them via VFIO passthrough, the driver must be unloaded and the devices rebinded — a process that typically requires manual intervention, module blacklisting, and often a reboot.

The Message: Structure and Content

Message 5758 is deceptively brief for the amount of insight it contains. The assistant opens with a confident assessment: "This is already well set up." It then reveals that VM 131 (ml-pipelines) is already running with all 8 GPUs passed through using the pro6000 mapping, and the vfio-pci driver has successfully claimed the devices.

The message contains two key components. First, a conceptual explanation of how Proxmox handles the GPU ownership transition:

Proxmox handles the driver switch automatically via the PCI mapping. When a VM with hostpci entries starts, Proxmox unbinds the devices from nvidia and binds them to vfio-pci. When the VM stops, they go back.

Second, a verification step — the assistant runs an lspci command on the Proxmox host to confirm that the GPUs are indeed bound to vfio-pci rather than the NVIDIA driver. The output confirms this for two example devices (slots 01:00.0 and e1:00.0), showing Kernel driver in use: vfio-pci.

The Reasoning Process: From Question to Discovery

The assistant's thinking process in this message is a model of systematic investigation. Let's trace the logic.

Step 1: Challenge the assumption. The user assumed that GPU passthrough would require manual driver management — unloading the NVIDIA module, blacklisting it, and rebooting. The assistant's first move is to check whether this assumption holds by examining the actual state of the Proxmox host.

Step 2: Leverage prior reconnaissance. In the preceding messages (5754–5757), the assistant had already gathered critical information: IOMMU was enabled (iommu=pt in kernel cmdline), VFIO had been used before (dmesg showed VFIO activity), and VM 131 was running with hostpci0 through hostpci7 all mapped to pro6000. The assistant had also discovered the PCI mapping configuration in /etc/pve/mapping/pci.cfg, which defined the pro6000 mapping for the Blackwell GPUs.

Step 3: Connect the dots. The crucial insight was that VM 131 (ml-pipelines) was already running and already had the GPUs. If the NVIDIA driver were still holding the devices, the VM couldn't have started successfully. The fact that the VM was running meant the driver switch had already occurred — automatically, at VM boot time.

Step 4: Verify empirically. Rather than stopping at the logical deduction, the assistant runs lspci -ks to check the kernel driver in use for specific GPU slots. This confirms the devices are on vfio-pci, providing concrete evidence for the conclusion.

Assumptions Made and Corrected

This message is particularly interesting for what it reveals about assumptions — both the user's and the assistant's.

The user's primary assumption was that GPU passthrough is a manual, fragile process requiring driver unloading, blacklisting, and rebooting. This is a reasonable assumption based on common experience: VFIO passthrough traditionally requires the host to release the device, which means the NVIDIA driver must not claim it at boot. Many guides recommend adding the GPU device IDs to the VFIO driver's allowlist and blacklisting the NVIDIA driver to prevent it from grabbing the devices first.

The assistant's initial assumption (visible in message 5753) was more nuanced but still incomplete. When the user first asked about the switch, the assistant responded with a question: "For GPU passthrough to a VM you need the host to not bind the GPUs to the nvidia driver — they need to be on vfio-pci instead." This suggests the assistant initially believed the NVIDIA driver was currently holding the GPUs and would need to be released. The assistant asked for clarification about how many GPUs were being passed through and whether the LXC container needed to keep running — a reasonable question, but one that assumed the current state was the NVIDIA driver binding.

The discovery that VM 131 was already running with all 8 GPUs corrected both assumptions. Proxmox's PCI mapping system handles the rebinding transparently: when the VM starts, Proxmox unbinds the devices from whatever driver currently holds them (even NVIDIA) and rebinds them to vfio-pci. When the VM stops, the devices return to the NVIDIA driver. No blacklisting, no reboot, no manual module unloading.

Input Knowledge Required

To fully understand this message, several pieces of prior knowledge are necessary:

  1. Proxmox virtualization architecture: Understanding that Proxmox uses QEMU/KVM for VMs and LXC for containers, and that PCI passthrough is configured through hostpci entries in VM configuration files.
  2. VFIO and PCI passthrough mechanics: Knowing that vfio-pci is a driver that allows userspace access to PCI devices, enabling them to be passed through to VMs, and that the kernel driver in use determines which subsystem controls the device.
  3. NVIDIA driver stack: Recognizing the nvidia, nvidia_modeset, nvidia_drm, and nvidia_uvm kernel modules and understanding that they must be unloaded or not claim a device for VFIO passthrough to work.
  4. Proxmox PCI mapping system: The pro6000 mapping referenced in the VM config and the PCI mapping file at /etc/pve/mapping/pci.cfg are Proxmox-specific abstractions that simplify GPU assignment across multiple hosts.
  5. The Blackwell GPU hardware: The device ID 10de:2bb5 corresponds to the NVIDIA RTX PRO 6000 Blackwell GPU, and the vfio-pci driver in use confirms passthrough is active.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Proxmox automates GPU driver switching: The key takeaway is that Proxmox's PCI mapping system handles the NVIDIA-to-VFIO driver transition automatically at VM start and stop. This is a significant operational insight — it means no manual intervention, no blacklisting, no rebooting is needed for the switch.
  2. The existing VM 131 is a working reference: The ml-pipelines VM serves as proof that the configuration works. Its config file (examined in message 5756) shows exactly how to set up the passthrough: eight hostpci entries all mapped to pro6000, with OVMF BIOS and Q35 machine type.
  3. The LXC container is already stopped: The pct list output in message 5755 showed CT 129 (llm-two) was already stopped, meaning no conflict exists between the container and the VM for GPU access.
  4. The NVIDIA modules remain loaded but harmless: The lsmod output in message 5754 showed NVIDIA modules were still loaded on the host, but this doesn't prevent VFIO passthrough — the modules just don't claim the specific devices that VFIO has taken.

The Verification Step: Why It Matters

The lspci -ks commands at the end of the message are not mere decoration. They serve a critical function: empirical validation. The assistant could have simply stated the conclusion based on logical deduction, but instead chose to verify with concrete data. This is a hallmark of good systems engineering — never trust that a configuration is working without checking the actual state.

The output shows two GPU slots (01:00.0 and e1:00.0) both bound to vfio-pci, with the NVIDIA modules listed as available but not in use. This confirms that:

Broader Implications for Infrastructure Management

This message illustrates a principle that extends far beyond this specific configuration: the most elegant solution to an operational problem is often the one that already exists. The user was preparing to implement a complex, manual workflow for GPU switching — unloading modules, editing blacklist files, potentially rebooting — when the infrastructure platform had already automated the entire process.

For teams managing GPU-rich infrastructure, this has practical implications:

Conclusion

Message 5758 represents a moment of discovery that transformed a potentially complex operational task into a non-issue. By systematically investigating the Proxmox host's current state, the assistant uncovered that the GPU passthrough was not only possible but already operational. The user's concern about driver unloading, blacklisting, and rebooting was rendered moot by Proxmox's automatic driver switching mechanism.

The message is a testament to the value of empirical investigation over assumption-driven planning. Rather than accepting the premise that manual intervention was needed, the assistant checked the actual state of the system and found a working solution already in place. The lspci verification at the end provides the concrete evidence that transforms a hypothesis into a confirmed fact.

In the broader narrative of the coding session, this message represents a pivot point. With the GPU passthrough question resolved effortlessly, the team could proceed with their VM experiment without the overhead of manual driver management. The discovery that Proxmox handles the switch automatically meant the tangent experiment could begin immediately, without the hours of configuration and debugging that the user had anticipated.

This is infrastructure work at its most satisfying: when the answer to "how do we make this painless?" is "it already is."