Unbinding the A6000s: A Critical PCI Passthrough Operation in a Multi-GPU Proxmox Environment

In the sprawling infrastructure of a machine learning cluster, few operations are as deceptively simple—and as fraught with consequence—as telling the kernel to release a PCI device from one driver so another can claim it. Message [msg 6770] captures exactly such a moment: the assistant, having just installed the NVIDIA driver 580.126.09 on a Proxmox host called kpro5, discovers that all five GPUs on the machine are still bound to the vfio-pci driver for virtual machine passthrough, and must surgically unbind only the two RTX A6000s intended for the host's own use while leaving the three RTX 3090s untouched for their running VMs.

The message is brief—a reasoning preamble followed by a single bash invocation—but it represents the culmination of a careful diagnostic chain spanning the previous 22 messages, and it embodies a class of systems integration problem that is endemic to GPU cluster management: the tension between virtualization and direct GPU compute, mediated by the Linux kernel's PCI driver binding mechanism.

The Context That Demanded This Message

To understand why this message was written, one must trace the thread back to its origin. The user's request in [msg 6747] was straightforward: a host called kpro6 had been decommissioned, its LXC container CT129 had been moved to kpro5, and the user wanted to spin up that container with two available RTX A6000 GPUs to deploy Qwen3.6-27B, a dense 27-billion-parameter language model. The assistant's first task was to survey the new host.

What it found in [msg 6749] was a machine with a mixed GPU topology: two RTX A6000s (GA102, 48GB each) at PCI addresses 4f:00.0 and ce:00.0, and three RTX 3090s at 53:00.0, 57:00.0, and d6:00.0. Critically, there was no NVIDIA driver installed—nvidia-smi returned "command not found"—and the nouveau open-source driver was loaded instead. Over the next several messages, the assistant methodically installed kernel headers, blacklisted nouveau, downloaded and compiled the NVIDIA 580.126.09 driver via DKMS, and verified that the driver modules loaded successfully.

But then came the discovery in [msg 6767]: nvidia-smi listed no GPUs. The driver had compiled and loaded, but it couldn't see any hardware. The reason became clear when the assistant checked each PCI device's driver binding:

PCI 4f:00.0 driver: vfio-pci
PCI 53:00.0 driver: vfio-pci
PCI 57:00.0 driver: vfio-pci
PCI ce:00.0 driver: vfio-pci
PCI d6:00.0 driver: vfio-pci

Every single GPU was bound to vfio-pci, the Linux kernel's driver for virtual function I/O passthrough. This is standard practice on Proxmox hosts: GPUs assigned to virtual machines must be bound to vfio-pci so that VMs can access them via PCI passthrough. But it creates a problem when the host itself needs to use a GPU—the NVIDIA driver cannot claim a device that is already bound to another driver.

The Reasoning: Surgical Unbinding Under Constraints

The assistant's reasoning in [msg 6770] reflects a careful triage of constraints. The first paragraph lays out the layout with explicit clarity:

- A6000 #1: 0000:4f:00 (IOMMU group 16) — not used by any running VM - A6000 #2: 0000:ce:00 (IOMMU group 43) — not used by any running VM - 3090s: used by running VMs (125, 128) — must stay on vfio-pci

This triage is the product of the investigation in [msg 6768] and [msg 6769], where the assistant checked which VMs were running and which GPUs they used. VM 125 and VM 128 were running with 3090s assigned; VMs 120 and 130 had the A6000s assigned but were stopped. The conclusion is correct and critical: the A6000s can be safely unbound from vfio-pci without disrupting any running workloads, while the 3090s must remain bound.

The assistant's stated plan is then: "I need to unbind the 2 A6000s (and their audio functions) from vfio-pci, then load the nvidia driver. The driver compiled successfully — it just couldn't find any unbound GPUs." This reveals the core assumption: that the NVIDIA driver, once loaded, will automatically claim any GPU that is not already bound to another driver. This is generally correct for the proprietary NVIDIA driver—it scans the PCI bus and binds to any compatible device not claimed by another driver—but it depends on the driver being loaded after the unbinding occurs, or on the driver being reloaded. The assistant does not explicitly reload the NVIDIA driver in this message, which is a subtle point worth examining.

The Execution: Unbinding via sysfs

The bash command in [msg 6770] performs the unbinding through the Linux kernel's sysfs interface, the standard mechanism for manipulating driver bindings at runtime:

echo "0000:4f:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind

This writes the PCI device address to the unbind attribute of the vfio-pci driver, instructing the kernel to detach the driver from that device. Four devices are unbound: the two A6000 GPUs (4f:00.0 and ce:00.0) and their corresponding audio function endpoints (4f:00.1 and ce:00.1). The audio functions are standard companions on NVIDIA GPUs—they provide HDMI/DisplayPort audio—and they are also bound to vfio-pci because they sit on the same PCI device. Unbinding them is necessary because the NVIDIA driver may need to claim the entire PCI device tree.

The verification loop that follows attempts to check whether each device now has a driver:

for d in 4f:00.0 4f:00.1 ce:00.0 ce:00.1; do
  echo -n "PCI $d: "
  ls -l /sys/bus/pci/devices/0000:$d/driver 2>/dev/null | awk -F/ "{print \$NF}" || echo "unbound"
done

The output is ambiguous:

PCI 4f:00.0: PCI 4f:00.1: PCI ce:00.0: PCI ce:00.1: 

No driver name appears after any of the PCI addresses, and the fallback "unbound" message is also absent. This suggests that the ls command succeeded (so there is a driver symlink) but the awk pipeline produced empty output, or there is a subtle quoting issue with the awk script inside the SSH command. In either case, the unbinding operations themselves produced no error output, which is a strong signal that they succeeded—writing an invalid PCI address to the unbind file would produce an "Invalid argument" or "No such device" error.

Assumptions and Their Validity

Several assumptions underpin this message, and examining them reveals the depth of systems knowledge required:

Assumption 1: Unbinding from vfio-pci is sufficient for the NVIDIA driver to claim the device. This is generally true, but it depends on timing. If the NVIDIA driver modules were loaded before the unbinding, they may not automatically re-scan the PCI bus. The assistant's plan mentions "then load the nvidia driver," but the command in this message only performs the unbinding. The actual driver loading or rescanning would need to happen in a subsequent message. This is consistent with the assistant's workflow—it acts in rounds, and the next round would handle the next step.

Assumption 2: The A6000s are safe to unbind because no running VM uses them. This is well-verified. The assistant checked VM configs in [msg 6768] and confirmed that VMs 120 and 130 (which have the A6000s) are stopped. However, there is a subtle risk: if a VM is configured to auto-start or if someone starts VM 120 or 130 after the unbinding, the VM will fail to boot because its PCI device is no longer bound to vfio-pci. The assistant does not address this persistence concern in this message.

Assumption 3: The audio function endpoints must also be unbound. This is correct practice. The NVIDIA driver expects to manage the entire PCI device, including its audio function. Leaving the audio function bound to vfio-pci could cause driver conflicts or incomplete device initialization.

Assumption 4: The driver version 580.126.09 is compatible with the RTX A6000 (Ampere/GA102). This is well-established. The A6000 uses the GA102 architecture, which is supported by all NVIDIA drivers from the 450 series onward. The 580.126.09 driver is a production branch driver from 2025, fully compatible.

The Deeper Significance

This message, while small in scope, illuminates a fundamental challenge in GPU cluster management: the tension between virtualization and direct compute. In a Proxmox environment, GPUs are a scarce resource that must be partitioned between virtual machines (for isolation, legacy OS support, or multi-tenant workloads) and the host itself (for high-performance ML serving). The vfio-pci driver is the mechanism that enables this partitioning, but it creates a binary choice: a GPU is either for VMs or for the host, not both.

The assistant's approach—surgically unbinding only the target GPUs while leaving others untouched—is the correct strategy for this scenario. An alternative would be to reconfigure the vfio-pci bindings at the kernel parameter level (via vfio-pci.ids), but that would require a reboot and would affect all GPUs. The sysfs approach is dynamic and minimally disruptive.

The message also demonstrates a pattern that recurs throughout the opencode session: the assistant operates as a systematic debugger, tracing a problem from symptom (no GPUs visible to nvidia-smi) through diagnosis (all GPUs on vfio-pci) to targeted intervention (unbinding only the A6000s). Each step builds on the previous one, and the reasoning is made explicit for the user to validate.

Input and Output Knowledge

To fully understand this message, a reader needs input knowledge of: the Linux PCI driver model and sysfs interface; the vfio-pci driver's role in GPU passthrough; the NVIDIA proprietary driver's device claiming behavior; the Proxmox GPU mapping and VM configuration system; and the specific hardware topology of kpro5 (two A6000s, three 3090s, their PCI addresses and IOMMU groups).

The message creates output knowledge: the A6000s are now unbound from vfio-pci and available for the NVIDIA driver to claim. This is a prerequisite for the next steps—loading the NVIDIA driver (or triggering a rescan), verifying that nvidia-smi sees both A6000s, configuring the LXC container CT129 for GPU passthrough, and ultimately deploying Qwen3.6-27B.

Conclusion

Message [msg 6770] is a small but critical pivot point in a larger infrastructure deployment. It represents the moment when the assistant transitions from software installation (drivers, kernel modules) to hardware configuration (PCI device binding), and it demonstrates the kind of low-level systems engineering that is often invisible in high-level ML workflows. The unbinding of two PCI devices via sysfs is not glamorous, but it is the kind of operation that, if done incorrectly, can silently undermine an entire deployment. The assistant's careful triage, explicit reasoning, and surgical execution reflect a deep understanding of the Linux kernel's device model and the practical realities of managing a multi-GPU cluster.