Unbinding the A6000s: Navigating Proxmox GPU Passthrough Topology for Model Migration

Introduction

In the sprawling infrastructure of a multi-host GPU cluster, few operations are as delicate as reassigning hardware accelerators between virtualization domains. Message 6769 captures one such moment: the assistant, having just installed NVIDIA drivers on the kpro5 Proxmox host, now faces the critical task of unbinding two RTX A6000 GPUs from the vfio-pci driver so they can be passed through to an LXC container for model serving. This message is the bridge between driver installation and actual deployment—a moment where the abstract topology of PCI devices, IOMMU groups, and VM configurations must be precisely understood before any GPU can be made available to the inference stack.

The message itself is deceptively simple: a single bash command querying the Proxmox PCI mapping configuration, preceded by a brief summary of which VMs own which GPUs. But beneath this surface lies a complex reasoning process about virtualization boundaries, device ownership, and the careful choreography required to repurpose hardware without disrupting running workloads.

The Migration Context

The broader session is a migration of the Qwen3.6-27B model deployment from a decommissioned host (kpro6) to kpro5. The user's instruction in [msg 6747] set the stage: kpro6 was gone, CT129 (the LXC container) had been moved to kpro5, and the goal was to deploy the new dense 27B model using two available RTX A6000 GPUs. But kpro5 is not a dedicated machine—it hosts multiple VMs, some running critical workloads on RTX 3090 GPUs, and the A6000s themselves were previously configured for VM passthrough via vfio-pci.

The assistant had already completed several prerequisite steps by message 6769: installing the NVIDIA driver 580.126.09, blacklisting the nouveau driver, installing kernel headers, and confirming that nvidia-smi works. But a critical discovery in [msg 6766] revealed that all GPUs on the host—both the 3090s and the A6000s—were bound to the vfio-pci driver. This is the standard Proxmox pattern for GPU passthrough to VMs, but it means the GPUs are invisible to the host's NVIDIA driver stack. Before the CT129 LXC can access the A6000s via nvidia-smi and CUDA, those devices must be unbound from vfio-pci and bound to the NVIDIA kernel driver instead.

What the Message Actually Does

The message opens with a reasoning summary:

VMs 120 and 130 (both stopped) have the A6000s assigned. VMs 125 and 128 (running) use 3090s. The A6000s are bound to vfio-pci but not actively used by running VMs. Let me check the PCI mappings and unbind the A6000s.

This statement reflects several layers of inference. The assistant had previously queried the VM configurations in [msg 6768], discovering which VMs had hostpci entries. By cross-referencing the PCI addresses from lspci with the VM config files, the assistant determined that VMs 120 and 130 (both in stopped state) were configured to use the A6000s at PCI addresses 4f:00.0 and ce:00.0, while VMs 125 and 128 (running) used the 3090s at 53:00.0, 57:00.0, and d6:00.0.

The key insight here is that the A6000s are assigned to stopped VMs but not actively used. This distinction matters because unbinding a device from vfio-pci while a VM is actively using it would cause the VM to crash or lose its GPU. Since the A6000 VMs are stopped, the assistant can safely proceed.

The command itself queries two sources of PCI mapping information:

cat /etc/pve/mapping/pci.cfg 2>/dev/null || echo "No mapping file"
pvesh get /cluster/mapping/pci 2>/dev/null

The first checks for a local PCI mapping configuration file, while the second uses Proxmox's API tool (pvesh) to query the cluster-level PCI mapping. The output reveals a well-organized mapping system with named entries for each GPU:

The Reasoning Process

The assistant's thinking in this message reveals a careful, methodical approach to infrastructure management. Several cognitive steps are visible:

First, establishing ownership. Before any action, the assistant needed to definitively know which VMs owned which GPUs. This required cross-referencing multiple data sources: the lspci output showing PCI addresses, the VM configuration files showing hostpci entries, and the VM status (running vs. stopped). The assistant had already gathered these in previous messages and now synthesizes them into a clear picture.

Second, assessing risk. The critical question is whether unbinding the A6000s would disrupt running workloads. The assistant explicitly checks that VMs 120 and 130 are stopped, and that VMs 125 and 128 (which use the 3090s) are running but unaffected. This risk assessment is essential—unbinding the wrong device could crash a production VM.

Third, verifying the mapping layer. Rather than blindly proceeding to unbind, the assistant first queries the PCI mapping configuration. This is a wise precaution because Proxmox's PCI mapping system can be complex, with mappings stored at both the local file level (/etc/pve/mapping/pci.cfg) and the cluster level. Understanding the mapping structure is necessary to correctly update it after unbinding.

Fourth, planning the unbind sequence. The assistant's stated intent—"Let me check the PCI mappings and unbind the A6000s"—implies a planned sequence: first understand the mapping, then perform the unbind operation, and finally update the mapping to reflect the new device assignment.

Assumptions and Their Validity

Several assumptions underpin the assistant's reasoning in this message:

Assumption 1: The PCI mapping file is authoritative. The assistant queries both the local file and the cluster API, implicitly assuming that one or both will provide a complete picture. This is reasonable for a well-configured Proxmox cluster, but the output shows that the local file doesn't exist (the || echo "No mapping file" fallback triggers), while the cluster API returns data. This reveals that the mappings are stored at the cluster level, not locally—an important architectural detail.

Assumption 2: Stopped VMs mean safe unbinding. The assistant assumes that because VMs 120 and 130 are stopped, unbinding the A6000s from vfio-pci will not cause issues. This is generally correct, but there's a subtlety: if the VMs are configured to autostart, or if someone starts them after the unbind, they would fail to find their GPU. The assistant doesn't check for autostart configurations or document the change for future reference.

Assumption 3: The NVIDIA driver will automatically claim the devices after unbinding. The assistant's plan implies that after unbinding from vfio-pci, the NVIDIA kernel driver (nvidia and nvidia-uvm) will automatically bind to the A6000s. This is true if the NVIDIA driver is loaded and the device IDs are in its supported list, but it's not guaranteed—sometimes a manual driver override or nvidia-xconfig is needed. The assistant doesn't explicitly verify this assumption in this message.

Assumption 4: The 3090s can remain on vfio-pci. The assistant correctly identifies that the 3090s are in use by running VMs and should be left alone. This is sound, but it means the host's NVIDIA driver will only see the A6000s, not the 3090s. This is fine for the CT129 use case, but it's worth noting that the host itself won't have unified GPU access.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. Proxmox VE architecture: Understanding that Proxmox uses vfio-pci for GPU passthrough to VMs, that PCI mappings are stored in the cluster file system, and that pvesh is the command-line API tool.
  2. GPU device identification: Knowing that 10de:2204 is the PCI vendor:device ID for the RTX 3090 (NVIDIA GA102) and 10de:2230 is for the RTX A6000 (also GA102, but a different SKU). The subsystem IDs (1043:87d5 for ASUS, 10de:1459 for NVIDIA reference) further distinguish the cards.
  3. IOMMU groups: Understanding that IOMMU groups define which PCI devices can be passed through together. The fact that each GPU has its own IOMMU group (16, 20, 24, 43, 50, 72) indicates they are on separate PCIe root ports, which is ideal for passthrough.
  4. The LXC vs. KVM distinction: The CT129 container will access GPUs differently than a VM—LXCs can use the host's NVIDIA driver directly via nvidia-smi and device node passthrough, while VMs require vfio-pci with a guest driver. This is why unbinding from vfio-pci is necessary for LXC use.
  5. The broader deployment context: Understanding that this is part of a model migration from kpro6 to kpro5, and that the Qwen3.6-27B model requires two GPUs with sufficient VRAM (the A6000s have 48GB each, totaling 96GB, which fits the ~55GB BF16 model).

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A complete PCI topology map of kpro5: The output documents which GPUs are at which PCI addresses, their IOMMU groups, their device IDs, and their subsystem IDs. This is essential for any future hardware management.
  2. Confirmation of the mapping storage location: The fact that /etc/pve/mapping/pci.cfg doesn't exist but pvesh get /cluster/mapping/pci returns data confirms that mappings are stored in the Proxmox cluster file system (likely /etc/pve/mapping/pci.cfg on the cluster FS, which is a symlink to the PMXFS database). This is an important architectural finding.
  3. Validation of the safe-to-unbind assessment: The assistant's explicit reasoning that A6000 VMs are stopped and 3090 VMs are running creates a clear audit trail for why the unbind operation is safe.
  4. A baseline for the unbind operation: The output provides the exact PCI paths (0000:4f:00 and 0000:ce:00) and IOMMU groups (16 and 43) that will need to be manipulated during the unbind process.

Broader Significance

This message exemplifies a class of infrastructure operation that is common in GPU cluster management but rarely documented: the transition of a GPU from VM passthrough to LXC/host-native access. In many organizations, GPUs are initially provisioned for VM workloads, but as AI inference workloads grow, the same hardware needs to be repurposed for container-native access. The unbind operation is the critical pivot point.

The assistant's methodical approach—establish ownership, assess risk, verify mappings, then act—is a template for safe hardware reconfiguration. The explicit reasoning about which VMs are running versus stopped, and which GPUs belong to which VM, demonstrates the kind of situational awareness that prevents costly mistakes.

The message also highlights the complexity of modern GPU virtualization. A single Proxmox host with five GPUs, three VMs, and one LXC container creates a topology where device ownership must be precisely tracked. The PCI mapping system provides the necessary abstraction, but it requires careful interpretation to understand the actual runtime state.

Conclusion

Message 6769 is a quiet but crucial step in the model deployment pipeline. It doesn't involve model downloads, configuration files, or inference benchmarks. Instead, it's about understanding the physical and virtual topology of GPU resources—a prerequisite for any meaningful GPU workload. The assistant's careful reasoning about device ownership, VM states, and mapping configurations demonstrates the kind of systems thinking that separates successful infrastructure management from costly errors. By the end of this message, the path forward is clear: the A6000s can be safely unbound from vfio-pci and made available to the CT129 LXC for Qwen3.6-27B inference. The unbind operation itself will come in the next message, but the groundwork—the understanding—has been laid.