The Hot-Plug Pivot: Rebinding GPUs at Runtime on a Proxmox Hypervisor

In the lifecycle of any serious machine learning infrastructure, moments arise when hardware topology must be reshaped without the luxury of a full teardown. Message [msg 6050] captures one such moment: the precise instant when an AI assistant reaches into the Linux kernel's PCI subsystem and surgically moves four NVIDIA RTX PRO 6000 Blackwell GPUs from the nvidia driver to vfio-pci, live, on a running Proxmox host. This single message — a short reasoning preamble followed by a bash command — represents the operational crux of a larger infrastructure reconfiguration that would ultimately determine how 8 GPUs are split between an LXC container and a confidential-computing VM.

Context: The GPU Split Mandate

The conversation leading up to [msg 6050] had been a marathon of ML environment optimization. Over dozens of messages, the assistant had built and tuned a SGLang inference server running the massive Qwen3.5-397B-A17B-NVFP4 model across 8 Blackwell GPUs, achieving impressive throughput. But at [msg 6034], the user pivoted the objective: "I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)."

This was not a simple request. The Proxmox host — a dual-socket AMD machine — had its 8 GPUs split across two NUMA domains: NUMA 0 (GPUs at PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) and NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). The user wanted NUMA 0's GPUs to remain with the nvidia driver for the existing LXC container running SGLang, while NUMA 1's GPUs would be handed over to vfio-pci for passthrough into a separate VM — one presumably configured with AMD SEV-SNP for confidential computing.

The assistant's response was methodical. It first stopped the SGLang server and the LXC container ([msg 6044]–[msg 6046]). It then verified that vfio-pci was loaded and examined each GPU's IOMMU group ([msg 6048]–[msg 6049]). This last step was critical: IOMMU groups define the smallest set of devices that can be isolated for passthrough, and misidentifying group boundaries would either prevent passthrough from working or, worse, cause system instability.

The Message: Reasoning and Execution

Message [msg 6050] opens with a concise piece of reasoning that reveals the assistant's mental model of PCI topology:

Each IOMMU group has the GPU plus its PCIe bridge (root port). The bridge devices don't need to be rebound — only the GPU endpoint does.

This statement is the distillation of the IOMMU group analysis from the previous message. The assistant had discovered that each GPU's IOMMU group contained three devices: the upstream PCIe root port (e.g., 0000:80:01.0), its associated PCIe capability register (.1), and the GPU endpoint itself (0000:81:00.0). The insight — that only the endpoint needs to move — is non-trivial. A less experienced operator might attempt to rebind the entire IOMMU group, which would involve moving the root port away from its native driver, potentially disrupting the entire PCIe hierarchy below it. The assistant correctly recognizes that the bridge devices are infrastructure that should remain untouched; only the leaf device (the GPU) needs to change drivers.

The bash command that follows is a textbook implementation of Linux's PCI driver hot-plug mechanism. For each of the four NUMA 1 GPUs, it performs three operations:

  1. Unbind from nvidia: Writing the PCI address to /sys/bus/pci/drivers/nvidia/unbind tells the nvidia driver to release the device. This is a hot-unplug operation — the driver cleans up its state and detaches from the device without requiring a system reboot.
  2. Register with vfio-pci: Writing 10de 2bb5 (the vendor and device IDs for the RTX PRO 6000 Blackwell) to /sys/bus/pci/drivers/vfio-pci/new_id tells the vfio-pci driver to recognize this device. The 2>/dev/null suppression is pragmatic: if the ID was already registered, the write returns an error, but that's harmless.
  3. Bind to vfio-pci: Writing the PCI address to /sys/bus/pci/drivers/vfio-pci/bind completes the handoff, attaching the device to vfio-pci. The verification step — reading the symlink at /sys/bus/pci/devices/$dev/driver and extracting the driver name — confirms that each GPU now reports vfio-pci as its driver. The output shows all four succeeded.

Technical Depth: What Makes This Work

This operation is remarkable for what it does without doing. It does not require:

Assumptions and Risks

The assistant made several assumptions in this message, all of which were validated by the subsequent results:

That the bridge devices can be left in place. This is correct — the PCIe root port continues to route traffic to the GPU endpoint; only the endpoint's driver changes. The IOMMU group remains intact from the hardware perspective; vfio-pci simply takes ownership of the endpoint within that group.

That vfio-pci was already loaded. Verified in [msg 6048] — lsmod | grep vfio confirmed all vfio modules were present. Without this, the new_id and bind operations would fail with -ENODEV.

That hot-unbinding from nvidia is safe. The nvidia driver's PCI remove path must properly release BAR mappings, tear down GPU state, and notify user-space consumers (like nvidia-smi). In this case, the SGLang server was already stopped and the LXC container was halted, so no user-space consumers were active. If a CUDA process had been running, the unbind could have caused a kernel panic or GPU lockup.

That the device ID 10de:2bb5 is correct. This was established earlier in the conversation from lspci -nn output. An incorrect ID would cause vfio-pci to silently ignore the new_id write (or bind to the wrong device).

The Significance in the Larger Arc

Message [msg 6050] is the hinge point of the entire GPU reconfiguration. Before it, the assistant had gathered information, stopped services, and planned the topology. After it, the assistant would update the LXC config to only mount nvidia0–nvidia3 ([msg 6055]), create a new PCI mapping pro6000-vm for the VM ([msg 6058]), and set up a systemd service to persist the binding across reboots ([msg 6061] onward). The actual VM passthrough would then work because the GPUs were already on vfio-pci.

But the story doesn't end cleanly. Later in the same chunk, when the VM boots with SEV-SNP enabled and full IOMMU translation active, GPU-to-GPU P2P DMA would break — every transfer producing corrupted data, causing NCCL to hang during distributed initialization. The assistant would diagnose this via IO_PAGE_FAULTs in dmesg and a CUDA P2P test, then fix it by disabling NCCL P2P entirely (NCCL_P2P_DISABLE=1), forcing NCCL to use shared memory transport instead. This workaround would cost some performance but enable the server to load successfully, achieving 108 tok/s single-stream and up to 2,800 tok/s at high concurrency.

The hot-plug rebinding in [msg 6050] is thus the first step in a chain of infrastructure decisions that reveal the deep complexity of modern GPU virtualization. The assistant's ability to reason about IOMMU groups, PCI topology, and driver binding at runtime — and to execute that reasoning through precise sysfs manipulations — demonstrates a level of systems engineering sophistication that goes far beyond typical ML deployment work. It is the kind of knowledge that lives at the intersection of kernel internals, virtualization, and high-performance computing, and it is exactly the kind of expertise that separates a working deployment from a broken one.