The Asymmetric GPU Switching Problem: Documenting the LXC-VM Dance on Proxmox

In the middle of an intense optimization session for deploying large language models across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a seemingly simple operational question emerged: how do you reliably switch GPU ownership between a Proxmox LXC container and a virtual machine? The answer, as the assistant discovered through direct experimentation, was anything but symmetric. Message [msg 5788] captures the moment this knowledge was codified — a concise operational summary that distills several rounds of trial-and-error into a reusable workflow.

Context: Two Environments, One Set of GPUs

The user was running a production SGLang inference server for the Kimi-K2.5 model inside an LXC container (CT 129) on a Proxmox host. The host had eight NVIDIA RTX PRO 6000 Blackwell GPUs, which required the open kernel module (nvidia-dkms-590-open) rather than the proprietary driver — a fact that had been discovered and fixed earlier in the session when the VM initially failed to see the GPUs ([msg 5764] through [msg 5771]). The user wanted to experiment with passing all eight GPUs to a VM (131, ml-pipelines) for a tangent experiment.

The assistant had already established that Proxmox's PCI passthrough mapping (pro6000) handled the forward direction automatically: when qm start 131 was issued, Proxmox unbinded the GPUs from the nvidia driver and bound them to vfio-pci, making them available to the VM. This worked seamlessly, as confirmed by lspci -ks showing "Kernel driver in use: vfio-pci" for all eight GPUs.

But the return journey proved more treacherous.

The Discovery: Stopping the VM Doesn't Release the GPUs

After the VM experiment concluded, the user attempted to switch back to the LXC container and found that nvidia-smi inside the container reported "No supported GPUs were found" ([msg 5777]). The assistant investigated and discovered that even after qm stop 131, the GPUs remained stubbornly bound to vfio-pci:

01:00.0 3D controller: NVIDIA Corporation Device 2bb5 (rev a1)
    Kernel driver in use: vfio-pci

Proxmox had not automatically rebinded them to the nvidia driver. This was the critical finding: the hypervisor handles binding GPUs to vfio-pci when a VM starts, but does not handle the reverse operation when the VM stops. The nvidia driver, which was still loaded in the kernel (confirmed via lsmod | grep nvidia), did not automatically reclaim the devices either.

The assistant solved this with a manual sysfs operation — iterating over the eight PCI device addresses, unbinding each from vfio-pci, and binding it to nvidia:

for dev in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0; do
  echo "$dev" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
  echo "$dev" > /sys/bus/pci/drivers/nvidia/bind 2>/dev/null
done

This immediately restored GPU visibility on the host ([msg 5784]). After restarting the container, nvidia-smi showed all eight GPUs, and the SGLang service auto-started and began loading the model.

The Message: Codifying the Workflow

Message [msg 5788] is the assistant's summary and documentation of this entire episode. It opens with a confirmation — "All back. The sglang service is loading the model" — establishing that the system has returned to its desired state. Then it presents the two workflows as reference material.

The LXC-to-VM transition is elegantly simple:

pct stop 129
qm start 131

Two commands. No manual intervention. Proxmox handles the PCI rebinding automatically as part of VM startup.

The VM-to-LXC transition is more involved:

qm stop 131
# rebind GPUs from vfio-pci to nvidia
for dev in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0; do
  echo "$dev" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
  echo "$dev" > /sys/bus/pci/drivers/nvidia/bind 2>/dev/null
done
pct start 129

The asymmetry is striking. Going one way requires two commands and zero manual driver management. Going the other requires stopping the VM, manually rebinding eight PCI devices via sysfs, and then starting the container. The assistant ends with a practical offer: "Want me to save these as scripts on the host?" — recognizing that this workflow will be needed repeatedly and should be automated rather than memorized.

Assumptions, Corrections, and Lessons Learned

Several assumptions were challenged during this episode, and the message implicitly carries these lessons:

Assumption 1: VM stop triggers automatic GPU release. This was incorrect. Proxmox's PCI passthrough mapping binds devices to vfio-pci when the VM starts, but there is no corresponding hook to rebind them to the original driver on VM stop. The devices remain in vfio-pci limbo — visible to lspci but inaccessible to the nvidia driver.

Assumption 2: The nvidia driver automatically reclaims orphaned devices. Also incorrect. Even though the nvidia kernel modules were loaded (as confirmed by lsmod showing nvidia, nvidia_uvm, nvidia_modeset, and nvidia_drm), they did not automatically bind to devices that had been released from vfio-pci. The driver requires an explicit bind operation via sysfs. This is a safety feature — the kernel cannot assume which driver should claim a device after it is released, as multiple drivers may be capable of managing it.

Assumption 3: The container would need manual GPU configuration after switching. This was correct — the container needed to be stopped and restarted to pick up the GPUs. However, the SGLang service was configured to auto-start (via the systemd service created earlier in the session), so the model began loading without additional intervention.

Assumption 4: The host reboot (mentioned in [msg 5773]) complicated the switching process. The reboot was initially concerning, but the assistant discovered that the GPUs were still properly configured — the issue was purely the vfio-pci binding, not a driver configuration problem. The open kernel modules survived the reboot and were ready to claim the GPUs once unbinded from vfio-pci.

Knowledge Created

This message creates several forms of output knowledge that extend beyond the immediate operational context:

Operational knowledge: The asymmetric switching workflow is now documented. Anyone managing this Proxmox cluster can follow the two workflows without rediscovering the manual rebind step through trial and error. The eight PCI addresses are listed explicitly, removing ambiguity about which devices need rebinding.

System knowledge: The behavior of Proxmox's PCI passthrough is now better understood. The hypervisor handles the forward binding (nvidia → vfio-pci) automatically as part of VM startup via QEMU's device assignment. But the reverse (vfio-pci → nvidia) is not automated because there is no standard "cleanup" hook that runs after VM stop — the device is simply released from vfio-pci, and the host must explicitly reclaim it.

Scripting opportunity: The assistant's offer to save these as scripts recognizes that operational procedures are best automated. The eight-device rebind loop is error-prone when done manually — a single typo in a PCI address (e.g., 0000:e1:00.0 vs 0000:el:00.0) could leave a GPU unbound, causing mysterious failures. A script also reduces cognitive load during emergency switches.

Debugging methodology: The episode demonstrates a systematic approach to GPU visibility problems: check the host's PCI binding state (lspci -ks), verify kernel module loading (lsmod), attempt manual rebinding via sysfs, and verify with nvidia-smi at each layer (host, container, VM). This methodology is reusable for any GPU passthrough troubleshooting scenario.

Broader Implications

This message, while seemingly a simple operational note, reveals deeper truths about GPU virtualization with Proxmox. The asymmetry between LXC-to-VM and VM-to-LXC transitions is not a bug but a consequence of how PCI passthrough works at the architectural level. LXC containers share the host kernel and thus inherit the host's driver bindings — they see whatever the host sees. VMs, with their separate kernel and device model, require dedicated device assignment via vfio-pci, which completely detaches the device from the host driver stack.

The forward transition (nvidia → vfio-pci) is automated because Proxmox's QEMU integration handles vfio-pci binding as part of VM startup — it's a well-defined operation with a clear trigger. The reverse transition (vfio-pci → nvidia) is not automated because there is no standard trigger — VM stop releases the device from vfio-pci, but the kernel has no mechanism to automatically rebind it to the previously used driver. The device enters an unbound state, and it is the administrator's responsibility to rebind it.

For operators managing hybrid LXC/VM GPU workloads, this knowledge is essential. Without it, a simple "stop VM, start container" sequence would silently fail, presenting as "no GPUs found" with no obvious cause. The error message — "Failed to initialize NVML: No supported GPUs were found" — gives no hint that the GPUs are simply bound to the wrong driver. The assistant's documentation of the manual rebind step transforms an opaque failure into a manageable, repeatable procedure.

Conclusion

Message [msg 5788] is a masterclass in operational documentation. It captures a hard-won discovery about Proxmox's asymmetric GPU switching behavior and presents it as a clean, reusable workflow. The message demonstrates that effective system administration is not just about fixing problems, but about codifying the fix so it never needs to be rediscovered. The assistant's closing offer — to save the workflows as scripts — shows an understanding that the best documentation is automation, and that operational knowledge should be transformed from tribal knowledge into executable artifacts.

The message also illustrates a broader principle of infrastructure management: the simplest operations (start VM, stop VM) often conceal complex driver-level mechanics that only reveal themselves at the boundary between virtualization layers. Understanding these boundaries — where the hypervisor's automation ends and manual intervention begins — is what separates routine operation from reliable operation.