The Art of the Graceful Handoff: Rebinding GPUs Between VM and Container

In the high-stakes world of production AI inference, the physical hardware doesn't change—but its allegiance must. Message 5785 of this opencode session captures what appears to be a trivial operation: restarting an LXC container after its GPUs were returned from a VM. But beneath the surface of a single bash command lies a rich story about device management, driver binding, and the careful choreography required to share expensive hardware between virtualized environments.

The Message

The assistant writes:

GPUs back on host. Now restart the container:

>

``bash ssh root@10.1.2.6 'pct stop 129 2>/dev/null; sleep 2; pct start 129 && echo "CT 129 started"' ``

>

CT 129 started

This is the culmination of a multi-step recovery operation triggered when a host reboot disrupted the GPU binding state. The message itself is deceptively simple—a single command with a status output—but it represents the final act in a drama that began when the user discovered their LXC container could no longer see any GPUs.

The Context: A Tangled Web of Device Ownership

To understand why this message exists, we must trace the events that led to it. The session involves a Proxmox hypervisor (host 10.1.2.6) managing both an LXC container (CT 129, named llm-two) and a QEMU/KVM virtual machine (VM 131, named ml-pipelines). Both need access to the same pool of eight NVIDIA RTX PRO 6000 Blackwell GPUs—but they cannot share them simultaneously.

The user had been running their Kimi-K2.5 INT4 model inference server inside the LXC container, using all eight GPUs. Then they pivoted to a "tangent experiment": passing all eight GPUs through to a VM. This required the host to unbind the GPUs from the NVIDIA kernel driver (nvidia.ko) and rebind them to the vfio-pci driver, which Proxmox handles automatically when a VM with hostpci mappings starts.

The VM experiment succeeded—the GPUs were visible inside the guest after installing the open kernel module (nvidia-dkms-590-open) instead of the proprietary one, which is required for Blackwell (SM120) architecture support. But then the host rebooted. And after the reboot, when the user tried to switch back to the LXC container, they encountered the dreaded error: "Failed to initialize NVML: No supported GPUs were found."

This is where the real work began.

The Root Cause: Why the Container Couldn't See Its GPUs

The host reboot created a state mismatch. When Proxmox boots, the NVIDIA kernel modules load and bind to compatible devices. But if a VM was previously running with GPU passthrough, the devices may still be claimed by vfio-pci in the kernel's device tree—even after the VM stops. The assistant's investigation revealed exactly this: after stopping VM 131, the GPUs remained bound to vfio-pci rather than reverting to the NVIDIA driver. The kernel modules (nvidia, nvidia_uvm, nvidia_modeset, nvidia_drm) were loaded with refcount zero—present but not managing any devices.

The fix required manual intervention: unbinding each GPU from vfio-pci and rebinding it to the NVIDIA driver by writing to the sysfs PCI driver bind/unbind interfaces. This is a low-level kernel operation that most users never encounter, but it's essential knowledge for anyone managing GPU passthrough in production.

The assistant's command in message 5785—pct stop 129; sleep 2; pct start 129—is the final step after this rebinding was completed. The container itself needed to be restarted because its device access was configured at startup. When the container originally started after the reboot, the GPUs weren't available to the NVIDIA driver, so the container's /dev/nvidia* device nodes were never created. Restarting the container after the GPUs were properly bound ensures the container runtime re-probes the device availability and makes the GPUs accessible to processes inside.

Assumptions and Implicit Knowledge

This message makes several assumptions that are worth examining. First, it assumes that stopping and starting the container is sufficient to re-establish GPU access. This is correct for LXC containers on Proxmox, where device passthrough is configured in the container configuration file (typically /etc/pve/lxc/129.conf) and the container runtime binds the host devices into the container's namespace at startup. However, it assumes the container configuration already includes the necessary GPU device entries—if those were missing, the restart alone wouldn't help.

Second, the message assumes the NVIDIA kernel modules are properly loaded and functional on the host. The preceding investigation confirmed this: lsmod | grep nvidia showed all modules loaded, and nvidia-smi -L on the host listed all eight GPUs. Without this confirmation, restarting the container would be futile.

Third, the 2>/dev/null on the pct stop command reveals an assumption: the container might already be stopped. This is a defensive coding practice—suppressing error output for the stop command in case it fails because the container isn't running, which is harmless. The && after pct start ensures the success message only prints if the start succeeds, providing clear feedback.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible across the messages leading up to this one, follows a systematic diagnostic pattern:

  1. Observe the symptom: The user reports "Failed to initialize NVML: No supported GPUs were found" inside the LXC container.
  2. Check the host state: Are the GPUs bound to the NVIDIA driver or to vfio-pci? The answer was vfio-pci—still claimed by the VM.
  3. Release the GPUs: Stop VM 131 to release the vfio-pci binding.
  4. Verify the release: Check again—GPUs still on vfio-pci. The auto-rebind didn't happen.
  5. Force the rebind: Manually write each PCI device address to the NVIDIA driver's bind interface via sysfs.
  6. Verify the rebind: Confirm lspci shows "Kernel driver in use: nvidia" and nvidia-smi -L lists all GPUs.
  7. Restart the consumer: Restart the LXC container so it picks up the now-available devices. This is a textbook example of layered debugging: starting from the symptom, moving to increasingly lower-level diagnostics, and finally performing a manual operation that the normal tooling should have handled automatically.

Input Knowledge Required to Understand This Message

A reader needs significant context to grasp why this simple command matters. They must understand:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A verified workflow for GPU rebinding: The sequence of stopping the VM, manually rebinding via sysfs, and restarting the container is now a documented, tested procedure. This can be scripted for future use.
  2. Confirmation that the container configuration is correct: The fact that pct start 129 succeeded and the GPUs became available inside proves the container's device passthrough configuration is properly set up.
  3. A baseline for troubleshooting: If the GPUs fail to appear again, the team now knows to check the host driver binding first, then the container runtime state.
  4. The sleep 2 pattern: The two-second delay between stop and start is a pragmatic workaround—giving the container time to fully release its resources before re-acquiring them. This is the kind of heuristic that emerges from experience with race conditions in container lifecycle management.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption in this sequence was that Proxmox would automatically rebind GPUs from vfio-pci back to the NVIDIA driver when the VM stopped. The assistant's investigation in message 5782 revealed this didn't happen: even after qm stop 131, the GPUs remained on vfio-pci. This required the manual sysfs rebinding workaround.

Why didn't the auto-rebind work? Several possibilities exist. The host reboot may have left the VFIO driver in a state where it didn't properly release the devices. Or the NVIDIA driver's hot-plug detection may not have triggered because the devices were never physically removed—they were just reassigned at the driver level. This is a known edge case in GPU passthrough setups, and the manual rebind approach is the standard workaround.

Another subtle issue: the assistant assumed that stopping the container with pct stop 129 would cleanly release resources. The 2>/dev/null suppression suggests awareness that this might fail silently if the container was already in a bad state. In practice, the stop succeeded, and the subsequent start brought the container up with GPU access restored.

Broader Implications

This message, for all its brevity, illustrates a fundamental tension in AI infrastructure management: the conflict between flexibility and stability. The ability to switch GPUs between a VM (for experimental workloads) and a container (for production inference) is enormously valuable—it allows a single hardware pool to serve multiple purposes. But the switching mechanism is fragile, relying on kernel-level driver rebinding that can fail in subtle ways.

The solution demonstrated here—manual sysfs rebinding followed by container restart—is not elegant, but it is robust. It works because it operates at the right level of abstraction: not trying to fix the automatic rebinding, but instead performing the operation explicitly when the automatic mechanism fails. This is a recurring theme in systems administration: when automation breaks, fall back to the primitive operations that the automation was supposed to abstract away.

For the production deployment that follows this message (the Kimi-K2.5 INT4 model running as a systemd service inside the LXC container), this workflow means the team can confidently switch between VM experimentation and container-based inference without fear of permanent device loss. The GPUs are always recoverable—it just takes a few manual commands and a container restart.