The Art of GPU Rebinding: A Sysfs Journey Between VMs and Containers

Introduction

In the complex ecosystem of GPU-accelerated machine learning infrastructure, few moments are as fraught with tension as the transition between virtualization domains. Message 5783 captures one such moment: a brief but technically dense intervention where an AI assistant manually rebinds eight NVIDIA RTX PRO 6000 Blackwell GPUs from the vfio-pci driver (used for VM passthrough) back to the proprietary nvidia driver (used for LXC container access). This message, though only a few lines long, encapsulates a deep understanding of Linux PCI device management, the Proxmox virtualization stack, and the practical realities of sharing expensive GPU hardware across competing workloads.

The full message reads:

GPUs are still on vfio-pci — they didn't auto-rebind to nvidia after VM stop. Need to manually unbind from vfio and rebind to nvidia:

>

``bash for dev in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0; do echo "$dev" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null echo "$dev" > /sys/bus/pci/drivers/nvidia/bind 2>/dev/null done echo "Done" lspci -ks 01:00.0 | grep "driver in use" ``

>

Kernel driver in use: nvidia

This article examines the reasoning, context, assumptions, and technical knowledge embedded in this single message, unpacking why such a seemingly simple command sequence represents a critical piece of infrastructure engineering.

The Context: A Machine Learning Infrastructure Balancing Act

To understand why this message was written, one must first understand the broader infrastructure context. The conversation leading up to message 5783 reveals a sophisticated multi-GPU setup: a Proxmox hypervisor (hostname 10.1.2.6) managing eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) that must be shared between two fundamentally different execution environments.

The first environment is an LXC container (CT 129, named llm-two) running an SGLang-based inference server for the Kimi-K2.5 INT4 model. This container requires direct access to the GPUs through the NVIDIA driver stack — nvidia.ko, nvidia_uvm.ko, nvidia_modeset.ko, and nvidia_drm.ko — to perform CUDA computations for large language model inference. The second environment is a KVM virtual machine (VM 131, named ml-pipelines) that needs GPU passthrough via the vfio-pci driver for its own workloads.

The tension is inherent: a PCI device can only be bound to one driver at a time. When the VM is running, the GPUs are bound to vfio-pci, making them invisible to the host's NVIDIA driver stack. When the VM stops, the expectation is that the GPUs will automatically rebind to the nvidia driver, making them available to the LXC container. Message 5783 is written precisely because this automatic rebinding failed.

The Diagnostic Journey: Why Auto-Rebind Failed

The assistant's opening statement — "GPUs are still on vfio-pci — they didn't auto-rebind to nvidia after VM stop" — is the culmination of a diagnostic chain that began several messages earlier. The user reported that after rebooting the Proxmox host and then stopping the VM, the LXC container could not see any GPUs: nvidia-smi returned "Failed to initialize NVML: No supported GPUs were found."

The assistant's first hypothesis (in message 5778) was that the NVIDIA driver modules might not have reloaded properly after the host reboot. But examination of the host state revealed a more specific problem: lsmod showed the NVIDIA modules were loaded (nvidia, nvidia_uvm, nvidia_modeset, nvidia_drm), and lspci confirmed the GPUs were still bound to vfio-pci. The modules were present but not bound to any devices — a driver-device affinity mismatch.

This is a subtle failure mode. Normally, when a VM with PCI passthrough stops, Proxmox's QEMU process releases the VFIO device, the kernel's driver core notices the device is no longer claimed by vfio-pci, and it probes available drivers to see if any will bind. The NVIDIA driver should claim the device. But this automatic probing can fail for several reasons: the NVIDIA driver's probe() function might reject the device if initialization conditions aren't met, the driver might be in an error state from a previous failed attachment, or the device might need a reset cycle that wasn't triggered.

The assistant's decision to bypass the automatic mechanism and perform a manual rebind through sysfs demonstrates a pragmatic understanding that when the kernel's driver auto-probing fails, the sysfs interface provides a lower-level escape hatch that can force the issue.

The Technical Mechanism: Sysfs Driver Binding

The command executed in message 5783 is a masterclass in Linux PCI device management. Let's dissect each component:

The PCI device addresses: 0000:01:00.0, 0000:11:00.0, 0000:61:00.0, 0000:71:00.0, 0000:81:00.0, 0000:91:00.0, 0000:e1:00.0, 0000:f1:00.0. These are the full PCI bus addresses (domain:bus:device.function) for the eight RTX PRO 6000 GPUs. The 0000 domain is standard for most systems. The bus numbers (01, 11, 61, 71, 81, 91, e1, f1) reveal the physical slot layout — these are spread across multiple PCIe root ports, which is typical for a dual-socket server motherboard with multiple PCIe slots.

The unbind operation: echo "$dev" > /sys/bus/pci/drivers/vfio-pci/unbind. Writing a PCI address to the unbind file of a driver tells the kernel to detach that device from the driver. The 2>/dev/null suppresses errors for devices that might already be in the process of unbinding or that have already been released. This is a safety measure — if any GPU has already partially transitioned, the command won't fail the entire loop.

The bind operation: echo "$dev" > /sys/bus/pci/drivers/nvidia/bind. This writes the PCI address to the NVIDIA driver's bind file, instructing the kernel to attach the device to the NVIDIA driver. The NVIDIA driver's probe function will then initialize the GPU, set up the device state, and make it available through the standard NVIDIA device nodes (/dev/nvidia0, /dev/nvidia-uvm, etc.).

The verification: lspci -ks 01:00.0 | grep "driver in use". This checks a single GPU (the first one, at address 01:00.0) to confirm the driver change took effect. The -k flag shows kernel drivers, -s shows the device in a tree format. The output confirms "Kernel driver in use: nvidia".

Assumptions and Potential Pitfalls

The assistant makes several implicit assumptions in this message:

Assumption 1: The NVIDIA driver modules are already loaded and functional. This is critical — writing to the nvidia/bind sysfs file only works if the NVIDIA driver is loaded in the kernel. If the driver modules weren't loaded, the bind would fail silently (the 2>/dev/null would hide the error). The assistant had verified in message 5778 that lsmod | grep nvidia showed all four NVIDIA modules loaded, so this assumption was valid.

Assumption 2: The GPUs are in a clean state for rebinding. After being used by a VM, the GPUs might have residual state — pending DMA operations, uncleared PCIe configuration space, or stuck interrupt handlers. Writing to the unbind file might not fully reset the device. In some cases, a full PCIe bus reset or even a power cycle is needed. The assistant implicitly assumes that the VFIO release path (triggered when the VM stopped) left the devices in a state where the NVIDIA driver can successfully initialize them.

Assumption 3: The sysfs unbind/bind sequence is atomic enough. There's a race condition: if the unbind completes but the NVIDIA driver's probe fails, the device could be left in a driverless state. The assistant doesn't check intermediate states — it unbinds and binds in rapid succession, hoping the NVIDIA driver claims the device before any other driver (like nouveau, the open-source reverse-engineered driver) can grab it.

Assumption 4: All eight GPUs have the same PCI address pattern. The addresses follow a consistent format (0000:XX:00.0 where XX varies), but the assistant doesn't dynamically discover them — they're hardcoded from earlier lspci output. If the PCI topology changed (e.g., after a host reboot with different BIOS settings), these addresses could be wrong.

Assumption 5: The 2>/dev/null suppression is safe. By hiding errors, the assistant risks missing a failure on one or more GPUs. The final verification only checks one GPU (01:00.0), so if GPU at address 11:00.0 failed to bind, the assistant wouldn't immediately know. The user would discover this only when trying to use that specific GPU.

The Input Knowledge Required

To understand and execute this message, one needs:

  1. Linux PCI device model: Understanding that PCI devices are identified by bus addresses and managed by kernel drivers through a binding system.
  2. Sysfs driver interface: Knowledge that /sys/bus/pci/drivers/<driver>/ contains bind and unbind files that accept PCI addresses for manual driver attachment/detachment.
  3. VFIO and GPU passthrough: Understanding that vfio-pci is the driver used for passing PCI devices through to KVM VMs, and that it must be released before the NVIDIA driver can claim the device.
  4. Proxmox virtualization: Awareness that Proxmox uses both LXC containers (which share the host kernel and drivers) and KVM VMs (which get dedicated PCI devices via VFIO), and that switching between them requires driver rebinding.
  5. NVIDIA driver stack: Knowledge that the NVIDIA driver consists of multiple kernel modules (nvidia.ko, nvidia_uvm.ko, nvidia_modeset.ko, nvidia_drm.ko) and that all must be loaded and functional for GPUs to work.
  6. Blackwell GPU requirements: From earlier context, the RTX PRO 6000 Blackwell GPUs require the open kernel module variant (nvidia-dkms-590-open), not the proprietary one. This was already fixed in the VM (messages 5764-5771).

The Output Knowledge Created

This message creates several forms of knowledge:

  1. A working GPU rebinding procedure: The immediate output is a functional set of eight GPUs now bound to the NVIDIA driver, ready for use by the LXC container. The lspci verification confirms this.
  2. A documented sysfs workflow: The command sequence serves as a reusable procedure for manual driver rebinding. Any system administrator facing similar "stuck on vfio-pci" issues can adapt this pattern.
  3. Evidence of a kernel behavior gap: The fact that automatic rebinding failed is itself valuable knowledge. It suggests that the Proxmox/KVM VFIO release path doesn't reliably trigger NVIDIA driver reattachment, at least for Blackwell GPUs with the open kernel module. This could indicate a kernel bug, a driver issue, or a configuration problem.
  4. PCI topology documentation: The eight PCI addresses, extracted and hardcoded into the script, serve as a map of the system's GPU layout. This is useful for any future PCIe configuration tasks.

The Thinking Process: A Window into Infrastructure Debugging

The reasoning visible in this message reveals a structured approach to infrastructure debugging:

Step 1: Observe the symptom. The user reports nvidia-smi fails in the LXC container. The assistant doesn't jump to conclusions — it gathers data.

Step 2: Verify the host state. The assistant checks whether the VM is still running (it is), whether the NVIDIA modules are loaded (they are), and what driver each GPU is bound to (vfio-pci).

Step 3: Identify the mismatch. The GPUs are on vfio-pci but the container needs them on nvidia. The VM must be stopped first.

Step 4: Stop the VM. The assistant stops VM 131 (message 5781).

Step 5: Re-check and find the failure. After the VM stops, the assistant checks again (message 5782) and finds the GPUs are still on vfio-pci. The automatic rebinding didn't happen.

Step 6: Apply the manual fix (message 5783). The assistant doesn't try to debug why auto-rebind failed — that would be time-consuming and might require kernel tracing. Instead, it applies the direct sysfs manipulation, which is fast, reliable, and bypasses whatever mechanism is broken.

Step 7: Verify. A quick lspci check confirms the fix worked.

The elegance of this approach is its pragmatism. The assistant doesn't get lost in the "why" of the auto-rebind failure. It identifies the desired state (GPUs on nvidia driver), identifies the mechanism to achieve it (sysfs bind/unbind), and executes. The "why" can be investigated later if the problem recurs; for now, the infrastructure needs to be operational.

The Broader Significance: PCI Driver Management as Infrastructure Skill

This message exemplifies a class of infrastructure knowledge that is increasingly valuable in the age of GPU-accelerated computing. As organizations deploy more GPU hardware, the ability to flexibly allocate GPUs between different execution environments — bare metal, containers, VMs — becomes critical. The sysfs driver binding interface is a low-level tool that enables this flexibility when higher-level orchestration fails.

The technique shown here is not specific to NVIDIA or VFIO. The same /sys/bus/pci/drivers/<driver>/bind and unbind interface works for any PCI device and any kernel driver. Network interfaces, storage controllers, FPGA accelerators — all can be manually rebound using this mechanism. It's a universal Linux skill that pays dividends across many hardware management scenarios.

Conclusion

Message 5783 is deceptively simple. On its surface, it's a shell loop that writes eight PCI addresses to two sysfs files. But beneath that simplicity lies a deep understanding of Linux kernel internals, virtualization architecture, and practical infrastructure engineering. The assistant correctly diagnosed a failure in automatic driver rebinding, identified the manual sysfs bypass, executed it efficiently, and verified the result — all in a single message.

The message also serves as a reminder that even in an era of sophisticated orchestration tools (Kubernetes, Nomad, Slurm), the lowest-level kernel interfaces remain essential for infrastructure management. When the automatic mechanisms fail — and they will fail — knowing how to manipulate /sys/bus/pci/drivers/ directly can save hours of debugging and keep expensive GPU hardware productive.

For the machine learning engineer or infrastructure administrator managing multi-GPU systems, this message is a case study in pragmatic problem-solving: observe the symptom, identify the mismatch, apply the minimal intervention, verify the result, and move on. The GPUs are now bound to the NVIDIA driver, the LXC container can see them, and the Kimi-K2.5 inference server can resume its work. The infrastructure lives to serve another day.