The Moment the GPU Handoff Breaks: Diagnosing a Failed Switch from VM Passthrough to LXC

A Single Line of Failure

In the middle of an intense multi-day optimization session spanning speculative decoding, CUDA graph tuning, and production deployment of large language models, a single user message arrived that stopped the momentum cold:

Now switching back to llm-two container; root@llm-two:~# nvidia-smi Failed to initialize NVML: No supported GPUs were found Unable to determine the number of GPUs

This is message <msg id=5777>, and at first glance it looks like a simple error report. But in context, it represents a critical inflection point: the moment a carefully orchestrated GPU switching workflow between a Proxmox LXC container and a virtual machine collapsed, threatening to undo hours of configuration work and leave eight NVIDIA RTX PRO 6000 Blackwell GPUs inaccessible to either environment.

The Context: A Delicate Dance of GPU Ownership

To understand why this message matters, we must reconstruct the chain of events that led to it. The user had been running a production SGLang server serving the Kimi-K2.5 INT4 model inside an LXC container (CT 129, named llm-two) on a Proxmox VE host (10.1.2.6). This container had full access to eight Blackwell GPUs via the host's NVIDIA driver stack. The setup was working well—the server was deployed as a systemd service with hierarchical KV cache, EAGLE-3 speculative decoding, and all the optimizations accumulated over dozens of prior messages.

Then the user decided to run a "tangent experiment" ([msg 5752]): pass all eight GPUs to a VM (VM 131, ml-pipelines) instead. This required the GPUs to be bound to the vfio-pci driver on the host rather than the NVIDIA driver, so the VM could claim them via PCI passthrough. Proxmox handles this automatically when a VM with hostpci mappings starts—it unbinds the devices from the NVIDIA driver and rebinds them to vfio-pci. When the VM stops, the reverse happens.

The experiment hit a snag immediately: the VM guest had the proprietary NVIDIA driver (nvidia-dkms-590) installed, but Blackwell GPUs require the open kernel module (nvidia-dkms-590-open). The assistant diagnosed this ([msg 5764]), installed the correct package ([msg 5768]), rebooted the VM ([msg 5769]), and confirmed all eight GPUs were visible inside the guest ([msg 5771]).

Then the user reported they "had to reboot host" ([msg 5773]). The assistant checked and found everything still working—VM 131 was running, GPUs were on vfio-pci, and inside the VM all eight GPUs were visible (<msg id=5774-5776>).

The Switchback That Failed

Message &lt;msg id=5777&gt; is the user's next report. They stopped the VM (or at least attempted to switch back to the LXC container) and ran nvidia-smi inside llm-two. The result: "No supported GPUs were found."

This is the moment where theory meets reality. The Proxmox GPU switching mechanism should have worked automatically—stop the VM, and the GPUs rebind to the NVIDIA driver on the host, making them available to the LXC container. But something went wrong. The NVIDIA driver loaded, but it couldn't initialize any of the eight GPUs.

Why This Happened: The Open vs. Proprietary Driver Mismatch

The root cause traces back to the VM experiment. When the assistant installed nvidia-dkms-590-open inside the VM guest, that only fixed the VM's driver. But the host—the Proxmox node running the LXC container—still had the proprietary nvidia-dkms-590 driver loaded. Here's the critical detail: the host's NVIDIA kernel modules were loaded before the VM started, but all eight GPUs were bound to vfio-pci while the VM was running. When the VM stopped and the GPUs rebinded to the NVIDIA driver, the proprietary driver modules (nvidia.ko) tried to initialize the Blackwell GPUs—and failed, because the proprietary driver doesn't support SM120 architecture GPUs.

The host needed the open kernel modules too. But the host had never been updated, because the original LXC setup worked fine with the proprietary driver (or so everyone thought). The assistant's earlier investigation ([msg 5754]) showed the host had nvidia kernel modules loaded with a refcount of zero (no devices bound), but it never checked whether the host's driver was open or proprietary. This was an oversight.

Assumptions Made and Broken

Several assumptions underpinned the workflow that message &lt;msg id=5777&gt; shattered:

Assumption 1: The host's NVIDIA driver works with Blackwell GPUs. The original LXC setup had the GPUs working with the proprietary driver, so everyone assumed the host driver was fine. But the LXC container shares the host's kernel—if the host's driver modules can't initialize the GPUs, the container can't use them either.

Assumption 2: The vfio-pci rebinding is transparent. The Proxmox mechanism of unbinding/rebinding devices between drivers is supposed to be seamless. But it assumes the target driver (nvidia) can actually claim and initialize the device after rebinding. If the driver doesn't support the hardware, rebinding is meaningless.

Assumption 3: Only the VM guest needs the open driver. The assistant's diagnosis focused entirely on the VM guest, because that's where the error messages appeared. The host's driver stack was never re-examined after the VM experiment began. This tunnel vision is understandable—the symptom ("no devices found" inside the VM) pointed to the VM's driver, and fixing it made the VM work. But the host's driver compatibility was a latent issue that only surfaced when the GPUs returned to the host.

Assumption 4: The switchback would "just work." The assistant had described the switching workflow optimistically ([msg 5759]): "Stop the VM → GPUs rebind to nvidia → start CT 129. No blacklisting or rebooting needed." This was correct in mechanism but wrong in outcome, because it assumed the NVIDIA driver on the host could handle the GPUs.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Proxmox PCI passthrough mechanics: How vfio-pci driver binding works, how hostpci mappings in VM configs trigger automatic driver switching, and the relationship between host kernel modules and LXC container device access.
  2. NVIDIA driver architecture: The distinction between the proprietary (nvidia.ko) and open (nvidia-open.ko) kernel modules, and critically, that Blackwell (SM120) GPUs require the open variant. This is a relatively recent change in the NVIDIA driver ecosystem—older GPUs (Ampere, Hopper) work with either.
  3. The conversation's hardware context: Eight RTX PRO 6000 Blackwell Server Edition GPUs, a Proxmox host with IOMMU enabled, an LXC container (CT 129) for production inference, and a VM (ID 131) for experiments.
  4. The prior optimization work: The extensive tuning of SGLang, EAGLE-3 speculative decoding, CUDA graphs, NCCL allreduce, and hierarchical KV cache that made the LXC deployment valuable enough to want to return to.

Output Knowledge Created

This message creates several pieces of critical knowledge:

  1. A documented failure mode for GPU passthrough workflows: The combination of Blackwell GPUs + Proxmox + LXC + VM creates a specific failure path where the host's driver type must match the GPU architecture, even if the GPUs are normally passed through to a VM.
  2. A gap in the switching procedure: The original plan assumed stopping the VM and starting the LXC container was sufficient. This message proves an additional step is needed: ensuring the host's NVIDIA driver supports the GPUs that will rebind to it.
  3. A diagnostic signal: "No supported GPUs were found" after a VM-to-host GPU handoff specifically points to a driver compatibility issue, not a PCIe enumeration problem or a VFIO configuration error. This narrows the troubleshooting space considerably.
  4. A priority signal for the assistant: The message implicitly asks for a fix. The assistant now needs to either install the open kernel modules on the host (requiring a host reboot) or find an alternative way to make the GPUs accessible to the LXC container.

The Thinking Process Revealed

While the user message itself is terse—just a command output pasted from the terminal—it reveals a specific thinking process on the user's part:

The user had just been told everything was working. The assistant confirmed all eight GPUs were visible inside the VM ([msg 5776]). The user then logically attempted to return to the previous working state: stop the VM experiment, go back to the LXC container where the production server normally runs. This is a natural "cleanup" step—the experiment is done, restore normal operations.

The user ran nvidia-smi as the first verification step. This is the standard "are my GPUs there?" command in any NVIDIA environment. Getting "No supported GPUs were found" is particularly jarring because it's different from "no devices found" or "driver not loaded." It means the driver is loaded and sees the devices, but refuses to claim them. This is a subtle but important diagnostic distinction that the user captured faithfully by pasting the exact output.

The fact that the user reported this immediately, without trying other diagnostic commands or rebooting, shows they recognized this as an unexpected failure that needed expert intervention. They didn't waste time guessing—they escalated to the assistant with the raw evidence.

Broader Implications

This message, for all its brevity, encapsulates a fundamental truth about GPU-accelerated infrastructure: the handoff between virtualization layers is never as clean as the documentation suggests. Each GPU generation adds new wrinkles—Blackwell's open-kernel requirement, the death of MIG on certain architectures, changes in NVLink topology, and so on. The Proxmox PCI passthrough mechanism is elegant in theory, but it depends on driver compatibility at both ends of the handoff, and those dependencies are easy to overlook when the GPUs are "working" in one environment.

The fix that this message implicitly calls for—installing nvidia-dkms-590-open on the Proxmox host—would require a host reboot, which would kill the running VM and any other workloads. This is a high-cost operation, which is why the assistant had been reluctant to suggest it earlier. Message &lt;msg id=5777&gt; makes it unavoidable.

In the end, this single line of terminal output—"Failed to initialize NVML: No supported GPUs were found"—represents the collision between two correct configurations (VM with open driver, host with proprietary driver) that together produce an incorrect system state. It's a reminder that in complex infrastructure, correctness is not compositional: two things that each work alone may not work together.