The Moment of Truth: Verifying GPU Stack Coherence in an LXC Migration

[assistant] [bash] ssh root@10.1.2.5 'pct exec 129 -- nvidia-smi' 2>&1
Sat May  9 08:44:28 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|       ...

At first glance, message [msg 6786] appears to be nothing more than a routine health check — an assistant running nvidia-smi inside an LXC container and receiving the standard GPU status table. But in the context of the surrounding conversation, this single command represents a critical verification milestone: the successful culmination of a complex, multi-step infrastructure migration that involved unbinding GPUs from virtual machine passthrough, installing a matching NVIDIA driver on a Proxmox host, resolving a version mismatch between kernel modules and userspace libraries, and correctly configuring device passthrough into an unprivileged container. The message is brief, but the chain of dependencies it validates is anything but.

The Context: Migrating a GPU Workload Between Hosts

The conversation leading up to this message documents a migration of the CT129 LXC container from a decommissioned host (kpro6) to a new host (kpro5). This is a common but nontrivial operation in GPU-accelerated environments, particularly when the hardware configuration differs between hosts. Kpro5 has a different GPU topology than kpro6 — two RTX A6000s alongside three RTX 3090s, rather than the four-GPU setup the container was originally configured for. The 3090s are already in use by running virtual machines, so only the A6000s are available for the container.

The assistant's first task is to prepare the host. Messages [msg 6759] through [msg 6773] document the installation of NVIDIA driver 580.126.09 on the Proxmox host, including the critical step of unbinding the two A6000 GPUs from the vfio-pci driver (which had them reserved for VM passthrough) so the native NVIDIA kernel module can claim them. This is a delicate operation on a live hypervisor — the 3090s must remain bound to vfio-pci for the running VMs, while the A6000s need to be released to the NVIDIA driver. The assistant accomplishes this by writing directly to the PCI driver sysfs interface (/sys/bus/pci/drivers/vfio-pci/unbind), a low-level operation that requires precise knowledge of the PCI bus addresses and IOMMU group topology.

The Version Mismatch Problem

Once the host driver is installed and working, the assistant updates the CT129 LXC configuration ([msg 6775]-[msg 6777]) to reflect the new GPU count — removing references to nvidia2 and nvidia3 that no longer exist, updating the cgroup device major numbers (which differ between kernel versions), and adjusting resource allocations. The container is started successfully ([msg 6778]), but the first attempt to verify GPU access inside the container fails with a classic error ([msg 6779]):

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 590.48

This is one of the most common and frustrating issues in GPU-accelerated Linux environments. The NVIDIA driver stack consists of two components: the kernel module (nvidia.ko, nvidia-uvm.ko, etc.) which runs in kernel space and is loaded by the host, and the userspace libraries (libnvidia-compute, libnvidia-ml.so, nvidia-smi, etc.) which run in user space and communicate with the kernel module via device nodes. These two components must have exactly matching versions. In this case, the host has the newly installed kernel module at version 580.126.09, but the container inherited the old userspace libraries at version 590.48 from its previous life on kpro6.

The mismatch arises because LXC containers share the host's kernel but have their own filesystem. The NVIDIA device nodes (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm) are passed through from the host, and the kernel module is the host's, but the userspace libraries that nvidia-smi and other tools link against live inside the container's filesystem. When the container was migrated, its root filesystem still contained the old 590.48 libraries — a perfect trap for anyone who assumes that "the driver is installed on the host, so the container should work."

Diagnosing and Repairing the Stack

The assistant's response to the mismatch is methodical. First, it inspects the container's installed NVIDIA packages ([msg 6780]), finding the remnants of the 590.48 installation alongside an orphaned 565.x package. It then purges the old 590-series packages ([msg 6781]) and searches for the matching 580.126.09 versions in the Ubuntu package repository ([msg 6782]-[msg 6784]).

A subtle dependency issue emerges: the libnvidia-compute-580 and nvidia-utils-580 packages depend on a virtual package nvidia-kernel-common-580-580.126.09, which has no installation candidate in the repository. The assistant works around this by installing nvidia-kernel-common-580 first at the exact version pin, then installing the compute and utils packages ([msg 6785]). This is a pragmatic solution to what is essentially a packaging metadata issue — the virtual package name doesn't resolve, but the concrete packages can be installed by specifying the version explicitly.

The Verification

Message [msg 6786] is the payoff. The assistant runs nvidia-smi inside the container again, and this time it succeeds. The output shows:

What This Message Reveals About the System

This message is a verification milestone — a point where the assistant confirms that a complex chain of dependencies is correctly assembled. The chain includes:

  1. Host-level kernel module: The NVIDIA kernel driver (580.126.09) must be loaded and functional on the Proxmox host.
  2. Device unbinding: The GPUs must be released from vfio-pci and claimed by the NVIDIA driver.
  3. Device node creation: The host's udev rules must create /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, and /dev/nvidia-uvm with the correct major/minor numbers.
  4. LXC configuration: The container config must specify the correct cgroup device permissions (c 195:* rwm, c 502:* rwm, c 505:* rwm) and mount entries for each device node.
  5. Container userspace libraries: The container's libnvidia-compute, nvidia-utils, and related packages must match the host kernel module version exactly.
  6. NVML initialization: The NVML library must be able to open the device nodes and communicate with the kernel module without version mismatch errors. Any one of these links breaking would cause nvidia-smi to fail, and the error messages produced by failures at different levels are often indistinguishable to an inexperienced operator. The "Driver/library version mismatch" error from [msg 6779] could easily be mistaken for a host-level driver problem, leading an operator down the wrong troubleshooting path.

Assumptions and Knowledge Required

To understand and execute this migration correctly, the assistant relies on several key pieces of knowledge:

Output Knowledge Created

This message produces concrete, actionable knowledge:

  1. Confirmation that the GPU stack is coherent: The container has working GPU access with matching driver versions.
  2. The exact driver version and CUDA version: 580.126.09 with CUDA 13.0, which determines what CUDA toolkit and PyTorch versions are compatible going forward.
  3. The GPU count and model: Two RTX A6000s (48GB each), which constrains what models can be deployed and what tensor parallelism configurations are possible.
  4. A validated migration procedure: The sequence of steps (unbind GPUs → install host driver → update LXC config → install matching userspace libs → verify) is now a documented, repeatable process for future migrations. The message also implicitly validates the assistant's troubleshooting methodology. The initial failure at [msg 6779] was correctly diagnosed as a userspace library version mismatch rather than a kernel module problem, the host driver installation, and the container configuration were all correct. The fix targeted the right layer of the stack.

The Broader Significance

In the context of the full session (Segment 43, Chunk 0), this migration is the enabling step for deploying the Qwen3.6-27B model with speculative decoding on the kpro5 host. The two A6000 GPUs, each with 48GB of VRAM, are sufficient for a 27B-parameter BF16 model with MTP speculation (as the assistant later confirms, achieving 73.5 tok/s). Without this verification step, none of the subsequent work — the DFlash investigation, the DDTree benchmarks, the hidden state extraction pipeline — would be possible.

The message also illustrates a broader truth about infrastructure engineering: the most important commands are often the simplest. nvidia-smi is the first thing any GPU operator learns to run, but its output encodes an enormous amount of information about the health of the system. A successful nvidia-smi inside a container means the kernel module loaded, the device nodes were created, the cgroup permissions were granted, the bind mounts worked, the userspace libraries matched, and NVML initialized without error. It is the moment of truth for a whole chain of dependencies, and when it works, it means the foundation is solid enough to build upon.