The Driver/Library Mismatch: A Single Diagnostic Command That Revealed a Container Migration Problem

The Message

The subject message is deceptively simple — a single bash command and its error output:

[assistant] [bash] ssh root@10.1.2.5 'pct exec 129 -- nvidia-smi' 2>&1
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 590.48

In just three lines, this message captures a moment of verification that unexpectedly reveals a fundamental incompatibility. The assistant had just finished migrating an LXC container (CT129) from a decommissioned host (kpro6) to a new host (kpro5). The container was started successfully, and the natural next step was to confirm that the NVIDIA GPUs were accessible inside the container. Instead of a clean nvidia-smi output showing two RTX A6000s, the assistant received a classic version mismatch error: the NVML library inside the container reported version 590.48, but the host kernel module was running version 580.126.09.

Why This Message Was Written

This message was written as a verification step — a sanity check that the entire migration effort had succeeded. To understand why this specific command was chosen, we need to trace the assistant's reasoning through the preceding messages.

The assistant had just completed a multi-step migration of the Qwen3.6-27B model deployment from kpro6 to kpro5. This involved: installing the NVIDIA driver 580.126.09 on the Proxmox host, unbinding two RTX A6000 GPUs from vfio-pci (which had been reserving them for virtual machines), updating the CT129 LXC configuration to reference the correct device nodes (nvidia0 and nvidia1 instead of the old nvidia0 through nvidia3), adjusting cgroup device permissions for the correct major numbers (195 for control, 502 for UVM, 505 for caps), and finally starting the container. Each step built on the previous one, and the container started successfully — pct status 129 returned "running."

At this point, the assistant had every reason to believe the migration was complete. The GPUs were visible on the host (nvidia-smi on the host showed two A6000s), the device nodes were properly bound into the container via LXC mount entries, and the container was running. The natural verification step was to run nvidia-smi inside the container to confirm that the GPU stack was fully operational from the application's perspective. This is the standard "does it work?" test for any NVIDIA GPU setup — if nvidia-smi runs successfully, the CUDA stack is functional.

The choice of pct exec 129 -- nvidia-smi rather than a more complex test is deliberate. pct exec runs a command directly inside a Proxmox LXC container, similar to lxc exec or docker exec. It's the simplest, most direct way to verify GPU accessibility from within the container. The assistant could have checked for device file presence (ls -la /dev/nvidia* inside the container), but nvidia-smi is a more comprehensive test — it validates not just that the device files exist, but that the entire NVIDIA Management Library (NVML) stack is functional, including the userspace-kernel interface.

The Assumption That Failed

The assistant made a reasonable but incorrect assumption: that the container's userspace NVIDIA libraries would be compatible with the host's kernel driver. This assumption was rooted in the migration workflow. The container CT129 had been originally set up on kpro6, which ran NVIDIA driver version 590.48. When the container was migrated to kpro5 (by moving its rootfs and reconfiguring the LXC settings), the container's filesystem — including all installed CUDA libraries, NVML libraries, and libcuda.so — came along unchanged. The host's kernel module, however, was now version 580.126.09.

The NVIDIA driver stack has a strict requirement: the userspace libraries (libcuda.so, libnvidia-ml.so, etc.) must match the kernel module version. A container running 590.48 userspace libraries cannot communicate with a 580.126.09 kernel module. This is a fundamental constraint of the NVIDIA driver architecture, and it's a common pitfall when migrating containers between hosts with different driver versions.

The assistant's assumption was not careless — it was a natural oversight in a complex migration. The container had been reconfigured for the new host's GPU topology (device nodes, cgroup permissions), the host driver was installed and working, and the container started without errors. The mismatch only manifested when the userspace libraries attempted to communicate with the kernel module, which is a runtime check that no earlier step would have caught.

Input Knowledge Required

To understand this message, one needs several pieces of contextual knowledge:

Proxmox LXC architecture: The pct exec command runs commands inside a Proxmox LXC container. LXC containers share the host kernel but have isolated filesystems. This means the container's userspace libraries are independent of the host's — they come from the container's own operating system image, not the host's.

NVIDIA driver stack layering: The NVIDIA GPU driver consists of a kernel module (nvidia.ko, nvidia-uvm.ko) loaded by the host kernel, and userspace libraries (libcuda.so, libnvidia-ml.so, libnvidia-ml.so — NVML) that applications link against. The kernel module and userspace libraries must be from the same driver version. When they mismatch, nvidia-smi (which uses NVML) fails with the "Driver/library version mismatch" error.

Container migration semantics: When an LXC container is migrated between hosts, its root filesystem is moved as-is. Any software installed inside the container — including NVIDIA driver userspace components — remains at their original versions. The container does not automatically adapt to the new host's driver version.

The error message format: "Failed to initialize NVML: Driver/library version mismatch / NVML library version: 590.48" — the error explicitly tells you which library version is present (590.48) and implies that the kernel module version differs. The kernel module version is not shown in this error, but from the context we know it's 580.126.09 (the driver installed on kpro5 in [msg 6764]).

Output Knowledge Created

This message created several critical pieces of knowledge:

The container has stale userspace libraries: The 590.48 NVML library version confirms that the container's CUDA stack was installed under the old kpro6 host's driver. The container needs its NVIDIA userspace libraries updated to match the new host's 580.126.09 driver.

The migration is not yet complete: Despite the container starting successfully and the host GPUs being functional, the application stack inside the container cannot use the GPUs. An additional step — updating the container's NVIDIA userspace libraries — is required before the Qwen3.6-27B model can be deployed.

The host driver installation is correct: The error specifically says "Driver/library version mismatch," not "Failed to initialize NVML: Cannot find device" or "No devices were found." This distinction is important — it means the host-side setup (device nodes, cgroup permissions, kernel module loading) is working correctly. The kernel module is loaded, the device files are accessible, and NVML can communicate with the kernel module — it just gets rejected due to version incompatibility. This narrows the problem space considerably.

A specific remediation path: The fix is well-understood: install the matching userspace NVIDIA libraries inside the container. On Ubuntu, this typically means adding the NVIDIA CUDA repository and installing cuda-drivers or libnvidia-compute-580 (or the equivalent package for the host driver version). Alternatively, the container could use the host's NVIDIA libraries via bind mounts, though this is less common for LXC.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the choice of this particular diagnostic command at this particular moment. The sequence of actions reveals a systematic verification approach:

  1. Install host driver → verify with nvidia-smi on host ([msg 6772])
  2. Configure LXC device nodes and cgroup permissions ([msg 6777])
  3. Start the container → verify with pct status ([msg 6778])
  4. Verify GPU access inside container → pct exec 129 -- nvidia-smi (this message) Step 4 is the logical culmination of the verification chain. Each step validates a different layer of the stack: the kernel module (step 1), the device access configuration (step 2), the container lifecycle (step 3), and the full userspace stack (step 4). The error at step 4 indicates that while layers 1-3 are correct, layer 4 (the container's userspace libraries) is out of sync. The assistant did not, for example, try to run python -c "import torch; print(torch.cuda.is_available())" or any other higher-level test. Running nvidia-smi is the minimal, most diagnostic test — it isolates the NVML layer without involving Python, PyTorch, or any application code. If nvidia-smi fails, there's no point testing higher-level frameworks.

Broader Significance

This message is a perfect illustration of a class of problems that arise in containerized GPU workloads, particularly in migration scenarios. The NVIDIA driver stack's strict kernel-userspace version coupling is a well-known source of friction in environments where containers move between hosts — Kubernetes clusters with heterogeneous GPU drivers, CI/CD pipelines that rebuild container images on different hardware, and Proxmox LXC migrations like this one.

The error message itself is famously unhelpful in its phrasing. "Driver/library version mismatch" sounds like a host-side configuration problem, but in container environments, it almost always means the container's userspace libraries are wrong. The NVML library version (590.48) is the key diagnostic clue — it tells you what version the container expects, and you can compare it to the host's nvidia-smi output to confirm the mismatch.

The assistant's response to this error (which would come in the next message) would need to address the container's userspace libraries. The options include: installing the matching driver packages inside the container, using the host's libraries via bind mounts (e.g., mounting /usr/lib/x86_64-linux-gnu/libcuda.so* from the host into the container), or rebuilding the container with the correct driver version baked in. Each approach has trade-offs in complexity, reproducibility, and maintenance burden.

This single diagnostic command, despite its brevity, encapsulates the entire challenge of maintaining GPU-accelerated containerized workloads across heterogeneous infrastructure. It's a reminder that in the world of NVIDIA GPUs, the kernel module is only half the story — the userspace libraries must follow suit, and containers carry their own library stacks wherever they go.