The Driver Mismatch: A Case Study in LXC Container GPU Migration

Introduction

In the world of machine learning infrastructure, few things are as deceptively simple—and as catastrophically disruptive—as a driver version mismatch. Message <msg id=6781> captures a pivotal moment in a complex migration of an AI serving stack from one Proxmox host to another. The assistant, having just started an LXC container (CT129) on the new kpro5 host, discovers that the container's NVIDIA userspace libraries are version 590.48 while the host kernel module is version 580.126.09. The result is a single, damning error: Failed to initialize NVML: Driver/library version mismatch. This message documents the first step in resolving that mismatch: purging the incompatible 590.x libraries from inside the container.

The Message

The subject message reads:

Ubuntu 24.04 inside, with 590.x libs. Need to replace with 580.x. Let me remove the old ones and install matching.

>

[bash] ssh root@10.1.2.5 'pct exec 129 -- bash -c "apt-get purge -y nvidia-utils-590 libnvidia-compute-590 libnvidia-decode-590 nvidia-firmware-590-590.48.01 nvidia-kernel-common-590 2>&1 | tail -5"' 2>&1

>

Processing triggers for libc-bin (2.39-0ubuntu8.7) ... (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Readi...

At first glance, this appears to be a straightforward package removal. But beneath the surface lies a cascade of decisions, assumptions, and infrastructure knowledge that makes this message a rich case study in real-world AI deployment.

Context: The Migration from kpro6 to kpro5

To understand why this message exists, we must trace the events that led to it. The assistant was tasked with migrating the Qwen3.6-27B model deployment from a decommissioned host (kpro6) to a new host (kpro5). This migration involved:

  1. Setting up the host: Installing NVIDIA driver 580.126.09 on kpro5, a Proxmox Virtual Environment (PVE) host running kernel 6.8.12-9-pve. This required blacklisting the open-source nouveau driver, installing matching Proxmox kernel headers, and running the NVIDIA .run installer.
  2. Unbinding GPUs from vfio-pci: The kpro5 host had five GPUs—two RTX A6000s and three RTX 3090s—all bound to the vfio-pci driver for VM passthrough. The assistant had to unbind only the two A6000s (leaving the 3090s for existing running VMs) and then re-run the NVIDIA installer so it could detect and bind them.
  3. Configuring the LXC container: The CT129 container, previously configured for kpro6's 4-GPU setup, needed its configuration updated for 2 GPUs. The assistant updated device mount entries, cgroup device permissions, and resource limits (32 cores, 64GB RAM).
  4. Starting the container: After starting CT129, the assistant ran nvidia-smi inside the container and hit the mismatch error.

Why This Message Was Written: The Reasoning and Motivation

The message exists because of a fundamental architectural constraint of NVIDIA's GPU stack: the kernel module (nvidia.ko) and the userspace libraries (libcuda.so, libnvidia-compute, etc.) must be version-matched. The kernel module is loaded by the host operating system, while the userspace libraries live inside the container. When the container was migrated from kpro6 (which had driver 590.48.01) to kpro5 (which has driver 580.126.09), the container's filesystem—including its NVIDIA userspace libraries—came along unchanged.

The error message NVML library version: 590.48 told the assistant exactly what was wrong: the container's libnvidia-ml.so (NVML) was from the 590.48 driver, but the host kernel module was 580.126.09. NVIDIA's driver architecture requires these to match because the userspace libraries communicate with the kernel module through a versioned interface. A mismatch means every API call fails.

The assistant's motivation was clear: restore GPU functionality inside the container as quickly as possible. The purge command was the first step in a two-phase fix: (1) remove the incompatible libraries, (2) install the matching 580.x libraries. The assistant chose to purge the old packages rather than overlay or ignore them because apt would refuse to install the new 580.x packages if conflicting 590.x packages were present.

How Decisions Were Made

Several decisions are embedded in this single command:

Decision 1: Purge vs. remove vs. overlay. The assistant chose apt-get purge -y rather than apt-get remove (which leaves configuration files) or manual file deletion. Purge is the most thorough option, removing both packages and their configuration files. This was the right call because the container had stale configuration from the kpro6 era that could interfere with the new driver installation.

Decision 2: Which packages to purge. The assistant explicitly listed five packages:

Assumptions Made

The message reveals several assumptions:

Assumption 1: The 580.x packages exist in Ubuntu's repositories. The assistant assumed that Ubuntu 24.04's package repositories would have NVIDIA 580.x userspace packages available. This was not guaranteed—NVIDIA driver versions in Ubuntu's repos often lag behind the latest releases, and 580.126.09 was a relatively new driver. The next message ([msg 6782]) shows the assistant checking this assumption by searching for 580.x packages, finding only -server variants (e.g., libnvidia-compute-580-server), which may or may not be compatible with the non-server driver.

Assumption 2: The container has network access to Ubuntu repos. The apt-get purge command succeeded, confirming the container had working network access and functional apt infrastructure. If the container had been offline or misconfigured, the purge would have failed.

Assumption 3: Purging these packages won't break dependencies for other software. The assistant assumed that no other installed software depended on the 590.x NVIDIA packages. This was a reasonable assumption for a container dedicated to ML inference, but it was untested.

Assumption 4: The host's NVIDIA kernel module (580.126.09) is stable and correct. The assistant had successfully installed the driver on the host and verified it with nvidia-smi ([msg 6772]), showing both A6000 GPUs. The assumption was that the driver installation was complete and correct.

Mistakes or Incorrect Assumptions

Potential mistake: Purging nvidia-kernel-common-590. This package may have contained configuration files or udev rules that the container needed for device node management. However, since the container was using bind-mounted device nodes from the host (via lxc.mount.entry), this was likely safe—the host handles device node creation, not the container.

Potential oversight: Not checking for nvidia-utils-580 availability first. The assistant purged the 590.x packages before confirming that 580.x packages were available in the Ubuntu repos. If the repos only had -server variants (as suggested by msg [msg 6782]), the assistant would need an alternative installation method (e.g., extracting libraries from the host's .run installer). The purge was not irreversible—the container could still use the host's libraries via bind mounts—but it did remove the fallback option of the old 590.x libraries.

Assumption that proved partially incorrect: Ubuntu repos would have matching 580.x packages. The next message reveals that only -server variants of 580.x were found (libnvidia-compute-580-server, etc.). The -server packages are designed for NVIDIA's data center driver branch and may not be fully compatible with the GeForce/RTX driver (580.126.09) running on the host. This forced the assistant to consider alternative approaches.

Input Knowledge Required

To understand this message, one needs:

  1. NVIDIA driver architecture: Knowledge that the NVIDIA GPU stack has two components—a kernel module (loaded by the OS) and userspace libraries (used by applications)—and that they must be version-matched. This is a common source of confusion for developers new to GPU computing.
  2. LXC container technology: Understanding that LXC containers share the host's kernel but have their own filesystem. Device nodes like /dev/nvidia0 can be bind-mounted from the host, but the userspace libraries (.so files) must be present inside the container's filesystem.
  3. Proxmox PVE specifics: Knowledge of pct exec for running commands inside LXC containers, the lxc.mount.entry configuration syntax, and the cgroup device permission system (lxc.cgroup2.devices.allow).
  4. Package management on Ubuntu: Understanding apt-get purge vs. remove, package naming conventions for NVIDIA drivers (e.g., libnvidia-compute-590 vs. nvidia-utils-590), and the relationship between driver version numbers and package names.
  5. The migration context: Awareness that CT129 was migrated from kpro6 (driver 590.48.01) to kpro5 (driver 580.126.09), carrying its old userspace libraries with it.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. The purge succeeded: The truncated output shows Processing triggers for libc-bin, confirming that apt-get purge completed without errors. The container's 590.x NVIDIA packages have been removed.
  2. The container is now without NVIDIA userspace libraries: Immediately after this command, the container has no NVIDIA libraries. Any attempt to run GPU-accelerated applications will fail until matching libraries are installed.
  3. The path forward is constrained: The assistant has committed to installing 580.x libraries. The next step is to find and install them, which may require using Ubuntu's repos (if available), extracting from the host's .run installer, or using an alternative source.
  4. The container's package state is now clean: The rc (removed but config files remain) state for the 565.x packages and the now-purged 590.x packages means the container has no conflicting NVIDIA packages. This is a prerequisite for a clean installation.

The Thinking Process Visible in the Message

The assistant's reasoning is telegraphically clear in the message's opening line: "Ubuntu 24.04 inside, with 590.x libs. Need to replace with 580.x." This two-sentence diagnosis captures:

  1. Observation: The container OS is Ubuntu 24.04 (confirmed by cat /etc/os-release in the previous message).
  2. Observation: The installed NVIDIA libraries are version 590.x (confirmed by dpkg -l | grep nvidia).
  3. Inference: These must be replaced with 580.x to match the host's kernel module.
  4. Action plan: "Let me remove the old ones and install matching." The thinking is linear and pragmatic: diagnose the mismatch, identify the incompatible components, remove them, and prepare for replacement. There is no exploration of alternative approaches (e.g., bind-mounting the host's libraries, using a container with matching libraries, or downgrading the host driver). The assistant commits to the simplest path: make the container match the host. The command construction itself reveals careful thinking. The assistant uses pct exec 129 -- bash -c "..." to run inside the container, with the entire command quoted for the outer SSH session. The package list is explicitly enumerated rather than using a wildcard (e.g., *590*), which could have caught unintended packages. The tail -5 output filter shows awareness of output volume—the assistant knows apt-get purge produces verbose progress output and only needs the final lines.

Broader Implications

This message, while small, illustrates a recurring challenge in ML infrastructure: the tension between container portability and hardware-specific dependencies. Containers promise "build once, run anywhere," but NVIDIA's driver version matching requirement means that a container built on one host may not work on another host with a different driver version. This is a fundamental limitation of the current GPU virtualization model.

The migration from kpro6 to kpro5 was necessitated by hardware changes (kpro6 was being decommissioned), but the driver versions differed because the two hosts had different GPU architectures (kpro6 had Blackwell GPUs requiring driver 590.x, while kpro5 has Ampere A6000s running driver 580.x). The container, designed to be portable, carried driver-specific libraries that tied it to its original host.

This is why many production ML deployments use container images that are rebuilt for each target environment, or use host-mounted NVIDIA libraries rather than container-bundled ones. The assistant's approach—purging and reinstalling—is a pragmatic middle ground: modify the container in place rather than rebuilding from scratch.

Conclusion

Message <msg id=6781> is a small but critical step in a complex infrastructure migration. It demonstrates the practical challenges of GPU-accelerated container deployment, where a version mismatch between a kernel module and a userspace library can bring an entire AI serving stack to a halt. The assistant's response is swift and methodical: diagnose, identify, purge, and prepare for replacement. While not all assumptions proved correct (the Ubuntu repos only had -server variants of 580.x), the purge was a necessary first step that cleared the way for alternative installation methods. This message serves as a microcosm of the broader migration effort—a reminder that in GPU computing, the smallest version numbers can have the largest consequences.