The Missing Dependency: Tracing an NVIDIA Version Mismatch Across Host and Container

In the complex dance of deploying large language models on GPU-accelerated infrastructure, few problems are as deceptively simple—and as frustrating—as a driver version mismatch. Message 6784 captures a pivotal diagnostic moment in an ongoing effort to migrate a Qwen3.6-27B model deployment to a new Proxmox host. The assistant, having successfully installed the NVIDIA 580.126.09 driver on the host and unbounded two RTX A6000 GPUs from vfio-pci for container passthrough, finds the LXC container unable to initialize NVML. The error message is terse: "Failed to initialize NVML: Driver/library version mismatch." The host kernel module reports version 580.126.09, but the container's userspace libraries are from an older 590.48 installation, left over from the previous host.

The immediate response is straightforward: install matching userspace libraries inside the container. But when the assistant attempts to install libnvidia-compute-580=580.126.09-0ubuntu0.24.04.2 and nvidia-utils-580=580.126.09-0ubuntu0.24.04.2, the package manager rejects the request with an unmet dependency: these packages require nvidia-kernel-common-580-580.126.09, which is nowhere to be found. This is where message 6784 enters the narrative.

The Diagnostic Command

The assistant executes a single, carefully constructed command:

ssh root@10.1.2.5 'pct exec 129 -- bash -c "apt-cache policy nvidia-kernel-common-580-580.126.09 2>/dev/null && echo --- && apt-cache policy nvidia-kernel-common-580 2>/dev/null"'

This command chains three layers of remote execution: an SSH connection to the Proxmox host, a pct exec to run inside the LXC container (ID 129), and a bash -c subshell to execute two apt-cache policy queries. The 2>/dev/null redirects suppress any stderr noise, ensuring only the clean policy output is returned. The && operator ensures both queries run sequentially, separated by a visual --- delimiter.

The apt-cache policy command is the apt equivalent of asking "what versions of this package are available, and which one would be installed by default?" It queries all configured repositories and displays the installation candidate along with all available versions and their priority weights. This is the standard diagnostic tool for understanding why a package cannot be installed.

What the Output Reveals

The output delivers a clear diagnosis:

For nvidia-kernel-common-580-580.126.09 (the package with the exact version baked into the name):

libnvidia-compute-580 : Depends: nvidia-kernel-common-580-580.126.09
nvidia-utils-580 : Depends: nvidia-kernel-common-580-580.126.09

The assistant recognized this as a missing-dependency problem and immediately moved to investigate the availability of that specific package. The apt-cache policy command is the correct diagnostic tool for this situation. But the assistant did not stop at checking the missing package—it also queried the generic nvidia-kernel-common-580 package in the same command. This second query was the key insight: by checking both the version-encoded package name and the generic package name, the assistant could determine whether the dependency was truly unavailable or merely packaged under a different naming convention.

This dual-query approach reflects an understanding of NVIDIA's packaging history. NVIDIA has shifted between packaging strategies across driver versions, sometimes bundling kernel common files into a version-specific package name and sometimes using a single package with versioned releases. The assistant's suspicion that the Ubuntu archive might not have the exact version-specific package, but that the NVIDIA CUDA repository might offer it under the generic name, proved correct.

Input Knowledge Required

To understand this message, one needs familiarity with several layers of infrastructure:

Proxmox LXC containers: The pct exec 129 -- bash -c syntax executes a command inside a specific container. The assistant is running inside the host's shell (via SSH) and using Proxmox's container management tool to reach the container's environment.

NVIDIA driver architecture: The NVIDIA driver has two components—the kernel module (nvidia.ko, nvidia-uvm.ko) loaded by the host, and userspace libraries (libcuda.so, libnvidia-compute, nvidia-smi, etc.) used by applications. These must match exactly. The kernel module version is baked at compile time, and the userspace libraries check the kernel module's version string at initialization. Any mismatch causes NVML initialization failure.

Ubuntu and NVIDIA packaging: The Ubuntu archive packages NVIDIA libraries under versioned names like libnvidia-compute-580 with Ubuntu-specific version strings. The NVIDIA CUDA repository uses a different scheme with -1ubuntu1 suffixes. The nvidia-kernel-common-580-580.126.09 package is a metapackage that exists only in the Ubuntu archive's packaging scheme, while the NVIDIA repo's nvidia-kernel-common-580 serves the same role under a different name.

apt-cache policy semantics: This command shows the installation candidate (the version apt would install by default) and all available versions with their priorities. A "(none)" candidate with an empty version table means the package is not available from any configured repository at all.

The Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The Ubuntu archive does not contain nvidia-kernel-common-580-580.126.09. This package name, which the libnvidia-compute-580 and nvidia-utils-580 packages from the Ubuntu archive depend on, simply does not exist. This means those Ubuntu archive packages cannot be installed on this system without additional repository configuration.
  2. The NVIDIA CUDA repository does contain nvidia-kernel-common-580 version 580.126.09-1ubuntu1. This is the functional equivalent of the missing package, packaged under NVIDIA's own naming convention. It is available with priority 600, meaning apt would prefer it if the package name matched.
  3. The solution path is now clear: Install nvidia-kernel-common-580 from the NVIDIA CUDA repository first, which will satisfy the dependency, then install libnvidia-compute-580 and nvidia-utils-580. Alternatively, install all userspace components directly from the NVIDIA CUDA repository rather than the Ubuntu archive, avoiding the dependency chain entirely.
  4. The container's repository configuration is intact: The NVIDIA CUDA repository at https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 is correctly configured inside the container and returning package metadata. This confirms that the repository setup survived the container migration from kpro6 to kpro5.

Assumptions and Their Validity

The assistant makes several assumptions in this message. First, it assumes that the NVIDIA CUDA repository's nvidia-kernel-common-580 package at version 580.126.09-1ubuntu1 is functionally compatible with the Ubuntu archive's libnvidia-compute-580 and nvidia-utils-580 packages. This is a reasonable assumption—both packages are from NVIDIA, targeting the same driver version, and the version strings are aligned (580.126.09). However, there is a subtle risk: the Ubuntu archive packages might have been compiled against specific library versions or file paths that differ from the NVIDIA repo's kernel-common package. In practice, this usually works, but it is not guaranteed.

Second, the assistant assumes that the version 580.126.09-1ubuntu1 from the NVIDIA repo exactly matches the host kernel module's 580.126.09. The host driver was installed from the NVIDIA .run installer, which produces kernel module version 580.126.09. The NVIDIA repo's -1ubuntu1 suffix indicates a packaging revision, not a driver version change, so the kernel module interface should be identical. This assumption is almost certainly correct.

Third, the assistant implicitly assumes that the container's apt cache is up to date. The apt-cache policy command reads from the locally cached package metadata. If the cache is stale, the output might not reflect the latest repository state. However, the presence of version 580.126.09-1ubuntu1 with priority 600 suggests the cache is current—the NVIDIA repo updates frequently, and an outdated cache would likely show older versions or nothing at all.