The Bridge Between Driver and Deployment: Configuring GPU Passthrough for an LXC Container
Introduction
In the sprawling ecosystem of modern AI infrastructure, the moment when raw hardware transitions into a usable compute resource is often invisible—a silent handoff between kernel modules and container runtimes that passes without ceremony. Yet this transition is where the most brittle assumptions live, where device major numbers must align with cgroup permissions, where a single missing --optional flag can strand a 27-billion-parameter language model without GPU access. Message <msg id=6774> captures exactly this moment: the assistant, having just finished installing the NVIDIA driver 580.126.09 on the kpro5 host and verified that two RTX A6000 GPUs are visible via nvidia-smi, now pivots to the next critical task—reconfiguring the LXC container CT129 so that the Qwen3.6-27B model deployment can actually use those GPUs.
The Message
The subject message is concise and action-oriented:
Now update CT129 config. Currently it has nvidia0-3 (from kpro6's 4-GPU setup). We only have 2 GPUs here, so we need nvidia0 and nvidia1. Let me also check the device major/minor numbers.
>
```bash ssh root@10.1.2.5 'ls -la /dev/nvidia* && echo "---" && nvidia-smi -L' 2>&1 crw-rw-rw- 1 root root 195, 0 May 9 10:42 /dev/nvidia0 crw-rw-rw- 1 root root 195, 1 May 9 10:42 /dev/nvidia1 crw-rw-rw- 1 root root 195, 255 May 9 10:42 /dev/nvidiactl crw-rw-rw- 1 root root 502, 0 May 9 10:42 /dev/nvidia-uvm crw-rw-rw- 1 root root 502, 1 May 9 10:42 /dev/nvidia-uvm-tools
>
/dev/nvidia-caps: total 0 drwxr-xr-x 2 root root 80 May 9 10:42 . drwxr-xr-x 21 root root 6380 May 9 10:42 .. cr-------- 1 root root 505, 1 May 9 10:42 /dev/nvidia-cap1 cr--r--r-- 1 root r... ```
At first glance, this reads as a straightforward verification step: check that the device nodes exist, note their major/minor numbers, and proceed. But beneath the surface, this message represents a delicate bridge between two fundamentally different GPU ownership models—the raw host-level access that the NVIDIA driver provides, and the mediated, permission-gated access that an LXC container requires.
Context and Motivation
To understand why this message exists, we must trace the arc of the preceding twenty-five messages. The kpro5 host (10.1.2.5) was being prepared to replace the decommissioned kpro6 as the deployment target for Qwen3.6-27B, a dense 27-billion-parameter model requiring approximately 55 GB of VRAM in BF16 precision. The host had two RTX A6000 GPUs (48 GB each, Ampere architecture) that were previously bound to the vfio-pci driver for virtual machine passthrough. Messages <msg id=6749> through <msg id=6773> covered the full installation pipeline: installing Proxmox kernel headers, blacklisting the Nouveau open-source driver, downloading and compiling the NVIDIA 580.126.09 driver via DKMS, unbinding the A6000s from vfio-pci, and finally confirming that nvidia-smi reported both GPUs with CUDA 13.0 support.
With the driver installed and GPUs visible, the assistant now faced a new problem. The CT129 LXC container—a privileged Ubuntu container with 128 cores, 260 GB of memory, and an 800 GB root filesystem—had been configured for a different machine (kpro6) with four GPUs. Its configuration file still contained mount entries for /dev/nvidia2 and /dev/nvidia3, devices that simply did not exist on kpro5. Before the container could be started, its configuration needed to be pruned to match the two-GPU reality of the new host.
This is the core motivation for <msg id=6774>: the assistant is performing a reconciliation between the container's declared hardware dependencies and the actual hardware available. It is not installing software, not debugging a crash, not tuning performance—it is aligning configuration with physical reality, a task that is simultaneously trivial and absolutely unforgiving of mistakes.## The Device Node Landscape
The output of ls -la /dev/nvidia* reveals a carefully orchestrated hierarchy of device nodes that the NVIDIA driver creates during its initialization. Each node is a character device with a specific major number that identifies the kernel driver, and a minor number that identifies the specific resource within that driver's domain.
The major number 195 corresponds to the NVIDIA proprietary driver's control interface. /dev/nvidia0 and /dev/nvidia1 are the compute devices—one per physical GPU—while /dev/nvidiactl (minor 255) is the control device used by the CUDA runtime for driver version negotiation and context management. The major number 502 belongs to the NVIDIA Unified Memory (UVM) driver, which manages the shared virtual memory space between CPU and GPU. /dev/nvidia-uvm is the primary UVM interface, and /dev/nvidia-uvm-tools is a debugging and profiling auxiliary. The /dev/nvidia-caps/ directory contains capability descriptors (major 505) that expose GPU hardware capabilities to userspace without requiring direct device access.
This structure encodes a critical assumption: the NVIDIA driver expects to own these major numbers globally. When an LXC container is configured to access these devices, the container's cgroup device controller must explicitly allow character devices with these major numbers. This is why the CT129 config contains lxc.cgroup2.devices.allow: c 195:* rwm—it grants read-write-mknod access to all minor devices under major 195. Without this line, the container's processes would receive "permission denied" errors when attempting to open CUDA contexts, even if the device nodes are bind-mounted.
The assistant's decision to run ls -la /dev/nvidia* is not merely a sanity check. It is a deliberate act of knowledge acquisition: the major numbers must be confirmed because they can vary between driver versions or kernel configurations. The output confirms major 195 for control devices, 502 for UVM, and 505 for caps—consistent with the standard NVIDIA driver layout. This confirmation allows the assistant to proceed with confidence that the existing cgroup permission line (c 195:* rwm) in CT129's config remains valid.
The Assumption of Continuity
A subtle but important assumption underlies this message: that the CT129 container's internal CUDA installation is compatible with the host's driver version 580.126.09. The container will inherit the host's device nodes via bind mounts, but it must have matching userspace libraries (libcuda.so, libnvidia-ml.so, etc.) installed inside the container image. The assistant does not verify this in <msg id=6774>—it assumes that because CT129 was previously running on kpro6 with a similar driver stack, the userspace libraries are either already present or will be installed later.
This assumption is reasonable but not risk-free. If the container's CUDA toolkit version mismatches the host driver's CUDA version (13.0 in this case), the containerized applications may fail with cryptic "CUDA driver version is insufficient" errors. The assistant is implicitly betting that the container's Ubuntu image will either have compatible libraries or that the mismatch will be caught and fixed in a subsequent step.
The Thinking Process
The message reveals the assistant's mental model through its structure. The first sentence—"Now update CT129 config. Currently it has nvidia0-3 (from kpro6's 4-GPU setup). We only have 2 GPUs here, so we need nvidia0 and nvidia1"—establishes the delta between the current state (a config designed for four GPUs) and the desired state (a config for two GPUs). The assistant has already read the CT129 config in a previous round (visible in the context as <msg id=6775>'s predecessor) and knows exactly which lines need to change.
The second sentence—"Let me also check the device major/minor numbers"—is a defensive maneuver. The assistant could have simply assumed that the major numbers match the previous installation on kpro6, but it chooses to verify. This is characteristic of robust infrastructure automation: never assume that device numbering is stable across driver installations, kernel versions, or hardware platforms. The verification costs only a single SSH command but could save hours of debugging if the numbers had changed.
The bash command itself is carefully composed. It uses ls -la /dev/nvidia* to show both the device nodes and their permissions, then nvidia-smi -L to list the GPUs by name. The && chaining ensures that if ls fails (indicating no NVIDIA devices at all), the nvidia-smi command won't execute—avoiding a potentially confusing error cascade. The output is truncated in the message (the final line cuts off with "cr--r--r-- 1 root r..."), but the essential information—the major/minor pairs for nvidia0, nvidia1, nvidiactl, nvidia-uvm, and nvidia-uvm-tools—is fully captured.
What This Message Creates
The output of this message is knowledge: a confirmed mapping of device nodes to their major/minor numbers, and a verified list of available GPUs. This knowledge feeds directly into the next action—editing the CT129 configuration file to remove the stale nvidia2 and nvidia3 mount entries, and ensuring the cgroup permission line covers the correct major numbers.
But the message also creates a subtle form of confidence. By verifying the device nodes before modifying the container config, the assistant reduces the risk of a blind edit. If the container were started with a config referencing nonexistent devices, the LXC runtime would either fail to start or silently skip the missing bind mounts—both outcomes that could lead to confusing partial failures later. The verification step ensures that the config edit is grounded in observed reality rather than remembered assumptions.
The Broader Pattern
This message exemplifies a recurring pattern in infrastructure work: the verification-before-modification cycle. Before changing any configuration, the assistant checks the current state of the relevant system. This is the operational equivalent of "measure twice, cut once." In the context of the larger session, this pattern appears repeatedly—checking GPU availability before installing the driver, checking driver compilation logs before loading the module, checking device nodes before editing the container config.
The pattern is particularly important in heterogeneous environments like Proxmox, where a single host may manage GPUs across multiple VMs and containers simultaneously. The kpro5 host has five GPUs total (two A6000s and three RTX 3090s), with the 3090s bound to vfio-pci for running virtual machines. The assistant must be precise about which GPUs are being configured for the container and which must remain untouched. A mistake here—unbinding a 3090 from vfio-pci, or binding an A6000 to the wrong container—could disrupt production VMs.
Conclusion
Message <msg id=6774> is a small but essential gear in a much larger machine. It does not install software, train models, or serve inference requests. It simply checks that the device nodes exist and records their numbers. Yet without this step, the entire deployment of Qwen3.6-27B on kpro5 would be built on an unverified foundation. The message reminds us that the most critical infrastructure work is often the least glamorous: the careful, deliberate verification of basic assumptions before proceeding to more complex tasks. In a world of 27-billion-parameter models and speculative decoding pipelines, the humble ls -la /dev/nvidia* remains an indispensable tool.