The Verification That Mattered: Confirming GPU Passthrough in an LXC Container

In the sprawling, multi-session effort to deploy the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments were as quietly pivotal as a single bash command executed at message index 457. The message itself is deceptively simple — a two-line shell command followed by its output listing device nodes. But this message represents the first successful checkpoint in an entirely new architectural approach, one born from the failure of every previous strategy. It is the moment where the assistant confirms that the LXC container can see the GPUs, and by doing so, validates a workaround that had been built from scratch over the preceding dozen messages.

The Message in Full

The assistant executed the following command on the Proxmox host:

ssh root@10.1.2.6 "pct status 129 && pct exec 129 -- ls -la /dev/nvidia* 2>&1"

And received the response:

status: running
crw-rw-rw- 1 root root 504,   0 Feb 19 03:34 /dev/nvidia-uvm
crw-rw-rw- 1 root root 504,   1 Feb 19 03:34 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Feb 19 03:34 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Feb 19 03:34 /dev/nvidia1
crw-rw-rw- 1 root root 195,   2 Feb 19 03:34 /dev/nvidia2
crw-rw-rw- 1 root root 195,   3 Feb 19 03:34 /dev/nvidia3
crw-rw-rw- 1 root root 195,   4 Feb 19 03:34 /dev/nvidia4
crw-rw-rw- 1 root root 195,   5 Feb 19 03:34 /dev/nvidia5
crw-rw-rw- 1 root root 195,   6 Feb 19 03:34 /dev/nvidia6
crw-rw-rw- 1 root root 195,   7 Feb 19 03:34 /dev/nvidia7
crw-rw-rw- 1 root root 195, 255 Feb 19 03:34 /dev/nvidiactl

Eleven device nodes. Eight GPUs, one control device, one UVM (Unified Virtual Memory) device, and one UVM tools device. All present, all with the correct major numbers matching the host. The container was running, and the GPUs were visible from inside it.

Why This Message Was Written: The Critical Checkpoint

To understand why this verification was necessary, we must trace the reasoning that led to this moment. The session had been wrestling with a fundamental performance bottleneck: the eight Blackwell GPUs, when passed through to a KVM virtual machine via VFIO, appeared with a PHB (PCIe Host Bridge) topology in nvidia-smi topo -m. This meant that Peer-to-Peer (P2P) DMA between GPUs — essential for tensor-parallel inference at scale — would route through the host bridge rather than directly between GPUs, incurring severe latency penalties. The VM approach, despite successfully running the model, was fundamentally crippled by the virtualization layer's IOMMU groups.

The assistant and user had explored numerous remedies: modifying Proxmox host kernel parameters, migrating the VM from i440FX to Q35 chipset, fixing BAR allocation with pci=realloc, and attempting ACS (Access Control Services) disable to merge IOMMU groups. All failed. The hardware topology was immutable — each GPU sat on its own PCIe root complex, and no amount of kernel parameter tuning could change that.

This led to a strategic pivot: instead of trying to fix P2P within the VM, why not eliminate the VM entirely? An LXC container, running directly on the Proxmox host, would see the bare-metal GPU topology without any VFIO/IOMMU abstraction. The GPUs would appear as they do on a native Linux installation, with NODE links between GPUs on the same NUMA node and SYS links across NUMA boundaries — exactly the topology needed for efficient P2P communication.

But this approach required something the host did not have: an NVIDIA driver. The Proxmox host ran the stock PVE kernel (6.8.12-9-pve) with no GPU drivers installed — the GPUs had previously been bound to VFIO for passthrough to the VM. So the assistant embarked on a rapid setup sequence: installing build-essential and pve-headers, blacklisting the nouveau driver, downloading the 590.48.01 NVIDIA driver (the same version that worked in the VM), and installing it with DKMS for kernel persistence. All eight GPUs appeared on the host, and nvidia-smi topo -m confirmed the coveted bare-metal topology.

The next step was configuring the LXC container. The existing container 129 (llm-two) was unprivileged, which would complicate GPU device access. The assistant converted it to privileged mode, rewrote its configuration file with cgroup device permissions and bind-mount entries for every NVIDIA device node, and started it. Message 456 was the start command. Message 457 — the subject of this article — was the first verification that the entire chain had worked.

How Decisions Were Made

The structure of the command reveals the assistant's decision-making process. The use of && is deliberate: pct status 129 runs first, and only if it reports "running" does the pct exec command execute. This is a defensive pattern — if the container had failed to start (due to a configuration error, a missing device node on the host, or a ZFS uid mapping issue from the unprivileged-to-privileged conversion), the error would be immediately visible without attempting to execute inside a non-running container.

The choice of pct exec over SSH is equally intentional. At this point, the assistant had just switched the container from unprivileged to privileged mode, which shifts the uid mapping for the container's filesystem. SSH keys stored in /root/.ssh/ would have incorrect ownership (uid 100000 instead of 0), making SSH authentication fail. Indeed, the very next message (msg 458) shows exactly this failure: Permission denied (publickey,password). By using pct exec, the assistant bypasses the network stack entirely and runs commands directly inside the container via the Proxmox host's container runtime — a more reliable approach when the container's network configuration is in flux.

The output confirms that the bind-mount approach works. Each /dev/nvidia* device inside the container has the same major number as on the host (195 for nvidia devices, 504 for nvidia-uvm). The minor numbers map directly: nvidia0 through nvidia7 correspond to the eight GPUs, nvidiactl (195, 255) is the control device, and nvidia-uvm (504, 0) and nvidia-uvm-tools (504, 1) handle unified virtual memory management. The crw-rw-rw- permissions show world-readable/writable access, which is acceptable since the container is privileged and trusted.

Assumptions Embedded in This Message

The assistant makes several assumptions at this point, most of which are reasonable but some of which will prove incomplete:

First, the assistant assumes that device visibility inside the container is sufficient for CUDA to function. This is a necessary condition but not sufficient — the NVIDIA userspace driver stack (libraries like libcuda.so, libnvidia-ml.so) must also be installed inside the container, and the kernel module must support the Blackwell architecture's GSP (GPU System Processor) firmware requirements. The assistant's next planned step (visible in msg 458) is precisely to install the userspace driver.

Second, the assistant assumes that the privileged container conversion did not introduce filesystem issues beyond SSH key ownership. The ZFS subvolumes used for unprivileged containers apply a uid/gid shift (typically +100000) to all files. When the container is switched to privileged mode, this shift is no longer applied, meaning files created while unprivileged now appear with uid 100000 instead of their intended owners. The assistant is aware of this — msg 456 explicitly mentions "the unprivileged->privileged ownership change" — but the verification in msg 457 does not check for it. That check comes in msg 459, where ls -la /root/.ssh/ reveals the shifted ownership.

Third, the assistant assumes that the cgroup2 device permissions (lxc.cgroup2.devices.allow: c 195:* rwm) are correctly configured. The fact that the devices appear in the container's /dev/ directory confirms the bind-mounts work, but cgroup permissions control whether processes inside the container can actually open and read/write those devices. The assistant does not test this at this stage — that requires running an NVIDIA tool like nvidia-smi inside the container, which in turn requires the userspace driver.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

Output Knowledge Created

This message produces concrete, actionable knowledge:

  1. The container started successfully after the unprivileged-to-privileged conversion. The configuration file written in msg 455 (with cgroup permissions and bind-mount entries) is syntactically correct and accepted by the LXC runtime.
  2. All eight GPU device nodes are visible inside the container. The bind-mounts work correctly, and the device major/minor numbers match the host. This means the physical GPU devices are accessible from within the container's filesystem namespace.
  3. The create=file directive in the bind-mount entries works. When LXC starts a container and encounters lxc.mount.entry with create=file, it creates the target file in the container's /dev/ if it doesn't exist. The output shows all device files present, confirming this mechanism functions correctly.
  4. No device nodes are missing. All expected devices — nvidia0 through nvidia7, nvidiactl, nvidia-uvm, nvidia-uvm-tools — are present. Notably, the nvidia-caps directory (also configured in the bind-mounts) does not appear in the listing, but that's because ls -la /dev/nvidia* only matches files starting with "nvidia", and a directory would require ls -la /dev/nvidia-caps/ to list its contents.

The Broader Significance

This message, for all its brevity, represents a successful architectural transition. The assistant had spent the previous segment (segment 3) exhaustively trying to make P2P work within a KVM VM — modifying kernel parameters, changing chipset emulation, fixing BAR allocation, disabling ACS — and hit a wall. The hardware topology simply would not cooperate. The pivot to LXC was a gamble: it required installing the NVIDIA driver on the Proxmox host (which had never had one), converting the container to privileged mode, and hoping that the bare-metal topology would translate into functional P2P DMA.

Message 457 is the first evidence that the gamble paid off at the device level. The GPUs are visible. The container is running. The next steps — installing the userspace driver, running nvidia-smi, checking the topology, and ultimately running the model — all depend on this foundation.

Of course, the story does not end here. The very next phase of the session reveals a devastating blocker: CUDA initialization fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED) both on the host and inside the container, despite nvidia-smi detecting all eight GPUs. The root cause is a GSP firmware incompatibility between the NVIDIA driver 590.48.01 and the Blackwell architecture on the PVE kernel. The LXC approach, so promising in this verification message, ultimately founders on a driver compatibility issue that neither the assistant nor the user anticipated.

But that does not diminish the significance of message 457. It remains the moment where the assistant confirmed that the LXC path was technically viable at the device level — a necessary checkpoint that, had it failed, would have sent the entire effort back to the drawing board. In the high-stakes world of deploying large language models across multi-GPU infrastructure, such checkpoints are the difference between methodical progress and blind debugging.