The Moment of Truth: Verifying GPU Passthrough in an LXC Training Container

In the high-stakes world of large-scale machine learning infrastructure, few moments carry as much tension as the first boot of a freshly provisioned GPU compute container. The hardware has been assembled, the kernel has been custom-built from source, the NVIDIA drivers have been compiled against it, and the LXC configuration has been meticulously crafted with device passthrough entries for each of the eight NVIDIA RTX PRO 6000 Blackwell GPUs. But none of that matters until the container starts and the GPUs are actually visible inside it. Message [msg 8528] captures this exact moment — a concise but critical verification step that transforms configuration into capability.

The Message

The assistant writes:

Now start the container and verify GPUs are visible:

>

``bash ssh -o ConnectTimeout=10 root@10.1.2.6 ' pct start 200 sleep 3 pct status 200 echo "---" # Check GPU devices inside container pct exec 200 -- ls -la /dev/nvidia* 2>&1 echo "---" # Get the container IP pct exec 200 -- ip addr show eth0 2>&1 | grep "inet " ' ``

>

Output: `` status: running --- crw-rw-rw- 1 root root 509, 0 May 15 17:33 /dev/nvidia-uvm crw-rw-rw- 1 root root 509, 1 May 15 17:33 /dev/nvidia-uvm-tools crw-rw-rw- 1 root root 195, 0 May 15 17:33 /dev/nvidia0 crw-rw-rw- 1 root root 195, 1 May 15 17:33 /dev/nvidia1 crw-rw-rw- 1 root root 195, 2 May 15 17:33 /dev/nvidia2 crw-rw-rw- 1 root root 195, 3 May 15 17:33 /dev/nvidia3 crw-rw-rw- 1 root root 195, 4 May 15 17:33 /dev/nvidia4 crw-rw-rw- 1 root root 195, 5 May 15 17:33 /dev/nvidia5 crw-... ``

The output continues to show all eight GPUs (/dev/nvidia0 through /dev/nvidia7), the NVIDIA UVM (Unified Virtual Memory) device, and the container's IP address 10.1.2.200. The three-second sleep before verification is a deliberate pause to allow the container's init process to complete and all device nodes to be populated.

Why This Message Matters: The Context

This message sits at a critical juncture in a much larger infrastructure effort. The host machine, kpro6 (10.1.2.6), is a dual-socket AMD EPYC 9335 server with 64 cores, 504 GB of RAM, and eight RTX PRO 6000 Blackwell GPUs — each with 96 GB of GDDR7 memory. Getting to this point required an extraordinary amount of work documented across the preceding segments of the conversation.

The journey to this container boot included: recovering from a bricked system caused by a glibc incompatibility that required an Arch Linux ISO rescue ([msg 8516]); building a custom Proxmox VE 6.14 kernel from source because community kernel packages caused toolchain disasters; compiling NVIDIA's open-gpu-kernel-modules version 595.71.05 from source against that custom kernel; and debugging a kernel boot panic with the cryptic message "no working init found." The assistant and user had already rejected multiple failed approaches — a jaminmc community 6.19 kernel built with GCC 14, a trixie gcc-14 package that pulled incompatible libc6, and even a glibc shim hack using LD_PRELOAD that ultimately bricked the system.

The immediate predecessor to this message ([msg 8527]) was the configuration of GPU passthrough in the LXC container's config file at /etc/pve/lxc/200.conf. That configuration added lxc.cgroup2.devices.allow entries for the NVIDIA device major numbers (195 for the GPUs, 509 for UVM, 226 and 234 for NVIDIA caps) and lxc.mount.entry lines to bind-mount each device node from the host into the container. This is a delicate operation — LXC device passthrough requires matching the device major and minor numbers exactly, and any mismatch results in invisible or non-functional GPUs inside the container.

The Verification Strategy

The assistant's approach to verification is methodical and reveals a clear mental model of the failure modes at play. The command sequence executes four distinct checks in a single SSH session:

  1. Container startup: pct start 200 initiates the container. If this fails, nothing else matters — the configuration is fundamentally broken.
  2. Status confirmation: pct status 200 after a three-second sleep confirms the container entered the "running" state. The sleep 3 is not arbitrary; it accounts for the time needed for the container's init system to boot, networking to come up via DHCP, and device nodes to be populated by the host-side lxc.mount.entry directives. Too short a sleep risks a false negative.
  3. GPU device enumeration: pct exec 200 -- ls -la /dev/nvidia* lists all NVIDIA device nodes inside the container. This is the core verification. The assistant is looking for: - All eight /dev/nvidia0 through /dev/nvidia7 devices (major number 195) - The /dev/nvidia-uvm and /dev/nvidia-uvm-tools devices (major number 509) - The correct minor numbers matching the physical GPU indices
  4. Network connectivity: pct exec 200 -- ip addr show eth0 | grep "inet " retrieves the container's IP address. This confirms the virtual Ethernet (veth) interface is functional and DHCP succeeded, which is essential for subsequent SSH-based setup steps. The output confirms all checks pass. The container is running, all eight GPUs are visible with their correct device nodes, and the container has obtained IP 10.1.2.200 on the vmbr0 bridge.

Assumptions Embedded in the Verification

Several assumptions underpin this verification and are worth examining. First, the assistant assumes that device node visibility inside the container is a sufficient proxy for full GPU functionality. While seeing /dev/nvidia0 through /dev/nvidia7 is necessary, it is not sufficient — the NVIDIA kernel modules (nvidia, nvidia-uvm) must also be properly loaded on the host, and the CUDA driver version must match the userspace libraries that will later be installed inside the container. The assistant is implicitly trusting that the host-side NVIDIA setup (which was verified with nvidia-smi in [msg 8522]) is correct and that the passthrough mechanism faithfully exposes the devices.

Second, the assistant assumes the three-second sleep is adequate. On a heavily loaded host or with a slow storage backend, container initialization could take longer. The choice of three seconds is a heuristic — long enough for typical boots but short enough to avoid unnecessary delay. A more robust approach might loop with a retry, but the assistant opts for simplicity here.

Third, the assistant assumes the container's networking will work via DHCP on vmbr0 without additional configuration. The LXC configuration specified net0 name=eth0,bridge=vmbr0,ip=dhcp, which delegates IP assignment to the Proxmox host's DHCP infrastructure. This works because the host has a functioning network bridge, but it introduces a dependency on the host's DHCP server being operational.

The Deeper Significance: From Infrastructure to Training

This message represents the transition from infrastructure provisioning to software deployment. With the container running and GPUs visible, the assistant can now proceed to the next steps: installing NVIDIA userspace libraries and CUDA toolkit inside the container, setting up the Python environment with PyTorch, transformers, FLA (Flash Linear Attention), and wandb, and finally deploying the DFlash training pipeline.

The todo list from [msg 8521] shows the roadmap: "Install NVIDIA userspace + CUDA toolkit inside container" and "Install Python env (uv), PyTorch, transformers, fla, wandb" follow immediately after container creation. The verification in this message is the gate that must pass before any of that work can begin.

What makes this moment particularly significant is the history of the session. The assistant and user have already spent enormous effort getting to this point — recovering from a bricked host, building kernels and drivers from source, debugging boot panics. A failure here — GPUs not visible, container not starting — would have meant going back to debug the LXC configuration or, worse, the host-level GPU setup. The clean output is a quiet triumph after a long infrastructure battle.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Container 200 is operational: The container named dflash-train is running and healthy on kpro6.
  2. All eight GPUs are accessible: The device nodes /dev/nvidia0 through /dev/nvidia7 are present inside the container, along with /dev/nvidia-uvm and /dev/nvidia-uvm-tools. The GPU indices match the physical GPU numbering on the host.
  3. Container networking is functional: The container obtained IP 10.1.2.200 via DHCP on the vmbr0 bridge, making it accessible for SSH-based setup and monitoring.
  4. The LXC GPU passthrough configuration is correct: The device major numbers (195 for GPUs, 509 for UVM) and the bind-mount entries in the LXC config are validated. This configuration can serve as a template for future GPU containers on this host.
  5. The path is clear for software installation: With hardware verification complete, the assistant can proceed to install the NVIDIA userspace stack, CUDA toolkit, Python environment, and training scripts without hardware-level debugging distractions.

Conclusion

Message [msg 8528] is a brief but pivotal verification step in a much larger infrastructure saga. In just a few lines of bash, it confirms that weeks of kernel building, driver compilation, system recovery, and configuration work have succeeded. The eight GPU device nodes visible inside the container represent the culmination of a complex engineering effort — a custom-built kernel, source-compiled NVIDIA drivers, a carefully configured LXC container, and a host recovered from a bricked state. For the reader who has followed the session from the beginning, this message carries the quiet satisfaction of a system that works as designed. For the training pipeline that will run for days on these GPUs, it is the foundation upon which everything else depends.