The Moment of Apparent Success: Verifying GPU Device Passthrough in an LXC Container
Introduction
In any complex systems debugging session, there comes a moment when everything looks like it's working — the devices are visible, the connections are live, the surface-level checks pass. Message 466 in this opencode session captures exactly such a moment. It is a deceptively simple bash command and its output: an SSH connection test into an LXC container, verifying that GPU device nodes are present after an extensive reconfiguration of the Proxmox host. But this message, like many in the trenches of systems engineering, is far more significant than its brevity suggests. It represents the culmination of hours of driver installation, container conversion, uid shifting, and device bind-mount configuration — and it sets the stage for a deeper, more insidious problem that is about to surface.
The Message Itself
The assistant writes:
SSH is starting. Let me wait a moment:
>
``bash sleep 5 && ssh root@10.1.230.174 "whoami && hostname && ls /dev/nvidia* 2>&1" 2>&1 ``
>
``` root llm-two /dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia0 /dev/nvidia1 /dev/nvidia2 /dev/nvidia3 /dev/nvidia4 /dev/nvidia5 /dev/nvidia6 /dev/nvidia7 /dev/nvidiactl
>
/dev/nvidia-caps: nvidia-cap1 nvidia-cap2 ```
On its face, this is a verification step. The assistant waits five seconds for the SSH daemon to finish starting inside the container, then connects and runs three commands: whoami to confirm the user identity, hostname to confirm which container is being accessed, and ls /dev/nvidia* to list the NVIDIA device nodes. The output confirms that SSH is working, the user is root, the hostname is llm-two (the LXC container), and crucially, all GPU device nodes are present.
The Road to This Point
To understand why this message matters, one must appreciate the journey that preceded it. The session had been wrestling with a fundamental performance bottleneck: GPU-to-GPU communication (P2P DMA) was limited to PHB (PCIe Host Bridge) topology inside a KVM virtual machine, meaning each GPU-to-GPU transfer had to traverse the host bridge rather than using direct peer-to-peer DMA. This imposed severe latency penalties for the multi-GPU inference workload (GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs).
The team had already attempted numerous workarounds within the KVM VM — switching attention backends, tuning server parameters, evaluating expert parallelism — but the PCIe bottleneck was fundamental. The Proxmox host's IOMMU groups and VFIO passthrough architecture prevented the GPUs from communicating directly.
The LXC container approach was the next logical gambit. Unlike a KVM VM, an LXC container shares the host kernel and can access hardware devices directly. If the GPUs could be bind-mounted into a privileged container, they would see the true bare-metal PCIe topology — NODE within the same NUMA socket and SYS across sockets — rather than the virtualized PHB topology. This would, in theory, enable P2P DMA and eliminate the bottleneck.
The preceding messages in the session (messages 438–465) document the laborious setup: installing the NVIDIA driver (590.48.01) on the Proxmox host, blacklisting the nouveau driver, configuring the container from unprivileged to privileged, writing the LXC configuration file with bind-mount entries for all eight GPU device nodes, fixing the uid/gid ownership shift caused by the unprivileged-to-privileged conversion, and waiting for SSH to become available. Message 466 is the first verification that this entire chain of dependencies has been correctly assembled.
What the Output Actually Tells Us
The output from ls /dev/nvidia* reveals several important facts:
All eight GPUs are visible. The devices /dev/nvidia0 through /dev/nvidia7 correspond to the eight RTX PRO 6000 Blackwell GPUs. Each is a character device with major number 195, and each has been successfully bind-mounted from the host into the container.
The control device is present. /dev/nvidiactl (major 195, minor 255) is the NVIDIA control device, required for CUDA API calls that manage GPU contexts and device enumeration. Without it, the CUDA runtime cannot function.
The UVM devices are present. /dev/nvidia-uvm and /dev/nvidia-uvm-tools (major 504) are the Unified Virtual Memory devices, essential for CUDA's unified memory model, which allows seamless access to host and device memory. These are particularly important for the ML workload, which relies on UVM for efficient memory management across GPUs.
The NVIDIA capabilities interface is available. The /dev/nvidia-caps/ directory contains nvidia-cap1 and nvidia-cap2, which expose GPU capabilities to userspace. This is a newer feature of the NVIDIA driver stack, used by the proprietary userspace libraries to query hardware capabilities without requiring direct GPU access.
The SSH connection itself is also significant. The previous message (msg 465) showed that SSH was in "activating (start-pre)" state — the daemon was starting but not yet accepting connections. The sleep 5 in this message was a pragmatic wait, and the successful connection confirms that the container's networking is functional and that the uid/gid ownership fix did not break essential system services.
The Assumptions Embedded in This Message
Every verification step carries implicit assumptions, and this message is no exception. The assistant's reasoning — visible in the comment "SSH is starting. Let me wait a moment" — reveals a straightforward mental model: the SSH daemon needs time to initialize, then we test connectivity and device visibility. If both work, we can proceed to the next step: installing the NVIDIA userspace libraries (CUDA toolkit, cuDNN, etc.) inside the container and running the actual ML workload.
The critical assumption is that device node visibility implies CUDA functionality. This is a reasonable assumption in normal circumstances: if the kernel module is loaded, the device nodes exist, and the permissions are correct, the CUDA runtime should be able to initialize. The assistant had already verified on the host that nvidia-smi detects all eight GPUs and reports the correct driver version (590.48.01) and CUDA version (13.1). The device nodes were created by the NVIDIA kernel module during driver installation. Everything appeared consistent.
A second assumption is that the host-level driver installation is sufficient for container-level CUDA usage. In an LXC container, the kernel module runs on the host, and the container only needs the userspace libraries (libcuda.so, libnvidia-ml.so, etc.) to be installed or bind-mounted. The assistant's plan was to install the NVIDIA userspace driver inside the container, which should have been straightforward given that the device nodes were accessible.
The Dramatic Irony: What the Assistant Doesn't Yet Know
This is where the message becomes genuinely interesting from a narrative perspective. The chunk summary reveals that immediately after this apparent success, CUDA initialization fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED) — both on the host and inside the container. The device nodes are present, nvidia-smi works, but cuInit fails.
The root cause turns out to be a GSP (GPU System Processor) firmware compatibility issue. The NVIDIA driver 590.48.01 includes GSP firmware files only for the ga10x (Ampere) and tu10x (Turing) architectures — not for Blackwell (GB202). The Proxmox VE kernel (6.8.12-9-pve) is older than the kernels typically used with Blackwell GPUs, and the GSP firmware initialization fails silently, leaving the GPUs in a state where nvidia-smi can query their basic properties but the CUDA runtime cannot establish a working context.
This is a fundamentally different failure mode from what the KVM VM experienced. In the VM, the guest's own NVIDIA driver stack handled the GPUs via VFIO passthrough, and the GSP firmware was loaded within the guest's kernel context. The host-level driver never needed to interact with the Blackwell GSP. But in the LXC approach, the host kernel module must handle GSP initialization directly — and it lacks the necessary firmware support.
The message thus stands at a pivot point. It documents a verification that should have been the green light to proceed, but instead becomes the prelude to a deeper investigation. The device nodes are there, the SSH connection works, the topology is correct — but the foundation is rotten.
Technical Depth: What Device Nodes Really Mean
To fully appreciate this message, it helps to understand what each device node represents in the NVIDIA driver stack. The Linux kernel's NVIDIA driver (nvidia.ko) registers several classes of devices:
NVIDIA GPU devices (/dev/nvidiaN, major 195): Each GPU gets a unique minor number (0–7 in this case). These are the primary interfaces for GPU commands, memory management, and execution. The CUDA driver opens these devices to send work to the GPU.
NVIDIA control device (/dev/nvidiactl, major 195, minor 255): This is a singleton device used for global CUDA operations — enumerating GPUs, querying capabilities, and managing driver state. Every CUDA application opens this device first.
NVIDIA UVM devices (/dev/nvidia-uvm, /dev/nvidia-uvm-tools, major 504): The Unified Virtual Memory driver enables CUDA's unified memory model, where host and device memory can be accessed through a single pointer. UVM is critical for modern ML frameworks like PyTorch and TensorFlow, which rely on unified memory for efficient tensor operations across CPU and GPU.
NVIDIA capabilities devices (/dev/nvidia-caps/nvidia-capN, major 507): These expose hardware capabilities to userspace without requiring direct GPU access. They were introduced in newer driver versions to support containerized environments where the full device node set may not be available.
The presence of all these device nodes in the container confirms that the bind-mount configuration was correct and that the NVIDIA kernel module is functioning at the device node level. The assistant's configuration file (written in msg 455) included entries for each of these device types, and the output validates that configuration.
The Thinking Process Visible in the Message
Although the message is short, it reveals the assistant's reasoning process through its structure. The comment "SSH is starting. Let me wait a moment" shows an understanding of timing dependencies — the assistant knows that the SSH daemon was in the process of starting (as shown in msg 465) and needs time to become ready. The sleep 5 is a pragmatic heuristic, not a precise wait — it reflects the reality of systems debugging where exact timing is unpredictable.
The choice of commands inside the SSH session is also revealing. whoami and hostname are identity checks — they confirm that the connection reached the correct container and that the uid/gid fix didn't break the root user. ls /dev/nvidia* is a comprehensive device listing that covers all NVIDIA device types in a single command. The 2>&1 redirect ensures that any error messages (e.g., "No such file or directory") are captured in the output.
The assistant does not check nvidia-smi inside the container at this point — that would come later. The focus is on the foundational layer: SSH connectivity and device node visibility. This is a sensible incremental approach: verify the transport layer first, then the device layer, then the runtime layer.
Conclusion
Message 466 is a snapshot of a system at a particular moment — a moment when the hard work of configuration and troubleshooting appears to have paid off. The GPU devices are visible, the container is accessible, and the path forward seems clear. But in the broader narrative of this debugging session, it is also a moment of dramatic irony. The reader (and the chunk summary) knows that CUDA initialization is about to fail, revealing a GSP firmware compatibility issue that no amount of device node verification could have predicted.
This message teaches an important lesson about systems debugging: surface-level success does not guarantee deep-level functionality. The device nodes are the interface, but the firmware is the foundation. And sometimes, the most frustrating bugs are the ones that let you get almost all the way there before pulling the rug out from under you.