The Critical Glue: Configuring GPU Passthrough for an LXC Training Container on kpro6

In the sprawling effort to provision a production-grade deep learning training machine, few steps are as deceptively simple — and as easy to get wrong — as configuring GPU passthrough for an LXC container. Message <msg id=8527> captures this exact moment: a single SSH command that bridges the gap between a bare-metal Proxmox host with eight NVIDIA RTX PRO 6000 Blackwell GPUs and the Ubuntu container that will soon drive a DFlash speculative decoding training pipeline at 25 Ktok/s. This message, on its surface just a heredoc appended to a configuration file, represents the culmination of days of kernel building, driver compilation, and system recovery — and the gateway to the actual training workload.

The Broader Context: Why This Message Exists

To understand why <msg id=8527> was written, one must first understand the journey that led to it. The assistant had been provisioning kpro6 (10.1.2.6), a new Proxmox VE 8.4 host equipped with 2× AMD EPYC 9335 processors (64 cores total), 504 GB of RAM, and eight NVIDIA RTX PRO 6000 Blackwell GPUs — each with 96 GB of GDDR7 memory, totaling 783 GB of VRAM. This machine was destined to become the primary training node for a DFlash block-diffusion speculative decoding drafter, a project that had already consumed dozens of messages across multiple sessions.

The provisioning path had been treacherous. The assistant had built a custom Proxmox VE 6.14 kernel from source after a community kernel bricked the system with incompatible glibc. NVIDIA's open-gpu-kernel-modules (version 595.71.05) had been compiled from source against that custom kernel. The system had been rescued from a boot failure using an Arch ISO. By the time we reach <msg id=8527>, the host was stable with all eight GPUs recognized and CUDA 13.2 operational.

The user's instruction was clear: "Spin up a new container with ubuntu on kpro6, deploy the train workload there (7-1 GPUs, 1 training, rest inference hidden layer extraction)." The assistant had already executed the first part of this instruction in preceding messages: downloading the Ubuntu 24.04 LXC template (<msg id=8523>), creating the container (CT 200) with 480 GB RAM, 64 cores, and a 1 TB rootfs on the scratch ZFS pool (<msg id=8525>), and inspecting the bare config that resulted (<msg id=8526>). That config showed a container with no GPU access whatsoever — just CPU, memory, and networking. Message <msg id=8527> is the step that transforms that generic container into a GPU-enabled training powerhouse.

The Anatomy of the Command

The message executes a single bash command over SSH, using a heredoc to append GPU passthrough entries to /etc/pve/lxc/200.conf. The structure reveals careful thinking about what LXC needs to expose NVIDIA GPUs to a containerized workload.

The configuration has two parts. First, cgroup2 device permissions:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm

These lines grant the container read-write-mknod access to four device major numbers. Major number 195 is the standard NVIDIA GPU device (/dev/nvidia[0-7]). Major 509 is the NVIDIA UVM (Unified Virtual Memory) device (/dev/nvidia-uvm), essential for CUDA memory management across GPUs. Major 226 covers the NVIDIA caps devices (/dev/nvidia-caps/nvidia-cap*), which expose hardware capabilities. Major 234 is the NVIDIA control device (/dev/nvidiactl).

Second, bind mount entries:

lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
...

Each of the eight GPUs gets its own bind mount, plus entries for the control device, UVM device, UVM tools, and both capability devices. The optional flag means the container starts even if a device is temporarily absent, and create=file ensures the device node is created in the container's /dev namespace.

Decisions Made and Their Rationale

Several design decisions are embedded in this message, each reflecting deep knowledge of both LXC mechanics and NVIDIA's device architecture.

Why individual bind mounts for each GPU rather than a wildcard? LXC does not support wildcard bind mounts for device files. Each GPU must be explicitly listed. The assistant lists all eight (nvidia0 through nvidia7), which is correct but creates a maintenance burden if GPUs are added or removed. This is a deliberate trade-off: explicitness over flexibility, because the hardware configuration is fixed.

Why these four major numbers? The assistant had verified the host device nodes in <msg id=8522>, which showed crw-rw-rw- 1 root root 195, 0 for each GPU, 509, 0 for nvidia-uvm, and so on. The major numbers were read directly from the host's /dev listing, ensuring accuracy. This is a critical point: the assistant did not guess these numbers or rely on documentation — it inspected the actual running system.

Why cgroup2 rather than cgroup? Proxmox VE 8.x uses cgroup v2 by default. Using the wrong cgroup version would silently fail to grant permissions, and the container would see Permission denied when trying to access the devices. The assistant correctly uses lxc.cgroup2.devices.allow.

Why --unprivileged 0 on container creation? In <msg id=8525>, the container was created with --unprivileged 0, meaning it runs as a privileged container. This is necessary because NVIDIA GPU passthrough requires access to host device nodes, which unprivileged containers cannot do with standard bind mounts. This decision was made at container creation time and is a prerequisite for the GPU passthrough configuration in <msg id=8527> to function.

Assumptions Embedded in the Message

The message makes several assumptions, most of which are well-founded but worth examining.

Assumption 1: The device major numbers are stable across reboots. Linux device major numbers are assigned by the kernel driver and are stable for a given driver version. Since the assistant had built the NVIDIA 595.71.05 kernel modules from source and confirmed they loaded correctly, this assumption is safe — but it would break if the NVIDIA driver were ever upgraded to a version that reassigns major numbers.

Assumption 2: All eight GPUs are identical and use the same device major numbers. The RTX PRO 6000 Blackwell GPUs are identical server-class cards, so this holds. A heterogeneous GPU setup (e.g., mixing consumer and workstation cards) could have different major numbers for different GPUs, which would require additional cgroup entries.

Assumption 3: The container will always have exactly eight GPUs. The bind mounts are hardcoded to nvidia0 through nvidia7. If a GPU fails and is removed, the optional flag prevents the container from failing to start, but the training script would need to handle fewer GPUs gracefully.

Assumption 4: The LXC config file path is /etc/pve/lxc/200.conf. This is the standard path for Proxmox LXC container configurations, where 200 is the container ID. The assistant confirmed this path existed in <msg id=8526> before appending to it.

What Input Knowledge Was Required

To write this message, the assistant needed to synthesize knowledge from several domains:

  1. Proxmox LXC configuration syntax: Understanding of lxc.cgroup2.devices.allow, lxc.mount.entry, the bind,optional,create=file mount options, and the config file format.
  2. NVIDIA device architecture: Knowledge that NVIDIA GPUs expose multiple device types (compute, control, UVM, caps) and their respective major numbers. This includes understanding that nvidia-uvm is required for CUDA memory management and that the caps devices are needed for certain GPU capabilities.
  3. Linux device numbering: Understanding of major/minor device numbers and how to read them from ls -la /dev/nvidia*.
  4. The specific hardware configuration: Knowing that kpro6 has exactly eight GPUs, that they are all RTX PRO 6000 Blackwell cards, and that the driver version is 595.71.05.
  5. The training workload requirements: Knowing that the DFlash training pipeline requires all eight GPUs (seven for target model inference, one for drafter training), which justifies the effort of passing through every GPU.
  6. Container creation history: Knowing that the container was created with --unprivileged 0 and --features nesting=1, both of which are prerequisites for GPU passthrough.

What Output Knowledge Was Created

The immediate output is a modified LXC configuration file that enables GPU access. But the message also creates several forms of knowledge:

  1. A reproducible configuration: The appended config block serves as documentation of exactly what GPU passthrough looks like for this system. If the container needs to be recreated, the config can be reused verbatim.
  2. Verification of correctness: The command echoes the full config after modification, and the output shows it was applied correctly. The assistant can visually confirm that all entries are present and properly formatted.
  3. A foundation for the next step: The following message (<msg id=8528>) starts the container and verifies that all eight GPUs are visible inside it. The output shows ls -la /dev/nvidia* returning all eight GPU devices plus UVM devices — confirming the passthrough works.
  4. An audit trail: The message is logged in the conversation history, providing a record of exactly what configuration was applied and when. This is valuable for debugging if GPU access issues arise later.

The Thinking Process Visible in the Message

While the message itself is a single bash command, the reasoning behind it is visible in the surrounding context. In <msg id=8523>, the assistant examined the GPU topology using nvidia-smi topo -m and noted: "504 GB RAM, GPU topology: 4+4 across 2 NUMA nodes (GPUs 0-3 on NUMA 0, GPUs 4-7 on NUMA 1). For 7-1 topology, the drafter should be GPU 7 (last on NUMA 1) so the 7 target GPUs span both nodes but the drafter stays cleanly on one side." This shows the assistant was thinking about GPU assignment before the container was even created.

The choice to pass through all eight GPUs rather than a subset reflects the assistant's understanding that the training pipeline needs flexibility. The 7-1 topology (seven GPUs for target model forward passes, one for drafter training) could be configured at the software level inside the container, not at the hardware level. Passing through all eight gives maximum flexibility.

The use of optional in the bind mount entries shows defensive thinking: if a GPU fails or is temporarily removed for maintenance, the container should still start. The training script would crash when trying to use the missing GPU, but the container itself wouldn't be stuck in a failed state.

Conclusion

Message <msg id=8527> is a textbook example of the kind of infrastructure work that makes machine learning at scale possible. It is not glamorous — it is a heredoc appended to a config file — but it represents the culmination of days of system building and the gateway to a training pipeline that will consume hundreds of GPU-hours. The assistant's careful attention to device major numbers, cgroup version, bind mount syntax, and defensive error handling reflects a deep understanding of the Linux container ecosystem and the specific requirements of NVIDIA GPU workloads. Without this message, the container would be a CPU-only sandbox; with it, it becomes a training powerhouse capable of driving eight Blackwell GPUs at full utilization.