The LXC Container Configuration: A Critical Pivot in the Pursuit of GPU P2P DMA

In the sprawling, multi-session effort to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 455] represents a quiet but decisive pivot point. It is not a dramatic message—no debugging breakthroughs, no performance benchmarks, no model inference. It is a configuration file being written to a Proxmox host. Yet within this single message lies the culmination of a strategic shift: the abandonment of KVM-based virtualization for GPU access in favor of LXC containers, driven by the relentless pursuit of peer-to-peer (P2P) DMA communication between GPUs. This message is the bridge between diagnosing a bottleneck and implementing a workaround.

The Context: Why LXC Instead of KVM?

To understand message [msg 455], one must first understand the problem it was trying to solve. Earlier in the session (segment 3), the assistant and user had been investigating why GPU-to-GPU P2P DMA was not working inside a KVM virtual machine running on Proxmox. The nvidia-smi topo -m output inside the VM showed a PHB (PCIe Host Bridge) topology—meaning every GPU appeared to be connected through the host bridge, with no direct peer-to-peer links. This is a well-known limitation of VFIO passthrough: when GPUs are passed through to a VM via VFIO, the hypervisor interposes itself between the devices, and the virtual PCIe topology presented to the guest does not reflect the physical topology. For workloads like all-to-all communication in MoE (Mixture of Experts) models, the lack of P2P DMA is a severe performance bottleneck.

The assistant and user had spent considerable effort trying to fix this within the KVM framework: modifying Proxmox host kernel parameters, migrating the VM from i440FX to Q35 chipset, fixing BAR allocation, disabling ACS (Access Control Services) to merge IOMMU groups. But they ultimately discovered that the hardware topology itself was the fundamental obstacle—each GPU sat on its own PCIe root complex, and no amount of software configuration could merge what the hardware separated.

This led to the LXC gambit. The reasoning was elegant: if the VFIO layer is the problem, bypass it entirely. An LXC container on the Proxmox host shares the host kernel, so GPU device nodes passed into the container should see the real PCIe topology. The host's nvidia-smi topo -m in message [msg 452] had already confirmed this: inside the host (and therefore inside an LXC container), the topology showed NODE within sockets and SYS across sockets—the true bare-metal topology. P2P DMA should work.

What the Message Actually Does

Message [msg 455] is a single bash command executed via SSH on the Proxmox host (kpro6). It writes a complete LXC container configuration file to /etc/pve/nodes/kpro6/lxc/129.conf, replacing the existing unprivileged container config. The changes are:

  1. Privileged mode: unprivileged: 0 — the container is switched from unprivileged to privileged. This is necessary because unprivileged containers have restricted device access capabilities, and passing through eight GPUs with full control requires elevated privileges.
  2. Increased memory: The memory allocation is raised from 400000 MB to 460000 MB. This likely reflects an understanding that the ML workload (running SGLang serving GLM-5-NVFP4) needs more RAM, possibly for KV cache allocation or model weights.
  3. cgroup device permissions: Three lxc.cgroup2.devices.allow lines grant access to character devices with major numbers 195, 504, and 507. These correspond to the NVIDIA driver (nvidia), the NVIDIA UVM (Unified Virtual Memory) driver (nvidia-uvm), and the NVIDIA caps device (nvidia-caps). The rwm permissions (read, write, mknod) give the container full control over these device classes.
  4. Device bind mounts: Eleven lxc.mount.entry lines bind-mount individual device nodes from the host into the container. Eight for the GPUs (/dev/nvidia0 through /dev/nvidia7), plus the control device (nvidiactl), the UVM device (nvidia-uvm), the UVM tools device (nvidia-uvm-tools), and the caps directory (nvidia-caps). Each uses optional,create=file (or create=dir for the caps directory), meaning the mount is attempted but won't fail if the device doesn't exist yet, and the target file/directory will be created automatically.

The Reasoning Behind Each Decision

Every line in this configuration file encodes a specific decision based on knowledge accumulated over the preceding messages.

Why privileged mode? The assistant had already considered the alternative. In message [msg 453], the host's device nodes showed permissions crw-rw-rw- (world-readable and writable), which might suggest unprivileged access could work. However, the NVIDIA UVM driver and certain control operations require capabilities that unprivileged containers lack. The cgroup2.devices.allow mechanism works differently for privileged vs unprivileged containers—privileged containers have a simpler path to device access. The assistant's choice reflects a pragmatic trade-off: security isolation is less important than functional correctness when the goal is scientific computing.

Why specific major numbers? The major numbers 195, 504, and 507 were discovered empirically in message [msg 452]. The assistant ran ls -la /dev/nvidia* on the host and saw:

Assumptions Embedded in the Message

Several assumptions underpin this configuration, some explicit and some implicit:

The device major numbers are stable. The assistant assumes that 195, 504, and 507 will remain the same after reboot or driver reload. This is generally true for NVIDIA drivers on a given kernel, but a kernel upgrade or driver reinstallation could change them. The bind mounts would fail silently (due to optional) if the devices don't exist, but the cgroup permissions would still grant access to whatever devices share those major numbers—potentially a security concern if the numbers changed.

The container will start successfully with this config. The assistant hasn't tested this configuration yet. The container was stopped in message [msg 454], and this message writes the new config. The assumption is that Proxmox's LXC implementation will accept the privileged mode switch and the device mounts without conflict.

The NVIDIA userspace stack is already installed in the container. The assistant's todo list (visible in earlier messages) includes "Install NVIDIA userspace driver in container" as a separate step. This configuration only provides device access; the container still needs libcuda.so, libnvidia-ml.so, and other userspace libraries. The assumption is that these can be installed from the NVIDIA CUDA toolkit or by bind-mounting them from the host.

Eight GPUs will all be accessible simultaneously. The bind mounts for all eight GPUs assume that the host's device enumeration (nvidia0 through nvidia7) is stable and that all GPUs will be available when the container starts. If a GPU is reset or fails, the bind mount will be skipped (due to optional), but the container might see an incomplete GPU set.

What Knowledge Was Required

To write this message, the assistant needed:

  1. Proxmox LXC configuration syntax: The lxc.cgroup2.devices.allow and lxc.mount.entry directives are specific to Proxmox's LXC implementation. The optional flag and create=file/create=dir options are LXC mount entry features.
  2. NVIDIA device node architecture: Understanding that NVIDIA GPUs expose multiple device types (compute, control, UVM, caps) with different major numbers, and that all are needed for full GPU functionality.
  3. The difference between privileged and unprivileged containers: Knowing that GPU access requires privileged mode or extensive cgroup configuration.
  4. The host's current device topology: The specific major numbers and device node names came from message [msg 452], which was the result of a deliberate exploration.
  5. The container's existing configuration: The assistant read the original config in message [msg 436] and preserved most settings (arch, cores, hostname, network, ostype, rootfs) while modifying only the GPU-related lines.

What Knowledge Was Created

This message produced:

  1. A deployable LXC configuration that, when the container starts, will have access to all eight GPUs with the correct device topology.
  2. A record of the decision to use privileged mode and the specific device mappings, which serves as documentation for future debugging.
  3. A testable hypothesis: that LXC containers can bypass the VFIO P2P bottleneck. This configuration is the experiment's setup; the results will come in subsequent messages when the container starts and CUDA initializes.
  4. A template for future GPU LXC configurations: The pattern of cgroup permissions plus bind mounts is reusable for any Proxmox host with NVIDIA GPUs.

The Broader Significance

Message [msg 455] is interesting precisely because it is so mundane. It is a configuration file—a list of device paths and permission flags. Yet it represents the convergence of several threads of investigation: the PCIe topology analysis, the IOMMU group mapping, the ACS disable attempts, the Q35 chipset migration. All of that prior work led to the conclusion that KVM virtualization was the wrong approach for this hardware, and LXC containers were the right one.

The message also reveals something about the assistant's working style: it is methodical and documentation-oriented. The configuration is written with a heredoc that echoes the config back for verification. The optional flags suggest defensive programming—the assistant anticipates that some devices might not be available and wants the container to start anyway. The comments in the heredoc explain the purpose of each section.

In the broader narrative of the session, this message is the calm before the storm. The LXC container will start, and then the real test begins: will CUDA initialize? Will P2P DMA work? The answer, as revealed in subsequent messages, is a frustrating "no"—CUDA initialization fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED), and the assistant will spend the rest of segment 4 debugging a host-level NVIDIA driver incompatibility with the Proxmox VE kernel. But that failure does not diminish the significance of this message. It was the right strategic call, executed carefully, that failed for reasons no amount of LXC configuration could fix.

Conclusion

Message [msg 455] is a testament to the fact that in complex systems engineering, the most important decisions are often encoded in the most prosaic artifacts. A container configuration file, barely twenty lines long, encapsulates a strategic pivot from KVM to LXC, embodies hours of PCIe topology investigation, and sets the stage for the next phase of debugging. It is a message that, on its surface, does nothing but write text to a file. But that text represents the accumulated understanding of the system's hardware constraints and the creative workaround that, while ultimately unsuccessful in this case, was the correct engineering response to the problem at hand.