The Pivot Point: Choosing Privileged Containers for GPU P2P on Proxmox

A Single Decision That Defined an Entire Debugging Session

In any complex engineering session, there are moments that function as hinges — small decisions upon which entire subsequent chains of investigation swing. Message [msg 454] in this opencode session is precisely such a hinge. It is deceptively brief: the assistant observes that an LXC container is unprivileged, weighs two options, makes a choice, and executes a single bash command to stop the container. Yet this message encodes a cascade of reasoning about GPU architecture, Linux container security models, device node management, and performance optimization that had been building across dozens of prior exchanges.

The message reads in full:

[assistant] The container is unprivileged. I need to either switch it to privileged or handle the device permissions carefully. For simplicity and maximum GPU performance, let me switch it to privileged and add the GPU mounts. Let me stop the container first: [bash] ssh root@10.1.2.6 "pct stop 129 2>&1; sleep 2; pct status 129" status: stopped

To understand why this message was written, one must understand the problem it was trying to solve. The session had been wrestling with a persistent and frustrating bottleneck: deploying the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs in a Proxmox virtualized environment. Earlier in the session, the team had discovered that GPU-to-GPU communication (Peer-to-Peer DMA, or P2P) was operating at severely degraded speeds — the topology inside the KVM virtual machine showed PHB (PCIe Host Bridge) connections between GPUs, meaning every cross-GPU data transfer had to traverse the host's PCIe root complex rather than direct GPU-to-GPU links. This was a crippling limitation for a model that relies heavily on tensor parallelism across devices.

The Reasoning Behind the Decision

The assistant's reasoning, visible in the concise prose of the message, reflects a careful trade-off analysis. The container in question — container 129, named llm-two — was configured as an unprivileged container. In Proxmox/LXC, unprivileged containers use user namespace mapping (user ID mapping) to provide security isolation. This means that root inside the container is mapped to a non-root user outside, and device access is mediated through cgroup device permissions rather than direct ownership.

The assistant explicitly names two paths: "switch it to privileged or handle the device permissions carefully." The first option — switching to privileged — removes the user namespace isolation and grants the container direct access to host devices with the same permissions as host-root. The second option — handling device permissions carefully — would involve keeping the container unprivileged but adding precise cgroup device allow rules (lxc.cgroup2.devices.allow) for each GPU device node, while also ensuring the device file permissions inside the container were correctly set up through bind mounts.

The choice to go with the privileged route is justified by two stated criteria: "simplicity" and "maximum GPU performance." This reveals an important assumption: that the overhead of cgroup device mediation in an unprivileged container could introduce latency or throughput degradation for GPU operations. While this assumption is reasonable — cgroup device controllers do add a layer of permission checking on every device access — it is worth noting that for GPU compute workloads, the actual performance impact of cgroup device mediation is typically negligible compared to the GPU kernel execution time. The real performance concern was likely about the complexity of debugging device permission issues later, rather than raw throughput.

The Context That Made This Message Necessary

To fully appreciate this message, one must trace back through the preceding work. The session had already accomplished a remarkable amount. The assistant had installed the NVIDIA driver 590.48.01 directly on the Proxmox host — a non-trivial operation requiring the addition of the Proxmox no-subscription repository, installation of pve-headers and build-essential, blacklisting the nouveau open-source driver, and running the proprietary NVIDIA installer with --dkms --silent --no-x-check. The result was confirmed in [msg 451]: all 8 GPUs visible via nvidia-smi, with driver version 590.48.01 and CUDA 13.1.

More critically, [msg 453] had revealed the bare-metal GPU topology on the host:

• GPU0-3 (NUMA 0): NODE to each other — indicating direct P2P capability
• GPU4-7 (NUMA 1): NODE to each other — same within the second socket
• Cross-NUMA: SYS — indicating system-level interconnect (slower but still better than PHB) This was the breakthrough the session had been seeking. Inside the KVM VM, the topology had shown PHB for all cross-GPU connections, which was the root cause of the performance bottleneck. The LXC approach promised to bypass the VFIO/IOMMU layer entirely, giving the container direct access to the physical GPU devices and their native P2P capabilities. But this promise could only be realized if the container could actually access those devices. And that access hinged on the decision being made in message [msg 454].

Input Knowledge Required

Understanding this message requires substantial domain knowledge spanning several areas:

LXC security models: The distinction between privileged and unprivileged containers is fundamental to Proxmox/LXC. Unprivileged containers use user namespace mapping (subuid/subgid), which means that UID 0 inside the container maps to a non-privileged UID on the host. This prevents privilege escalation from the container to the host but complicates device access, because device nodes have specific ownership and permission requirements that may not match the mapped UIDs.

NVIDIA device node architecture: The NVIDIA driver creates multiple device node types. The nvidia devices (major number 195, minors 0-7 for each GPU plus 255 for the control device) are the primary GPU interfaces. The nvidia-uvm device (major 504) handles Unified Virtual Memory operations. The nvidia-caps devices (major 507) provide NVIDIA capability interfaces. Each of these must be accessible inside the container for CUDA to function.

GPU P2P and PCIe topology: The entire reason for pursuing the LXC approach was the P2P bottleneck. Understanding why PHB topology is bad and NODE topology is good requires knowledge of how PCIe switches, root complexes, and GPU memory access patterns interact in multi-GPU systems.

Proxmox container configuration syntax: The assistant is working with /etc/pve/nodes/kpro6/lxc/129.conf, the Proxmox LXC configuration file format, which uses lxc.cgroup2.devices.allow for cgroup v2 device permissions and lxc.mount.entry for bind mounts.

The Decision's Consequences

The decision to switch to a privileged container had immediate and far-reaching consequences. In the very next message ([msg 455]), the assistant rewrote the container configuration with unprivileged: 0 and added bind-mount entries for all 8 GPU device nodes, the control device, the UVM device, and the capabilities directory. The cgroup device allow rules were also added for completeness, though they are technically redundant for privileged containers.

This configuration was then applied, and the container was started. Inside the container, nvidia-smi topo -m confirmed the bare-metal topology — the very goal of the exercise. The model cache was copied from the VM's ZFS zvol to a shared dataset and bind-mounted into the container, avoiding a 296GB re-download.

However — and this is where the story takes its dramatic turn — a major blocker emerged. CUDA runtime initialization (cuInit) failed with error code 3 (CUDA_ERROR_NOT_INITIALIZED) both on the host and inside the container. Despite nvidia-smi detecting all 8 GPUs correctly, the CUDA runtime could not initialize. The root cause appeared to be a driver compatibility issue with the Proxmox VE kernel (6.8.12-9-pve): the driver 590.48.01 lacked Blackwell GSP firmware files, and the older PVE kernel may not have supported the Blackwell architecture's GSP requirements.

This failure was not caused by the privileged-container decision — it would have occurred regardless of the container type, since it was a host-level driver issue. But the decision did set up the expectation that the LXC approach would work, making the subsequent CUDA initialization failure more surprising and requiring further investigation.

Assumptions and Potential Mistakes

The message reveals several assumptions worth examining:

Assumption 1: Privileged containers provide maximum GPU performance. This is plausible but unverified. Modern cgroup v2 device controllers are lightweight, and the performance difference for GPU workloads between privileged and unprivileged containers is likely negligible. The real advantage of privileged containers is operational simplicity — fewer permission errors to debug.

Assumption 2: The host NVIDIA driver installation would be sufficient for container GPU access. This turned out to be partially incorrect. While nvidia-smi worked on the host and inside the container, CUDA runtime initialization failed, suggesting that the driver stack was not fully functional for compute workloads. The assumption that a working nvidia-smi implies a working CUDA stack was a reasonable heuristic that proved misleading.

Assumption 3: Switching to privileged is reversible and safe. In a production environment, privileged containers raise security concerns — they break the isolation that LXC is designed to provide. However, in this debugging context on a dedicated ML server, the security trade-off was acceptable.

The Output Knowledge Created

This message created several forms of output knowledge:

Procedural knowledge: The exact sequence of steps to convert an unprivileged LXC container to privileged, stop it, and prepare it for GPU bind-mounts. This is captured in the bash command and the subsequent configuration rewrite.

Decision rationale: The explicit reasoning that privileged containers are preferable for GPU workloads "for simplicity and maximum GPU performance." This rationale can inform future container configuration decisions.

A testable hypothesis: The hypothesis that LXC containers can bypass VFIO/IOMMU P2P bottlenecks was about to be tested. The message set up the conditions for this test, even though the test would ultimately reveal a different class of problem (driver/GSP firmware incompatibility).

The Thinking Process Visible in the Message

The message is short, but its structure reveals a clear thinking process:

1. Observation: "The container is unprivileged." — A factual statement about the current state.
2. Problem framing: "I need to either switch it to privileged or handle the device permissions carefully." — The assistant identifies two possible approaches, acknowledging that the current state (unprivileged without GPU mounts) is insufficient.
3. Decision: "For simplicity and maximum GPU performance, let me switch it to privileged and add the GPU mounts." — The assistant applies criteria (simplicity, performance) to choose between the options.
4. Action: "Let me stop the container first." — The assistant recognizes that configuration changes require the container to be stopped, and executes the stop command. This is textbook structured problem-solving: observe, frame, decide, act. The brevity of the message belies the depth of reasoning that preceded it — the entire prior session had been building toward this moment, gathering the information needed to make this decision confidently.

Conclusion

Message [msg 454] is a study in how a single, apparently simple decision can encapsulate hours of prior reasoning and set the direction for hours of subsequent investigation. The choice between privileged and unprivileged containers for GPU access is not merely a checkbox in a configuration file — it reflects trade-offs in security, performance, debuggability, and operational complexity. The assistant's decision to prioritize simplicity and performance was reasonable given the context: a dedicated ML server, a known P2P bottleneck to overcome, and a debugging session where clarity was paramount.

The fact that this decision ultimately led to a different dead end (CUDA initialization failure due to GSP firmware) does not make it the wrong decision. It was a necessary step in the process of elimination — ruling out container configuration as a variable, allowing the team to focus on the real issue of host-level driver compatibility. In engineering, knowing what doesn't work is often as valuable as knowing what does.