Bridging the Gap: Infrastructure Decisions in the LXC GPU Passthrough Journey

Introduction

In the sprawling, multi-session saga of deploying the GLM-5-NVFP4 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 488] stands as a quiet but pivotal infrastructure moment. It is not a message of breakthrough discoveries or dramatic debugging triumphs. There are no NaN crashes being resolved here, no topology matrices being analyzed, no CUDA initialization errors being wrestled with. Instead, this message is about something more mundane yet equally critical: the careful, deliberate work of assembling the environment so that the ML stack can function at all.

The message reads:

Done. nvtop installed in both the container and host.

Now let me continue setting up the ML stack. I need to bind-mount the model cache into the container and install PyTorch/sglang:

[bash command that unmounts the VM disk, stops the container, adds a bind mount for shared storage, and restarts the container]

This single exchange, sandwiched between the breakthrough of confirming bare-metal GPU topology inside an LXC container ([msg 473]) and the looming blocker of CUDA initialization failure ([msg 489]), represents the unglamorous but essential work of infrastructure assembly. To understand why this message matters, we must understand the full context of the struggle that precedes it and the assumptions that underpin it.

The Broader Context: Escaping the VFIO/IOMMU Bottleneck

By the time we reach message [msg 488], the conversation has traversed a long and winding road. The user and assistant began in Segment 0 with a clean Ubuntu 24.04 installation, installing NVIDIA drivers, CUDA Toolkit, and wrestling with flash-attn compilation issues. They deployed the GLM-5-NVFP4 model using SGLang across 8 GPUs with tensor parallelism, only to encounter persistent NaN crashes during decode. After extensive debugging of attention backends and quantization compatibility, they achieved a working baseline but identified a critical performance bottleneck: PCIe P2P (Peer-to-Peer) latency caused by virtualization overhead in the Proxmox KVM environment.

The topology inside the KVM VM showed PHB (PCIe Host Bridge) connections between all GPU pairs — the worst possible topology for P2P DMA, meaning all GPU-to-GPU communication had to go through the host's PCIe root complex rather than directly between devices. This is a known issue with VFIO passthrough: the hypervisor interposes on PCIe transactions, preventing direct GPU-to-GPU communication.

Segment 3 was a deep dive into fixing this at the hypervisor level: modifying Proxmox host kernel parameters, migrating the VM from i440FX to Q35 chipset, fixing BAR allocation, and attempting to disable ACS (Access Control Services). But ultimately, the hardware topology itself was the enemy — each GPU sat on its own PCIe root complex, making P2P fundamentally impossible regardless of software configuration.

Segment 4, where message [msg 488] lives, represents a strategic pivot: instead of trying to fix the VM's PCIe topology, the assistant proposed using an LXC container on the Proxmox host. LXC containers share the host kernel and have direct access to hardware devices, bypassing the VFIO/IOMMU layer entirely. The critical validation came in [msg 473], where nvidia-smi topo -m inside the container showed the true bare-metal topology: NODE within sockets and SYS across sockets — exactly what P2P DMA requires.

Why This Message Was Written: The Motivation and Reasoning

Message [msg 488] is written to accomplish two distinct goals, one reactive and one proactive.

The reactive goal: responding to the user's request. In [msg 486], the user interjected with a simple request: "btw install nvtop." The assistant acknowledges this in the opening line — "Done. nvtop installed in both the container and host." This is a quick side-task, but it reveals something about the workflow: the user is thinking about monitoring and observability. nvtop is the NVIDIA GPU equivalent of htop or top — it provides real-time visualization of GPU utilization, memory usage, temperature, and process information. Installing it in both the container and the host suggests the assistant is thinking ahead: the host might need monitoring for the driver-level GPU state, while the container will be where the actual ML workloads run.

The proactive goal: continuing the ML stack setup. The bulk of the message is a bash script executed on the Proxmox host (10.1.2.6) that performs three operations:

Unmount the VM disk — cleanup from the previous step where the assistant mounted the VM 128's ZFS zvol to copy the HuggingFace model cache ([msg 484]). This is good hygiene: leaving filesystems mounted when they're no longer needed risks data corruption or conflicts.
Stop the container and add a bind mount — This is the core operation. The assistant adds mp0: /shared,mp=/shared to the LXC configuration file at /etc/pve/nodes/kpro6/lxc/129.conf. This creates a bind mount from the host's /shared directory into the container at the same path. The /shared directory is a ZFS dataset created in [msg 479] specifically to hold the HuggingFace model cache.
Restart the container — After modifying the config, the container must be stopped and started (not just restarted via pct reboot) because LXC mount points are processed during container initialization.

The Decisions Made and Their Rationale

Several implicit and explicit decisions are embedded in this message.

Decision 1: Bind-mount the model cache rather than re-downloading. The GLM-5-NVFP4 model is approximately 296GB (the copy operation in [msg 485] showed a final size of 405GB, though this may include download artifacts or multiple versions). Re-downloading this over the network would take hours and consume bandwidth. The VM 128 already had the model cached in its HuggingFace hub directory. By mounting the VM's ZFS zvol read-only ([msg 484]), copying the model to a shared ZFS dataset, and then bind-mounting that dataset into the LXC container, the assistant avoids a massive redundant download. This is a classic infrastructure optimization: reuse cached artifacts across environments.

Decision 2: Using a shared ZFS dataset rather than directly mounting the VM's zvol. The VM's disk is a ZFS zvol (rpool/data/vm-128-disk-0) with a GPT partition table. While it could theoretically be mounted read-only indefinitely, this creates a dependency on the VM's disk remaining intact and unmounted by the VM itself. A shared dataset is cleaner — it's a dedicated storage location that won't be affected by VM operations.

Decision 3: Stopping the container to modify its configuration. Proxmox LXC containers cannot have their mount points modified while running. The pct stop and pct start cycle is necessary. The assistant waits 2 seconds after stopping and 3 seconds after starting, which is a reasonable heuristic for the container to transition states.

Decision 4: Appending to the config file rather than using pct set. The assistant uses cat >> to append the mount point entry directly to the configuration file. This is a lower-level approach than using the pct set command, which would be the "proper" Proxmox API method. This decision is likely driven by convenience — the assistant is already in a bash heredoc and can simply append text. However, it carries a subtle risk: if the mount point entry already existed (e.g., from a previous attempt), this would create a duplicate. In this case, it's the first mount point being added, so the risk is minimal.

Assumptions Embedded in This Message

Every technical decision rests on assumptions, and message [msg 488] is no exception.

Assumption 1: The LXC container approach will ultimately work. The assistant is investing significant effort in setting up the ML stack inside the container — installing CUDA toolkit, creating a Python venv, copying the model cache, configuring bind mounts. This assumes that the CUDA initialization issue (which emerges in the very next message) can be resolved. At this point in the conversation, the assistant has seen that nvidia-smi works perfectly inside the container, detecting all 8 GPUs, and the topology looks correct. The assumption is that the userspace CUDA libraries will work once properly installed. As we know from subsequent messages, this assumption proves incorrect — CUDA initialization fails with error code 3 on both the host and container, ultimately traced to a GSP firmware compatibility issue between the NVIDIA driver and the Proxmox VE kernel.

Assumption 2: The /shared dataset has sufficient space. The model cache is 405GB. The assistant created the shared dataset in [msg 479] without specifying a size limit. On the rpool ZFS pool, which has 792G available (as shown in [msg 477] for the subvol-129-disk-0, though the pool itself may have more), 405GB is a significant but manageable allocation. However, the ML stack will need additional space for Python venvs, compiled packages (flash-attn, vllm, sglang), and temporary build artifacts.

Assumption 3: The container's SSH and networking will survive the stop/start cycle. The container has IP 10.1.230.174 on the nv bridge. Stopping and starting the container will cause it to re-acquire its DHCP lease. The assistant's earlier troubleshooting (<msg id=464-466>) showed that SSH takes a moment to become available after container start. The 3-second sleep after pct start is optimistic — in practice, SSH might not be ready yet, but the assistant doesn't attempt to SSH into the container in this message, so it's not immediately problematic.

Assumption 4: The bind mount path /shared doesn't conflict with anything inside the container. The container is an Ubuntu 24.04 system. The /shared path is not a standard Linux directory, so it's unlikely to conflict with system files. However, if the container's root filesystem already has a /shared directory (e.g., from a previous configuration), the bind mount would overlay it.

Input Knowledge Required to Understand This Message

To fully grasp what's happening in message [msg 488], the reader needs knowledge spanning several domains:

Proxmox VE administration: Understanding that LXC containers can have bind mounts configured via mp0 entries in their config files, that containers must be stopped to modify certain configuration parameters, and that pct is the Proxmox container management tool.

ZFS storage concepts: Understanding that ZFS zvols are raw block devices that can contain partition tables and filesystems, that ZFS datasets can be created with specific mount points, and that ZFS snapshots and clones could provide alternative approaches.

NVIDIA GPU virtualization: Understanding the difference between VFIO passthrough (KVM) and device bind-mounts (LXC), and why the former introduces PCIe topology virtualization (PHB links) while the latter preserves bare-metal topology.

HuggingFace model caching: Understanding that HuggingFace stores downloaded models in ~/.cache/huggingface/hub/ with a specific directory naming convention (models--username--modelname), and that these caches can be shared across environments.

Linux filesystem mounting: Understanding that mount -o ro creates a read-only mount, that umount unmounts a filesystem, and that bind mounts (--bind) make a directory tree accessible at another location.

Output Knowledge Created by This Message

This message produces several concrete outcomes:

A shared HuggingFace model cache at /shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4 on the Proxmox host, accessible to any container or host process that mounts /shared.
An updated LXC container configuration with the mp0 bind mount entry, ensuring the model cache persists across container restarts.
A cleanly unmounted VM disk, avoiding potential ZFS conflicts.
A running container with the model cache available at /shared inside the container, ready for the ML stack installation that follows.
Documentation of the approach — the sequence of commands in this message serves as a reproducible recipe for sharing storage between Proxmox host and LXC containers.

The Thinking Process Visible in This Message

While the assistant's reasoning is not explicitly shown in this message (there is no visible chain-of-thought or deliberation), the structure of the bash command reveals a clear mental model:

The assistant is thinking in terms of dependencies and ordering. The VM disk must be unmounted before the container is stopped (though the order doesn't strictly matter here — the VM disk mount and the container are independent). The container must be stopped before the config is modified. The config must be modified before the container is started. This sequential dependency chain is reflected in the linear structure of the bash script.

The assistant is also thinking about state management. It checks the container status after starting (pct status 129) to confirm the operation succeeded. It displays the updated config (cat /etc/pve/nodes/kpro6/lxc/129.conf) to provide visibility into what was changed. These are debugging affordances — if something goes wrong later, the user (or the assistant in a future message) can see exactly what state the system was left in.

The decision to use cat >> with a heredoc rather than pct set is also revealing. The assistant is operating at the level of raw configuration files, not the Proxmox API. This suggests a mental model that treats the Proxmox host as a Linux system first and a hypervisor second — a pragmatic approach that prioritizes getting things done over using the "proper" interfaces.

The Broader Significance: Infrastructure as Enabler

Message [msg 488] is, on its face, unremarkable. It's a routine infrastructure operation: mount a shared filesystem, restart a container. But in the context of the multi-session struggle to deploy GLM-5-NVFP4 across 8 Blackwell GPUs with P2P DMA, it represents a critical enabling step.

The P2P DMA bottleneck in the KVM VM was a hard wall. The LXC container approach showed promise — the topology was correct, the GPUs were visible, the driver was installed. But without the model cache, the container was an empty shell. The ML stack (PyTorch, SGLang, flash-attn) could be installed from packages, but the 296GB model itself had to be available. By bridging the model cache from the VM's storage to the container, this message closes the gap between "the GPUs work" and "the model can run."

It's also a testament to the flexibility of the Proxmox platform. The ability to mix KVM VMs and LXC containers, to mount ZFS zvols read-only, to create shared datasets, and to bind-mount host directories into containers — these are all features that enable the kind of iterative experimentation seen in this conversation. When one approach (KVM with VFIO) hits a fundamental hardware limitation, another approach (LXC with device bind-mounts) can be tried without starting from scratch.

Conclusion

Message [msg 488] is a bridge — between the VM that held the model and the container that would run it, between the breakthrough of correct GPU topology and the eventual deployment of the ML workload. It's a reminder that in complex infrastructure work, the mundane tasks — unmounting filesystems, editing config files, restarting containers — are just as important as the dramatic debugging victories. The assistant's methodical approach, its attention to state management, and its pragmatic use of low-level tools all reflect the kind of thinking that makes large-scale ML deployments possible.

The message also embodies a key lesson about infrastructure debugging: when you hit a fundamental limitation (like hardware topology preventing P2P DMA in a VM), the solution is often not to fight the limitation but to change your approach entirely. The pivot from KVM to LXC, and the careful work of porting the environment across that boundary, is what makes the rest of the story possible — even if the next message reveals yet another obstacle in the form of CUDA initialization failure.