The Config That Bridges Host and Container: Updating LXC GPU Passthrough for a Migrated ML Workload

Introduction

In the world of high-performance machine learning infrastructure, the difference between a working deployment and a broken one often comes down to a handful of configuration lines. Message [msg 6775] captures one such moment: a brief but critical step in migrating a Qwen3.6-27B model deployment from a decommissioned host (kpro6) to a new one (kpro5). The message is deceptively simple — the assistant reads an LXC container configuration file to assess what needs changing — but it sits at the intersection of GPU virtualization, Proxmox administration, and the practical challenges of moving ML workloads between machines with different hardware topologies.

The Message

The assistant writes:

Good — nvidia0 and nvidia1 exist. Major 195 for control, 502 for UVM, 505 for caps. Now update CT129 config — remove nvidia2/nvidia3, update the cgroup device major numbers.

Then it executes a bash command to read the current LXC configuration:

ssh root@10.1.2.5 'cat /etc/pve/lxc/129.conf' 2>&1

The output reveals the full configuration of container 129 — a Proxmox LXC container named llm-two with 128 cores, 260 GB of memory, an 800 GB root filesystem, and a set of GPU device passthrough entries inherited from its previous life on a 4-GPU host.

The Context: A Migration Story

To understand why this message matters, we need to step back. The assistant has been migrating the Qwen3.6-27B model — a 27-billion-parameter dense model requiring roughly 55 GB in BF16 precision — from kpro6, which is being decommissioned, to kpro5. But kpro5 has a different GPU layout. It hosts five NVIDIA GPUs total: two RTX A6000s (48 GB each, Ampere architecture) and three RTX 3090s. The 3090s are already bound to vfio-pci and assigned to running virtual machines (VMs 125 and 128), leaving only the two A6000s available for the ML workload.

The assistant's work in the preceding messages ([msg 6750] through [msg 6774]) involved a complete NVIDIA driver installation on the Proxmox host: downloading and installing driver version 580.126.09, blacklisting the open-source nouveau driver, unbinding the A6000s from vfio-pci so the NVIDIA driver could claim them, and verifying that the device nodes /dev/nvidia0 and /dev/nvidia1 appeared correctly. The final verification showed two RTX A6000s with 49 GB each, running CUDA 13.0.

But the container that will run the model — CT129, an LXC container — was originally configured for kpro6's 4-GPU setup. Its configuration file still contained mount entries for /dev/nvidia2 and /dev/nvidia3, devices that no longer exist on this host. If the container were started without updating this config, it would either fail to mount the non-existent devices or, worse, silently ignore the error and leave the container with an incomplete view of the GPU topology.

Why LXC GPU Passthrough Is Delicate

Proxmox LXC containers handle GPU passthrough through two mechanisms that must work in concert. The first is lxc.mount.entry, which bind-mounts a device node from the host filesystem into the container's /dev directory. Each line like lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file tells LXC to make the host's /dev/nvidia0 accessible at the same path inside the container. The optional flag means the mount is skipped if the source doesn't exist — which is why the stale nvidia2 and nvidia3 entries wouldn't cause a hard failure, but they would leave the container unaware of the actual GPU count.

The second mechanism is lxc.cgroup2.devices.allow, which grants the container permission to access devices by their major and minor numbers. In Linux, device nodes are identified by major numbers (the driver type) and minor numbers (the specific device instance). NVIDIA GPUs use major number 195 for the control device (/dev/nvidiactl and /dev/nvidia0, /dev/nvidia1, etc.), major 502 for the UVM (Unified Virtual Memory) device, and major 505 for the NVIDIA capabilities device. The config already had c 195:* rwm, c 502:* rwm, and c 505:* rwm — broad permissions that cover all minor numbers under those majors.

The assistant's comment about "update the cgroup device major numbers" is slightly anticipatory — upon reading the config, it becomes clear that the cgroup entries are already correct. The real work is removing the stale mount entries for nvidia2 and nvidia3, which are artifacts of the 4-GPU configuration.

Assumptions and Knowledge

This message reveals several assumptions the assistant is making. First, it assumes that the LXC config was copied wholesale from kpro6 without modification — a reasonable assumption given that CT129 was originally created for the 4-GPU setup. Second, it assumes that removing the two stale mount entries is the only change needed; no other parameters (like memory, CPU allocation, or network config) require adjustment for the migration. Third, it assumes that the cgroup major numbers (195, 502, 505) are consistent across this host — which they are, as confirmed by the earlier ls -la /dev/nvidia* output in [msg 6774].

The input knowledge required to parse this message is substantial. A reader needs to understand:

The Broader Narrative

This message is a microcosm of the challenges in ML infrastructure migration. Moving a model from one host to another isn't just about copying files and restarting a service. It involves reconciling hardware differences — GPU counts, memory capacities, driver versions — with the configuration artifacts left behind by the previous setup. The two RTX A6000s on kpro5 provide 96 GB of combined VRAM, which is more than enough for the 55 GB Qwen3.6-27B model, but the container configuration still thinks there are four GPUs available.

The assistant's approach is methodical: verify the hardware (nvidia0 and nvidia1 exist), understand the current config (read the file), identify the discrepancies (nvidia2/nvidia3 are stale), and plan the fix (remove those entries). This pattern — observe, assess, act — is characteristic of reliable infrastructure work. The message captures the "assess" phase, where the assistant gathers the information needed to make the right edit.

There's also a subtle point about the optional flag in the mount entries. Because the stale entries use optional,create=file, the container wouldn't crash if started with the old config — it would simply skip mounting the non-existent devices. But the container would then have an incomplete GPU environment, potentially confusing the model serving software (SGLang or vLLM) that expects a consistent device topology. The assistant is preemptively fixing this rather than relying on the optional fallback, demonstrating a preference for correctness over tolerance.

Conclusion

Message [msg 6775] is a brief but essential step in a larger migration. It reads like a simple config check, but it encapsulates the complexity of GPU virtualization on Proxmox, the legacy of a 4-GPU configuration that no longer applies, and the meticulous work of aligning infrastructure with reality. The assistant's observation that "nvidia0 and nvidia1 exist" is both a confirmation of success (the driver installation worked) and a constraint (only two GPUs are available). The config file it reveals tells the story of a container built for a different machine, now being adapted for its new home. In the next messages, the assistant will edit this config, start the container, and proceed to deploy the Qwen3.6-27B model — but this message is where the path forward becomes clear.