The Unremarkable Command That Reveals Everything: Installing Build Tools in an LXC Container for GPU Passthrough

In the sprawling narrative of deploying a GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, most of the dramatic tension comes from spectacular failures: NaN crashes during model decode, PCIe P2P bandwidth bottlenecks, CUDA initialization errors, and driver incompatibilities with the Proxmox VE kernel. But sometimes the most revealing moments come from the quiet, mundane steps — the commands that seem almost too trivial to mention. Message [msg 469] is one such moment.

The message itself is deceptively simple:

[assistant] [bash] ssh root@10.1.230.174 "apt update 2>&1 | tail -5 && apt install -y build-essential wget curl 2>&1 | tail -5"

>

Get:35 http://archive.ubuntu.com/ubuntu noble-security/multiverse amd64 c-n-f Metadata [396 B] Fetched 20.8 MB in 11s (1857 kB/s) Reading package lists... Building dependency tree... 170 packages can be upgraded. Run 'apt list --upgradable' to see them.

>

[bash metadata] bash tool terminated command after exceeding timeout 60000 ms

On its face, this is an assistant SSHing into a container and running a package manager update followed by an installation of three basic development tools. The output shows the update succeeded — 20.8 MB fetched in 11 seconds — and then the tool was killed after a 60-second timeout. But this message sits at a critical inflection point in a much larger story, and understanding why it was written, what it reveals, and what went wrong requires unpacking layers of context that span the entire session.

The Strategic Pivot: Why LXC?

To understand message [msg 469], we must first understand the strategic situation that led to it. The session had been pursuing GPU passthrough via KVM virtualization, where the Proxmox host assigned physical GPUs to a virtual machine using VFIO. This approach worked — the VM could see and use all eight GPUs — but it came with a fatal flaw: the PCIe topology inside the VM showed all GPUs as being behind a PCIe Host Bridge (PHB), rather than their true physical arrangement of NODE (within a socket) and SYS (across sockets). This virtualized topology prevented Peer-to-Peer (P2P) DMA transfers between GPUs, which is essential for high-performance distributed inference in large models like GLM-5-NVFP4.

The assistant had spent significant effort trying to work around this: modifying host kernel parameters, migrating the VM from i440FX to Q35 chipset, fixing BAR allocation, and attempting to disable ACS (Access Control Services). All of these efforts ultimately failed because the hardware topology — each GPU on its own PCIe root complex — fundamentally prevented P2P within the VM.

The LXC container approach was the next logical gambit. Unlike a full KVM virtual machine, LXC containers share the host kernel directly. This means the PCIe topology visible inside the container should match the bare-metal topology of the host. If the assistant could get the NVIDIA driver working on the Proxmox host and then bind-mount the GPU device nodes into the container, the GPUs would appear with their true physical topology, and P2P DMA should work.

Messages [msg 440] through [msg 468] document this pivot: installing the NVIDIA driver (590.48.01) on the Proxmox host, converting the existing unprivileged LXC container to privileged, configuring bind-mounts for all eight GPU device nodes, fixing the uid/gid mapping that broke when switching from unprivileged to privileged mode, and finally verifying SSH access to the container. By message [msg 468], the assistant had copied the NVIDIA driver .run file into the container's filesystem. The next logical step — message [msg 469] — was to install the NVIDIA userspace driver inside the container.

What the Message Actually Does

The command runs two operations sequentially via SSH into the container at IP 10.1.230.174:

  1. apt update: Refresh the package index from Ubuntu's repositories. The output confirms this succeeded — 20.8 MB fetched in 11 seconds from noble-security (Ubuntu 24.04, codename Noble). The tail shows 170 packages are upgradable.
  2. apt install -y build-essential wget curl: Install three packages. build-essential provides the GNU C/C++ compiler toolchain (gcc, g++, make, etc.). wget and curl are HTTP download tools. These are prerequisites for running the NVIDIA driver installer, which is a self-extracting .run archive that needs to compile kernel modules (or, in this case, install userspace components). The 2>&1 | tail -5 on each command pipes stderr to stdout and shows only the last 5 lines, keeping the output concise. The command timed out after 60 seconds. The apt update completed in about 11 seconds (as shown by the "Fetched 20.8 MB in 11s" line), but the apt install phase either hung or took longer than the remaining ~49 seconds. Message [msg 470] reveals what happened: the apt command was waiting for a lock held by another process (PID 669), and once it acquired the lock, all three packages were already installed.

The Hidden Complexity Behind a Simple Command

What makes this message interesting is not what it does, but what it represents. The assistant is making several assumptions:

Assumption 1: The container's package manager is healthy. After the unprivileged-to-privileged conversion and the uid/gid shifting operation (messages [msg 460]-[msg 462]), the container's filesystem had been modified extensively. The assistant assumes that apt still functions correctly — that the package database wasn't corrupted, that repository configurations survived, and that network access works. The successful apt update confirms this assumption was correct.

Assumption 2: Build tools are needed for the NVIDIA driver. The NVIDIA .run installer typically needs a compiler toolchain even when using --no-kernel-module mode (which skips kernel module compilation). The installer may need to compile helper binaries or check system compatibility. This is a reasonable assumption, though message [msg 470] shows the packages were already installed, making this step technically redundant.

Assumption 3: The container has network access to Ubuntu repositories. This was verified earlier when the container was first set up, but the assistant is re-verifying it. The 11-second fetch time confirms good network connectivity.

Assumption 4: SSH access works reliably. The assistant had just fixed SSH access in message [msg 466] after the uid/gid shift broke it. Running another SSH command tests that the fix is durable.

The Timeout: A Minor Glitch with Deeper Implications

The 60-second timeout is the most visible "failure" in this message, but it's a mundane one. The apt install command was blocked by a lock held by another process (likely a background apt or dpkg process from the container's startup or a previous command). This is a common occurrence in container environments where multiple setup commands might overlap.

But the timeout also reveals something about the assistant's operating constraints: it has a 60-second timeout on bash commands. This is a practical limit — the assistant cannot wait indefinitely for a command to complete. In the context of GPU setup, where driver compilation can take minutes, this timeout is a significant constraint that shapes the assistant's strategy. It's why the assistant uses --no-kernel-module in the container (message [msg 471]) — kernel module compilation would almost certainly exceed the timeout.

Knowledge Required to Understand This Message

To fully grasp what's happening in message [msg 469], a reader needs:

  1. Understanding of LXC containers: That they share the host kernel and can access host devices via bind-mounts, unlike KVM VMs which have virtualized hardware.
  2. Knowledge of NVIDIA driver architecture: The distinction between the kernel-mode driver (nvidia.ko, nvidia-uvm.ko) and the userspace driver (libcuda.so, nvidia-smi, etc.). The kernel module is installed on the host; the userspace components must be installed in any container that needs CUDA.
  3. Awareness of the P2P DMA problem: Why the virtualized PCIe topology in the KVM VM was a blocker, and why the LXC approach was expected to solve it.
  4. Familiarity with the session's history: The failed attempts to enable P2P in the VM, the driver installation on the host, and the container configuration steps.
  5. Understanding of the uid/gid mapping issue: Why converting an unprivileged container to privileged breaks file ownership, and why fixing it was necessary before SSH would work.

Knowledge Created by This Message

Message [msg 469] produces several pieces of actionable knowledge:

  1. The container's package manager is functional after the uid/gid fix. This confirms the container conversion didn't corrupt the system.
  2. Network connectivity is good — 20.8 MB in 11 seconds is a healthy transfer rate.
  3. The container is running Ubuntu 24.04 (Noble) with 170 upgradable packages. This matches the expected OS version.
  4. The apt install command may need retrying due to the lock contention, which the assistant does in message [msg 470].
  5. The build tools are prerequisites for the NVIDIA driver installation — this step must complete before the driver can be installed, which happens in message [msg 471].

The Broader Significance

In the grand narrative of this session, message [msg 469] is a hinge point. It's the moment when the assistant transitions from host-side configuration to container-side software installation. Everything before this message was about getting the GPUs visible and accessible in the container. Everything after will be about whether the CUDA runtime actually works inside the container — and as the chunk summary reveals, it ultimately fails with CUDA_ERROR_NOT_INITIALIZED due to Blackwell GSP firmware incompatibility with the PVE kernel.

The message is also a testament to the assistant's methodical approach. It doesn't jump straight to installing the NVIDIA driver. It first ensures the basic toolchain is present, verifies network access, and confirms the container is in a healthy state. This step-by-step debugging methodology — establish a working baseline before adding complexity — is essential when dealing with the intricate failure modes of GPU virtualization.

The timeout, while minor, is a reminder that even the most carefully planned operations encounter friction. The assistant doesn't panic or overreact; it simply retries in the next message. This resilience in the face of small failures is characteristic of effective system administration, whether performed by a human or an AI.

Ultimately, message [msg 469] is a message about foundations. It's not glamorous. It doesn't produce dramatic output or solve a hard problem. But it lays the groundwork for everything that follows — and in a session where the stakes are deploying a state-of-the-art language model across eight cutting-edge GPUs, the quality of the foundations determines whether the whole structure stands or falls.