The Critical Pivot: Installing NVIDIA Userspace with `--no-kernel-module` in an LXC Container

Message Overview

In message 471 of this opencode session, the assistant executes a single command inside an LXC container running on a Proxmox VE host:

ssh root@10.1.230.174 "chmod +x /root/NVIDIA-Linux-x86_64-590.48.01.run && /root/NVIDIA-Linux-x86_64-590.48.01.run --no-kernel-module --silent --no-x-check 2>&1"

The output confirms the installer ran successfully, decompressing the 397MB driver archive. On the surface, this is a routine software installation step. But in the context of the broader session — a multi-day effort to deploy the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs — this message represents a decisive architectural pivot. The --no-kernel-module flag is the key that unlocks an entirely different approach to GPU virtualization, one chosen specifically to bypass a crippling P2P (peer-to-peer) DMA bottleneck that had stymied the team's previous KVM-based deployment.

Why This Message Was Written: The P2P Bottleneck Crisis

To understand why this message exists, one must trace the thread back through the preceding segments. The team had successfully deployed the GLM-5-NVFP4 model on 8 Blackwell GPUs using SGLang within a KVM virtual machine. They had resolved NaN decode crashes, established baseline throughput, and identified the core performance bottleneck: PCIe P2P latency between GPUs. In a KVM VM using VFIO passthrough, the GPU topology reported by nvidia-smi topo -m showed PHB (PCIe Host Bridge) interconnects rather than the NODE or PIX topology that enables direct GPU-to-GPU DMA. This virtualization-induced topology degradation meant that tensor-parallel inference — which relies heavily on fast P2P transfers for all-reduce operations during model sharding — was fundamentally limited by host-side PCIe traversal through the hypervisor.

Segment 3 of the session documented a deep investigation into enabling P2P DMA for GPUs in the Proxmox VM. The team attempted host kernel parameter modifications, migration to the Q35 chipset, BAR allocation fixes, and ACS (Access Control Service) disabling — all to merge IOMMU groups and allow direct GPU communication. The investigation concluded with a sobering discovery: the hardware topology placed each GPU on its own PCIe root complex, making P2P fundamentally impossible within the constraints of VFIO virtualization. The IOMMU groups could not be merged because the physical PCIe hierarchy itself prevented it.

This is where segment 4 — and message 471 — enters the picture. Faced with a hard architectural limit in the KVM approach, the team pivoted to an LXC container strategy. LXC containers share the host kernel, meaning the NVIDIA kernel module loaded on the Proxmox host is directly available inside the container. The GPU device nodes can be bind-mounted from the host into the container. Crucially, inside an LXC container, nvidia-smi topo -m shows the true bare-metal topology — NODE within sockets, SYS across sockets — because the container does not interpose a virtual PCIe layer. This means P2P DMA should theoretically work at native hardware speeds.

The Decision Embedded in `--no-kernel-module`

The --no-kernel-module flag is the linchpin of the entire LXC strategy. In a standard NVIDIA driver installation on a bare-metal machine or a full VM, the .run installer compiles and installs a kernel module (nvidia.ko, nvidia-uvm.ko, etc.) that interfaces with the GPU hardware. This module must match the exact kernel version and configuration. In the KVM VM approach used earlier in the session, the guest VM had its own kernel and required a full driver installation including the kernel module.

But in the LXC approach, the kernel module is already loaded on the Proxmox host. The container shares the host's kernel — that's the fundamental nature of containerization. Installing a kernel module inside the container would be not only redundant but potentially destructive: it would attempt to load a second copy of the module into the same kernel, causing conflicts. The --no-kernel-module flag instructs the installer to skip kernel module compilation and installation entirely, deploying only the userspace components: the CUDA runtime libraries (libcuda.so), the NVIDIA Management Library (libnvidia-ml.so), the OpenGL/GLX libraries, the CUDA compute toolchain, and the various utility binaries (nvidia-smi, nvidia-cuda-mps-control, etc.).

This decision reflects a deep understanding of Linux container architecture. The assistant recognized that the container's device nodes — /dev/nvidia0 through /dev/nvidia7, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, and /dev/nvidia-caps/ — are bind-mounted from the host. These device nodes are managed by the host's kernel module. The container only needs the userspace libraries to communicate with these devices via ioctl() calls and CUDA driver API entry points. The --no-kernel-module flag is thus not an optimization or a convenience — it is a correctness requirement.

Assumptions Made

The message rests on several critical assumptions, most of which were validated in preceding messages but deserve examination:

Assumption 1: The host kernel module is fully functional. The NVIDIA driver 590.48.01 had been installed on the Proxmox host in message 450 using --dkms --silent --no-x-check. The host's nvidia-smi output in message 451 confirmed all 8 GPUs were visible and the driver was operational. However, the host kernel was the Proxmox VE kernel (6.8.12-9-pve), not a standard Ubuntu kernel. The driver's DKMS (Dynamic Kernel Module Support) integration had compiled the module against this specific kernel, and it loaded successfully. The assumption was that this host-side kernel module would service CUDA calls from the container transparently.

Assumption 2: Device node major numbers are consistent. The container's device nodes were bind-mounted from the host. The assistant had verified in message 452 that the host's device nodes had major numbers 195 (nvidia), 504 (nvidia-uvm), and 507 (nvidia-caps). The container configuration in message 455 explicitly allowed these major numbers via lxc.cgroup2.devices.allow entries. The assumption was that the userspace libraries inside the container would open these same device nodes and communicate correctly.

Assumption 3: The userspace driver version matches the kernel module version. The installer being run inside the container was the exact same .run file (590.48.01) that was used on the host. This is essential because the CUDA driver API is a versioned interface between libcuda.so and the kernel module. Mismatched versions can cause CUDA_ERROR_NOT_INITIALIZED or other initialization failures.

Assumption 4: The container has sufficient userspace dependencies. The container is running Ubuntu 24.04 (the same OS as the KVM VM). The preceding message (msg 470) confirmed that build-essential, wget, and curl were already installed. The NVIDIA userspace installer requires standard shared library dependencies (glibc, libstdc++, etc.) which are present in any standard Ubuntu installation.

Assumption 5: SSH access to the container works. The command is executed via ssh root@10.1.230.174. This IP address was confirmed in message 465. The assistant had just verified SSH access in message 466, where whoami && hostname && ls /dev/nvidia* returned successfully.

Mistakes and Incorrect Assumptions

The most significant error — which becomes apparent in the subsequent messages beyond message 471 — is the assumption that the host-side NVIDIA driver stack is fully compatible with the Blackwell architecture (RTX PRO 6000) when running on the Proxmox VE kernel. The chunk summary for segment 4 reveals that after this installation, CUDA runtime initialization (cuInit) fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED) both on the host and inside the container. The root cause is a driver compatibility issue: the 590.48.01 driver lacks Blackwell GSP firmware files (only gsp_ga10x.bin and gsp_tu10x.bin exist), and the older PVE kernel (6.8.12) may not support the Blackwell architecture's GSP (GPU System Processor) requirements.

This is a subtle but critical failure mode. The NVIDIA driver since the Turing architecture has used a GSP firmware coprocessor on the GPU to handle initialization, power management, and other tasks. Blackwell GPUs require specific GSP firmware that may not be included in driver 590.48.01 (which was released before Blackwell's full firmware stack was finalized). The nvidia-smi tool can enumerate the GPUs because it uses a different communication path (NVML library → kernel module → PCIe config space), but CUDA initialization requires the GSP firmware to be loaded and operational.

The assistant's assumption that "the driver works because nvidia-smi works" was reasonable but incomplete. This is a classic pitfall in GPU computing: nvidia-smi detection does not guarantee CUDA operational readiness, especially on newer GPU architectures with firmware-based initialization paths.

A secondary issue is the assumption about the Proxmox VE kernel's compatibility. The PVE kernel is based on Debian's kernel with additional patches for virtualization features. It may lack certain NVIDIA driver backports or compatibility layers that the standard Ubuntu kernel (used in the KVM VM) provides. The KVM VM succeeded because the guest's own NVIDIA driver stack handled the GPUs via VFIO passthrough without requiring host-level GSP firmware — the GPU was presented to the guest as a raw PCIe device, and the guest's driver handled all firmware interactions.

Input Knowledge Required

To understand this message fully, one needs:

LXC container architecture: Understanding that containers share the host kernel and that device bind-mounts provide access to hardware without requiring in-container kernel modules.
NVIDIA driver architecture: Knowledge that the NVIDIA driver has two components — a kernel module (nvidia.ko) that manages GPU hardware access and device nodes, and userspace libraries (libcuda.so, libnvidia-ml.so) that applications link against. The --no-kernel-module flag only makes sense with this mental model.
The P2P bottleneck context: Awareness that the team had spent segments 2 and 3 debugging PCIe P2P limitations in a KVM VM, and that the LXC approach was chosen specifically to expose bare-metal GPU topology.
The hardware topology: Understanding that 8 RTX PRO 6000 Blackwell GPUs are distributed across two NUMA sockets (GPUs 0-3 on NUMA 0, GPUs 4-7 on NUMA 1) with NODE connectivity within each socket and SYS connectivity across sockets.
The Proxmox environment: Familiarity with Proxmox VE, LXC configuration syntax, ZFS subvolumes, and the unprivileged-to-privileged container conversion that preceded this message.
The model deployment context: Awareness that this infrastructure supports GLM-5-NVFP4 inference using SGLang with tensor parallelism across 8 GPUs, which requires efficient P2P communication.

Output Knowledge Created

This message creates several important pieces of knowledge:

A validated LXC installation procedure: The command demonstrates that NVIDIA's userspace driver can be installed inside an LXC container using --no-kernel-module, with the installer completing successfully. This procedure can be replicated for other GPU-accelerated LXC deployments.
Confirmation of the architectural pivot: The successful installation confirms that the LXC path is viable from a software installation standpoint. The device nodes are accessible, the installer runs without errors, and the userspace libraries are deployed.
A testable hypothesis: The installation sets up the conditions to test whether CUDA applications can initialize and whether P2P DMA works at native speeds inside the container. The subsequent cuInit failure (revealed in the chunk summary) becomes the next debugging target.
A reference point for driver version consistency: By using the exact same driver version (590.48.01) on both host and container, the message establishes the principle that kernel module and userspace driver versions must match in an LXC GPU setup.

The Thinking Process

The assistant's reasoning is visible in the sequence of messages leading to this point. The progression shows a systematic debugging methodology:

Problem identification (Segments 2-3): The KVM VM's PHB topology prevents P2P DMA, limiting tensor-parallel performance.
Root cause analysis (Segment 3): The physical PCIe topology places each GPU on its own root complex, making IOMMU group merging impossible.
Solution design (Segment 4, messages 442-470): Switch to LXC containers, which share the host kernel and see bare-metal topology. Install NVIDIA driver on host, configure device bind-mounts, fix uid mapping, install userspace components in container.
Execution (Message 471): The final step of the userspace installation, using --no-kernel-module as the key architectural decision. The assistant does not explain the --no-kernel-module flag explicitly in the message — the reasoning is implicit in the context. The brevity of the message ("Good, they're already installed. Now install the NVIDIA userspace driver:") reflects confidence that the reader (the user) understands why this flag is necessary. The assistant has internalized the architectural distinction between kernel module and userspace components and applies it without fanfare. This is characteristic of expert-level technical work: the most critical decisions are often encoded in a single flag or parameter, invisible to the novice but laden with meaning for those who understand the system architecture. The --no-kernel-module flag is not just an installation option — it is the distilled expression of the entire LXC strategy, a recognition that containerization changes the fundamental relationship between software and hardware.

The Critical Pivot: Installing NVIDIA Userspace with --no-kernel-module in an LXC Container