The Pivot Point: Researching NVIDIA Driver Installation on Proxmox Host for LXC GPU Passthrough

In the ongoing struggle to achieve performant multi-GPU inference with 8 RTX PRO 6000 Blackwell GPUs, a critical pivot occurred at message [msg 431]. This message, a brief research dispatch from the assistant, represents the moment when the team abandoned the KVM/VFIO path and began investigating the LXC container alternative in earnest. Though it contains only two web fetches, this message is the hinge upon which the entire subsequent segment of the conversation turns — it is the bridge between the question "Would I get better results in Proxmox LXC containers?" and the comprehensive implementation guide that follows.

The Context: Why LXC Was Suddenly the Best Option

To understand why this message matters, one must appreciate the dead end the conversation had reached. The team had spent multiple segments deploying the GLM-5-NVFP4 model on 8 Blackwell GPUs through a KVM virtual machine on Proxmox. While they successfully got inference running, they discovered a fundamental performance bottleneck: the VFIO/IOMMU layer that enables GPU passthrough to the VM also prevents true GPU-to-GPU Peer-to-Peer (P2P) DMA communication. Each GPU-to-GPU transfer required a staged copy through host RAM, incurring a ~13µs latency floor that crippled NCCL performance for small messages — precisely the kind of traffic that MoE (Mixture of Experts) dispatch generates.

The team had explored every conceivable workaround within the KVM paradigm. They tried ACS override patches to merge IOMMU groups. They investigated IOMMUFD, the newer device-centric VFIO backend. They consulted NCCL issues on GitHub (<msg id=422-423>) confirming that P2P in VMs was "not planned" for support. They even attempted bare-metal-like configurations with Q35 chipset migration and BAR allocation fixes. None of it worked. The fundamental topology — each GPU on its own PCIe root complex, in its own IOMMU group — made P2P impossible under VFIO.

Then, in [msg 427], the user asked a question that changed the trajectory: "Would I get better results in Proxmox LXC containers?" The assistant's initial response in [msg 428] was cautiously optimistic but hedged, saying LXC "won't help with P2P either" before correcting itself mid-thought. Upon reflection, the assistant realized that LXC containers share the host kernel and access devices via bind mounts, not VFIO — meaning no IOMMU translation layer intercepts DMA. P2P should work natively. The user's follow-up in [msg 429] — "How do I setup proxmox with nvidia driver and LXC in this way?" — triggered the research phase captured in this message.

What the Message Actually Contains

Message [msg 431] is a pure research message. It contains two tool calls executed in parallel:

1. A webfetch to a Proxmox forum thread: https://forum.proxmox.com/threads/getting-nvidia-drivers-to-install-on-host-for-use-in-lxc-with-cuda.150143/

This thread, titled "Getting Nvidia drivers to install on host for use in LXC with CUDA," is directly on point. It addresses the exact scenario the team faces: installing NVIDIA drivers on the Proxmox host (not inside a VM) so that LXC containers can leverage GPU acceleration. The forum thread likely contains community-tested procedures, workarounds for common pitfalls (like DKMS failures on the PVE kernel), and configuration examples for bind-mounting device nodes.

2. An exa_web_search_exa with the query: &#34;Proxmox PVE kernel NVIDIA driver DKMS install \&#34;pve-headers\&#34; 590 Blackwell RTX 6000&#34;

This search is more targeted. It combines several critical constraints:

The Reasoning Behind the Search Strategy

The assistant's choice of these two searches reveals a methodical research approach. Rather than blindly following generic tutorials, the assistant is seeking out:

  1. Community-validated procedures: The Proxmox forum thread provides real-world experience from users who have already navigated the NVIDIA-on-Proxmox-host installation. This is crucial because Proxmox is not a standard Linux distribution — it's a specialized virtualization platform with its own kernel, package management, and boot process. Generic NVIDIA installation guides for Ubuntu or Debian may not account for PVE-specific quirks like the custom kernel headers path or the pve-enterprise vs pve-no-subscription repository differences.
  2. Hardware-specific compatibility data: The second search explicitly targets the Blackwell RTX 6000 + driver 590 combination on the PVE kernel. This is forward-looking risk mitigation. The assistant knows from earlier research (the Level1Techs forum thread in [msg 422]) that Blackwell GPUs have had P2P bugs and driver compatibility issues. By searching for this specific combination, the assistant hopes to surface any known incompatibilities between the PVE kernel and the Blackwell NVIDIA driver before attempting the installation.

Assumptions Embedded in This Research

Several assumptions underlie the assistant's research at this point:

Assumption 1: The NVIDIA driver can be installed on the Proxmox host. This is not trivial. Proxmox is built on Debian but uses its own kernel. NVIDIA's official driver supports only specific kernel versions. If the PVE kernel is too old or too new, DKMS compilation may fail. The assistant is implicitly assuming that the PVE kernel 6.8.12-9 (as revealed later in the segment) is compatible with driver 590.48.01.

Assumption 2: The host can run the NVIDIA driver while the VFIO-pci driver is also loaded. This turns out to be a critical oversight. The GPUs are currently claimed by vfio-pci for the KVM VM passthrough. The NVIDIA driver and vfio-pci cannot simultaneously own the same device. The assistant later realizes (in [msg 433]) that the VM must be stopped and the vfio-pci bindings removed before the NVIDIA driver can claim the GPUs. This means LXC and KVM GPU access are mutually exclusive — a significant operational constraint that is not immediately obvious.

Assumption 3: The PVE kernel has the necessary firmware files for Blackwell GPUs. This assumption proves incorrect in the most painful way. Later in the segment, the team discovers that driver 590.48.01 lacks Blackwell GSP firmware files (only gsp_ga10x.bin and gsp_tu10x.bin exist), and the older PVE kernel may not support the Blackwell architecture's GSP requirements at all. This becomes the blocker that ultimately stalls the LXC approach.

Assumption 4: The container will see the same GPU topology as bare metal. This assumption is correct — and is later confirmed when nvidia-smi topo -m inside the container shows NODE and SYS topology instead of the PHB seen in the VM. But it turns out that seeing the right topology is useless if CUDA cannot initialize.

Input Knowledge Required to Understand This Message

To fully grasp what is happening in [msg 431], a reader needs:

  1. Understanding of the P2P bottleneck: The entire motivation hinges on the VFIO/IOMMU P2P problem. Without knowing that KVM passthrough prevents GPU-to-GPU direct communication, the urgency of finding an alternative makes no sense.
  2. Knowledge of LXC vs KVM architecture: The key insight — that LXC containers share the host kernel and access devices via bind mounts rather than VFIO — is what makes the LXC approach promising. A reader needs to understand that containers are not virtual machines; they share the host kernel's device drivers and memory management.
  3. Familiarity with Proxmox terminology: Terms like "PVE kernel," "pve-headers," "DKMS," and "bind mount" are specific to the Proxmox ecosystem. The assistant's search queries are crafted with this vocabulary.
  4. Awareness of the Blackwell GPU context: The RTX PRO 6000 is a recent (2025) Blackwell-architecture GPU. Its driver requirements, GSP firmware dependencies, and known P2P bugs are all relevant to understanding why the specific driver version and kernel compatibility matter.

Output Knowledge Created by This Message

This message does not produce a guide or commands — it produces raw research data. The output is:

  1. Forum thread content: The webfetch retrieves the Proxmox forum discussion, which likely contains step-by-step instructions, error messages encountered by other users, and configuration examples. This becomes the raw material for the comprehensive guide in [msg 433].
  2. Search result snippets: The exa search returns summaries of pages discussing DKMS installation on PVE kernels with NVIDIA drivers. These snippets help the assistant gauge whether the installation path is well-documented and whether any Blackwell-specific issues have been reported.
  3. Confidence assessment: By seeing that multiple forum threads and tutorials exist for this exact scenario, the assistant gains confidence that the LXC approach is viable and well-documented, justifying the investment of time in writing a detailed guide.

The Thinking Process Visible in the Message

While the message itself contains only tool calls, the thinking is visible in the choice of what to fetch. The assistant is not randomly browsing — it is systematically covering two dimensions:

The Broader Significance

Message [msg 431] is a reminder that in complex technical debugging, the most important step is often not the fix itself but the research that identifies the fix. The assistant could have guessed at the installation steps based on general Linux knowledge, but instead it sought out community-validated procedures specific to the Proxmox + NVIDIA + Blackwell combination. This discipline — seeking out targeted, hardware-specific information rather than relying on generic knowledge — is what separates effective troubleshooting from guesswork.

The message also illustrates the iterative nature of the assistant's reasoning. The LXC idea did not come fully formed; it emerged from the user's question, was initially dismissed, then reconsidered, researched, and eventually implemented. Message [msg 431] captures the moment when that reconsideration solidifies into action. It is the pivot point — and pivot points, in any narrative, are where the most interesting dynamics occur.