The Download That Changed Everything: A Pivot Point in GPU Passthrough Strategy

Introduction

In the middle of an intense debugging session spanning multiple days, a seemingly mundane command was issued: wget downloading a 397MB NVIDIA driver file onto a Proxmox host. Message <msg id=448> appears, at first glance, to be one of the most trivial moments in the conversation — a simple file download with progress bars scrolling by. But this message represents a critical inflection point: the moment when an entire strategy for GPU-accelerated machine learning deployment was abandoned and a new one was embraced. The download of NVIDIA-Linux-x86_64-590.48.01.run onto the Proxmox host kpro6 was the first concrete step in a high-stakes pivot from KVM-based virtual machine passthrough to LXC container-based GPU access, driven by the discovery of a fundamental hardware limitation that no amount of software configuration could overcome.

The Message

The subject message is a single bash command executed via SSH on the Proxmox host:

[assistant] [bash] ssh root@10.1.2.6 "cd /root && wget -q --show-progress https://us.download.nvidia.com/XFree86/Linux-x86_64/590.48.01/NVIDIA-Linux-x86_64-590.48.01.run 2>&1"

     0K .......... .......... .......... .......... ..........  0% 4.74M 84s
    50K .......... .......... .......... .......... ..........  0% 4.97M 82s
   100K .......... .......... .......... .......... ..........  0% 6.87M 74s
   150K .......... .......... .......... .......... ..........  0% 8.74M 67s
   200K .......... .......... .......... .......... ..........  0% 9.55M 62s
   250K .......... .......... .......... .......... ..........  0% 7.50M 60s
   300K .......... .......... ..........

The output shows the familiar wget progress display — dots representing downloaded chunks, percentages, download speeds, and estimated time remaining. It is the kind of output that any system administrator has seen thousands of times. Yet in this context, those progress bars tell a story of commitment. Each dot represents data flowing from NVIDIA's servers to a machine that is about to undergo a fundamental change in how it relates to its eight RTX PRO 6000 Blackwell GPUs.

The Strategic Context: Why This Download Was Necessary

To understand why this simple download is significant, one must understand the journey that led to it. The session had been attempting to deploy the GLM-5-NVFP4 large language model across eight RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism. Initially, this was done through a KVM virtual machine on Proxmox, with GPUs passed through via VFIO (Virtual Function I/O). This approach worked — the model could be loaded and inference performed — but a critical performance bottleneck was discovered.

The bottleneck was PCIe Peer-to-Peer (P2P) DMA. In the KVM VM, nvidia-smi topo -m showed PHB (PCIe Host Bridge) topology between GPUs, meaning that GPU-to-GPU communication had to traverse the host bridge rather than using direct P2P DMA. This added significant latency and reduced throughput for tensor-parallel inference, where GPUs must constantly exchange intermediate results. The root cause was that each GPU was on its own PCIe root complex, and the VFIO/IOMMU layer in KVM could not bridge them for P2P.

The assistant and user had explored numerous workarounds: switching to Q35 chipset, fixing BAR allocation with pci=realloc, disabling ACS (Access Control Services) to merge IOMMU groups, and even modifying the Proxmox host kernel parameters. All failed. The hardware topology was fundamentally incompatible with P2P DMA in a virtualized environment.

The LXC container approach emerged as the alternative. In an LXC container, the GPUs would be visible with their bare-metal topology because the container shares the host's kernel — there is no VFIO layer, no IOMMU translation, and no hypervisor between the GPU driver and the hardware. The nvidia-smi topo -m output inside an LXC container would show NODE (same NUMA socket) and SYS (cross-socket) connections, which are the true hardware topology and should support P2P DMA natively.

But LXC access requires the NVIDIA driver to be installed on the host, not just in the guest. In the KVM approach, the host only needed VFIO to pass the GPU through; the guest had its own NVIDIA driver stack. For LXC, the host must own the GPUs directly, load the nvidia kernel module, and expose the device nodes (/dev/nvidia0 through /dev/nvidia7, /dev/nvidiactl, /dev/nvidia-uvm, etc.) to the container via bind-mounts. This is a fundamentally different relationship between the host and its GPUs.

Decisions Made in This Message

Several decisions crystallized in this single wget command:

The decision to commit to the LXC path. Until this moment, the LXC approach was theoretical — a plan documented in the comprehensive guide the assistant had written in <msg id=433>. Downloading the driver onto the host made it real. Once the NVIDIA driver was installed on the Proxmox host, the GPUs would be claimed by the nvidia kernel module, making them unavailable for VFIO passthrough to the KVM VM. This was a one-way door: the host could no longer simultaneously run the GPU-intensive KVM VM.

The decision to use driver version 590.48.01. This was the same version that had been working inside the KVM VM. The assistant verified the URL existed in <msg id=447> before initiating the download. The choice to match versions was deliberate — it minimized the variables in the experiment. If something broke, it wouldn't be because of a driver mismatch between the host and what the container would eventually use.

The decision to download directly to /root on the host. This was a practical choice for a headless server. The file would be executed with --dkms to register it with Dynamic Kernel Module Support, ensuring it would survive kernel updates.

The decision to proceed despite unresolved risks. The assistant knew that installing the NVIDIA driver on a Proxmox host was non-trivial. The PVE kernel (6.8.12-9-pve) is not a standard upstream kernel; it has Proxmox-specific patches. DKMS builds against the kernel headers, and there was a risk that the NVIDIA kernel module would fail to compile or would behave unexpectedly. The assistant had already encountered and resolved an apt repository issue (the enterprise repo was returning 401 Unauthorized) and had to add the no-subscription repository to install pve-headers. These were warning signs that the environment was not a standard target for NVIDIA's installer.

Assumptions Embedded in the Download

The message carries several assumptions, some of which would prove incorrect:

That the NVIDIA driver 590.48.01 supports Blackwell GPUs on the PVE kernel. The driver had been working inside the KVM VM, but that was a different kernel (the VM's guest kernel) with a different driver stack. The host's PVE kernel might have different compatibility characteristics. The RTX PRO 6000 Blackwell (device ID 10de:2bb5) is a very new architecture, and driver support can be kernel-version dependent.

That DKMS would successfully build the kernel module. The pve-headers package had been installed, but the build process for the NVIDIA kernel module involves compiling against the kernel's internal API, which can change between versions. A mismatch could cause the build to fail silently or produce a malfunctioning module.

That the GSP (GPU System Processor) firmware bundled with the driver would work on the host. The Blackwell architecture uses a GSP coprocessor for certain management functions. The driver package includes firmware files like gsp_ga10x.bin and gsp_tu10x.bin, but as would later be discovered, the Blackwell-specific GSP firmware files were absent from driver 590.48.01. This would become a critical blocker.

That nvidia-smi working would imply CUDA working. This is a common assumption that often holds true, but in this case it would not. The driver's kernel module can enumerate GPUs and report their status (which nvidia-smi does) while the CUDA runtime library (libcuda.so) fails to initialize because of deeper firmware or driver interface issues.

That the LXC approach would solve the P2P problem. The assistant assumed that if the topology showed NODE/SYS instead of PHB, P2P DMA would work. While the topology is a necessary condition, it is not sufficient — there could be other barriers at the driver, firmware, or hardware level.

Input Knowledge Required

To understand this message, one needs knowledge of:

  1. The Proxmox virtualization stack: VFIO passthrough, IOMMU groups, PCIe topology, and the difference between KVM and LXC GPU access models.
  2. NVIDIA driver architecture: The distinction between the kernel-mode driver (nvidia.ko), the userspace CUDA driver (libcuda.so), and tools like nvidia-smi. The role of DKMS for kernel module compilation.
  3. PCIe P2P DMA: How GPUs communicate directly over the PCIe fabric without host memory involvement, and why this is critical for multi-GPU ML workloads using NCCL (NVIDIA Collective Communications Library).
  4. The Blackwell GPU architecture: The RTX PRO 6000's device ID (2bb5), its GSP requirements, and the driver version compatibility landscape.
  5. The GLM-5-NVFP4 deployment context: Why tensor parallelism across 8 GPUs requires low-latency GPU-to-GPU communication, and why P2P is essential for acceptable performance.
  6. Proxmox package management: The enterprise vs. no-subscription repository distinction, and how to install pve-headers for DKMS builds.

Output Knowledge Created

This message, combined with its surrounding context, creates several pieces of knowledge:

  1. A documented procedure for installing NVIDIA drivers on Proxmox hosts for LXC GPU access, including the prerequisite steps of blacklisting nouveau, installing build dependencies, and handling repository configuration.
  2. Evidence that driver 590.48.01 is downloadable and accessible from NVIDIA's servers, confirming its availability for the RTX PRO 6000 Blackwell on Linux x86_64.
  3. A record of the strategic pivot point — the exact moment when the session committed to the LXC approach over the KVM approach, which would later prove to have its own insurmountable challenges.
  4. The foundation for a negative result: The download and subsequent installation would lead to a situation where nvidia-smi works but cuInit fails, demonstrating that driver visibility and CUDA initialization are separate concerns — a finding that would be valuable for anyone attempting a similar setup.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in the messages leading up to this download reveals a methodical, research-driven approach. In <msg id=430>, the assistant began by researching Proxmox LXC GPU passthrough techniques, fetching multiple forum threads and tutorials. In <msg id=433>, it synthesized this research into a comprehensive step-by-step guide. When the user said "Implement this, you have access" in <msg id=434>, the assistant pivoted from planning to execution.

The assistant's exploration in <msg id=435-447> shows careful state assessment: checking the kernel version, the existing LXC configuration, whether any NVIDIA driver is present, whether VFIO is binding the GPUs, and whether build dependencies are installed. Each step builds on the previous one. The discovery that the enterprise Proxmox repo requires authentication (returning 401) is handled pragmatically by switching to the no-subscription repo. The discovery that nouveau is loaded is handled by blacklisting and unloading it.

The verification step in <msg id=447> — using wget -q --spider to check if the URL exists before downloading — shows a cautious approach. The assistant is not blindly executing commands; it's verifying prerequisites. This is the thinking of an experienced systems engineer who knows that a failed download mid-operation can leave the system in an inconsistent state.

What Follows: The Unforeseen Blocker

The download in <msg id=448> succeeds, and the driver installation in subsequent messages appears to work perfectly. nvidia-smi shows all 8 GPUs with the correct driver version and CUDA 13.1 support. The device nodes are created with the expected major numbers (195 for NVIDIA, 234 for nvidia-uvm). The topology inside the LXC container shows NODE and SYS connections — exactly what was hoped for.

But then comes the blocker: cuInit fails with error code 3 (CUDA_ERROR_NOT_INITIALIZED). The open-source kernel module sees the GPUs but CUDA cannot initialize. The proprietary kernel module makes the GPUs invisible entirely. The root cause traces to missing Blackwell GSP firmware files in the driver package and possible kernel version incompatibilities.

This negative result does not diminish the significance of message <msg id=448>. On the contrary, it elevates it. The download was the first step on a path that needed to be explored to discover that the LXC approach, while promising in theory, was blocked by a different class of problem — one at the driver/firmware level rather than the virtualization layer. The session would need to pivot yet again, but that pivot was only possible because this download was executed.

Conclusion

Message <msg id=448> is a masterclass in how the most operationally simple actions can carry the heaviest strategic weight. A wget command — one of the most routine operations in system administration — becomes the fulcrum on which an entire deployment strategy turns. The download of the NVIDIA driver onto the Proxmox host was the point of no return for the LXC experiment, the moment when theory became practice, and the first step on a path that would reveal new and unexpected challenges. In the narrative of this coding session, it is the quiet before the storm — a few seconds of progress bars that masked the complexity of what was to come.