The Prerequisite Check: A Methodical Approach to GPU Driver Installation on a Migrated Proxmox Host

Introduction

In the sprawling infrastructure of a multi-host GPU cluster, the decommissioning of a primary node and the migration of its workloads to a secondary host is a delicate operation. This article examines a single message from an opencode coding session—message index 6751—where an AI assistant, tasked with deploying the Qwen3.6-27B language model on a newly migrated Proxmox LXC container, pauses before taking action to conduct a systematic reconnaissance of the target host's readiness for NVIDIA driver installation. The message is deceptively simple: a brief observation followed by a single bash command and its output. Yet within this exchange lies a rich demonstration of methodical troubleshooting, prerequisite awareness, and the kind of systems-thinking that separates a hasty installation from a robust deployment.

Context: The Migration from kpro6 to kpro5

To understand why this message exists, we must first understand the broader context. The user had just informed the assistant that kpro6 (IP 10.1.2.6)—a machine with 8× RTX PRO 6000 Blackwell GPUs that had been the primary deployment target for much of the session—was being decommissioned from the cluster. Its LXC container, CT129, had been moved to kpro5 (IP 10.1.2.5), a host with a heterogeneous GPU mix: two RTX A6000s (48 GB each, Ampere architecture) available for the container, alongside several RTX 3090s already bound to other virtual machines. The user's request was straightforward: set up the NVIDIA driver on the host, configure CT129 with the two A6000s, and deploy Qwen3.6-27B—a new dense 27B-parameter model—with tool calling, thinking parsers, and MTP (Multi-Token Prediction) speculation enabled.

The assistant had already taken initial reconnaissance steps in the preceding messages ([msg 6749], [msg 6750]), discovering that kpro5 had no NVIDIA driver installed whatsoever—no packages, no kernel modules, no nvidia-smi binary. The modprobe nvidia command had failed silently. This set the stage for the message under analysis.

The Subject Message: A Pause for Prerequisite Checking

The message reads:

No NVIDIA packages at all. Need to install from scratch. Let me check the Proxmox repositories and kernel headers situation.

This is followed by a bash command that executes remotely on kpro5, probing four specific aspects of the system:

  1. Kernel headers — whether pve-headers or linux-headers packages are installed, which are required for DKMS (Dynamic Kernel Module Support) to build the NVIDIA kernel module against the running kernel.
  2. APT sources — what package repositories are configured, to determine whether Proxmox-specific or standard Debian repositories are available for driver installation.
  3. IOMMU status — whether Input-Output Memory Management Unit is enabled, which is relevant for GPU passthrough to LXC containers and had been a source of significant complications on the previous host (kpro6).
  4. Nouveau driver status — whether the open-source Nouveau driver is loaded, which would conflict with the proprietary NVIDIA driver and must be blacklisted before installation.

Why These Four Checks?

Each of these checks reflects a specific concern derived from the assistant's accumulated experience in this session. The kernel headers check is standard practice: NVIDIA's driver installer uses DKMS to compile a kernel module tailored to the exact running kernel version. Without the appropriate headers, the installation will fail or produce a non-functional module. The assistant had already confirmed that no NVIDIA packages were installed; the next question was whether the infrastructure for installing them was present.

The APT sources check serves a dual purpose. First, it reveals whether the host is configured with Proxmox VE repositories (which would provide pve-headers packages matching the custom Proxmox kernel) or standard Debian repositories. Second, it informs the assistant about which package sources are available for installing build dependencies, the driver itself (if using the NVIDIA CUDA repository), or alternative installation methods.

The IOMMU check carries particular weight given the session's history. On kpro6, the assistant had invested enormous effort diagnosing P2P DMA corruption issues under SEV-SNP IOMMU, ultimately discovering that Blackwell GPUs require IOMMU DMA translation to function (the FSP boot fails with error 0x177 under identity mode). While kpro5 uses different GPUs (RTX A6000, Ampere architecture, not Blackwell), the assistant is checking whether IOMMU is configured at all—a relevant factor for GPU passthrough to LXC containers.

The Nouveau check is perhaps the most critical. The open-source Nouveau driver, if loaded, will hold a lock on the NVIDIA GPU devices, preventing the proprietary driver from claiming them. The nvidia-installer and dkms processes will fail if Nouveau is active. Blacklisting Nouveau is a prerequisite step that must happen before the proprietary driver installation can begin.

What the Output Reveals

The command output is revealing in its silences. The "=== Headers ===" section produces no matching packages—no pve-headers, no linux-headers. This is a significant finding: the host lacks the kernel headers needed for DKMS-based driver installation. The assistant will need to install them before proceeding.

The APT sources output shows standard Debian Bookworm repositories (ftp.pl.debian.org for main and updates, security.debian.org for security updates). Notably absent are any Proxmox VE repositories (deb https://enterprise.proxmox.com/debian/enterprise bookworm enterprise or deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription). This absence is meaningful: without Proxmox repositories, pve-headers packages cannot be installed from standard apt sources, meaning the assistant will need to either add the Proxmox repository or find an alternative approach to obtaining matching kernel headers.

The "=== IOMMU ===" section is empty—no IOMMU-related messages appear in the kernel ring buffer (dmesg). This could mean IOMMU is not enabled in the BIOS, or that it is enabled but has generated no log messages. Either way, it suggests a simpler GPU passthrough situation than the fraught IOMMU configuration on kpro6.

The "=== Nouveau ===" section confirms that Nouveau is loaded and active, occupying approximately 2.9 MB of kernel memory, along with several dependent modules (mxm_wmi, drm_gpuvm, drm_exec, gpu_sched, drm_ttm_helper, ttm). This is a clear signal that Nouveau must be blacklisted before the NVIDIA driver installation can proceed.

Assumptions Embedded in the Command

The assistant makes several assumptions in crafting this command. First, it assumes a Proxmox VE environment (checking for pve-headers specifically), which is reasonable given that kpro5 runs the 6.8.12-9-pve kernel. Second, it assumes that the standard approach to NVIDIA driver installation on Proxmox involves DKMS and kernel headers—an assumption validated by common practice but worth verifying. Third, it assumes that IOMMU configuration is relevant to the GPU passthrough setup, drawing on the painful lessons learned from kpro6. Fourth, it assumes that Nouveau is the primary obstacle to driver installation, which is almost always true on systems that have never had the proprietary driver installed.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure of the command itself. The four checks are ordered by logical dependency: first, verify the build infrastructure (headers); second, verify the package sources; third, check for potential complications from virtualization features (IOMMU); fourth, check for the most common driver conflict (Nouveau). This ordering reflects a mental model of the installation process as a pipeline with prerequisites that must be satisfied in sequence.

The brief prefatory statement—"No NVIDIA packages at all. Need to install from scratch. Let me check the Proxmox repositories and kernel headers situation."—reveals the assistant's assessment of the situation. The word "scratch" is significant: this is not a repair or upgrade of an existing driver installation, but a greenfield installation on a host that has never had NVIDIA drivers. The assistant correctly recognizes that a greenfield installation requires more thorough prerequisite checking than an upgrade would.

What This Message Creates: Output Knowledge

This message produces several pieces of actionable knowledge:

  1. Kernel headers are absent — The host needs linux-headers-6.8.12-9-pve or equivalent installed before driver installation can proceed.
  2. No Proxmox repositories are configured — The assistant must either add the Proxmox VE repository or find kernel headers through other means.
  3. IOMMU is not producing visible messages — Either disabled or silently functioning; either way, less likely to be an obstacle than on kpro6.
  4. Nouveau is loaded and must be blacklisted — A blacklist nouveau entry in /etc/modprobe.d/ will be required, followed by a reboot or module removal.
  5. The host runs Debian Bookworm — Package management commands should target Bookworm-compatible packages.

Conclusion

Message 6751 is a textbook example of the "measure twice, cut once" philosophy in systems administration. Rather than blindly running the NVIDIA installer and debugging failures as they arise, the assistant pauses to gather critical information about the host's readiness. The four checks—kernel headers, APT sources, IOMMU, Nouveau—each address a distinct failure mode that could derail the installation. By performing these checks upfront, the assistant transforms the installation task from a leap of faith into a well-understood sequence of steps with known prerequisites. This methodical approach, born from experience with the complexities of GPU passthrough and driver management on the previous host, exemplifies the kind of systematic thinking that distinguishes robust infrastructure work from trial-and-error debugging.