The Clean Slate: Installing NVIDIA Userspace After a Source-Based Kernel and Driver Build

Introduction

In the sprawling narrative of provisioning a high-performance machine-learning host—kpro6, a Proxmox VE server equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs and a 14 TB NVMe scratch pool—message [msg 8493] represents a quiet but decisive turning point. It is the moment when the assistant, having just recovered from a catastrophic system failure, executes the final installation step of a carefully rebuilt software stack. The message itself is brief: a single bash command that installs the NVIDIA userspace driver components using the proprietary .run installer, deliberately skipping kernel module installation because those modules have already been built from source and installed by hand. But this brevity belies the immense weight of context it carries. To understand why this message matters, one must understand the disaster that preceded it, the engineering philosophy shift that followed, and the precise sequence of decisions that made this simple command possible.

The Context: From Bricked System to Clean Rebuild

The story of message [msg 8493] begins not with success but with failure. Earlier in the session, the assistant had attempted to install a community-built 6.19 kernel on the Proxmox host—a kernel compiled on Debian Trixie with GCC 14, while the host ran Debian Bookworm with GCC 12. This toolchain mismatch triggered a cascade of increasingly desperate workarounds: patched kernel headers, rebuilt gendwarfksyms and objtool binaries, and finally a GLIBC_2.38 shim library injected into /usr/local/lib. The shim poisoned the system's dynamic linker, bricking SSH access entirely. The machine had to be physically rescued from a live ISO.

After recovery, the user issued a clear directive: no more hacks. Build everything from source with the native toolchain. The assistant pivoted completely. It removed the community kernel and all NVIDIA artifacts. It cloned the official Proxmox VE kernel repository (branch bookworm-6.14) and built the kernel from source using the system's native GCC 12.2.0. It then cloned NVIDIA's open-gpu-kernel-modules repository and compiled the 595.71.05 kernel modules against the freshly built kernel headers. Both builds completed with zero errors and zero patches—a stark validation of the source-based approach.

By message [msg 8492], the kernel modules were installed and registered with depmod. The system had a clean, self-built 6.14.11-9-bpo12-pve kernel with matching NVIDIA kernel modules, all compiled with a single, consistent GCC 12.2.0 toolchain. What remained was the userspace layer: nvidia-smi, the CUDA libraries, the OpenGL and Vulkan drivers, and the X server integration. These components cannot be built from the open-gpu-kernel-modules repository; they ship as proprietary binaries in NVIDIA's .run installer package.

The Message Itself: A Deliberate, Minimal Command

The subject message executes a single SSH command on the remote host:

ssh -o ConnectTimeout=10 root@10.1.2.6 '
# Blacklist nouveau before reboot
cat > /etc/modprobe.d/blacklist-nouveau.conf << "EOF"
blacklist nouveau
options nouveau modeset=0
EOF

# Install userspace only
/root/NVIDIA-Linux-x86_64-595.71.05-no-compat32.run \
    --silent \
    --no-kernel-modules \
    --no-questions \
    --ui=none 2>&1 | grep -E "ERROR|WARNING|Install" | head -10
echo "---EXIT: $?---"
echo "---NVIDIA-SMI---"
ls -la /usr/bin/nvidia-smi 2>/dev/null
'

The command does two things. First, it creates a modprobe configuration file to blacklist the open-source nouveau driver, which would otherwise conflict with the proprietary NVIDIA driver at boot. This is a standard precaution when installing NVIDIA drivers on Linux, but here it carries extra significance: after the earlier bricked-system incident, the assistant is being especially careful to ensure a clean boot.

Second, it executes the NVIDIA .run installer with the --no-kernel-modules flag. This flag is the key architectural decision embedded in this message. Normally, the NVIDIA .run installer would detect the running kernel, build matching kernel modules via DKMS, and install them. But in this case, the kernel modules have already been built from source and installed manually. Using --no-kernel-modules tells the installer to skip that step entirely and install only the userspace components: libraries, binaries, configuration files, and X driver support.

The output confirms a clean installation. The only warnings are cosmetic: the installer notes that it was "forced to guess the X library path" and X module path, which is harmless on a headless server that doesn't run a graphical desktop. No errors appear. The exit code is zero. The assistant then verifies that /usr/bin/nvidia-smi exists, confirming the userspace tool is in place.

The Reasoning Behind the Decisions

Every choice in this message reflects lessons learned from the earlier failure. The decision to build kernel modules from source rather than rely on the .run installer's DKMS mechanism was deliberate. DKMS would have compiled the modules against the running kernel's headers—but those headers were themselves built with GCC 12.2.0, so DKMS would have worked. However, the source-based approach gave the assistant complete control over the build process, including the ability to verify that the compiler versions matched exactly. As the assistant noted in [msg 8491]: "GCC versions match perfectly — both the kernel and host are gcc 12.2.0-14+deb12u1. No hacks needed."

The --no-kernel-modules flag also avoids a subtle risk. If the .run installer had attempted to build kernel modules via DKMS, it would have needed to detect the kernel source tree and configure the build. Any mismatch between the installer's expectations and the custom-built kernel could have introduced errors. By separating the kernel module build (done manually with explicit KERNEL_UNAME and SYSSRC parameters) from the userspace install (done with the .run file), the assistant created two independent, verifiable steps.

The proactive blacklisting of nouveau is another sign of careful system administration. On a fresh boot, if the kernel detects both the nouveau driver (built into the kernel or loaded as a module) and the NVIDIA proprietary driver, they can conflict, causing the NVIDIA driver to fail or the system to hang. By blacklisting nouveau before the first reboot into the new kernel, the assistant ensures that only the NVIDIA driver will attempt to control the GPUs.

Assumptions Embedded in This Message

The message makes several assumptions, most of them well-founded. It assumes that the .run installer file is still present at /root/NVIDIA-Linux-x86_64-595.71.05-no-compat32.run. This file was downloaded earlier in the session and survived the system rescue because it was stored on the root filesystem, which was preserved. It assumes that the userspace components from the .run installer are compatible with the kernel modules built from the open-gpu-kernel-modules source. This is a safe assumption because both come from the same NVIDIA driver release (595.71.05) and the open-gpu-kernel-modules repository is NVIDIA's own source for the kernel modules that ship in the .run package.

The assistant also assumes that the system will boot cleanly after reboot. This is not guaranteed—the custom-built kernel could have configuration issues, the NVIDIA kernel modules could fail to load, or the firmware mismatch that caused a boot panic earlier (mentioned in the chunk summary) could recur. The assistant addresses this by pinning the kernel for the next boot using proxmox-boot-tool in the subsequent message ([msg 8494]) and scheduling a reboot. The firmware issue is resolved separately by updating the initramfs.

What Input Knowledge This Message Requires

To understand this message, a reader needs to know several things. First, the distinction between kernel modules and userspace drivers in the NVIDIA stack: the kernel modules (nvidia.ko, nvidia-uvm.ko, etc.) handle low-level GPU control and memory management, while the userspace components (nvidia-smi, libcuda.so, etc.) provide the API that applications and frameworks like CUDA and PyTorch use. Second, the role of DKMS (Dynamic Kernel Module Support) in automatically rebuilding kernel modules when the kernel is updated. Third, the nouveau open-source driver and why it must be blacklisted when using the proprietary NVIDIA driver. Fourth, the Proxmox VE boot management system (proxmox-boot-tool) and how kernel pinning works on UEFI systems.

The reader also needs the broader context of the earlier failure: that a toolchain mismatch between a community kernel and the host system led to a bricked machine, and that the user explicitly directed the assistant to build everything from source with the native compiler. Without this context, the --no-kernel-modules flag might seem like an arbitrary optimization rather than a deliberate architectural choice born from painful experience.

What Output Knowledge This Message Creates

After this message executes, the system has a complete NVIDIA driver installation: kernel modules built from source and installed in /lib/modules/6.14.11-9-bpo12-pve/kernel/drivers/video/, userspace binaries and libraries installed by the .run installer, and nouveau blacklisted to prevent conflicts at boot. The system is ready for reboot. After rebooting (which occurs in [msg 8494]), the assistant verifies that all 8 GPUs are recognized and the system is stable.

This output knowledge is significant because it represents the successful completion of a complex, multi-step provisioning process. The system now has a pristine, high-performance training environment: Proxmox VE on a self-built 6.14 kernel, 8× NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each, 783 GB total) driven by the open-source 595.71.05 driver and CUDA 13.2, all compiled with a single, matching GCC 12.2.0 toolchain. The node is now prepared for creating an LXC container and resuming the DFlash drafter training workflow that was the original goal of the session.

The Thinking Process Visible in the Assistant's Reasoning

The assistant's thinking process in this message is methodical and checklist-driven. It follows a clear sequence: blacklist nouveau first (so the boot is clean), then install userspace (the last remaining component), then verify the installation. Each step is separated by echo markers for readability, and the exit code is captured. The assistant does not assume success—it checks for errors explicitly with grep -E "ERROR|WARNING" and verifies the output with ls.

This careful approach contrasts sharply with the earlier, more reckless attempt to patch a community kernel into submission. The assistant has learned from its mistake. It now treats the toolchain as an inviolable constraint: everything must be built with the same compiler, against the same headers, with no binary compatibility shims. The message is not just an installation command—it is a statement of engineering philosophy. Clean builds, matching toolchains, verifiable steps. No hacks.

Conclusion

Message [msg 8493] is the quiet capstone of a dramatic recovery. After a system was bricked by incompatible binaries and rescued from a live ISO, after a complete pivot to source-based builds, after hours of kernel compilation and module installation, this simple command installs the final piece of the puzzle. The NVIDIA userspace drivers drop into place, the nouveau driver is blacklisted, and the system stands ready for its first boot into a fully self-built software stack. The message is brief, but it represents the triumph of a disciplined engineering approach over the temptation of quick patches. It is a reminder that sometimes the fastest path to a working system is to throw away the shortcuts and build everything from scratch.