The First Clean Step: Building a Proxmox Kernel from Source After a Bricked System

Introduction

In the aftermath of a catastrophic system failure that rendered a Proxmox host with 8× Blackwell RTX PRO 6000 GPUs completely unreachable, message [msg 8477] represents a quiet but decisive turning point. After the user physically rescued the machine from a live ISO and issued a clear directive—"try not to do hacks; open nvidia can probably just be built locally without hacky repos, same with the kernel, all using correct gcc?"—the assistant pivoted from a series of increasingly desperate workarounds to a principled, source-based build strategy. This message, which simply runs make build-dir-fresh inside the official Proxmox VE kernel repository, is the first concrete step in that new direction. It is unremarkable in isolation but profoundly meaningful in context: it marks the moment when engineering discipline reasserted itself over patch-and-pray debugging.

The Context: A System Bricked by Toolchain Incompatibility

To understand why this message was written, one must first understand the disaster that preceded it. The assistant had been tasked with provisioning kpro6, a new Proxmox host equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs and a 14 TB NVMe scratch drive. The goal was to create a high-performance training environment for DFlash, a speculative decoding drafter model.

The initial approach was pragmatic but reckless. Rather than building a kernel from source, the assistant installed a community-built 6.19 kernel from an unofficial repository (jaminmc). This kernel had been compiled on Debian Trixie with GCC 14, while the host ran Debian Bookworm with GCC 12.2.0. The resulting binary incompatibility manifested in subtle ways: gendwarfksyms crashed, objtool failed, and kernel module builds produced corrupted .ko files.

The assistant's response was to patch around each symptom rather than address the root cause. It rebuilt gendwarfksyms and objtool from the kernel source tree, patched kernel headers to work around missing symbols, and ultimately created a GLIBC_2.38 compatibility shim—a shared library named libc_238_compat.so with soname=libc.so.6 placed in /usr/local/lib. This shim poisoned the system's dynamic linker, causing every binary (including bash and ssh) to fail with "version GLIBC_2.25 not found" errors. The machine became completely unreachable over SSH, requiring physical console access to rescue.

The User's Directive: A Clean Slate

After the user rescued the system using a Proxmox VE installation ISO and confirmed it was back online, they gave a remarkably clear and wise instruction (see [msg 8469]):

"proxmox back up; try not to do hacks; open nvidia can probably just be built locally without hacky repos, same with the kernel, all using correct gcc?"

This directive reframed the entire approach. The assistant acknowledged the error immediately in [msg 8470]:

"You're right. The whole mess came from trying to use a community kernel built on trixie and then hacking around glibc/gcc mismatches. The clean approach: 1. Build the 6.19 kernel ourselves on this bookworm system with its native GCC 12 — no glibc mismatch, no binary patching. 2. Build nvidia-open kernel modules from source against that kernel — same compiler, clean build."

The Preparation: Cleaning Up and Researching

Messages [msg 8471] through [msg 8476] document the cleanup and research phase. The assistant:

  1. Removed all community kernel artifacts: The jaminmc 6.19 kernel, the NVIDIA .run installer leftovers, the trixie APT repository, and the glibc shim files were all purged.
  2. Unpinned the boot kernel and refreshed the bootloader: The system was reverted to the stock Proxmox 6.8 kernel as a fallback.
  3. Researched the official PVE kernel build process: A task subagent ([msg 8475]) was dispatched to investigate the proper way to build the Proxmox VE kernel from source. The subagent discovered that the official repository is at git://git.proxmox.com/git/pve-kernel.git and that the bookworm-6.14 branch is the newest official PVE kernel targeting Debian Bookworm.
  4. Made a critical architectural decision: The assistant originally planned to build the 6.19 kernel from source, but the research revealed that Proxmox's official branches go 6.14 → 6.17 → 7.0—there is no official 6.19 branch. The bookworm-6.14 branch is the correct choice for a Bookworm-based system. The assistant reasoned that kernel 6.14 (from April 2025) has adequate Blackwell GPU support and that the NVIDIA 595 driver had already been confirmed to build against 6.14 headers without issues.
  5. Installed build prerequisites and cloned the repository: The devscripts, debhelper, equivs, and git packages were installed, and the PVE kernel repo was cloned into /scratch (the 14 TB NVMe pool) for fast compilation.

Message 8477: The First Build Step

With the repository cloned and the build environment prepared, message [msg 8477] executes the first actual build command:

ssh -o ConnectTimeout=10 root@10.1.2.6 '
cd /scratch/pve-kernel
echo "=== Preparing build directory ==="
make build-dir-fresh 2>&1 | tail -20
echo "---EXIT: $?---"
echo "---BUILD-DIR---"
ls -d proxmox-kernel-* 2>/dev/null
' 2>&1

The make build-dir-fresh target is a Proxmox-specific build system command. It performs the following steps:

  1. Creates a fresh build directory from the upstream kernel source tarball.
  2. Applies all Proxmox patches to the kernel source. The output shows patches being applied, including ZFS-related patches (fixing nfs_truncate_shares, zpool-status, arc-stat-summary, etc.) and systemd integration patches. These patches are what differentiate the Proxmox VE kernel from the upstream Linux kernel—they add ZFS compatibility, Proxmox-specific tooling, and virtualization optimizations.
  3. Prepares the debian packaging structure so that the subsequent make commands can build a proper .deb package. The output visible in the message shows the patch application process in action, with patches numbered 0006 through 0011 being applied via dpkg-source. This is a standard Debian packaging workflow: the kernel source is treated as a Debian source package, and patches are applied in order before compilation begins.

Why This Message Matters

On the surface, this message is mundane—a single SSH command that prepares a build directory. But in the narrative arc of this session, it carries enormous weight for several reasons:

1. It Represents a Complete Philosophical Pivot

The previous approach was characterized by what one might call "binary bricolage": taking pre-built components from different distributions and trying to force them to work together through patches, shims, and workarounds. This approach failed catastrophically. The new approach is "source sovereignty": building every component from source using a single, consistent toolchain. This message is the first execution of that philosophy.

2. It Demonstrates the Value of Official Channels

The assistant initially used an unofficial community kernel repository (jaminmc) because it offered a pre-built 6.19 package. This shortcut led directly to the toolchain mismatch that bricked the system. By switching to the official Proxmox git repository and building from source, the assistant eliminated the risk of binary incompatibility at the cost of compilation time. This is a classic engineering tradeoff: convenience vs. reliability. The bricked system proved that reliability must take precedence.

3. It Validates the Research Phase

The task subagent in [msg 8475] was critical. Without it, the assistant might have attempted to build the wrong kernel version or used the wrong build procedure. The research revealed that bookworm-6.14 is the correct branch, that the build process uses make build-dir-fresh followed by make, and that the resulting .deb packages can be installed directly. This upfront investment in research prevented what could have been another round of trial-and-error debugging.

4. It Sets the Stage for a Clean Driver Build

The ultimate goal is to have a system where the NVIDIA open-source kernel driver (version 595.71.05) is compiled against the same kernel that is running. This requires kernel headers that match the running kernel exactly. By building the kernel from source, the assistant ensures that the headers in /usr/src/linux-headers-* are an exact match for the compiled kernel binary. This eliminates the "header mismatch" class of bugs that plagued the earlier DKMS-based approach.

Assumptions and Knowledge

The message makes several implicit assumptions:

Conclusion

Message [msg 8477] is the quiet hinge point of a much larger story. It is the moment when a debugging spiral that nearly destroyed a $100,000+ GPU server was replaced by disciplined, source-based engineering. The make build-dir-fresh command is not dramatic—it produces no flashing lights, no breakthrough performance numbers. But it represents something more important: the recognition that in systems engineering, the correct toolchain is not a negotiable detail but a foundational requirement. The patches scroll by in the output—ZFS fixes, systemd integration, arc-stat-summary guards—and with each one, the system moves closer to stability. The real work, as it turns out, begins not with a clever hack but with the humble act of building from source.