The Pivot to Source: Building a Custom Proxmox Kernel for Blackwell GPUs

In the aftermath of a catastrophic system failure that bricked SSH access to a brand-new Proxmox host, a single message marks the turning point where engineering discipline reasserts itself over expedient hacks. Message [msg 8476] is the moment the assistant, having just recovered from a self-inflicted toolchain disaster, makes a deliberate strategic decision that defines the entire provisioning approach for kpro6 — a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs and a 14TB NVMe drive, destined for large-scale DFlash drafter training.

The Context of Crisis

To understand the weight of this message, one must appreciate what preceded it. The session had been provisioning kpro6, a new Proxmox host, when the assistant attempted to install a community-built 6.19 kernel from an unofficial repository (the "jaminmc" kernel, built on Debian Trixie with GCC 14) alongside the NVIDIA 595.71.05 open driver via DKMS. This created a fundamental toolchain mismatch: the community kernel was compiled with GCC 14 from Debian Trixie, while the host ran Debian Bookworm's native GCC 12.2.0. The assistant attempted a series of increasingly desperate workarounds — patching kernel headers, rebuilding gendwarfksyms and objtool binaries, and ultimately creating a GLIBC_2.38 shim library that it placed in /usr/local/lib. That shim, designed to satisfy symbol version requirements, poisoned the system's dynamic linker by masquerading as libc.so.6, causing every dynamically linked binary (including bash and ssh) to fail with version mismatch errors. The machine was effectively bricked for remote access.

The user physically rescued the system using a Proxmox VE ISO, then delivered a pointed instruction in [msg 8469]: "try not to do hacks; open nvidia can probably just be built locally without hacky repos, same with the kernel, all using correct gcc?" This directive reframes the entire approach: instead of patching binary incompatibilities, build everything from source using the system's native toolchain.

The Decision Point

Message [msg 8476] opens with the assistant synthesizing research from a subagent task ([msg 8475]) that investigated the official Proxmox VE kernel build process:

Now I have the picture. The bookworm-6.14 branch is the newest official PVE kernel targeting bookworm. There's no 6.19 branch — upstream went 6.14 → 6.17 → 7.0. The trixie-6.17 is the next option but it targets trixie (might need trixie build deps).

This paragraph contains the critical insight that reshapes the entire provisioning strategy. The assistant had previously been targeting kernel 6.19, operating under the assumption that a newer kernel was necessary for Blackwell GPU support. The research reveals that the official Proxmox kernel repository does not have a 6.19 branch at all — the upstream Linux kernel versioning went from 6.14 directly to 6.17 and then to 7.0. The community "jaminmc" 6.19 kernel was an unofficial fork, built on a different Debian release with a different compiler, and that mismatch was the root cause of the entire bricking incident.

The assistant now considers three options:

  1. bookworm-6.14 — the newest official PVE branch targeting Debian Bookworm (the host's OS)
  2. trixie-6.17 — a newer kernel but targeting Debian Trixie, which would require Trixie build dependencies and potentially reintroduce the same toolchain mismatch
  3. No 6.19 option at all — that path is a dead end

Why 6.14 Wins

The assistant's reasoning for selecting 6.14 is methodical and well-supported:

For our use case (Blackwell GPUs, EPYC 9335), 6.14 built from source with GCC 12 is the cleanest path. It's officially supported on PVE 8/bookworm, has Blackwell hardware support (6.14 is from Apr 2025), and we already confirmed the NVIDIA 595 driver builds against 6.14 with no issues (the DKMS build succeeded on 6.14 earlier).

Each clause addresses a specific requirement:

The Execution: Cloning and Building

Having made the strategic decision, the assistant immediately begins execution. The message contains two parallel tool calls: a bash command that installs build prerequisites and clones the repository, and the reasoning that led to those commands.

The bash command is carefully structured:

ssh -o ConnectTimeout=10 root@10.1.2.6 '
# Install build prerequisites
DEBIAN_FRONTEND=noninteractive apt-get install -y devscripts debhelper equivs git 2>&1 | tail -5
echo "---"
# Clone the PVE kernel repo into scratch (14TB NVMe, fast)
cd /scratch
git clone --branch bookworm-6.14 --single-branch git://git.proxmox.com/git/pve-kernel.git 2>&1 | tail -5
echo "---"
cd pve-kernel
git log --oneline -3
'

Several details reveal the assistant's thinking. First, it clones into /scratch — the 14TB NVMe ZFS pool created earlier in the session — explicitly noting it is "fast." This is a deliberate performance optimization: kernel builds are I/O and CPU intensive, and placing the source on the fastest available storage reduces build time. Second, it uses --single-branch to clone only the bookworm-6.14 branch, avoiding the overhead of fetching the entire repository history. Third, the DEBIAN_FRONTEND=noninteractive flag prevents any interactive prompts from blocking the automated installation.

The build prerequisites — devscripts, debhelper, equivs, and git — are the minimum toolchain for building Debian source packages. Notably absent are kernel-specific build dependencies like libelf-dev, bc, flex, bison, and pahole. This is not an oversight; the assistant is aware that mk-build-deps (called in the subsequent message [msg 8478]) will automatically resolve and install all build dependencies from the kernel package's debian/control file. The initial apt-get installs only the meta-tools needed to invoke that dependency resolution.

The git log output confirms success: three commits on the bookworm-6.14 branch, the most recent being ad58d14 update ABI file for 6.14.11-9-bpo12-pve. This is the official Proxmox kernel, version 6.14.11-9, backported to Debian Bookworm (the -bpo12 suffix indicates a backport to Debian 12).

Assumptions Embedded in This Message

Every decision in this message rests on a set of assumptions, some explicit and some implicit:

Explicit assumptions:

What This Message Creates

This message produces several forms of output knowledge:

  1. A confirmed build strategy: The decision to use bookworm-6.14 with native GCC 12 is now documented and justified. Any future provisioning of similar Blackwell-equipped Proxmox hosts can reference this decision.
  2. A cloned repository: The PVE kernel source is now available on kpro6's fast NVMe storage, ready for the build process that follows in subsequent messages.
  3. A validated toolchain: The successful installation of build prerequisites and the git clone confirm that the system's network and package management are functioning correctly after the rescue.
  4. A precedent for source-based driver installation: By choosing to build the kernel from source rather than relying on pre-built packages, the assistant establishes a pattern that will be followed for the NVIDIA driver — also built from source against the custom kernel headers.

The Broader Significance

This message represents more than just a kernel version decision. It is the moment the assistant internalizes the user's directive to "not do hacks" and translates it into concrete engineering practice. The previous approach — downloading a pre-built community kernel and patching around incompatibilities — was a form of technical debt accumulation. Each workaround (patching objtool, rebuilding gendwarfksyms, creating a glibc shim) added complexity and fragility. The shim library that ultimately bricked the system was the logical endpoint of that approach: a hack that metastasized until it broke the fundamental assumption that libc.so.6 is the system's C library.

The new approach is the opposite: start from source, use the native toolchain, accept no binary patches. It is slower (building a kernel takes hours even on 64 cores) but infinitely more reliable. The message's closing line — the git log showing three clean commits on the official Proxmox branch — is a quiet vindication of this philosophy. No hacks, no patches, no workarounds. Just source code and a compiler.

What Follows

The subsequent messages in this segment show the build proceeding: installing build dependencies via mk-build-deps ([msg 8478]), resolving missing packages, and ultimately compiling the kernel. The NVIDIA open driver is then built against the custom kernel headers with zero errors — a stark contrast to the endless patching required against the community 6.19 kernel. After fixing a firmware mismatch that causes a boot panic (a genuine hardware issue, not a toolchain problem), the system boots into the custom 6.14 kernel with all 8 Blackwell GPUs fully recognized.

The final configuration — Proxmox VE on a self-built 6.14 kernel, NVIDIA 595.71.05 open driver, all compiled with GCC 12.2.0 — is the direct result of the decision made in this single message. It is a textbook example of how a principled engineering choice, made at the right moment, can transform a project from a cascade of failures into a clean, reproducible success.