"It booted!" — The Climax of a Kernel and Driver Engineering Ordeal

"It booted!"

Three words. A single exclamation. But in message [msg 8514] of this opencode coding session, those three words carry the weight of a system that was bricked, rescued, rebuilt from source, debugged through a kernel panic, and finally brought online with all eight NVIDIA RTX PRO 6000 Blackwell GPUs fully recognized. This message is the climax of one of the most intense infrastructure provisioning sequences in the entire conversation — a moment where engineering persistence meets a clean, working result.

What the Message Contains

The message is deceptively simple. The assistant writes:

[assistant] It booted! [bash] ssh -o ConnectTimeout=10 root@10.1.2.6 'nvidia-smi' 2>&1 Fri May 15 19:33:27 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | ...

The output is truncated in the conversation data, but the header alone tells the story: Driver Version: 595.71.05, CUDA Version: 13.2. The subsequent message [msg 8515] reveals the full picture: eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 97,887 MiB of VRAM (96 GB), totaling 783 GB of GPU memory. All running on a custom-built Proxmox VE 6.14 kernel compiled from source with the system's native GCC 12.2.0 toolchain.

The Road to This Moment

To understand why this message matters, one must understand what preceded it. The journey to "It booted!" was anything but straightforward.

Earlier in the session, the assistant had been tasked with provisioning kpro6 — a new Proxmox host equipped with eight Blackwell-generation RTX PRO 6000 GPUs and a 14 TB NVMe drive. The initial approach was to install a community-built 6.19 kernel and the NVIDIA 595.71.05 open driver via DKMS (Dynamic Kernel Module Support). This seemed like a reasonable shortcut: use a pre-built kernel and let DKMS handle the driver compilation.

What followed was a debugging spiral of extraordinary complexity. The community kernel had been compiled with GCC 14 from Debian Trixie, while the host ran Debian Bookworm with GCC 12. This fundamental toolchain mismatch meant that kernel headers, build tools, and module binaries were all incompatible. The assistant attempted a series of increasingly desperate workarounds: patched kernel headers, rebuilt gendwarfksyms and objtool binaries, and finally a GLIBC_2.38 compatibility shim library. That shim proved catastrophic — it poisoned the system's dynamic linker, bricking SSH access entirely. The machine required physical rescue from a live ISO.

After the system was restored, the user issued a clear directive: no more hacks. Everything must be built from source with the correct, native toolchain. This was the turning point.

The Pivot to Native Source Builds

The assistant executed a complete pivot. It removed all traces of the community kernel and the jaminmc driver artifacts. It cloned the official Proxmox VE kernel repository from the bookworm-6.14 branch and compiled the kernel from source using the system's native GCC 12.2.0. The build completed with zero errors.

Following the same clean approach, the assistant cloned the NVIDIA open-gpu-kernel-modules repository and compiled the 595.71.05 kernel modules against the freshly built kernel headers. Again, zero errors, zero patches. The result was a pristine, fully compatible kernel and driver stack — all built with a single, matching GCC version.

The Kernel Panic and Debugging

But the story didn't end there. The first boot attempt into the new kernel resulted in a terrifying message from the user: "panic, no working init found;; up on old kernel" ([msg 8499]). The kernel had panicked during initialization.

The assistant's debugging across messages [msg 8500] through [msg 8510] is a textbook example of systematic kernel troubleshooting. The initial hypothesis was that ZFS modules were missing — Proxmox VE uses ZFS as its root filesystem, and without the ZFS kernel modules, the system cannot mount its root partition. However, investigation revealed that both zfs.ko and spl.ko were present in the kernel module directory and in the initramfs.

The root cause turned out to be a firmware mismatch. The system had a custom "jaminmc" firmware package (version 3.19-4-jaminmc) installed, which was incompatible with the official Proxmox 6.14 kernel build. The assistant downgraded to the official pve-firmware version 3.16-3 from the standard Proxmox repository, rebuilt the initramfs with update-initramfs, and refreshed both EFI System Partitions (ESPs) with proxmox-boot-tool refresh.

After a second reboot, the system was unreachable for over two minutes (messages [msg 8511] and [msg 8512] show "No route to host"). Then, in message [msg 8513], the kernel version check succeeded: 6.14.11-9-bpo12-pve. The system was alive.

The Significance of Message 8514

Message 8514 is the confirmation that everything works. The nvidia-smi output is the definitive proof point. It validates:

  1. The custom-built kernel boots successfully — the Proxmox VE 6.14 kernel compiled from source with GCC 12.2.0 is fully functional, including ZFS root filesystem support.
  2. The NVIDIA open driver 595.71.05 loads correctly — all five kernel modules (nvidia.ko, nvidia-modeset.ko, nvidia-drm.ko, nvidia-uvm.ko, nvidia-peermem.ko) compiled from source against the custom kernel headers are compatible and operational.
  3. All eight GPUs are recognized — the RTX PRO 6000 Blackwell Server Edition GPUs are visible to the driver, with CUDA 13.2 support.
  4. The "no hacks" engineering principle is validated — building everything from source with a consistent toolchain proved vastly more reliable than patching binary incompatibilities.

Assumptions and Decisions

This message reflects several key assumptions and decisions:

The assistant assumed that the firmware mismatch was the sole cause of the kernel panic, and that downgrading to the official pve-firmware would resolve it. This assumption proved correct, but it was reached only after ruling out other possibilities (missing ZFS modules, initramfs issues, dual ESP configuration problems).

The decision to rebuild the initramfs after the firmware downgrade was critical. Without this step, the old initramfs would still have referenced the incompatible firmware, and the boot would likely have failed again.

The decision to refresh both ESPs addressed the dual-boot configuration. The system had two EFI System Partitions (UUIDs 801D-3BEA and 8020-D63A), and both needed to have the correct kernel and initramfs. The assistant verified this explicitly.

The assumption that the system would come back online after the second reboot was not guaranteed. The assistant waited 90 seconds, then 60 seconds, then another 60 seconds before getting a response. The "No route to host" errors could have indicated a more fundamental problem.

Knowledge Created

This message creates several important pieces of knowledge:

The Broader Engineering Lesson

Message 8514 is more than just a status update. It encapsulates a fundamental engineering principle: when building system software, toolchain consistency is paramount. The initial approach of using a pre-built kernel from an incompatible distribution (Debian Trixie GCC 14 on a Bookworm GCC 12 system) created a cascade of failures that no amount of patching could fix. The shim library workaround didn't solve the problem — it made it catastrophically worse.

The successful approach was slower but infinitely more reliable: obtain the source code, compile it with the native toolchain, and verify every component against its dependencies. This is the difference between hacking and engineering. The three-word exclamation "It booted!" is the sound of that principle being vindicated.

For anyone building GPU compute infrastructure on custom kernels — whether for machine learning training, scientific computing, or any other purpose — this message and the journey that led to it serve as a powerful case study. The temptation to take shortcuts with pre-built binaries is strong, but the cost of incompatibility can far exceed the time investment of a clean source build. Sometimes the fastest path is the one that goes through the compiler.