The Art of Systematic Debugging: Diagnosing a Kernel Panic on a Custom-Built Proxmox Host

Introduction

In the high-stakes world of provisioning a production ML training node equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs, few events are as disheartening as a kernel panic on first boot. Message <msg id=8504> captures a pivotal moment in the kpro6 provisioning saga — a moment when the assistant, having just recovered from a catastrophic system bricking caused by toolchain incompatibility, faced a second boot failure and had to methodically diagnose why the custom-built 6.14 kernel refused to find its init process.

This message is a masterclass in systematic debugging under pressure. It is not a triumphant moment of success, but rather the quiet, disciplined work of ruling out hypotheses one by one. The assistant had already built a pristine kernel and NVIDIA driver stack from source using a consistent GCC 12.2.0 toolchain — a hard-won victory after the previous attempt with a community kernel had required physical rescue from a live ISO. But the reboot into this clean build had failed with "panic, no working init found," and the system had fallen back to the old kernel. Now, from the safety of that old kernel, the assistant needed to understand why.

The Message in Full

Let us examine the message exactly as it was written:

The ZFS modules ARE in the initramfs. The spl.ko and zfs.ko are both present. Let me check what else could cause "no working init found":

[bash] ssh -o ConnectTimeout=10 root@10.1.2.6 ' echo "=== init in new initrd ===" lsinitramfs /boot/initrd.img-6.14.11-9-bpo12-pve | grep -E "^(init$|sbin/init|usr/sbin/init|lib/systemd)" echo "=== init in old initrd ===" lsinitramfs /boot/initrd.img-6.8.12-9-pve | grep -E "^(init$|sbin/init|usr/sbin/init|lib/systemd)" echo "=== check systemd on root ===" ls -la /sbin/init file /sbin/init echo "=== kernel cmdline ===" cat /proc/cmdline echo "=== pve-firmware version installed ===" dpkg -l pve-firmware | tail -1 echo "=== zfs ABI match? ===" modinfo /lib/modules/6.14.11-9-bpo12-pve/zfs/zfs.ko 2>/dev/null | grep -E "vermagic|depends" ' 2>&1

The message then presents the results of these seven diagnostic probes, which we will analyze in depth.

Why This Message Was Written: The Debugging Context

To understand why this message exists, we must appreciate the chain of events that led to it. The assistant had been tasked with provisioning kpro6, a new Proxmox host with 8× Blackwell GPUs and a 14TB NVMe. The initial attempt to install a modern kernel and NVIDIA driver had gone spectacularly wrong: a community 6.19 kernel built with GCC 14 from Debian Trixie was incompatible with the host's Bookworm GCC 12 toolchain. Workarounds involving patched kernel headers, rebuilt binaries, and a GLIBC_2.38 shim library culminated in the shim poisoning the system's dynamic linker, bricking SSH access and requiring physical rescue from a live ISO.

After that disaster, the user explicitly directed the assistant to avoid "hacks" and build everything natively. The assistant complied, building the official Proxmox VE 6.14 kernel from source using the system's native GCC 12.2.0, then building the NVIDIA open-gpu-kernel-modules 595.71.05 against it. Everything compiled with zero errors. The kernel was installed, the initramfs was generated, and the system rebooted.

And then: silence. The machine went dark. Three consecutive SSH connection attempts failed (see <msg id=8495>, <msg id=8496>, <msg id=8497>). The user eventually reported: "panic, no working init found;; up on old kernel" (<msg id=8499>). The new kernel had panicked, and the bootloader had fallen back to the previous kernel.

The assistant's initial hypothesis, stated in <msg id=8500>, was that the ZFS modules were missing — Proxmox VE requires ZFS to mount its root filesystem, and a kernel built without ZFS support would panic when it couldn't find its root pool. But the diagnostic checks in <msg id=8501>, <msg id=8502>, and <msg id=8503> had already disproven this: both zfs.ko and spl.ko were present in /lib/modules/6.14.11-9-bpo12-pve/zfs/ and in the initramfs.

Message <msg id=8504> is the pivot point. The assistant has accepted that its initial hypothesis was wrong and is now conducting a broader diagnostic sweep. The opening sentence — "The ZFS modules ARE in the initramfs. The spl.ko and zfs.ko are both present. Let me check what else could cause 'no working init found'" — is a statement of intellectual honesty. The assistant is publicly acknowledging that it was wrong and is moving on.

The Diagnostic Strategy: Seven Probes in One Command

The assistant designed a single SSH command that runs seven independent checks, each targeting a different possible cause of the panic. This is efficient debugging: instead of running separate commands and waiting for each result, the assistant bundles all probes into one parallel execution. Let us examine each probe and what it reveals about the assistant's reasoning.

Probe 1: Is init present in the new initramfs? The "no working init found" error literally means the kernel couldn't find or execute an init process. The most obvious cause is that the initramfs doesn't contain init or /sbin/init or usr/sbin/init. The assistant checks both the new and old initramfs for comparison. Both contain init and usr/sbin/init — so the init binary is present.

Probe 2: Is the init binary on the root filesystem intact? Even if the initramfs has init, the kernel might fail to hand off to the real init on the root filesystem. The assistant checks /sbin/init on the currently running system: it's a symlink to /lib/systemd/systemd, which is normal for a Debian/Proxmox system. The file command confirms it's a valid symlink.

Probe 3: What is the kernel command line? This is perhaps the most revealing check. The output shows: initrd=\EFI\proxmox\6.8.12-9-pve\initrd.img-6.8.12-9-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs. This is the command line from the currently running old kernel, not the failed new kernel. But it reveals a critical detail: the bootloader configuration references the old kernel's initrd path. This suggests the bootloader configuration may not have been properly updated for the new kernel.

Probe 4: What version of pve-firmware is installed? The output shows pve-firmware 3.19-4-jaminmc. This is the custom firmware from the previous failed community kernel attempt — the "jaminmc" suffix indicates it's not the official Proxmox firmware. A firmware mismatch can cause the kernel to panic if it expects certain firmware files that aren't present or are incompatible.

Probe 5: What is the ZFS module's vermagic and dependency information? This check uses modinfo to see if the ZFS kernel module was compiled against the correct kernel version. If the vermagic doesn't match the running kernel, the module won't load, and the root filesystem won't mount. The output is truncated in the message, but the assistant would have seen this information.

Assumptions and Their Validity

The assistant made several assumptions in crafting this diagnostic command:

Assumption 1: The panic is caused by a missing or broken component, not a hardware issue. This is a reasonable assumption given that the old kernel boots fine on the same hardware. The problem is software.

Assumption 2: The initramfs is the correct place to look. The "no working init found" error is an initramfs-stage error. The kernel loads the initramfs into a tmpfs and tries to execute /init from it. If that fails, the kernel panics. So checking the initramfs contents is exactly the right approach.

Assumption 3: The firmware package could be the culprit. This is a more subtle insight. The pve-firmware package contains binary firmware blobs that kernel drivers may request at boot time. If the kernel expects firmware version X but only version Y is installed, drivers can fail to initialize, potentially cascading into a boot failure. The assistant's suspicion of the "jaminmc" firmware is well-founded given that it was installed as part of the earlier failed community kernel experiment.

Assumption 4: The ZFS module ABI must match the kernel. The modinfo check on vermagic verifies that the ZFS module was compiled against the correct kernel version. This is critical because the ZFS modules were built as part of the PVE kernel build process, not separately. If the build process produced modules incompatible with the kernel, they would fail to load silently, and the root filesystem would never mount.

Mistakes and Incorrect Assumptions

The assistant's initial assumption — that the panic was caused by missing ZFS modules — was incorrect. This is visible in the progression from <msg id=8500> through <msg id=8504>. In <msg id=8500>, the assistant states: "This is likely because our self-built kernel doesn't have ZFS built-in (the PVE kernel needs ZFS to mount the root filesystem). The ZFS modules are built separately as part of the PVE kernel build process, but they may not have been installed." This was a reasonable hypothesis, but it was wrong.

However, the assistant did not waste time on this incorrect assumption. Within three messages (<msg id=8501> through <msg id=8503>), it confirmed that the ZFS modules were present both on disk and in the initramfs, and pivoted to a broader diagnostic in <msg id=8504>. This rapid hypothesis rejection is a hallmark of effective debugging.

Another potential blind spot: the assistant did not check whether the bootloader (GRUB or systemd-boot) was actually configured to boot the new kernel. The kernel command line from the running system shows initrd=\EFI\proxmox\6.8.12-9-pve\initrd.img-6.8.12-9-pve, which is the old kernel's initrd. But this is from the currently running old kernel, which booted via fallback. The assistant would need to check the bootloader configuration separately to see if the new kernel was properly registered.

Input Knowledge Required

To fully understand this message, the reader needs:

Proxmox VE boot architecture: Proxmox uses ZFS for the root filesystem and a specialized boot process involving EFI system partitions (ESPs) managed by proxmox-boot-tool. The kernel and initramfs are stored on the ESP, and the bootloader loads them from there.
The initramfs concept: Linux kernels load an initial ram filesystem (initramfs) that contains the tools and modules needed to mount the real root filesystem. If initramfs is missing or broken, the kernel panics with "no working init found."
Kernel module vermagic: Each kernel module is stamped with a "vermagic" string that includes the kernel version, compiler version, and configuration flags. The kernel refuses to load modules whose vermagic doesn't match, as a safety measure.
The pve-firmware package: This Proxmox-specific package contains firmware blobs for various hardware. A mismatch between the firmware version the kernel expects and what's installed can cause driver initialization failures.
The history of the jaminmc firmware: Earlier in the session, a community 6.19 kernel was installed along with a custom firmware package (3.19-4-jaminmc). This firmware was built for a different kernel and may contain incompatible blobs.

Output Knowledge Created

This message produces several pieces of critical diagnostic information:

Confirmation that init exists in both initramfs images: Both the new and old initramfs contain init and usr/sbin/init. This rules out "missing init" as the cause.
Confirmation that /sbin/init is intact: The symlink to systemd is valid. The root filesystem's init binary is not the problem.
The kernel command line of the fallback boot: Shows that the system booted with initrd=\EFI\proxmox\6.8.12-9-pve\initrd.img-6.8.12-9-pve — the old kernel's initrd. This is expected for a fallback boot but doesn't directly diagnose the new kernel's failure.
Firmware version mismatch: The system has pve-firmware 3.19-4-jaminmc installed, which is the custom firmware from the previous failed community kernel attempt. This is a strong candidate for the root cause.
ZFS module vermagic: Though truncated in the output, this check would reveal whether the ZFS module is compatible with the kernel it was built for. This knowledge directly drives the next actions: the assistant will downgrade the firmware to the official version, rebuild the initramfs, and try again — which is exactly what happens in subsequent messages (<msg id=8506> through <msg id=8510>).

The Thinking Process

The assistant's reasoning in this message follows a clear pattern:

State the current understanding: "The ZFS modules ARE in the initramfs." This acknowledges the result of the previous diagnostic round.
Formulate the next question: "Let me check what else could cause 'no working init found'." This frames the diagnostic as a search through a space of possible causes.
Design a multi-probe experiment: Rather than guessing, the assistant runs seven checks simultaneously, covering init presence, init integrity, boot configuration, firmware version, and module compatibility.
Interpret results: The assistant doesn't explicitly interpret the results in this message (that happens in the next message, <msg id=8505>), but the choice of probes reveals its mental model of the boot process. The assistant is thinking like a systems engineer: it has a mental model of how Linux boots (firmware → bootloader → kernel → initramfs → root filesystem → init), and it's systematically checking each link in this chain. The ZFS module hypothesis was about the "root filesystem" link. Now it's checking the "initramfs" and "init" links, plus environmental factors (firmware, module compatibility).

Conclusion

Message <msg id=8504> is a textbook example of structured debugging in a complex systems engineering context. It demonstrates intellectual honesty (admitting a wrong hypothesis), efficient experimentation (bundling seven probes into one command), and systematic reasoning (checking each link in the boot chain). The assistant's ability to pivot quickly from a disproven hypothesis to a broader diagnostic sweep is precisely the skill that separates effective troubleshooting from aimless tinkering.

The message also reveals the fragility of custom-built system software. A single mismatched firmware package — left over from a previous failed experiment — can cause a kernel to panic on boot. The assistant's thoroughness in checking even this detail is what ultimately leads to the fix: downgrading the firmware, rebuilding the initramfs, and successfully booting into the custom 6.14 kernel with all 8 GPUs recognized.

In the end, the debugging process captured in this single message — the hypothesis testing, the multi-probe diagnostic, the systematic ruling-out of causes — is the quiet, unglamorous work that makes complex infrastructure possible. It is not the flashy part of the story, but it is the part that matters most.