The Diagnostic Pivot: Tracing a Kernel Panic Through the Initramfs

In the high-stakes world of provisioning a bare-metal machine with eight Blackwell RTX PRO 6000 GPUs, a kernel panic is not merely a setback—it is a puzzle that demands systematic forensic investigation. Message [msg 8503] captures a precise moment in that investigation: the assistant has just rebooted a freshly built Proxmox VE kernel (6.14.11-9-bpo12-pve) only to be greeted by the dreaded "no working init found" panic, forcing a fallback to the old 6.8 kernel. This message is the second step in a diagnostic chain, and it is a masterclass in how to isolate the root cause of a boot failure by comparing initramfs contents.

The Context of Failure

To understand why this message was written, we must first understand what led to it. The assistant had been tasked with provisioning a new Proxmox host, kpro6, equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. Earlier in the session, the assistant had attempted to install a community-built 6.19 kernel and the NVIDIA 595.71.05 driver, but a fundamental GCC version mismatch between the kernel (built with GCC 14 from Debian Trixie) and the host system (Debian Bookworm with GCC 12.2.0) caused a cascading series of failures. The assistant attempted workarounds—patching kernel headers, rebuilding gendwarfksyms and objtool, even creating a GLIBC shim library—but the shim ultimately poisoned the system's dynamic linker, bricking SSH access entirely. The system required physical rescue from a live ISO.

After recovery, the user gave a clear directive: no more hacks. Build everything from source with the native toolchain. The assistant pivoted completely. It cloned the official Proxmox VE kernel repository (branch bookworm-6.14) and built the kernel from source using the system's native GCC 12.2.0. It then cloned the NVIDIA open-gpu-kernel-modules repository and compiled the 595.71.05 driver against the freshly built kernel headers. Everything compiled with zero errors and zero patches. The assistant installed the kernel, pinned it with proxmox-boot-tool, refreshed the EFI System Partitions (ESPs), and rebooted with confidence.

Then came the panic.

The Initial Hypothesis

In [msg 8500], the assistant formulated its first hypothesis: "The new kernel panicked on init. This is likely because our self-built kernel doesn't have ZFS built-in (the PVE kernel needs ZFS to mount the root filesystem). The ZFS modules are built separately as part of the PVE kernel build process, but they may not have been installed."

This was a reasonable assumption. Proxmox VE uses ZFS as its root filesystem by default. The kernel itself does not have ZFS compiled in—ZFS is loaded as a kernel module (zfs.ko and its dependency spl.ko). If these modules are not present in the initramfs (the initial ram filesystem that the kernel loads before mounting the real root), the kernel cannot mount the ZFS pool and therefore cannot find the init binary. The result is exactly "no working init found."

The assistant's first diagnostic step ([msg 8501]) checked whether the ZFS modules existed on disk under /lib/modules/6.14.11-9-bpo12-pve/. They did—both zfs.ko and spl.ko were present in the zfs/ subdirectory. But presence on disk is not sufficient; the modules must also be included in the initramfs image that gets loaded at boot. The second diagnostic step ([msg 8502]) checked the initramfs contents for ZFS-related files and found ZFS userspace utilities and configuration files, but did not specifically check for the .ko files themselves. The output was ambiguous enough to warrant a more targeted check.

The Subject Message: A Targeted Diagnostic

Message [msg 8503] is the response to that ambiguity. The assistant executes a bash command over SSH that performs three checks:

  1. List ZFS .ko files in the new initramfs: lsinitramfs /boot/initrd.img-6.14.11-9-bpo12-pve | grep "\\.ko" | grep -E "zfs|spl|znvpair|zunicode|zlua|icp|zzstd|zavl"
  2. List ZFS .ko files in the old initramfs: The same command but for the working 6.8.12-9-pve initramfs
  3. Check depmod output: depmod -n 6.14.11-9-bpo12-pve | grep zfs | head -5 The command is carefully constructed. The lsinitramfs tool lists the contents of an initramfs image, and the double grep filters for kernel module files (.ko) that belong to the ZFS subsystem. The assistant includes all known ZFS-related module names—zfs, spl, znvpair, zunicode, zlua, icp, zzstd, zavl—to ensure nothing is missed. The comparison against the old, working initramfs provides a baseline: if the old initramfs has these modules and the new one does not, the hypothesis is confirmed. The results are revealing:
=== ZFS .ko in new initramfs ===
usr/lib/modules/6.14.11-9-bpo12-pve/kernel/drivers/net/ethernet/cavium/thunder/nicpf.ko
usr/lib/modules/6.14.11-9-bpo12-pve/kernel/drivers/net/phy/icplus.ko
usr/lib/modules/6.14.11-9-bpo12-pve/zfs/spl.ko
usr/lib/modules/6.14.11-9-bpo12-pve/zfs/zfs.ko
=== ZFS .ko in old initramfs ===
usr/lib/modules/6.8.12-9-pve/kernel/drivers/net/ethernet/cavium/thunder/nicpf.ko
usr/lib/modules/6.8.12-9-pve/kernel/drivers/net/phy/icplus.ko
usr/lib/modules/6.8.12-9-pve/zfs/spl.ko
usr/li...

Both initramfs images contain spl.ko and zfs.ko. The ZFS modules are present. The hypothesis is disproven.## The Reasoning Behind the Diagnostic

The assistant's thinking process in this message reveals several important assumptions and diagnostic strategies:

Assumption 1: The initramfs is the right place to look. The assistant correctly identifies that the initramfs is the critical artifact that bridges the kernel and the root filesystem. In a ZFS-booted Proxmox system, the initramfs must contain the ZFS kernel modules because the kernel itself does not have ZFS built-in. If the initramfs is missing these modules, the kernel cannot mount the root pool, and the boot fails with "no working init found."

Assumption 2: The old kernel's initramfs is a valid baseline. By comparing the new initramfs against the old one (which boots successfully), the assistant establishes a control condition. If the new initramfs has the same ZFS modules as the old one, then the ZFS module hypothesis is ruled out, and the search must continue elsewhere.

Assumption 3: The depmod check provides an independent verification. Running depmod -n (which outputs the module dependency resolution without writing to disk) and grepping for ZFS modules tests whether the kernel's module dependency database is properly configured. If depmod fails to resolve ZFS, that would indicate a deeper issue with the kernel's module metadata.

What the Message Achieves

This message produces critical output knowledge: the ZFS kernel modules are present in both the old and new initramfs. This conclusively rules out the missing-ZFS-modules hypothesis. The assistant can now move on to investigate other potential causes—the firmware package version mismatch, the dual-ESP configuration, or something else entirely.

The message also demonstrates a key engineering principle: falsify hypotheses systematically. Rather than guessing randomly, the assistant formulates a testable hypothesis, designs a command that produces clear evidence, and interprets the result. The evidence here is unambiguous: both initramfs images contain spl.ko and zfs.ko. The hypothesis is false.

The Input Knowledge Required

To understand this message, one needs to know:

  1. How Linux boot works: The kernel loads an initramfs, which contains the modules and scripts needed to mount the real root filesystem. If the initramfs lacks the necessary modules (like ZFS), the boot fails.
  2. How Proxmox VE uses ZFS: Proxmox typically boots from a ZFS pool, requiring zfs.ko and spl.ko to be in the initramfs.
  3. The lsinitramfs tool: This utility lists the contents of an initramfs image, allowing inspection of what modules are bundled.
  4. The ZFS module stack: ZFS on Linux consists of multiple kernel modules—spl (Solaris Porting Layer), zfs (the main filesystem), znvpair, zunicode, zlua, icp, zzstd, zavl. The assistant's grep pattern covers all of these.
  5. The depmod tool: This generates module dependency metadata. The -n flag outputs to stdout without modifying files, making it safe for diagnostic use.

The Broader Significance

This message sits at a turning point in the provisioning effort. The assistant had just recovered from a catastrophic failure (the bricked system caused by the GLIBC shim) and had successfully built a clean kernel and driver stack from source. The panic on reboot was a gut punch—it threatened to undo all that progress. But instead of panicking, the assistant methodically worked through the diagnostic tree.

The ZFS module check was the second node in that tree. The first node ([msg 8500]) checked whether the modules existed on disk. The second node ([msg 8501]) checked the initramfs for ZFS files broadly. The third node ([msg 8503], our subject) performed the precise check for the actual .ko files. Each step narrowed the search space.

When the ZFS hypothesis was ruled out, the assistant pivoted to checking the firmware package version and the dual-ESP configuration ([msg 8504]). The firmware package turned out to be the jaminmc custom build (3.19-4-jaminmc) rather than the official Proxmox firmware (3.16-3). The assistant downgraded the firmware and rebuilt the initramfs, which ultimately resolved the boot issue.

Mistakes and Incorrect Assumptions

The assistant's initial assumption—that ZFS modules were missing from the initramfs—was wrong. But it was a good wrong: it was the most likely cause given the information available, and it was efficiently tested and falsified. The mistake was not in the assumption itself but in the initial check ([msg 8502]), which only looked for ZFS-related files broadly (configuration files, scripts) without specifically checking for the .ko files. This created ambiguity that required the more targeted follow-up in [msg 8503].

Another subtle issue: the depmod -n check in the subject message appears to have been truncated in the output (the result cuts off after "usr/li..."). This is likely because the output was too long and got clipped by the SSH session or the tool's output capture. The assistant does not appear to have noticed this truncation, which could have led to missing important information about module dependency resolution.

Conclusion

Message [msg 8503] is a textbook example of systematic debugging in systems engineering. It formulates a clear hypothesis, designs a targeted diagnostic command, and interprets the results unambiguously. The ZFS modules are present in the initramfs—the hypothesis is disproven, and the search continues. This message, though small in scope, represents the disciplined thinking that separates effective troubleshooting from random trial and error. It is the kind of diagnostic that, in retrospect, seems obvious, but at the moment of crisis requires clarity of thought and deep knowledge of the boot process.

The assistant's ability to pivot from a failed hypothesis without frustration, and to immediately formulate the next diagnostic step, is what ultimately led to the successful resolution of the boot panic. The system was restored, the firmware was downgraded, and kpro6 booted into its custom-built 6.14 kernel with all 8 Blackwell GPUs fully recognized. The diagnostic chain that began with this message was a critical link in that success.