The Diagnostic Pivot: Investigating a Kernel Panic After a Clean-Source Build

The Message

In message 8501 of the conversation, the assistant executes the following bash command on a remote Proxmox host that has just suffered a boot panic:

ssh -o ConnectTimeout=10 root@10.1.2.6 '
echo "=== ZFS in new kernel ==="
find /lib/modules/6.14.11-9-bpo12-pve -name "zfs*" -o -name "spl*" -o -name "znvpair*" -o -name "zavl*" 2>/dev/null
echo "=== ZFS in old kernel ==="
find /lib/modules/6.8.12-9-pve -name "zfs*" | head -5
echo "=== Built debs ==="
ls /scratch/pve-kernel/*.deb
' 2>&1
=== ZFS in new kernel ===
/lib/modules/6.14.11-9-bpo12-pve/zfs
/lib/modules/6.14.11-9-bpo12-pve/zfs/zfs.ko
/lib/modules/6.14.11-9-bpo12-pve/zfs/spl.ko
=== ZFS in old kernel ===
/lib/modules/6.8.12-9-pve/zfs
/lib/modules/6.8.12-9-pve/zfs/zfs.ko
=== Built debs ===
/scratch/pve-kernel/linux-tools-6.14_6.14.11-9~bpo12+1_amd64.deb
/scratch/pve-kernel/linux-tools-6.14-dbgsym_6.14.11-9~bpo12+1_amd64.deb
/scratch/pve-kernel/proxmox-headers-6.14.11-9-bpo12-pve_6.14.11-9~bpo12+1_amd64.deb
/scratch/pve-kernel/...

This brief diagnostic command — only four lines of shell — sits at a pivotal moment in the provisioning of kpro6, a new Proxmox host equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. It represents the first step in a systematic debugging process after a boot failure, and the results it returns immediately falsify the assistant's initial hypothesis, forcing a deeper investigation that ultimately leads to the correct fix.

The Road to This Moment

To understand why this message was written, one must understand the harrowing sequence that preceded it. Earlier in the segment, the assistant had attempted to install a community-built 6.19 kernel on the Proxmox host. That kernel had been compiled with GCC 14 from Debian Trixie, while the host ran Debian Bookworm with GCC 12.2.0. The resulting toolchain mismatch triggered a cascade of increasingly desperate workarounds: patched kernel headers, rebuilt gendwarfksyms and objtool binaries, and finally a GLIBC_2.38 shim library that poisoned the system's dynamic linker, bricking SSH access entirely. The system required physical rescue from a live ISO.

After that near-disaster, the user gave a clear directive: no more hacks. Build everything from source with the native toolchain. The assistant pivoted completely, removing all community kernel and driver artifacts, cloning the official Proxmox VE kernel repository (branch bookworm-6.14), and building the kernel from source using the system's native GCC 12.2.0. The build completed with zero errors. Following the same clean approach, the assistant cloned the NVIDIA open-gpu-kernel-modules repository and compiled the 595.71.05 driver against the freshly built kernel headers — again, zero errors, zero patches. The kernel and drivers were installed, the bootloader was refreshed, and the system was rebooted with confidence.

The Panic

That confidence was premature. The user reported back with a terse but alarming message: "panic, no working init found;; up on old kernel" ([msg 8499]). The new kernel had failed to boot, and the system had fallen back to the previous 6.8 kernel via the bootloader's fallback mechanism.

In message 8500, the assistant formulated an initial hypothesis: the panic was likely caused by missing ZFS modules. Proxmox VE uses ZFS for its root filesystem, and if the ZFS kernel modules (zfs.ko, spl.ko) weren't present in the new kernel's module tree, the kernel would be unable to mount the root filesystem and would panic with "no working init found." The assistant ran a quick check from the old kernel:

find /lib/modules/6.14.11-9-bpo12-pve -name "zfs.ko*" 2>/dev/null

The output was empty — or so it seemed. The command returned nothing, which appeared to confirm the hypothesis. But this was a false negative, caused by the specific search pattern used. The find command searched for zfs.ko* (with a trailing asterisk), but the ZFS modules in the PVE kernel tree live under a zfs/ subdirectory, not directly under the module root. The initial check was too narrow.

The Subject Message: A Broader Diagnostic

Message 8501 is the correction. The assistant broadens the search, using multiple -name patterns ("zfs*", "spl*", "znvpair*", "zavl*") without restricting to the .ko* glob, and importantly, removes the 2>/dev/null suppression from the first find command (though it still suppresses stderr for the overall command). The command also adds a comparison against the old kernel's ZFS layout and lists the built .deb packages for completeness.

The results are immediately revealing:

  1. ZFS modules ARE present in the new kernel: /lib/modules/6.14.11-9-bpo12-pve/zfs/zfs.ko and spl.ko are both there, in a zfs/ subdirectory identical to the old kernel's layout.
  2. The built .deb packages are all present, confirming the build artifacts are intact. This is a classic debugging moment: the initial hypothesis is cleanly falsified. The ZFS modules exist. The panic has a different cause. The assistant must now pivot its investigation.

The Thinking Process Visible in the Message

The structure of this diagnostic command reveals the assistant's reasoning process. Three deliberate choices stand out:

First, the comparative design. The command doesn't just check the new kernel in isolation; it explicitly compares against the old kernel (find /lib/modules/6.8.12-9-pve -name "zfs*" | head -5). This is a thoughtful diagnostic pattern — by showing the known-good layout alongside the unknown, the assistant can immediately spot structural differences. The results show that both kernels store ZFS modules in a zfs/ subdirectory, confirming the layout is correct.

Second, the inclusion of the .deb listing. This is a sanity check that goes beyond the immediate ZFS question. By listing the built packages, the assistant verifies that the build process completed correctly and that the expected artifacts are in place. This rules out a class of build-failure hypotheses before they even need to be articulated.

Third, the careful use of find patterns. The previous check in message 8500 used -name "zfs.ko*" which failed to find modules in subdirectories. The subject message uses -name "zfs*" (without the .ko extension filter), which catches both the zfs/ directory and any files within it. This is a subtle but important correction — the assistant recognized that its earlier search pattern was too restrictive and adjusted accordingly.

Assumptions and Their Consequences

The primary assumption in this message is that the ZFS module presence is the critical diagnostic variable. This assumption was inherited from the previous message's hypothesis, and the assistant is now testing it directly. The assumption turns out to be incorrect — the ZFS modules are fine — but testing it is still the right move. In debugging, disproving a hypothesis is as valuable as proving one.

A secondary assumption is that the old kernel (6.8.12-9-pve) provides a reliable baseline for comparison. This is reasonable: the old kernel boots successfully, so its module layout is known-good. The comparison confirms that the new kernel's ZFS layout matches the old one, ruling out structural issues.

The message also implicitly assumes that the find command's output format is sufficient for diagnosis. It is — the results clearly show the module paths. But the command doesn't check module dependencies, vermagic strings, or other metadata that might reveal subtler incompatibilities (such as a module compiled against a different kernel version). That deeper investigation comes in subsequent messages.

Input Knowledge Required

To understand this message, the reader needs several pieces of context:

  1. Proxmox VE architecture: Proxmox uses ZFS for its root filesystem by default. The kernel must have ZFS modules available at boot time to mount the root pool. Without them, the kernel panics with "no working init found."
  2. The PVE kernel module layout: ZFS modules in Proxmox kernels live under /lib/modules/<version>/zfs/ (a subdirectory, not directly under kernel/). This is a PVE-specific convention.
  3. The boot failure history: The system had just been rebooted into a custom-built 6.14 kernel and failed. The user's report of "panic, no working init found" is the key symptom being investigated.
  4. The previous diagnostic attempt: Message 8500 ran a narrower search that returned empty results, leading to the ZFS-missing hypothesis. The subject message is a direct response to that inconclusive check.
  5. The clean-build context: The kernel was built from the official Proxmox git repository using the native GCC 12.2.0 toolchain, after a previous attempt with a community kernel had bricked the system. This context makes the boot failure particularly concerning — if a clean-source build also fails, the problem may be more fundamental.

Output Knowledge Created

This message produces three concrete pieces of knowledge:

  1. ZFS modules are present in the new kernel at /lib/modules/6.14.11-9-bpo12-pve/zfs/zfs.ko and spl.ko. This disproves the ZFS-missing hypothesis.
  2. The module layout matches the old kernel, confirming that the PVE build process correctly placed the ZFS modules in the expected location.
  3. The built packages are intact, ruling out a build failure as the cause. This output knowledge immediately redirects the debugging effort. Since ZFS modules are present, the assistant must look elsewhere: the initramfs contents, the firmware package version, the ESP configuration, or the kernel command line. In the subsequent messages (8502–8510), the assistant systematically checks each of these, eventually discovering that a firmware version mismatch (a custom "jaminmc" firmware package) was likely the culprit, and after fixing it and rebuilding the initramfs, the system boots successfully.

The Broader Engineering Lesson

This message exemplifies a fundamental debugging principle: test your hypotheses with the broadest possible signal before narrowing in. The assistant's first check (message 8500) was too narrow — it searched for zfs.ko* and found nothing, creating a false negative. The corrected search in message 8501 uses wildcard patterns that capture the full module tree structure, revealing the truth.

But there's a deeper lesson here about the relationship between toolchain consistency and system reliability. The entire kpro6 provisioning saga — from the bricked 6.19 kernel to the successful 6.14 build — demonstrates that building from source with a matching toolchain is not just a theoretical ideal but a practical necessity. When the assistant tried to patch around binary incompatibilities, the system collapsed. When it built everything from source with GCC 12.2.0, the kernel compiled cleanly, the NVIDIA driver compiled cleanly, and the only remaining issue was a firmware version mismatch — a comparatively trivial problem.

Message 8501 is the moment where the debugging process pivots from a simple hypothesis to a systematic investigation. It's a small command — four lines of shell — but it carries the weight of the entire provisioning effort. The results say: the kernel is good, the modules are in place, look somewhere else. And that "somewhere else" turns out to be a firmware package that needed downgrading to match the official PVE release. After that fix, the system boots cleanly, all 8 GPUs are recognized, and kpro6 joins the cluster as a fully operational training node.