The Diagnostic Pivot: Tracing a Kernel Panic Through Initramfs Analysis

In the high-stakes world of provisioning bare-metal machine learning infrastructure, a kernel panic on first boot is a moment of truth. After successfully building a custom Proxmox VE 6.14 kernel from source and compiling the NVIDIA 595.71.05 open GPU driver modules against it — all with a pristine, native GCC 12.2.0 toolchain — the assistant rebooted the new kpro6 host only to be met with a stark user report: "panic, no working init found" ([msg 8499]). The system had fallen back to the old 6.8 kernel. This is the context for message [msg 8502], a tightly focused diagnostic intervention that exemplifies systematic debugging under pressure.

The Moment of Crisis

The subject message opens with the assistant processing the results of its previous investigation. In [msg 8500], the assistant had hypothesized that the panic was caused by missing ZFS kernel modules — a reasonable guess, since Proxmox VE uses ZFS for its root filesystem, and a custom-built kernel might not have bundled the ZFS modules correctly. But [msg 8501] disproved that hypothesis: both zfs.ko and spl.ko (the Solaris Porting Layer that ZFS depends on) were present at /lib/modules/6.14.11-9-bpo12-pve/zfs/. The modules were on disk. The problem lay elsewhere.

Message [msg 8502] is the pivot. The assistant's reasoning is explicit in its opening line: "ZFS modules ARE there (zfs.ko, spl.ko). The issue is likely the initramfs — it may not have been rebuilt with ZFS support, or the ZFS userspace tools are out of sync." This is a textbook diagnostic maneuver: having ruled out one candidate cause (missing kernel modules), the assistant formulates a new hypothesis and designs an experiment to test it.

The initramfs — the initial RAM filesystem loaded by the kernel before mounting the real root — is the critical link. On a ZFS-root system, the initramfs must contain the ZFS kernel modules (zfs.ko, spl.ko) and the ZFS userspace tools (zpool, zfs, etc.) so that the kernel can import the ZFS pool and mount the root dataset. If the initramfs was built without ZFS support, or if it was built against a different kernel version, the kernel would have everything it needs on disk but nothing it needs in memory at boot time. The result: "no working init found."

The Diagnostic Tool: lsinitramfs

The assistant deploys lsinitramfs, a tool that lists the contents of a compressed initramfs image. The command is structured as a differential diagnosis: it inspects the new kernel's initramfs (/boot/initrd.img-6.14.11-9-bpo12-pve), filters for ZFS-related entries, and then performs the same inspection on the old kernel's initramfs (/boot/initrd.img-6.8.12-9-pve) for comparison.

The output reveals that the new initramfs does contain ZFS-related files: conf/conf.d/zfs, etc/default/zfs, etc/modprobe.d/zfs.conf, and the entire /etc/zfs/zed.d/ directory tree of ZFS Event Daemon scripts. These are the userspace configuration files and event handlers. But critically, the output is truncated — we see only the first 20 lines of ZFS-related content. The most important question — whether the actual .ko module files are inside the initramfs — is not yet answered by this truncated listing. The listing shows configuration and scripts, not kernel modules.

This truncation is significant. The assistant cannot yet conclude whether zfs.ko and spl.ko are physically present inside the initramfs image. The presence of ZFS configuration files suggests the initramfs build process attempted to include ZFS support, but the kernel modules themselves might have been omitted due to a path mismatch, a missing dependency, or a build-order issue. The assistant's next step — visible in the following messages — will be to check specifically for .ko files within the initramfs.

Assumptions and Their Validity

The assistant makes several implicit assumptions in this message. First, it assumes that the initramfs was rebuilt automatically when the kernel package was installed. On Debian-based systems, dpkg triggers update-initramfs via maintainer scripts, but this can fail silently if the initramfs-tools configuration is broken or if the ZFS hooks are not installed. Second, it assumes that the ZFS userspace tools on the root filesystem are compatible with the new kernel's ZFS modules — a version mismatch between the zfs.ko module and the zfs command-line tools could cause the initramfs to fail to import the pool even if both are present. Third, it assumes that the old kernel's initramfs is a valid reference standard — that whatever makes the old kernel boot successfully is what the new kernel needs.

These assumptions are reasonable but not infallible. The most significant risk is the first one: the initramfs might have been rebuilt, but with the wrong ZFS modules (e.g., modules from the old 6.8 kernel rather than the new 6.14 kernel). The lsinitramfs output in this message doesn't show the module paths, so the assistant cannot yet verify this.

Input Knowledge Required

To understand this message, the reader needs substantial Linux systems knowledge. The boot process for a UEFI system with ZFS root involves: the UEFI firmware loading the bootloader (systemd-boot, in Proxmox's case), which loads the kernel and initramfs from the EFI System Partition (ESP); the kernel decompresses the initramfs into memory and runs its /init script; the init script loads kernel modules (including ZFS), discovers and imports the ZFS pool, mounts the root dataset, and then hands off to the real init system (systemd). If any step in this chain fails — missing modules, wrong pool name, broken init script — the kernel panics with "no working init found."

The reader also needs to understand the Proxmox VE kernel build system. The PVE kernel is a Debian source package that applies patches to the upstream Linux kernel and builds both the kernel image and the ZFS modules as .deb packages. The proxmox-kernel-* deb installs the kernel image and initramfs, while the ZFS modules are included in the same package. The proxmox-boot-tool manages the ESPs and ensures the kernel and initramfs are copied to all boot partitions.

Output Knowledge Created

This message produces a critical piece of diagnostic evidence: the new kernel's initramfs does contain ZFS configuration files and event daemon scripts. This rules out the hypothesis that the initramfs was built without any ZFS awareness whatsoever. The ZFS userspace components are present. The remaining unknowns are: (1) whether the actual zfs.ko and spl.ko module files are inside the initramfs, and (2) whether the ZFS module versions match the kernel they're being loaded into.

The truncated output also implicitly teaches us something about the assistant's debugging methodology: it uses comparison as a diagnostic tool. By running the same command against both the working and non-working kernels, the assistant can spot differences that point to the root cause. This comparative approach is a hallmark of systematic troubleshooting.

The Broader Engineering Context

This message sits within a much larger narrative of building a pristine, high-performance ML training environment from scratch. The kpro6 host, equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each), represents a significant capital investment. The assistant had just recovered from a catastrophic failure — a previous attempt to install a community 6.19 kernel with a mismatched GCC 14 toolchain had bricked the system, requiring physical rescue from a live ISO. The user's explicit directive was to avoid "hacks" and build everything natively with the correct toolchain.

The kernel build in [msg 8486] through [msg 8490] was a triumph: the Proxmox VE 6.14 kernel compiled cleanly with the system's native GCC 12.2.0, producing .deb packages with zero errors. The NVIDIA driver build in [msg 8491] and [msg 8492] was equally clean. But the reboot failure in [msg 8495] through [msg 8498] (the repeated "No route to host" messages) showed that the system was unreachable — a kernel panic had occurred.

Message [msg 8502] is the first step in the recovery from that failure. It represents the transition from "something is broken" to "let me gather data to understand what." The assistant is methodically working through the boot chain: kernel modules → initramfs contents → bootloader configuration → firmware compatibility. Each step eliminates one class of failure and narrows the search space.

The Thinking Process

The assistant's reasoning in this message is a model of disciplined debugging. It begins by acknowledging the new evidence (ZFS modules are on disk), then explicitly states its revised hypothesis (initramfs issue), and finally designs a targeted command to test that hypothesis. The command is carefully structured: it checks both the new and old initramfs, uses grep -E "zfs|spl" to filter relevant content, and limits output with head -20 to avoid overwhelming the conversation.

The choice of lsinitramfs is itself revealing. This tool is not commonly used in everyday system administration — most administrators interact with initramfs through update-initramfs or mkinitramfs without ever inspecting the contents. The assistant's familiarity with this low-level tool indicates deep knowledge of the Linux boot process. It knows that the initramfs is a compressed cpio archive, that lsinitramfs can list its contents, and that the presence or absence of ZFS files within it is diagnostic.

The assistant also implicitly understands the temporal ordering of boot failures. A "no working init found" panic occurs late in the initramfs execution — after the kernel has decompressed the initramfs, after /init has started running, but before the real root filesystem has been mounted and the real init binary has been executed. This means the initramfs did start executing; the failure is in its logic, not in the kernel's ability to load it. This is a subtle but important distinction that informs the assistant's search.

Conclusion

Message [msg 8502] is a masterclass in diagnostic pivoting. When one hypothesis fails, the assistant does not flail or retry the same approach — it systematically formulates a new hypothesis, designs an experiment, and executes it. The message captures a moment of uncertainty being converted into actionable data. The initramfs contents are partially revealed, pointing toward a solution that will emerge in subsequent messages: a firmware mismatch between the custom kernel and the jaminmc firmware package, followed by a successful rebuild and reboot.

For the reader, this message illuminates the gap between "the kernel compiled" and "the system boots." In infrastructure engineering, compilation success is only the first milestone. The second milestone — a booting, stable system with all GPUs recognized — requires the kind of meticulous, hypothesis-driven debugging that this message exemplifies.