"Panic, No Working Init Found": A Diagnostic Turning Point in Custom Kernel Deployment

The message arrives as a fragment — seven words, two clauses separated by semicolons, delivered by a user who has just watched a server fail to boot:

panic, no working init found;; up on old kernel

This is message [msg 8499] in a long session dedicated to provisioning kpro6, a new Proxmox host equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had just completed an ambitious multi-hour effort: building a custom Proxmox VE 6.14 kernel from source using the system's native GCC 12.2.0, compiling the NVIDIA 595.71.05 open-gpu-kernel-modules from source against that kernel, installing both, and rebooting. Three consecutive SSH connection attempts ([msg 8495], [msg 8496], [msg 8497]) had returned "No route to host," leaving the assistant in the dark about what went wrong. This message is the user's intervention — a direct observation from the physical console that transforms the debugging landscape.

Why This Message Was Written

The user wrote this message because they were watching the server's boot process in real time, likely through IPMI, KVM, or a directly attached monitor. The assistant, operating remotely over SSH, had no visibility into the boot sequence — it only knew the machine wasn't reachable. The user's concise report serves two critical functions: it reports the exact kernel panic message, and it confirms that the system is recoverable (they've booted the old kernel). Without this message, the assistant would have continued polling SSH indefinitely, unable to diagnose the failure.

The semicolon-separated structure is telling. "Panic, no working init found" is the verbatim kernel panic message — the user is quoting what they saw on screen. "Up on old kernel" is the recovery action they took, restoring SSH access by selecting the previous kernel at the bootloader. This is a classic pattern in infrastructure debugging: the remote operator (the assistant) loses connectivity, and the on-site observer (the user) reports the failure mode and restores access.

The Reasoning and Motivation

The user's motivation is straightforward but important: they want the assistant to continue working. By reporting the exact error and restoring the old kernel, they've done three things simultaneously. First, they've provided the diagnostic key — "no working init found" is a specific failure that points to the initramfs, the root filesystem, or the firmware rather than a generic kernel crash. Second, they've re-established the SSH connection so the assistant can investigate. Third, they've implicitly communicated that the kernel build itself succeeded (it boots far enough to panic) but something in the boot chain is broken.

The user's decision to boot the old kernel rather than keep trying the new one shows practical judgment. They could have dropped into a recovery shell or attempted manual fixes at the console, but instead they chose the fastest path to restoring the assistant's ability to work. This is a collaboration pattern: the human handles physical access and observation, the AI handles analysis and remediation.

Assumptions Made

The user makes several assumptions in this brief message. They assume the assistant understands what "no working init found" means — that it will recognize this as a kernel panic related to the init process rather than, say, a driver crash or a filesystem error. They assume the assistant knows the context of the custom kernel build and can connect the error to the specific configuration. They assume the assistant can now SSH in (since the old kernel is booted) and perform diagnostics. And they assume that the problem is fixable — that this isn't a fundamental incompatibility but a configuration issue.

The assistant, for its part, had made an implicit assumption that was now proven wrong: that the custom-built kernel would boot cleanly. The kernel compiled without errors, the modules installed correctly, the initramfs was generated by the standard Debian tools — yet something was missing. The "no working init found" panic means the kernel successfully mounted the root filesystem but couldn't locate or execute /sbin/init, /bin/init, or the fallback /init from the initramfs. This is almost always a problem with the initramfs contents or the kernel's ability to access the root device.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was that the Proxmox kernel build process would produce a fully bootable kernel out of the box. The make deb command in the PVE kernel repository builds the kernel binary, headers, and tools, but it does not automatically install the ZFS modules into the initramfs or ensure that the firmware package is compatible. The assistant had installed the kernel and headers via dpkg -i but had not verified that the initramfs contained the necessary ZFS modules for the root filesystem.

A subtler issue was the firmware mismatch. The system had a custom "jaminmc" firmware package (version 3.19-4) left over from a previous community kernel experiment. The official PVE firmware package (3.16-3) contains binary blobs that the kernel expects to find at boot. While firmware issues typically cause hardware initialization failures rather than "no working init found," the mismatch contributed to the instability.

The assistant also assumed that update-initramfs would be triggered automatically by the kernel package installation. In this case, the initramfs was present and contained ZFS modules, but the boot process still failed — suggesting either a module dependency issue, a kernel command-line problem, or a firmware-related failure that prevented the root filesystem from being fully accessible.

Input Knowledge Required

To understand this message, one needs several layers of context. First, the technical meaning of "no working init found": this is a kernel panic that occurs when the kernel has mounted the root filesystem but cannot find or execute the init binary. It indicates that the root filesystem is accessible (the kernel didn't panic with "VFS: Unable to mount root fs") but something in the boot chain is broken — either the init binary is missing, the dynamic linker is broken, or critical libraries are absent.

Second, one needs to understand the Proxmox boot architecture: Proxmox uses ZFS as the root filesystem by default, which means the kernel must load ZFS modules (zfs.ko and spl.ko) from the initramfs before it can mount the root pool. If these modules are missing, incompatible, or incorrectly configured, the kernel may fail to find the root device or the init binary.

Third, one needs the session history: the assistant had just recovered from a bricked system caused by a GLIBC shim library, had cleaned up all remnants of the failed community kernel experiment, and had built everything from source with a consistent GCC 12.2.0 toolchain. The user's message is the first indication that this clean-slate approach still had issues.

Output Knowledge Created

This message transforms the debugging problem from a black-box "machine won't boot" to a targeted investigation. The assistant now knows exactly what error to look for and can focus on the initramfs, the ZFS module chain, and the firmware compatibility. The subsequent investigation ([msg 8500] through [msg 8509]) methodically checks each component: verifying ZFS modules exist in the initramfs, confirming the init binary is present, checking the kernel command line, examining the firmware package version, and rebuilding the initramfs with the official firmware.

The message also creates a new constraint: the fix must work on the next reboot, because the user has already had to intervene physically once. The assistant cannot afford another failed boot without wasting the user's time. This urgency drives the thorough pre-reboot sanity checks in [msg 8510], where the assistant verifies every component before attempting the second reboot.

The Thinking Process Visible in the Response

The assistant's response to this message ([msg 8500]) shows a clear diagnostic chain. It immediately identifies the most likely cause: "The new kernel panicked on init. This is likely because our self-built kernel doesn't have ZFS built-in." This is a reasonable first hypothesis — if the kernel can't find the ZFS modules, it can't mount the root filesystem, and it would indeed panic with "no working init found" because there's no root filesystem to find init on.

The assistant then checks the ZFS module location, the initramfs contents, the kernel command line, and the firmware version in rapid succession. Each check narrows the possibilities. When it finds that ZFS modules are present in both /lib/modules and the initramfs, it pivots to the firmware mismatch. When the firmware is corrected and the initramfs rebuilt, the second reboot succeeds ([msg 8513]).

The thinking visible here is systematic and hypothesis-driven: propose a cause, check the evidence, refine or pivot. The assistant doesn't guess randomly — it follows the boot chain from kernel to initramfs to root filesystem, verifying each link. The user's precise error message makes this possible. Without it, the assistant would have been stuck at "No route to host" with no way to proceed.

Conclusion

This seven-word message is a masterclass in effective debugging communication. The user provides exactly the information needed — the verbatim error and the recovery action — without extraneous detail. The assistant receives a clear diagnostic signal that transforms the problem from unreachable to solvable. The subsequent investigation and fix demonstrate the power of precise error reporting in human-AI collaboration: the human observes and reports, the AI analyzes and acts, and together they bring a complex system back to life.