The Dual ESP Diagnosis: Debugging a Kernel Panic on a Custom Proxmox Build

Introduction

In the high-stakes world of provisioning a new machine learning server, few moments are as tense as the first reboot after installing a custom-built kernel. Message 8506 captures exactly such a moment: the assistant, having just completed a clean, source-based build of the Proxmox VE 6.14 kernel and the NVIDIA 595.71.05 open driver on a machine called kpro6, discovers that the new kernel panics with "no working init found." The user, who had earlier directed the assistant to avoid "hacks" after a previous failed attempt bricked the system, now reports the failure and offers a clue: "There are 2 boot / esp partitions on 2 disks maybe?" This message is the assistant's response—a focused diagnostic probe that methodically investigates the dual-ESP hypothesis while simultaneously questioning whether a custom firmware package might be the culprit.

Context: The Road to kpro6

To understand why this message matters, we must appreciate what preceded it. The assistant had been working on DFlash drafter training infrastructure—a sophisticated asynchronous training pipeline for speculative decoding models running on NVIDIA Blackwell GPUs. The project required provisioning kpro6, a new Proxmox host equipped with 8× RTX PRO 6000 Blackwell GPUs (96 GB each) and a 14 TB NVMe drive.

The journey had been harrowing. An earlier attempt to install a community 6.19 kernel and the NVIDIA driver via DKMS had spiraled into a debugging nightmare due to a GCC version mismatch—the community kernel was built with GCC 14 from Debian Trixie, while the host ran Bookworm's GCC 12. The assistant attempted increasingly desperate workarounds: patched kernel headers, rebuilt gendwarfksyms and objtool binaries, and even a GLIBC_2.38 shim library. The shim ultimately poisoned the system's dynamic linker, bricking SSH access and requiring physical rescue from a live ISO.

After that catastrophe, the user gave a clear directive: no more hacks. Build everything natively with the correct toolchain. The assistant pivoted completely, cloning the official Proxmox VE kernel repository (branch bookworm-6.14) and building from source using the system's native GCC 12.2.0. The NVIDIA open-gpu-kernel-modules were compiled the same way—against the freshly built kernel headers. Zero errors, zero patches. The kernel and modules installed cleanly, and the system was rebooted into the new 6.14.11 kernel.

Then came the panic.

The Message: A Diagnostic Probe

Message 8506 opens with the assistant's reasoning: "The firmware package is the jaminmc one (3.19-4-jaminmc) — that might be mismatched. But more importantly, the dual ESP issue: the proxmox-boot-tool refresh may not have copied the new kernel's initramfs to both ESPs correctly."

This single sentence reveals the assistant's diagnostic branching. It considers two potential causes simultaneously:

  1. A firmware mismatch: The pve-firmware package installed on the system is version 3.19-4-jaminmc, which is a custom build (the "jaminmc" suffix indicates it was built by the user or from a non-standard repository). The assistant recognizes that firmware packages contain binary blobs for hardware initialization, and a mismatch between the firmware version and the kernel version can cause boot failures. However, it quickly deprioritizes this hypothesis—the firmware issue would more likely cause hardware initialization failures than a "no working init found" panic.
  2. A dual-ESP configuration problem: This is the primary hypothesis. The user's hint about "2 boot / esp partitions on 2 disks" is crucial. In Proxmox VE installations with multiple disks, each disk typically has its own EFI System Partition (ESP). The proxmox-boot-tool is responsible for keeping all ESPs synchronized with the latest kernel and initramfs. If the tool failed to copy the new kernel's initramfs to one of the two ESPs, and the system's UEFI firmware happened to boot from that ESP, the kernel would load but the initramfs would be missing or incorrect, leading to "no working init found."

The Bash Command: A Methodical Investigation

The assistant constructs a bash command that performs a systematic audit of the ESP configuration. Let us examine each step:

echo "=== ESP UUIDs ==="
cat /etc/kernel/proxmox-boot-uuids

This reads the list of ESP UUIDs that Proxmox is configured to manage. The file /etc/kernel/proxmox-boot-uuids is the authoritative source—it tells the system which partitions should receive kernel updates.

echo "=== ESP mounts ==="
proxmox-boot-tool status

This command queries the current state of all configured ESPs, showing which kernel versions are installed on each.

echo "=== ESP contents ==="
for uuid in $(cat /etc/kernel/proxmox-boot-uuids); do
    mountpoint=$(findmnt -rn -S UUID=$uuid -o TARGET 2>/dev/null)
    if [ -z "$mountpoint" ]; then
        mkdir -p /tmp/esp_$uuid
        mount UUID=$uuid /tmp/esp_$uuid 2>/dev/null
        mountpoint=/tmp/esp_$uuid
    fi
    echo "--- ESP $uuid at $mountpoint ---"
    ls -la $mountpoint/EFI/proxmox/ 2>/dev/null
    for kdir in $mountpoint/EFI/proxmox/*/; do
        echo "  $(basename $kdir): $(ls $kdir 2>/dev/null)"
    done
done

This loop is particularly elegant. For each ESP UUID, it first checks if the partition is already mounted (using findmnt). If not, it creates a temporary mount point and mounts it. Then it lists the contents of the EFI/proxmox/ directory on that ESP, and for each kernel version directory found, it lists the files within. This reveals exactly which kernels and initramfs images are present on each ESP.

The Output: What the Assistant Learned

The output confirms the dual-ESP setup:

Assumptions and Their Implications

The assistant operates under several assumptions in this message:

Assumption 1: The dual-ESP configuration is the most likely cause. This is a reasonable assumption given the user's hint and the fact that "no working init found" often points to initramfs issues. However, the assistant also keeps the firmware hypothesis alive as a secondary possibility.

Assumption 2: proxmox-boot-tool status accurately reflects what's on the ESPs. The status command shows that both ESPs are "configured with: uefi (versions: 6.14.11-9-bpo12-pve, 6.8.12-9-pve)." The assistant assumes this means the actual files are present and correct. The subsequent manual inspection of ESP contents is a validation step—the assistant is not taking the status output at face value.

Assumption 3: The initramfs is the critical component. The assistant focuses on whether the initramfs (the initial RAM filesystem that the kernel loads before mounting the real root) is present and correct. This is the right thing to check for a "no working init found" error, which occurs when the kernel successfully loads but cannot find an init binary to start the system.

Assumption 4: The ESPs can be mounted read-only without issues. The assistant mounts the ESPs temporarily, which is safe for diagnostic purposes but assumes the filesystem is healthy and not corrupted.

Potential Mistakes and Oversights

While the assistant's diagnostic approach is sound, there are several aspects worth examining critically:

The firmware hypothesis was too quickly dismissed. The "jaminmc" firmware package is a custom build. If the firmware binary blobs in this package are incompatible with the 6.14 kernel, they could cause hardware initialization failures that manifest as a boot panic. The assistant mentions this possibility but immediately pivots to the ESP issue. A more thorough investigation might have checked the firmware version against the kernel's expected firmware ABI.

The assistant did not check the kernel command line. In message 8504, the assistant examined /proc/cmdline from the old kernel and found root=ZFS=rpool/ROOT/pve-1 boot=zfs. But it did not verify what kernel command line the new kernel would use. If the UEFI boot entry for the new kernel has an incorrect or missing root= parameter, the kernel would not know where to find the root filesystem.

The assistant did not verify that the ZFS modules are loadable. While message 8501 confirmed that zfs.ko and spl.ko exist in the new kernel's module directory, and message 8503 confirmed they are in the initramfs, the assistant did not check whether these modules can actually be loaded against the new kernel. Module version magic mismatches are a common source of boot failures.

The assistant assumed proxmox-boot-tool handled the dual-ESP case correctly. Proxmox's boot tool is generally reliable, but edge cases with multiple ESPs can be tricky. The assistant's manual inspection of ESP contents is a good validation step, but it only checked one ESP (801D-3BEA) before the output was truncated. The second ESP (8020-D63A) was not inspected.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

Proxmox VE boot architecture: Understanding that Proxmox uses a kernel pinning mechanism (proxmox-boot-tool) to manage multiple kernels across potentially multiple EFI System Partitions. The tool copies kernel images, initramfs files, and boot configuration to each configured ESP.

EFI System Partition layout: Knowledge that ESPs use the FAT32 filesystem and contain an EFI/ directory with vendor-specific subdirectories (e.g., EFI/proxmox/ for Proxmox). Each kernel version gets its own subdirectory containing the kernel image (vmlinuz-*) and initramfs (initrd.img-*).

Linux boot process: Understanding that the kernel loads an initramfs from the ESP, which contains the necessary drivers and scripts to mount the real root filesystem. If the initramfs is missing, corrupted, or lacks the required modules (like ZFS), the kernel will panic with "no working init found."

ZFS as root filesystem: Proxmox typically uses ZFS for its root pool. The initramfs must contain the ZFS kernel modules (zfs.ko, spl.ko) and the ZFS userspace tools to import the pool and mount the root dataset.

Firmware packages: Understanding that pve-firmware contains binary firmware blobs that the kernel loads for various hardware components. A version mismatch can cause hardware initialization failures.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Confirmation of dual-ESP configuration: The system has two ESPs (UUIDs 801D-3BEA and 8020-D63A), both configured for UEFI boot with both kernel versions present.

Validation of proxmox-boot-tool operation: The tool correctly registered both kernels on both ESPs, suggesting the boot configuration step was successful.

ESP accessibility: The first ESP (801D-3BEA) is mountable and contains the expected kernel directories. This rules out filesystem corruption or missing partitions as the cause.

Narrowing of the problem space: Since both ESPs appear correctly configured, the "no working init found" error is likely caused by something other than a missing initramfs. Possible remaining causes include:

The Thinking Process: A Window into Diagnostic Reasoning

What makes this message particularly instructive is the visible reasoning process. The assistant demonstrates a classic diagnostic approach:

  1. Receive symptom: "panic, no working init found"
  2. Receive user hint: "2 boot / esp partitions on 2 disks"
  3. Formulate hypotheses: firmware mismatch OR dual-ESP issue
  4. Prioritize hypotheses: The dual-ESP issue is deemed "more important" because it directly relates to the boot path
  5. Design a test: A bash command that systematically inspects both ESPs
  6. Execute the test: Run the command remotely via SSH
  7. Interpret results: Both ESPs appear correctly configured The assistant's thinking is visible in the structure of the bash command itself. It doesn't just run proxmox-boot-tool status and trust the output—it manually mounts each ESP and inspects the files. This shows an understanding that status tools can report what should be there, but only direct filesystem inspection confirms what is there. The assistant also demonstrates good engineering judgment in how it handles the mounting. It uses findmnt to check if the ESP is already mounted (avoiding redundant mounts), creates temporary mount points in /tmp/, and mounts without specifying a filesystem type (letting the kernel auto-detect FAT32). These are small but important details that show experience with system administration.

Conclusion

Message 8506 is a textbook example of structured diagnostic reasoning in systems engineering. Faced with a boot failure on a freshly built custom kernel, the assistant methodically investigates the most likely cause—a dual-ESP configuration issue—while keeping alternative hypotheses (firmware mismatch) in reserve. The bash command it constructs is a model of clarity: it reads the authoritative configuration, queries the management tool, and then validates by direct inspection.

The message also reveals the assistant's growth from the earlier disaster. Where the previous attempt involved increasingly desperate hacks (shim libraries, patched binaries), this diagnostic is clean, systematic, and grounded in fundamental system administration principles. The assistant has learned that the path to reliability is not clever workarounds but methodical investigation and clean builds.

The story does not end here. The assistant will need to dig deeper—checking the initramfs contents more carefully, verifying ZFS module compatibility, examining the kernel command line, and potentially rebuilding the initramfs. But message 8506 marks the point where the diagnostic process shifts from "what went wrong with the build" to "what went wrong with the boot." It is the pivot point in a debugging journey that will ultimately lead to a stable, production-ready machine learning node with 8 Blackwell GPUs, ready for DFlash drafter training.