The Moment the Pivot Broke: Why mk-build-deps Failed and What It Revealed About Clean Engineering

The Message

ssh -o ConnectTimeout=10 root@10.1.2.6 '
cd /scratch/pve-kernel
# mk-build-deps needs the --build-dep flag or the control file directly
# Let me try the alternative approach
mk-build-deps -t "apt-get -y --no-install-recommends" -i proxmox-kernel-6.14.11/debian/control 2>&1 | tail -30
' 2>&1
E: You must put some 'deb-src' URIs in your sources.list
mk-build-deps: Unable to find package name in `apt-cache showsrc proxmox-kernel-6.14.11/debian/control'

At first glance, this is a forgettable moment: a failed command, a cryptic error, a dead end. But in the arc of the conversation, message [msg 8480] is a fulcrum. It is the third consecutive failure of the same tool (mk-build-deps), and it marks the precise instant when the assistant's strategy of "tweak the flags and retry" exhausts itself, forcing a fundamental re-evaluation of how to build the Proxmox VE kernel from source. This message is not just a failed command — it is a compressed lesson about tool assumptions, the cost of not diagnosing root causes, and the engineering philosophy that the user had just demanded: no more hacks.

Context: The Shadow of a Bricked System

To understand why this message matters, we must understand what led to it. The assistant was provisioning kpro6, a new Proxmox host equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs — a machine destined for large-scale ML training. Earlier in the session, the assistant had attempted to install a community-built 6.19 kernel (from Debian Trixie) and the NVIDIA 595.71.05 open driver via DKMS. This spiraled into a nightmare of GCC version mismatches: the community kernel was compiled with GCC 14 from Trixie, while the host ran Bookworm's GCC 12. The assistant tried patch after patch — rebuilt gendwarfksyms, recompiled objtool, and ultimately created a GLIBC_2.38 compatibility shim library that poisoned the system's dynamic linker. The result: every SSH invocation failed because bash itself couldn't load. The system was bricked. The user had to physically rescue it from a live ISO.

The user's response upon recovery was pointed and instructive (msg 8469): "try not to do hacks; open nvidia can probably just be built locally without hacky repos, same with the kernel, all using correct gcc?"

The assistant agreed wholeheartedly (msg 8470) and began a clean slate: removing the community kernel, purging NVIDIA leftovers, and preparing to build the official Proxmox VE kernel from source using the system's native GCC 12.2.0. This was the "correct" approach — the one that should have worked from the beginning.

The Assumption That Failed

The assistant had cloned the PVE kernel repository (bookworm-6.14 branch) into /scratch/pve-kernel and prepared the build directory. The next logical step was to install the build dependencies. In the Debian packaging ecosystem, the canonical tool for this is mk-build-deps, which reads a debian/control file, resolves the Build-Depends field against the package repository, and installs everything needed.

The assistant's assumption was straightforward: the PVE kernel is packaged as a Debian source package, so mk-build-deps should work. This assumption was wrong, but the assistant did not know it yet. The error message — "You must put some 'deb-src' URIs in your sources.list" — pointed at a surface-level problem: mk-build-deps needs deb-src entries in APT sources to query source package metadata. But the real problem was deeper: the PVE kernel is not distributed as a Debian source package at all. It is built from a git repository using a custom Makefile. There is no apt-cache showsrc entry for proxmox-kernel-6.14.11 because no such source package exists in any repository.

The assistant did not yet grasp this distinction. It interpreted the error as a configuration issue — missing deb-src lines — rather than a fundamental incompatibility between the tool and the project's distribution model.

The Thinking Process Visible in the Message

The comment in the bash command reveals the assistant's reasoning: "mk-build-deps needs the --build-dep flag or the control file directly." This is a diagnostic hypothesis — the assistant believes the previous two attempts (msg 8478 and msg 8479) failed because of incorrect flag combinations. In msg 8478, it used -ir flags. In msg 8479, it used --install --remove. Now in msg 8480, it tries -t "apt-get -y --no-install-recommends" -i. Each attempt varies the invocation syntax while keeping the core approach unchanged.

This is a classic debugging pattern: when a tool refuses to work, try different flag combinations. It is a reasonable heuristic when the problem is genuinely about syntax or options. But here, the problem is conceptual, not syntactic. The assistant is trapped in a local search of the flag space when the real solution requires escaping that space entirely.

The comment also reveals a subtle misunderstanding. The assistant writes "mk-build-deps needs the --build-dep flag or the control file directly" — but mk-build-deps does not accept a raw control file path as a positional argument in the way the assistant is using it. The tool expects either a source package name (which it resolves via apt-cache showsrc) or a .dsc file. Passing a debian/control file path is not a supported mode of operation. The assistant is essentially trying to use the tool in a way it was not designed for.

Why This Message Matters: The Pivot Point

Message [msg 8480] is the third consecutive failure. The pattern is now undeniable: three different invocations, three identical errors. The assistant has exhausted the "try different flags" strategy. Something fundamental must change.

What makes this message so instructive is what happens next. In msg 8481, the assistant tries adding a deb-src line to the Proxmox repository — but the Proxmox repo does not provide source packages, so it still fails. In msg 8482, the assistant finally reaches the correct diagnosis: "PVE does not have a source repo." It pivots to manually parsing the control file and installing dependencies with a raw apt-get install command. This works immediately (msg 8484), and the kernel build proceeds successfully (msg 8485-8486).

The entire detour — messages 8478 through 8481 — was caused by a single incorrect assumption: that mk-build-deps was the right tool for the job. The assistant spent four messages debugging a tool that was fundamentally unsuited to the task. The lesson is not about mk-build-deps syntax; it is about recognizing when a tool's assumptions about the world do not match reality.

Input Knowledge Required

To understand this message, the reader needs to know:

  1. The Debian packaging toolchain: mk-build-deps is part of the devscripts package. It resolves build dependencies by querying the APT source repository. It requires deb-src entries in sources.list because it uses apt-cache showsrc to find the source package corresponding to a binary package name.
  2. The PVE kernel distribution model: Proxmox VE kernels are not distributed as standard Debian source packages. They are maintained in a git repository on git.proxmox.com and built using a custom Makefile that produces .deb packages directly. There is no source package in any APT repository.
  3. The history of the session: The user had just rescued the system from a bricked state caused by toolchain hacks. The explicit mandate was "no hacks" — build everything cleanly from source with the native GCC. This creates a psychological pressure to use "proper" Debian tools rather than ad-hoc methods.
  4. The hardware context: The target machine has 8× Blackwell GPUs and a 14TB NVMe scratch pool. The kernel build is a prerequisite for compiling the NVIDIA open driver modules, which in turn are needed for ML training workloads. Every minute spent debugging mk-build-deps is a minute the GPUs sit idle.

Output Knowledge Created

This message produces negative knowledge — it teaches what does not work. Specifically:

  1. mk-build-deps cannot be used with the PVE kernel source tree. The tool fundamentally requires a source package in an APT repository, and the PVE kernel does not have one. This is not a configuration issue; it is a structural incompatibility.
  2. The PVE kernel's debian/control file is not a standalone package descriptor. It is part of a build system that expects to be invoked through the project's Makefile (make deb), not through Debian's dpkg-buildpackage pipeline. The control file exists for metadata and dependency tracking, not as an entry point for mk-build-deps.
  3. The correct approach is to parse the control file manually and install dependencies with a direct apt-get install command, or to use the project's own mk-build-deps invocation from within its Makefile (if one exists). The assistant ultimately did the former, extracting the Build-Depends field with grep and sed and passing it to apt-get install -y --no-install-recommends.

Assumptions and Their Consequences

The assistant made several assumptions, each of which proved costly:

| Assumption | Reality | Consequence | |---|---|---| | mk-build-deps accepts a debian/control file path | It expects a source package name or .dsc file | Three failed attempts | | The PVE kernel has a source package in an APT repo | It is distributed only via git | Adding deb-src lines didn't help | | Different flag combinations would fix the error | The error was about tool-reality mismatch | Four messages wasted | | The error message pointed to the real problem | The real problem was conceptual, not configurational | Chasing the wrong fix |

The most significant assumption was the last one. The error message was technically accurate — mk-build-deps does need deb-src URIs — but it was a symptom, not the disease. The assistant treated the symptom (missing deb-src lines) rather than diagnosing the disease (the PVE kernel is not a source package). This is a classic debugging pitfall: the error message tells you what the tool needs, but not whether the tool is appropriate for the task.

The Engineering Lesson

Message [msg 8480] is a microcosm of the entire segment's theme. The user had just demanded "no hacks" — build everything cleanly with the correct toolchain. But "clean" does not mean "use standard Debian tools regardless of fit." The cleanest approach is the one that matches the actual structure of the project, not the one that uses the most "proper" tool.

The PVE kernel is a git-based project with a custom build system. The cleanest way to build it is to follow the project's own build instructions: install the dependencies listed in its control file (however you choose to extract them) and run make deb. Using mk-build-deps was an attempt to impose a Debian-packaging workflow on a project that doesn't fit that mold. It was, ironically, a kind of hack — a misuse of a tool that was never designed for this purpose.

The assistant's eventual pivot — manually parsing the control file and installing deps with apt-get — was not elegant, but it was correct. It acknowledged the actual structure of the project rather than trying to force it into a preconceived toolchain. This is the deeper meaning of "no hacks": not the absence of manual commands, but the alignment of method with reality.

Conclusion

Message [msg 8480] is a small failure that contains a large lesson. In three lines of bash and one cryptic error, it captures the tension between tool-driven workflows and project-specific realities. The assistant's debugging loop — try different flags, fail identically, repeat — is a pattern familiar to every engineer who has ever insisted on using a tool that was wrong for the job. The pivot that follows this message is not just a technical correction; it is a conceptual breakthrough. The cleanest build is not the one that uses the most canonical tool, but the one that most faithfully reflects the structure of the thing being built.