The Build Dependency Trap: When mk-build-deps Meets a Missing deb-src

In the aftermath of a catastrophic system failure—where a glibc compatibility shim had poisoned the dynamic linker, bricking SSH access and requiring a physical rescue from a live ISO—a single assistant message captures the fragile transition from disaster recovery to disciplined engineering. The message, numbered 8478 in the conversation, is deceptively small: a bash command that attempts to install build dependencies for a custom Proxmox VE kernel, followed by an error. But this message sits at a critical inflection point where the assistant must prove it has learned the lesson that the user just spent hours teaching: no more hacks, no more binary patching, no more mixing toolchains. Everything must be built from source, with native tools, using the correct compiler.

The Message

The assistant executes the following command on the freshly rescued remote host:

ssh -o ConnectTimeout=10 root@10.1.2.6 '
# Install missing dh-python
DEBIAN_FRONTEND=noninteractive apt-get install -y dh-python python3-sphinx sphinx-common 2>&1 | tail -5
echo "---"
# Now install build deps properly
cd /scratch/pve-kernel
mk-build-deps -ir proxmox-kernel-6.14.11/debian/control 2>&1 | tail -20
' 2>&1

The output reveals that the first command succeeded—dh-python and the Sphinx documentation tools were installed—but the second command failed with a classic Debian packaging error:

E: You must put some 'deb-src' URIs in your sources.list
mk-build-deps: Unable to find package name in `apt-cache showsrc proxmox-kernel-6.14.11/debian/control'

Why This Message Was Written: The Full Context

To understand why this message exists, one must trace the narrative arc of the preceding hours. The assistant had been tasked with provisioning a new Proxmox host named kpro6, equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs and a 14TB NVMe scratch drive. The initial approach was to install a community-built 6.19 kernel (from the jaminmc repository) and then run NVIDIA's proprietary .run installer for the 595.71.05 driver. This quickly spiraled into a debugging nightmare.

The community kernel had been compiled on Debian Trixie with GCC 14, while the host ran Bookworm with GCC 12. This toolchain mismatch created cascading incompatibilities. The assistant attempted to patch around them—rebuilding gendwarfksyms, recompiling objtool, and ultimately creating a GLIBC_2.38 compatibility shim library. That shim, placed in /usr/local/lib/ with the soname libc.so.6, poisoned the system's dynamic linker. Every binary that loaded libc.so.6 from the standard search path picked up the shim instead of the real glibc, and since the shim only provided a subset of symbols, commands like bash and ssh immediately failed with version mismatch errors. The system was bricked for remote access.

The user rescued the machine via physical console (booting from a Proxmox VE ISO, importing the ZFS pool, and deleting the offending file). Upon recovery, the user delivered a clear directive in message 8469:

"proxmox back up; try not to do hacks; open nvidia can probably just be built locally without hacky repos, same with the kernel, all using correct gcc?"

This was not merely a suggestion—it was an engineering mandate born from painful experience. The assistant acknowledged the lesson in message 8470: "You're right. The whole mess came from trying to use a community kernel built on trixie and then hacking around glibc/gcc mismatches."

Message 8478 is the assistant's first concrete attempt to execute this new philosophy. It has already cleaned up the system (removing the jaminmc kernel, purging NVIDIA .run leftovers, deleting the shim libraries), researched the proper PVE kernel build process, cloned the official Proxmox kernel repository (git://git.proxmox.com/git/pve-kernel.git on the bookworm-6.14 branch), and successfully run make build-dir-fresh to prepare the source tree. Now it needs to install the build dependencies before it can compile the kernel. This message is the dependency-resolution step.## Assumptions Made and Their Consequences

The assistant made several assumptions in this message, and understanding them reveals the thinking process at work.

Assumption 1: mk-build-deps would work with the current apt configuration. The tool mk-build-deps is a Debian utility that reads a debian/control file, resolves the build dependencies listed there, and installs them automatically. It works by calling apt-cache showsrc to find source package entries. But this requires deb-src lines in /etc/apt/sources.list—the source package repositories. The assistant had not added these, and the standard Proxmox installation only configures binary (deb) repositories. The error message "You must put some 'deb-src' URIs in your sources.list" is a well-known Debian packaging pitfall.

Assumption 2: The build dependencies could be installed automatically without manual inspection of the control file. The assistant opted for the automated mk-build-deps -ir approach rather than reading the debian/control file and manually installing the required packages. This was a reasonable choice for efficiency—the kernel has dozens of build dependencies—but it introduced a dependency on the apt source configuration being correct.

Assumption 3: The previous step (installing dh-python and sphinx) was sufficient to make mk-build-deps succeed. The assistant correctly identified that the earlier make build-dir-fresh had failed due to missing dh-python (a required helper for Debian packaging). It installed that package plus Python Sphinx documentation tools. But it did not verify that the apt sources were configured for source packages before running mk-build-deps.

Mistakes and Incorrect Assumptions

The primary mistake was not checking the apt sources configuration before invoking mk-build-deps. The assistant could have run a simple verification:

grep -r 'deb-src' /etc/apt/sources.list /etc/apt/sources.list.d/

If this returned nothing, the fix would be trivial—either add deb-src lines matching the existing deb lines, or manually install the build dependencies by reading the control file.

A secondary oversight: the assistant used tail -20 to capture the output of mk-build-deps, which truncated the full error. The complete error output might have included the list of missing packages, which would have been useful for a manual fallback approach.

However, it is important to recognize what this message gets right. The assistant is following the user's mandate: it is building from source using the official Proxmox repository, on the correct branch (bookworm-6.14), with the system's native GCC 12.2.0. It is not downloading pre-built binaries from community repositories. It is not patching toolchain mismatches. The approach is architecturally sound even if the execution stumbles on a configuration detail.

Input Knowledge Required

To understand this message, one needs familiarity with several domains:

Output Knowledge Created

This message produces two outputs:

  1. Negative knowledge: The mk-build-deps approach fails on a system without deb-src configured. This is a useful diagnostic—it tells the assistant (and the reader) that the apt sources need modification before automated dependency resolution can work.
  2. Partial progress: The dh-python and python3-sphinx packages are now installed, which were indeed missing from the earlier build attempt. This is progress even though the overall command failed. The message also implicitly documents the correct next step: either add deb-src repositories and retry, or manually inspect the debian/control file and install the dependencies by hand. The assistant will need to choose one of these paths in the subsequent message.## The Thinking Process: What the Assistant's Reasoning Reveals The message contains two commented lines that serve as the assistant's reasoning notes:
# Install missing dh-python
# Now install build deps properly

These comments reveal a two-step mental model. First, the assistant recalls that the previous make build-dir-fresh step (executed in message 8477) had failed or produced warnings related to missing dh-python. This package is a Debian utility that provides dh commands used in debian/rules files—it is essential for building any Debian source package. The assistant correctly identifies this as a prerequisite and installs it along with python3-sphinx (needed for kernel documentation generation).

Second, the assistant attempts to install the full set of build dependencies using mk-build-deps, which reads the debian/control file and resolves all Build-Depends entries. This is the standard Debian way to set up a build environment for a source package. The assistant's reasoning is sound: rather than manually parsing the control file and installing dozens of packages one by one, use the automated tool.

The thinking process is visible in the sequencing: the assistant deliberately splits the work into two commands within a single SSH session. The first command (apt-get install) is expected to succeed and is verified by the echo "---" separator. The second command (mk-build-deps) is the main task. This structure shows that the assistant anticipated potential failure in the first step and wanted clear output separation for debugging.

What is not visible in the reasoning is any check of the apt source configuration. The assistant did not include a verification step like:

grep -q 'deb-src' /etc/apt/sources.list || echo "WARNING: no deb-src configured"

This omission is the critical gap. The assistant's mental model assumed that mk-build-deps would work on a standard Debian system, but it did not account for the fact that Proxmox VE, being a specialized hypervisor distribution, may have a non-standard apt configuration. In fact, Proxmox intentionally keeps the source repositories disabled by default to minimize the system's footprint.

The Broader Engineering Lesson

This message, for all its apparent simplicity, encapsulates a profound lesson about build systems and infrastructure engineering. The assistant is attempting to follow a clean, principled approach: build everything from source using native tools. This is the correct strategy, validated by the catastrophic failure of the previous hack-based approach. But the transition from "using pre-built binaries from random repositories" to "building from source" introduces a new class of problems: build dependency management.

Building from source is conceptually simple—you run make or dpkg-buildpackage—but the practical reality is that you must have every single build dependency installed, at the correct version, with the correct configuration. A missing deb-src line in sources.list is a trivial obstacle, but it can halt an entire infrastructure provisioning workflow. The assistant's error is not a failure of engineering judgment but a failure of environmental awareness: it assumed a standard Debian configuration where one did not exist.

The message also demonstrates a pattern that recurs throughout infrastructure automation: the tension between automation and verification. The assistant chose automation (mk-build-deps -ir) over manual verification (inspecting debian/control and installing packages by hand). Automation is faster and less error-prone when it works, but it creates a single point of failure when the environment doesn't match expectations. A hybrid approach—verify the apt source configuration first, then automate—would have been more robust.

How This Message Connects to the Segment's Narrative

This message is the pivot point in segment 49's narrative arc. The segment begins with the assistant attempting quick fixes (community kernel, binary driver installer, glibc shim) and suffering a catastrophic system bricking. After the user's intervention and explicit directive, the assistant completely changes strategy. Message 8478 is the first step in the new, disciplined approach.

The irony is that this "clean" approach immediately hits a mundane configuration error. This is realistic engineering: even when you do everything right, you encounter small obstacles. The story does not end here—the assistant will go on to fix the deb-src issue, successfully build the kernel from source, compile the NVIDIA open driver against it, and achieve a pristine system with all 8 GPUs recognized. But message 8478 captures the moment of transition, the first attempt at the new approach, and the inevitable small stumble that precedes every successful infrastructure deployment.

The message also validates the user's insight. The user said "build locally with correct gcc"—and the assistant is now doing exactly that, using the official Proxmox kernel repository, the native GCC 12.2.0, and standard Debian packaging tools. The deb-src error is a configuration detail, not a fundamental flaw in the approach. It is far easier to fix than a bricked system.

Conclusion

Message 8478 is a study in contrasts: a small command with large context, a simple error with deep roots, a moment of failure that is actually a sign of progress. The assistant has internalized the lesson of the bricked system and is now building from source with native tools. The deb-src error is not a regression to the old ways—it is a standard build system configuration issue that any Debian developer would recognize. The message captures the messy reality of infrastructure engineering: even the right approach encounters obstacles, and the measure of engineering maturity is not the absence of errors but the ability to recognize, diagnose, and fix them without abandoning the principled approach.