The Poisoned Library: How a GLIBC Shim Bricked a Proxmox Server

Introduction

In any complex systems engineering effort, there comes a moment where a clever workaround transforms into a catastrophic failure. Message 8459 of this opencode session captures precisely such a moment: the instant when an AI assistant realizes that a library it created to solve a specific build problem has silently poisoned the entire operating system, rendering the machine unreachable. The message is brief — a single SSH command and its cascading error output — but it represents the climax of a multi-hour debugging saga involving kernel compilation, NVIDIA driver installation, and the delicate machinery of the Linux dynamic linker.

The Scene: A System in Crisis

The subject message reads in full:

[assistant] The libc_238_compat.so with soname=libc.so.6 in /usr/local/lib is poisoning the system! Let me fix that immediately:

>

``` ssh -o ConnectTimeout=10 root@10.1.2.6 ' # Remove the dangerous shim libraries from /usr/local/lib rm -f /usr/local/lib/libc_238_compat.so /usr/local/lib/libglibc238_shim.so /usr/local/lib/libglibc238_shim.so.bak rm -f /etc/ld.so.conf.d/glibc238-shim.conf ldconfig echo "Fixed ldconfig"

>

# Re-run systemctl daemon-reload systemctl daemon-reload 2>&1 echo "systemctl: $?"

>

# Verify system is healthy sh -c "echo shell works" ' 2>&1 ```

>

Output: `` bash: /usr/local/lib/libc.so.6: version GLIBC_2.25' not found (required by bash) bash: /usr/local/lib/libc.so.6: version GLIBC_2.11' not found (required by bash) bash: /usr/local/lib/libc.so.6: version GLIBC_2.14' not found (required by bash) bash: /usr/local/lib/libc.so.6: version GLIBC_2.8' not found (required by bash) bash: /usr/local/lib/libc.so.6: version GLIBC_2.33' not found (required by bash) bash: /usr/local/lib/libc.so.6: version GLIBC_2.15' not found (required by bash) bash: /us... ``

The output is truncated, but the pattern is devastatingly clear: every GLIBC version symbol that bash requires — from GLIBC_2.8 through GLIBC_2.33 — is reported as "not found." The SSH connection itself cannot establish a shell because bash, the very first process that SSH spawns on the remote machine, crashes during its dynamic linking phase. The machine is effectively bricked for remote administration.

How We Got Here: The GLIBC_2.38 Problem

To understand why message 8459 happens, we must trace back through the preceding hour of work. The assistant was provisioning a new Proxmox host called kpro6, equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The goal was to install NVIDIA's 595.71.05 open-source driver on a custom kernel.

The assistant had opted to use a community-built Proxmox VE kernel (version 6.19.5-2-pve from the jaminmc repository) rather than building from source. This kernel's build tools — specifically objtool, modpost, insert-sys-cert, resolve_btfids, and gendwarfksyms — had been compiled on Debian Trixie (testing) against GLIBC 2.38. The target machine, however, ran Debian Bookworm with GLIBC 2.36. When the NVIDIA .run installer invoked these tools during kernel module compilation, they failed immediately with:

/usr/src/linux-headers-6.19.5-2-pve/tools/objtool/objtool: 
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found

This is a classic toolchain mismatch: binaries compiled against a newer version of glibc cannot run on an older glibc because the required version symbols (encoded in the ELF VERNEED section) don't exist in the older library.

The Shim Library Strategy

The assistant embarked on an increasingly elaborate series of workarounds ([msg 8440] through [msg 8444]). First, it tried hex-editing the binaries to change GLIBC_2.38 to GLIBC_2.17, but this failed because the specific symbols (__isoc23_strtoul, __isoc23_strtoull, __isoc23_strtol) didn't exist under the GLIBC_2.17 version tag. Then it tried creating a shim library that provided these symbols under the GLIBC_2.38 version tag, but the dynamic linker refused to use it because the version requirement was hard-bound to libc.so.6 specifically.

The breakthrough came in [msg 8444]: the assistant created a library called libc_238_compat.so with the same soname as the real libc (soname=libc.so.6). This library provided only the three missing __isoc23_* symbols under the GLIBC_2.38 version tag. When loaded via LD_PRELOAD, it satisfied the dynamic linker's version check for those specific symbols while the real libc.so.6 continued to provide everything else. The kernel build tools worked, and the NVIDIA driver compiled successfully.

But the assistant made a critical error: it placed this library in /usr/local/lib/ and ran ldconfig, which indexed it. Because the library's soname was libc.so.6, and /usr/local/lib appears in the default library search path before /lib/x86_64-linux-gnu, the dynamic linker began resolving all binaries' libc references to this stub library instead of the real one.## The Moment of Realization

Message 8459 is the moment of dawning horror. The assistant has just completed a successful NVIDIA driver build ([msg 8457]) and is running the userspace installer ([msg 8458]). It then realizes: "The libc_238_compat.so with soname=libc.so.6 in /usr/local/lib is poisoning the system!" The word "poisoning" is apt — like a chemical contaminant introduced for a specific purpose that then diffuses through the entire ecosystem, the fake libc has been cached by ldconfig and now shadows the real libc for every process on the system.

The assistant's attempted fix is logical but tragically too late: remove the dangerous files, update the linker cache, and verify the system is healthy. The command sequence is:

  1. rm -f the shim libraries and their configuration file
  2. Run ldconfig to rebuild the linker cache
  3. Run systemctl daemon-reload to restart system services
  4. Verify with sh -c "echo shell works" But step 4 never executes. The SSH connection itself fails because the remote shell (bash) cannot load. The error messages tell the story: bash requires dozens of GLIBC version symbols — GLIBC_2.25, GLIBC_2.11, GLIBC_2.14, GLIBC_2.8, GLIBC_2.33, GLIBC_2.15, and many more — and the stub library provides none of them. Every binary on the system is now broken.

The Mechanism of Failure

This failure is a masterclass in how the Linux dynamic linker works — and how fragile its assumptions can be. When a dynamically linked ELF binary starts, the kernel loads the dynamic linker (ld-linux-x86-64.so.2), which then:

  1. Reads the binary's DT_NEEDED entries to find required shared libraries
  2. Searches the library path (including ldconfig cache) for each needed library by soname
  3. For each library found, checks that all version requirements in the binary's VERNEED section are satisfied by the library's VERDEF section
  4. Resolves all symbol references The shim library had soname=libc.so.6, so the dynamic linker found it when searching for libc.so.6. Since /usr/local/lib typically precedes /lib/x86_64-linux-gnu in the search order, the shim was selected over the real libc. The shim defined only three symbols under GLIBC_2.38 — but bash (and every other binary) required dozens of symbols across dozens of version tags. The result: instant failure for every dynamically linked program.

Why This Is Hard to Fix Remotely

The tragedy of this situation is that the fix is trivial with physical access: just delete the file. But from SSH, the assistant cannot execute any command because SSH's first action on the remote machine is to spawn a shell, and the shell cannot load. This creates a catch-22: you need a working shell to delete the file, but you need to delete the file to have a working shell.

The assistant's subsequent attempts in [msg 8460] through [msg 8462] show escalating desperation. It tries using the absolute path to the real dynamic linker (/lib64/ld-linux-x86-64.so.2) to invoke rm directly, bypassing bash. But SSH always spawns a login shell first, which fails. It tries exec tricks and forced commands, but the SSH protocol itself requires the remote shell to start. The system is effectively bricked for remote access.

The Broader Lesson: Toolchain Consistency

This entire episode stems from a single root cause: the assistant chose to use a community kernel (jaminmc's 6.19.5-2-pve) that was compiled with a different toolchain than the host system. The kernel headers contained pre-built binaries (objtool, modpost, etc.) that were linked against GLIBC 2.38 from Debian Trixie, while the host ran Debian Bookworm with GLIBC 2.36.

The assistant's attempts to patch around this incompatibility — hex-editing version strings, creating shim libraries, using patchelf to redirect library paths — were all fundamentally hacks that addressed symptoms rather than the root cause. Each workaround introduced new complications, culminating in the system-wide poisoning.

The correct approach, which the assistant eventually adopted after this crisis (see [chunk 49.0]), was to build the kernel and all its tools from source using the host's native GCC 12.2.0 toolchain. This produced binaries that were perfectly compatible with the system's GLIBC 2.36, required no patches or shims, and resulted in a clean, stable system.

Input and Output Knowledge

To fully understand message 8459, one needs knowledge of:

The Thinking Process Visible in the Message

The assistant's reasoning in message 8459 reveals several assumptions:

  1. "The libc_238_compat.so ... is poisoning the system!" — The assistant correctly identifies the root cause. The shim library's soname matches the real libc, so it shadows the real libc for all processes.
  2. "Let me fix that immediately" — The assistant assumes it can still SSH into the machine to remove the files. This assumption is incorrect: the poisoning is already systemic.
  3. The command sequence — The assistant tries to remove the shim files, update ldconfig, restart system services, and verify health. This is the correct fix if it could be executed. But the fix requires a working shell, which the poisoning has destroyed.
  4. The truncated output — The error messages repeat for every GLIBC version that bash requires. The assistant sees the pattern but cannot act on it because the SSH session itself has failed. There is also a subtle mistake in the assistant's reasoning: it assumes the problem is only the files it explicitly created (libc_238_compat.so, libglibc238_shim.so). In [msg 8462], it discovers that the NVIDIA .run installer also placed a libc.so.6 in /usr/local/lib — a separate copy that further compounds the issue. The assistant's rm command in message 8459 would have missed this file, meaning even if the SSH command had succeeded, the poisoning might have persisted.

Conclusion

Message 8459 is a cautionary tale about the dangers of working around toolchain incompatibilities. The assistant's shim library was a clever solution to a specific problem — getting kernel build tools compiled against GLIBC 2.38 to run on a GLIBC 2.36 system — but it violated a fundamental invariant of the operating system: there must be exactly one libc.so.6, and it must be the real glibc. By creating a library with the same soname, the assistant inadvertently replaced the system's C library with a stub that provided almost none of the symbols that every program depends on.

The message also illustrates the brittleness of remote administration. A single bad file in /usr/local/lib can render a machine unreachable, requiring physical intervention or a rescue ISO to fix. The assistant's subsequent recovery — using a live ISO to delete the file — is described in [msg 8465] and following messages, but message 8459 stands as the moment of crisis, the point where all the accumulated technical debt of the workaround approach came due at once.

In the end, the lesson is clear: when building system software, use a consistent toolchain from source. Hacks that patch around binary incompatibilities may work temporarily, but they create hidden dependencies that can collapse catastrophically. The clean approach — building the kernel and NVIDIA driver from source with the native GCC — succeeded where all the workarounds failed, precisely because it maintained the integrity of the system's fundamental interfaces.