The Shim That Broke: Debugging a GLIBC Version Mismatch on Debian Bookworm

In the sprawling, multi-day effort to provision kpro6 — a new Proxmox host equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs — one message stands as a turning point where a clever workaround collided with the unforgiving reality of the Linux dynamic linker. Message [msg 8442] captures the assistant's attempt to resolve a GLIBC_2.38 version mismatch by rebuilding a compatibility shim library, a moment that crystallizes the difference between "it should work in theory" and "it works in practice."

The Context: A Kernel Built for a Different World

The story begins with the assistant's decision to install a community-built Proxmox VE kernel (version 6.19.5-2) from the jaminmc repository, rather than building the kernel from source. This kernel was compiled on Debian Trixie (testing), which ships with GCC 14 and GLIBC 2.38. The target machine, however, runs Debian Bookworm with GCC 12.2.0 and GLIBC 2.36. The kernel headers package — which contains the pre-built tool binaries needed to compile kernel modules — came with binaries linked against GLIBC 2.38 symbols that simply didn't exist on the Bookworm system.

Earlier in the session, the assistant had successfully rebuilt gendwarfksyms from source (see [msg 8424]), compiling it directly against Bookworm's libraries. But four other binaries — objtool, modpost, insert-sys-cert, and resolve_btfids — couldn't be rebuilt because the kernel headers package was a stripped version that lacked the full tools/build/ infrastructure ([msg 8430]). The assistant was stuck: it needed these binaries to compile the NVIDIA 595.71.05 open GPU kernel driver, but they demanded GLIBC 2.38.

The Shim Strategy: A Reasonable Starting Point

The assistant's initial approach was elegant in its simplicity. All four binaries needed only three functions — __isoc23_strtoul, __isoc23_strtoull, and __isoc23_strtol — which are C23 variants of the standard strtoul family. These functions differ only in their handling of whitespace and base detection; for the kernel build process, the standard versions would work identically. The assistant wrote a small C shim library that forwarded these calls to the existing GLIBC 2.36 functions ([msg 8439]).

The first attempt used LD_PRELOAD to inject the shim, but this failed because the dynamic linker checks version requirements (the VERNEED section) against libc.so.6 before loading any preloaded libraries. The version check is baked into the binary's ELF structure and cannot be circumvented by LD_PRELOAD alone.

Undeterred, the assistant tried binary-patching: using sed to replace the string "GLIBC_2.38" with "GLIBC_2.17" directly in the executable files ([msg 8440]). This is a known technique — both strings are exactly 10 bytes long, so the replacement doesn't corrupt the ELF structure. The idea was that GLIBC 2.17 is a base version present in virtually all modern glibc installations, so the version check would pass, and then LD_PRELOAD could supply the missing symbols.

But this also failed, and the assistant couldn't immediately see why.

The Subject Message: A Hypothesis Tested and Falsified

Message [msg 8442] opens with the assistant checking a critical fact: is GLIBC_2.17 actually available in Bookworm's libc? The previous message ([msg 8441]) had listed all available GLIBC version tags, and GLIBC_2.17 appeared in the output. So the assistant formulates a new hypothesis:

"GLIBC_2.17 IS available! Something else must be wrong. Let me check the strtoul symbol"

The reasoning here is sharp: the assistant knows that the binary patching changed the version tag from 2.38 to 2.17, and it knows that 2.17 exists in the system's libc. So why does the dynamic linker still reject the binary? The answer must lie in how the version tag interacts with the specific symbol.

The assistant checks what version tag strtoul itself is exported under in Bookworm's libc:

000000000004bab0  w   DF .text    0000000000000012  GLIBC_2.2.5 strtoul

strtoul is tagged as GLIBC_2.2.5, not GLIBC_2.17. This is a crucial detail. When the binary was patched to require __isoc23_strtoul@GLIBC_2.17, the dynamic linker doesn't just check whether GLIBC_2.17 exists as a version node in libc — it checks whether the specific symbol __isoc23_strtoul is available under that version tag. Since no such symbol exists under GLIBC_2.17 in Bookworm's libc, the link fails.

The assistant then attempts a different fix: rebuild the shim library itself to use the GLIBC_2.17 version tag instead of GLIBC_2.38. The idea is that if the shim exports __isoc23_strtoul under GLIBC_2.17, and the patched binary requires it under GLIBC_2.17, the dynamic linker will find a match.

This is where the message delivers its punchline. The build command uses a version script (glibc238.map) that declares:

GLIBC_2.17 {
    __isoc23_strtoul;
    __isoc23_strtoull;
    __isoc23_strtol;
};

But the C source file still contains .symver assembler directives that hardcode @GLIBC_2.38:

__asm__(".symver __isoc23_strtoul,__isoc23_strtoul@GLIBC_2.38");
__asm__(".symver __isoc23_strtoull,__isoc23_strtoull@GLIBC_2.38");
__asm__(".symver __isoc23_strtol,__isoc23_strtol@GLIBC_2.38");

The linker error is unambiguous:

/usr/bin/ld: /usr/local/lib/libglibc238_shim.so: version node not found for symbol __isoc23_strtol@GLIBC_2.38
/usr/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

The .symver directives still reference GLIBC_2.38, but the version script only declares GLIBC_2.17. The linker cannot find a version node matching GLIBC_2.38 in the version script, so it refuses to build the library. The shim approach has hit a dead end.

The Deeper Lesson: Understanding the Dynamic Linker's Versioning Model

This message reveals a subtle but important aspect of ELF symbol versioning that is easy to misunderstand. The GLIBC version tags in a binary's VERNEED section are not simple strings that the dynamic linker matches against a list of available versions. Each version requirement is a triple: (library soname, version tag, symbol name). The linker must find all three matching simultaneously in the loaded libraries.

When the assistant patched "GLIBC_2.38" to "GLIBC_2.17" in the binary, it changed the version tag but left the symbol name (__isoc23_strtoul) and the library reference (libc.so.6) intact. The dynamic linker then looked for __isoc23_strtoul@GLIBC_2.17 in libc.so.6 — and found nothing, because libc.so.6 doesn't export any symbol under the GLIBC_2.17 tag that isn't actually part of the GLIBC_2.17 release.

The attempted fix — rebuilding the shim with GLIBC_2.17 — failed because the .symver directives in the source code were inconsistent with the version script. But even if that inconsistency had been fixed, the approach still wouldn't have worked: the patched binary's DT_NEEDED entry points to libc.so.6, and the dynamic linker resolves version requirements against DT_NEEDED libraries, not against LD_PRELOAD libraries. The shim would need to either replace libc.so.6 entirely (impossible without breaking the system) or be injected as a DT_NEEDED library with the right soname.

The Knowledge Required to Understand This Message

To fully grasp what's happening in [msg 8442], the reader needs:

  1. ELF basics: Understanding of shared libraries, DT_NEEDED entries, and how the dynamic linker resolves symbols at runtime.
  2. GLIBC symbol versioning: The concept that GLIBC exports symbols under version tags (like GLIBC_2.2.5, GLIBC_2.36, GLIBC_2.38), and binaries can record which version of each symbol they require in their VERNEED section.
  3. The .symver assembler directive: This GCC/assembler feature allows a C source file to explicitly bind a symbol to a specific version tag, which is how glibc itself manages backward compatibility.
  4. The patchelf and sed binary patching technique: The idea that you can change a string in-place in an ELF binary as long as the replacement is the same length, without corrupting the file structure.
  5. The kernel module build process: Understanding that the NVIDIA .run installer invokes the kernel build system, which uses tools like objtool, modpost, and resolve_btfids from the kernel headers directory.
  6. The Debian version landscape: Knowing that Debian Bookworm ships GLIBC 2.36, while Debian Trixie ships GLIBC 2.38, and that mixing binaries compiled on different distributions can cause exactly this kind of version mismatch.

The Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The shim approach is definitively blocked for this particular configuration. The combination of .symver directives and version scripts creates an inconsistency that the linker rejects.
  2. The specific error message ("version node not found for symbol __isoc23_strtol@GLIBC_2.38") provides a clear diagnostic: the .symver directive and the version script disagree on which version tag to use.
  3. Confirmation that strtoul is available under GLIBC_2.2.5 in Bookworm's libc, which rules out the possibility that the underlying function is missing entirely.
  4. The binary patching + LD_PRELOAD strategy has a fundamental flaw: the version requirement is tied to a specific library soname (libc.so.6), and LD_PRELOAD cannot override version checks against DT_NEEDED libraries.

What Follows: The Pivot to a New Strategy

After this message, the assistant abandons the shim approach entirely. In subsequent messages ([msg 8443] onward), it tries progressively more radical solutions: creating a library with soname=libc.so.6 to intercept the version check, downloading GLIBC 2.38 from Debian Trixie into a private directory, and using patchelf to redirect the binaries to use the Trixie dynamic linker and libraries. This last approach eventually succeeds ([msg 8449]), allowing the NVIDIA driver to compile.

But the shim approach's failure in [msg 8442] is a critical learning moment. It demonstrates that ELF symbol versioning is not a simple string-matching exercise — it's a structured system where version tags, symbol names, and library identities form an inseparable triple. Patching one element without adjusting the others creates an inconsistency that the dynamic linker cannot resolve.

Engineering Lessons

The message illustrates several important engineering principles:

First, understand the abstraction before working around it. The dynamic linker's version checking may seem like a simple "does this version exist?" query, but it's actually a three-way join across library identity, version node, and symbol name. A workaround that only addresses one dimension will fail.

Second, when a workaround fails, verify your assumptions about why it should work. The assistant correctly checked whether GLIBC_2.17 existed in libc, but didn't check whether the specific symbol was available under that version. The assumption that "version exists → version check passes" was subtly wrong.

Third, source-based builds are more reliable than binary patching. The assistant's initial approach of rebuilding gendwarfksyms from source worked perfectly ([msg 8424]). The tools that couldn't be rebuilt from source — because the headers package was stripped — required increasingly fragile workarounds. The eventual solution (downloading a compatible glibc) was itself a form of binary patching, and it worked, but it added complexity that a source build would have avoided.

Fourth, recognize when a strategy has hit diminishing returns. The assistant spent several messages iterating on the shim approach, trying different version tags and different injection methods. Message [msg 8442] is where the approach definitively fails, and the assistant pivots. Recognizing that pivot point is valuable for anyone debugging similar issues.

Conclusion

Message [msg 8442] is a small but instructive moment in a much larger system provisioning effort. It captures the moment when a clever workaround — building a compatibility shim for three missing C23 functions — runs into the hard constraints of the ELF dynamic linking model. The shim itself was correct; the version script was correct; the binary patching was correct. But the interaction between them, mediated by the .symver directives and the VERNEED section, created a contradiction that no amount of incremental fixing could resolve.

For the reader, this message offers a window into real-world systems debugging: the iterative hypothesis testing, the careful verification of assumptions, and the moment of recognizing that a particular approach has hit a dead end. It's a reminder that in systems engineering, "should work" is never a substitute for "does work," and that the dynamic linker — that ancient, battle-tested piece of infrastructure — has its own unforgiving logic that must be respected.