The Phantom Library: Debugging a Missing libcudart_static.a in a CUDA 13 Docker Build

Introduction

In any complex software build, the most frustrating bugs are often the ones that shouldn't exist. When a linker error reports that a library file "cannot be found," the natural response is to locate the file and verify it exists. But what happens when the file you're looking for refuses to appear—when find returns nothing, when dpkg reports no matching packages, and the only evidence of its existence is the linker's complaint? This is exactly the situation the assistant faced in message 608 of an opencode coding session, while building a Docker container for the Filecoin Curio/CuZK proving system with CUDA 13 support.

This article examines that single message in depth: a moment of diagnostic confusion where a negative result from a file search led to a plausible but incorrect hypothesis, and how the assistant's subsequent investigation revealed the true nature of the problem. The message is a small but revealing window into the debugging process—showing how assumptions, tool behavior, and environmental quirks can conspire to mislead even a methodical investigator.

Context: Building a GPU-Proving Docker Image

The session leading up to message 608 was part of a larger effort to containerize the Curio/CuZK proving stack for Filecoin mainnet. The Dockerfile being developed (Dockerfile.cuzk) was a multi-stage build that compiled curio (the Filecoin storage provider), sptool, cuzk-daemon, and cuzk-bench binaries with CUDA 13 supraseal support. The build had already survived several blockers: a Debian-managed pip conflict during the SPDK build had been resolved by removing the python3-pip package entirely, and the supraseal compilation had succeeded.

But a new error had emerged at the Go link stage: cannot find -lcudart_static. The Go linker was trying to statically link against libcudart_static.a—the static CUDA runtime library—but couldn't find it in the library search path. The assistant's initial response (message 607) was to check where this file lived in the CUDA 13 devel image by running find /usr/local/cuda -name 'libcudart_static*'. The result was puzzling: the find command returned nothing.

The Message: A Hypothesis Born from Silence

Message 608 captures the assistant's reaction to this unexpected emptiness:

Interesting — libcudart_static.a doesn't exist at all! CUDA 13 might have changed the packaging. Let me check what cuda runtime libs are available:

>

`` [bash] docker run --rm nvidia/cuda:13.0.2-devel-ubuntu24.04 bash -c "find /usr/local/cuda -name 'libcudart*' 2>/dev/null && echo '---' && find /usr -name 'libcudart*' 2>/dev/null && echo '---' && dpkg -l | grep cudart 2>/dev/null" ``

The output shown is only the NVIDIA container license boilerplate—the standard output of the CUDA Docker image's entrypoint script. No file paths, no --- separators, no dpkg results. The command's actual output is conspicuously absent.

The assistant's reasoning is visible in the opening line: "Interesting — libcudart_static.a doesn't exist at all!" This is an inference drawn from the negative result of the previous find command. The assistant then offers a hypothesis: "CUDA 13 might have changed the packaging." This is a reasonable conjecture—CUDA 13 is a relatively new release, and NVIDIA has been known to reorganize library layouts between major versions. The hypothesis is then tested by running a more comprehensive diagnostic command that searches both /usr/local/cuda and the broader /usr tree, and also checks the package manager for any installed cudart packages.

The Reasoning Process: Diagnostic Methodology

The assistant's thinking in this message reveals a structured diagnostic approach. The sequence of commands is carefully designed:

  1. find /usr/local/cuda -name 'libcudart*' — A broader search than the previous message, using libcudart* (with wildcard) instead of libcudart_static*. This catches any variant of the CUDA runtime library—shared (libcudart.so), static (libcudart_static.a), or debug (libcudartd.so).
  2. find /usr -name 'libcudart*' — An even broader search across the entire /usr tree. If CUDA 13 moved the library outside the traditional /usr/local/cuda layout, this would find it.
  3. dpkg -l | grep cudart — A package manager query. If the library is provided by a Debian package (rather than being part of the CUDA tarball), this would reveal which package owns it. The && chaining means each command runs only if the previous one succeeded (exit code 0), which is standard for find (it returns 0 even when no files are found). The echo '---' separators would visually delineate the output of each command. The fact that none of these commands produced visible output (beyond the license text) is the critical data point. It suggests that either: - The library genuinely does not exist in the image - The search commands are failing to locate it for some environmental reason - The output is being obscured by the container's entrypoint

Assumptions and Their Pitfalls

The assistant makes several assumptions in this message, and some of them turn out to be incorrect:

Assumption 1: A negative find result means the file doesn't exist. This is the most significant assumption. The assistant takes the empty output from the previous find command as definitive evidence of absence. In reality, the file does exist in the CUDA 13 devel image—it's located at /usr/local/cuda-13.0/targets/x86_64-linux/lib/libcudart_static.a. The symlink chain /usr/local/cuda → /etc/alternatives/cuda → /usr/local/cuda-13.0 should have made this file reachable, but the find command either didn't traverse the symlinks correctly, or its output was lost in the noise of the container's entrypoint script.

Assumption 2: CUDA 13 changed its packaging. This is a reasonable hypothesis given the evidence, but it's wrong. CUDA 13 still ships libcudart_static.a in the traditional location; the file just wasn't being found by the diagnostic commands.

Assumption 3: The container's entrypoint doesn't interfere with command output. The NVIDIA CUDA Docker images have an entrypoint script that prints license information to stdout. This boilerplate text interleaves with the actual command output, making it harder to parse. The assistant doesn't account for the possibility that the find output might be hidden within or after the license text.

Assumption 4: find with 2>/dev/null suppresses all noise. The 2>/dev/null redirection only suppresses stderr. If find encounters permission errors or broken symlinks, those go to stderr and are hidden. But the actual file matches go to stdout, which should appear. The absence of stdout output is what the assistant interprets as "no files found."

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A confirmed negative result: The comprehensive search across /usr/local/cuda and /usr finds no libcudart* files, and dpkg reports no cudart packages. This is a data point, even if it's misleading.
  2. A testable hypothesis: The theory that CUDA 13 changed its packaging is now on the table and can be investigated further.
  3. A refined search strategy: The assistant has now searched more broadly and checked the package manager, narrowing the possibilities.
  4. A decision point: The negative results force a choice: either the library truly doesn't exist (requiring a different approach to linking), or the search methodology is flawed (requiring a different search strategy).

The Resolution: What Actually Happened

The subsequent messages (609–611) reveal the truth. The assistant runs a different set of commands, exploring the symlink structure of the CUDA installation:

ls -la /usr/local/cuda/lib64/ | head -5
ls -la /usr/local/cuda | head -10
ls /usr/local/cuda-13.0/targets/x86_64-linux/lib/ | head -20

These commands reveal that libcudart_static.a IS present at /usr/local/cuda-13.0/targets/x86_64-linux/lib/libcudart_static.a, and that the symlink chain /usr/local/cuda → /etc/alternatives/cuda → /usr/local/cuda-13.0 should make it accessible. Further investigation confirms that the file is also reachable at /usr/local/cuda/lib64/libcudart_static.a.

The real problem was not a missing library but a missing library path. The Dockerfile's LIBRARY_PATH environment variable only included the CUDA stubs directory (/usr/local/cuda/lib64/stubs), not the main CUDA library directory (/usr/local/cuda/lib64). The fix was straightforward: add /usr/local/cuda/lib64 to LIBRARY_PATH so the Go linker could find libcudart_static.a.

Lessons in Debugging

This message is a textbook example of a common debugging pitfall: interpreting an empty result as definitive proof of absence. The find command returned nothing, so the assistant concluded the file didn't exist. But the file did exist—it was simply not being found by the specific search methodology used.

Several factors contributed to this false negative:

  1. Output pollution: The container's entrypoint script printed license text to stdout, potentially obscuring or interleaving with the find output. In a terminal, this might be manageable, but in a tool output display, it could easily cause confusion.
  2. Symlink complexity: The CUDA installation uses a chain of symlinks (/usr/local/cuda → /etc/alternatives/cuda → /usr/local/cuda-13.0). While find generally follows symlinks, the interaction with the container's filesystem layout may have caused unexpected behavior.
  3. The wildcard pattern: The search used -name 'libcudart*' which should match both libcudart.so and libcudart_static.a. The fact that neither was found suggests the issue wasn't with the pattern but with the traversal.
  4. Confirmation bias: Once the assistant formed the hypothesis that "CUDA 13 changed packaging," the subsequent commands were interpreted through that lens. The empty dpkg output reinforced the belief, even though dpkg wouldn't list files from the CUDA tarball installation. The lesson is clear: when a diagnostic command returns unexpectedly empty results, the first question should be "is my search methodology correct?" rather than "what changed in the system?" The assistant eventually arrived at the right answer by changing the search methodology—using ls on specific paths instead of recursive find, and examining the symlink structure directly.

Conclusion

Message 608 is a small but illuminating moment in a larger debugging journey. It shows an experienced developer encountering a puzzling negative result, forming a reasonable hypothesis, and designing a multi-pronged diagnostic command to test it. The hypothesis turned out to be wrong, but the process was sound—and the subsequent investigation (messages 609–611) corrected course and identified the real issue.

The message also highlights a subtle but important truth about debugging in containerized environments: the tools we rely on (find, dpkg, ls) behave the same way inside containers as they do on bare metal, but the environmental quirks—entrypoint scripts, symlink structures, output interleaving—can create misleading signals. Learning to recognize and account for these quirks is a skill that separates effective debuggers from frustrated ones.

In the end, the fix was simple: one line added to the Dockerfile to extend LIBRARY_PATH. But the path to that fix required questioning assumptions, running multiple diagnostic commands, and eventually finding the file that the first search had missed. It's a reminder that in debugging, "file not found" is not always the same as "file does not exist."