The Missing Symlink: Diagnosing a CUDA Dynamic Linking Failure in a Multi-Stage Docker Build

Introduction

In the course of constructing a Docker container to bundle the Curio Go binary, the cuzk Rust/CUDA daemon, and all dependencies for mainnet 32GiB Filecoin proving, the assistant encountered a stubborn build failure. Message 582 captures the precise moment of diagnostic breakthrough—a short, almost terse observation that cuts through the noise of a failing compilation to identify the root cause. The message reads:

There's libcuda.so but no libcuda.so.1 symlink. The bellperson build script is dynamically linked and looks for libcuda.so.1 via the dynamic linker (not libcuda.so). I need to create the symlink. Let me add that to the Dockerfile.

This single message, comprising only three sentences and an edit action, represents the culmination of a multi-step debugging chain. It is a masterclass in how deep system knowledge—specifically, an understanding of Linux dynamic linking conventions, the CUDA toolkit's stub library structure, and the peculiarities of the bellperson build system—can resolve a cryptic build failure that might otherwise send a developer down countless dead ends.

The Broader Context: Containerizing a Heterogeneous Stack

To appreciate message 582, one must understand what the assistant was trying to accomplish. The project—a Filecoin Curio node with CUDA-accelerated proving via the cuzk engine—spans an extraordinary range of technologies: Go for the main node binary, Rust for the cuzk daemon, C++ for the Filecoin FFI (Foreign Function Interface), CUDA C++ for GPU kernels, and Python for build tooling (specifically the SPDK dependency setup). Containerizing such a stack is notoriously difficult because each layer has its own build-time dependencies, and the interactions between them are often undocumented.

The assistant had designed a two-stage Docker build (Dockerfile.cuzk). Stage 1 (builder) used nvidia/cuda:13.0.2-devel-ubuntu24.04 as its base image and was responsible for compiling everything: installing system packages (gcc-13, Python, nasm, libhwloc-dev, libssl-dev, libnuma-dev), installing Go 1.24.7 and Rust 1.86.0, running make deps to build the Filecoin FFI with CUDA and supraseal support, building the curio binary, and building the cuzk-daemon via Cargo. Stage 2 (runtime) used the slimmer nvidia/cuda:13.0.2-runtime-ubuntu24.04 image and only included runtime libraries and the compiled binaries.

The first build attempt (message 574) began promisingly. The metadata for both CUDA images was pulled, and the builder stage started executing. But as the build progressed, it hit a failure during the FFI compilation—specifically during the compilation of the neptune and bellperson Rust crates, which are cryptographic libraries that probe GPU capabilities at build time.

The Debugging Chain Leading to Message 582

The assistant's debugging process reveals a systematic narrowing of the problem space. In message 579, the assistant correctly identified that the neptune build script needed libcuda.so.1 at compile time to probe GPU capabilities. The CUDA devel image includes a stub libcuda.so in /usr/local/cuda/lib64/stubs/—this is a standard convention where NVIDIA provides stub libraries that allow code to link against CUDA at compile time without requiring a physical GPU or the full NVIDIA driver. The assistant's first fix was to add the stubs directory to LIBRARY_PATH and LD_LIBRARY_PATH during the build, reasoning that the linker would find libcuda.so there.

But this fix failed. Message 581 documents the assistant's realization: "The LD_LIBRARY_PATH approach isn't enough — the build script binary is linked at runtime and needs libcuda.so.1 specifically (not just libcuda.so)." The assistant then ran a Docker command to inspect the stubs directory directly:

docker run --rm nvidia/cuda:13.0.2-devel-ubuntu24.04 ls -la /usr/local/cuda/lib64/stubs/

This command pulled the devel image and listed the contents of the stubs directory. The output confirmed the suspicion.

The Content of Message 582: Diagnosis and Action

Message 582 is the assistant's response after seeing the stubs directory listing. It contains three distinct components, each serving a critical role:

The observation: "There's libcuda.so but no libcuda.so.1 symlink." This is the raw finding from the ls output. The stubs directory contains a file named libcuda.so but lacks the versioned symlink libcuda.so.1 that the dynamic linker expects.

The root cause analysis: "The bellperson build script is dynamically linked and looks for libcuda.so.1 via the dynamic linker (not libcuda.so)." This is the crucial insight. The assistant understands that Linux shared libraries use a versioned SONAME convention. When a program is linked against libcuda.so.1, the dynamic linker (ld.so) searches for that exact filename at runtime. The unversioned libcuda.so file is typically a symlink (or a linker script) used only during static linking with -lcuda. The bellperson build script produces a binary that is dynamically linked, meaning it records the SONAME libcuda.so.1 in its ELF headers and the dynamic linker resolves this at runtime. Simply having libcuda.so in LD_LIBRARY_PATH does not satisfy this requirement because the dynamic linker looks for the SONAME, not the generic name.

The action: "I need to create the symlink. Let me add that to the Dockerfile." The fix is straightforward once the root cause is understood: create a symlink from libcuda.so.1 to libcuda.so in the stubs directory so that the dynamic linker can resolve the SONAME during the build script's execution. The assistant then issues an edit command to modify the Dockerfile.

Assumptions and Mistakes

The debugging chain reveals several assumptions that were implicitly made and then corrected:

Assumption 1: The CUDA devel image's stubs directory would contain a complete set of symlinks. This was incorrect. The NVIDIA CUDA devel images include libcuda.so as a stub for linking purposes, but they do not include the libcuda.so.1 symlink because the stub is intended for compile-time linking (where -lcuda resolves to libcuda.so via the linker's -l flag logic), not for runtime dynamic linking where the SONAME is used.

Assumption 2: Setting LD_LIBRARY_PATH to the stubs directory would be sufficient for the build script. This was also incorrect. The assistant initially added the stubs directory to both LIBRARY_PATH (for the linker at compile time) and LD_LIBRARY_PATH (for the dynamic linker at runtime). But the dynamic linker resolves SONAMEs, not generic library names. The stub file is named libcuda.so, but the bellperson build script's binary expects libcuda.so.1. These are different strings from the dynamic linker's perspective.

Assumption 3: The build failure was in the linker step, not in the runtime execution of a build script. The assistant initially treated this as a compile-time linking issue (hence LIBRARY_PATH). But the actual failure was that the neptune/bellperson build scripts execute a binary that probes the GPU. That binary is dynamically linked against libcuda.so.1, and when the dynamic linker cannot find that SONAME, the binary fails to start, causing the build script to report an error. This distinction—between compile-time linking and runtime dynamic linking—is subtle but critical.

Input Knowledge Required

Understanding message 582 requires knowledge spanning several domains:

Linux dynamic linking conventions: The distinction between a library's filename (e.g., libcuda.so), its SONAME (e.g., libcuda.so.1), and the linker's -l flag resolution mechanism. The SONAME is embedded in the shared library's ELF headers and is what the dynamic linker uses at runtime. A program linked against libcuda.so.1 will fail to start if only libcuda.so is present, even if libcuda.so is a symlink to the same file.

CUDA toolkit structure: The NVIDIA CUDA devel images include stub libraries in /usr/local/cuda/lib64/stubs/. These stubs allow code to link against CUDA APIs at compile time without requiring the NVIDIA driver to be installed. They are not full implementations—they simply provide the symbol table entries needed for linking. The stubs directory typically contains libcuda.so (a linker script or symlink) but may not include versioned symlinks.

The bellperson/neptune build system: These Rust crates, part of the Filecoin proving stack, include build scripts that probe GPU capabilities by executing a small dynamically-linked binary. This binary links against libcuda.so.1 to call CUDA APIs for device detection. The build script runs this binary during compilation, and if the binary fails to execute (due to missing libraries), the build fails.

Docker build caching and layers: The assistant had to restart the Docker build multiple times after each Dockerfile edit. Understanding which layers are cached and which must be rebuilt is essential for efficient debugging in this context.

Output Knowledge Created

Message 582 produces both immediate and lasting knowledge:

Immediate fix: The edit to the Dockerfile adds a RUN ln -s ... command that creates the missing symlink. This is a one-line change, but it unblocks the entire build.

Diagnostic methodology: The message demonstrates a repeatable pattern for debugging dynamic linking failures in containerized builds: inspect the actual filesystem of the base image, check for SONAME mismatches, and understand whether the failure occurs at compile time or runtime.

Documentation of a CUDA devel image quirk: The fact that nvidia/cuda:13.0.2-devel-ubuntu24.04 lacks the libcuda.so.1 symlink in its stubs directory is now documented implicitly. This is not a bug in the NVIDIA image—the stubs are designed for compile-time linking, not runtime execution of dynamically-linked binaries—but it is a gotcha that anyone building CUDA-accelerated applications in Docker will encounter.

The Thinking Process

The assistant's reasoning in message 582 is concise but reveals a sophisticated mental model. The sequence is:

  1. Observe the discrepancy: The ls output shows libcuda.so but no libcuda.so.1. This is the raw data.
  2. Map to the failure mode: The bellperson build script produces a dynamically-linked binary. Dynamic linking uses SONAME resolution. The SONAME for the CUDA runtime library is libcuda.so.1. Therefore, the binary cannot start because the dynamic linker cannot find this exact filename.
  3. Distinguish from the previous attempt: The previous fix (adding to LD_LIBRARY_PATH) failed because LD_LIBRARY_PATH tells the dynamic linker where to search, but the dynamic linker is still searching for libcuda.so.1—which doesn't exist anywhere in the filesystem. Adding the stubs directory to the search path doesn't help if the target filename is absent.
  4. Formulate the minimal fix: Create a symlink. This is the simplest change that addresses the root cause. The symlink makes libcuda.so.1 point to libcuda.so, so the dynamic linker finds the SONAME it needs and loads the stub library.
  5. Execute: Edit the Dockerfile to add the symlink creation command. What is notable is what the assistant does not do. It does not try to install the full NVIDIA driver in the build container. It does not try to use a different base image. It does not try to modify the bellperson build script to use a different linking mode. It identifies the minimal, surgical fix that preserves the existing architecture.

Broader Significance

Message 582 exemplifies a class of debugging challenges that are increasingly common as software stacks grow more heterogeneous. When a single Docker build must compile Go, Rust, C++, CUDA, and Python components, the interactions between these ecosystems create failure modes that no single toolchain can diagnose. The Rust compiler reports a linker error, but the root cause is a missing symlink in a CUDA directory. The error message is misleading—it might say "cannot find -lcuda" or "error loading shared libraries"—but the actual problem is a versioned SONAME mismatch.

The assistant's ability to navigate this complexity stems from a deep understanding of how different layers of the system interact. The Go build system does not care about SONAMEs. The Rust build system, through its build scripts, does. The CUDA devel image provides stubs for one purpose (compile-time linking) but not for another (runtime execution of build scripts). The Docker build system caches layers, meaning that a failed build can leave partial state that influences subsequent attempts.

This message also illustrates the importance of looking at the actual filesystem rather than relying on documentation or assumptions. The assistant ran a docker run command to inspect the stubs directory directly, rather than assuming the contents based on general knowledge of CUDA toolkit structure. This empirical approach—checking what is actually there—is what allowed the diagnosis to proceed from "LD_LIBRARY_PATH should work" to "the file literally does not exist."

Conclusion

Message 582 is a small message with outsized significance. In three sentences, it captures the moment when a confusing, multi-layered build failure resolves into a single missing symlink. It demonstrates that containerizing a heterogeneous stack is not merely a matter of writing a Dockerfile and running a build—it requires deep system knowledge, iterative hypothesis testing, and the willingness to inspect the actual contents of base images rather than assuming they contain what one expects. The fix itself is trivial: one ln -s command. But the path to discovering that fix required understanding Linux dynamic linking, CUDA toolkit conventions, the bellperson build system's runtime behavior, and Docker's filesystem isolation—a combination of knowledge that is rare and valuable.