The Missing Symlink: Debugging a CUDA Docker Build at the Boundary of Toolchain and Runtime

In the intricate world of containerizing heterogeneous proving systems, the boundary between a successful build and a cryptic failure can be as thin as a missing symlink. Message [msg 581] captures a pivotal debugging moment in the construction of a Docker container for the Curio/cuzk mainnet proving stack — a system that weaves together Go, Rust, C++, CUDA, and Python build tools into a single deployable artifact. The message is brief, but it encapsulates a deeply technical insight about how CUDA's build-time stub libraries interact with runtime linker expectations, and it reveals the assistant's methodical, hypothesis-driven approach to diagnosing build failures in an environment it cannot directly inspect.

The Broader Context: Containerizing a Heterogeneous Proving Stack

To understand the significance of this single message, one must appreciate the broader effort underway. The assistant had been tasked with creating a Docker container (Dockerfile.cuzk) that bundles the Curio Go binary, the cuzk Rust/CUDA daemon, and all their dependencies for mainnet 32GiB proving. This was no ordinary containerization task. The Curio project depends on Filecoin FFI (C++/Rust bindings), supraseal (a CUDA-accelerated proving library), neptune (a GPU-accelerated Poseidon hash implementation), and bellperson (a GPU-accelerated SNARK proving library), among others. Each of these dependencies has its own build system, its own toolchain requirements, and its own peculiarities about what must be present at build time versus runtime.

The assistant had designed a multi-stage Docker build: a builder stage based on nvidia/cuda:13.0.2-devel-ubuntu24.04 that would compile everything, and a runtime stage based on nvidia/cuda:13.0.2-runtime-ubuntu24.04 that would contain only the minimal libraries needed to execute the binaries. This is a standard pattern for reducing final image size, but it places enormous pressure on getting the builder stage exactly right.

The Build Failure: Neptune's GPU Probe

The build had progressed through several stages successfully — Go installation, Rust toolchain setup, system dependencies — but had stalled during the compilation of the neptune crate, a GPU-accelerated Poseidon hash implementation used by the Filecoin proof system. The neptune build script, as the assistant discovered, probes for GPU capabilities at compile time by linking against libcuda and querying device properties. This is a common pattern in CUDA-accelerated Rust crates: they use a build script (a build.rs file) that runs as part of compilation to detect GPU features and generate optimized code paths.

The CUDA devel image includes a stubs directory at /usr/local/cuda/lib64/stubs/ that contains dummy libcuda.so libraries. These stubs provide the CUDA API symbols needed for linking without requiring an actual NVIDIA driver — they are designed specifically for build environments where no GPU is available. The assistant had attempted to make these stubs available by adding the stubs directory to LD_LIBRARY_PATH, a standard technique for influencing the dynamic linker at runtime. But the build continued to fail.

The Diagnostic Leap: From LD_LIBRARY_PATH to the Missing .so.1 Symlink

Message [msg 581] opens with the assistant's diagnosis:

Still failing. The LD_LIBRARY_PATH approach isn't enough — the build script binary is linked at runtime and needs libcuda.so.1 specifically (not just libcuda.so). The stubs directory has libcuda.so but may lack the .so.1 symlink. Let me check and create it.

This statement reveals a sophisticated understanding of how Linux shared library resolution works. On Linux, shared libraries follow a versioned naming convention: libcuda.so is typically a symlink to libcuda.so.1 (or libcuda.so.550 or whatever the driver version is). The .so file is the "linker name" used at compile time with -lcuda, while the .so.1 file is the "soname" used at runtime. When a binary is linked against libcuda, the linker embeds the soname (libcuda.so.1) into the binary's NEEDED entry. At runtime, the dynamic linker looks for libcuda.so.1, not libcuda.so.

The assistant's insight is that the build script's compiled binary — a temporary executable that neptune's build.rs produces to probe GPU capabilities — is linked at runtime (it's executed during the build), and therefore needs the soname libcuda.so.1 to resolve. Simply adding the stubs directory to LD_LIBRARY_PATH makes libcuda.so available for compile-time linking, but the runtime execution of the probe binary fails because libcuda.so.1 doesn't exist in the stubs directory.

This is a subtle distinction that would be easy to miss. Many developers would try adding the stubs directory to LD_LIBRARY_PATH and, if that failed, assume the approach was fundamentally wrong. The assistant correctly identified that the mechanism was right but the implementation was incomplete — the stubs directory had libcuda.so but was missing the libcuda.so.1 symlink that the runtime linker requires.

The Verification Step: Inspecting the Stubs Directory

The assistant then issues a command to verify this hypothesis:

[bash] docker run --rm nvidia/cuda:13.0.2-devel-ubuntu24.04 ls -la /usr/local/cuda/lib64/stubs/

This is a clean, isolated test. By running the base devel image directly (without any of the build context), the assistant can inspect the exact state of the stubs directory as it exists in the pristine image. This eliminates any possibility that the build process itself had modified or corrupted the directory. The --rm flag ensures no leftover containers, and the command is minimal — just listing the directory contents.

The message then shows Docker pulling the image layers. Notably, many layers are shown as "Already exists," indicating that this image (or its layers) had been partially pulled during the earlier build attempts. The output is cut off mid-pull, with the digest line being the last visible content. This is because the message ends at this point — the actual ls output would arrive in the next message (the following round), since the assistant must wait for the command to complete before it can act on the result.

Assumptions and Reasoning

The assistant makes several key assumptions in this message. First, it assumes that the build script binary is dynamically linked against libcuda.so.1 rather than statically linked or using dlopen with a different library name. This is a reasonable assumption given the standard CUDA toolchain behavior, but it's not verified until the ls output confirms the symlink is missing. Second, the assistant assumes that creating the missing symlink will resolve the build failure — an assumption that would be tested in the subsequent build attempt.

The reasoning follows a clear diagnostic pattern: observe the failure mode (build script crashes during GPU probe), identify the mechanism (runtime linking against libcuda), trace the missing dependency (libcuda.so.1 vs libcuda.so), and verify the root cause (inspect the stubs directory). This is textbook debugging methodology, applied to the uniquely challenging environment of a Docker build where each iteration requires a full rebuild.

Input and Output Knowledge

The input knowledge required to understand this message includes: familiarity with Linux shared library naming conventions (linker names vs. sonames), understanding of how CUDA's stub libraries work, knowledge of the neptune crate's build-time GPU probing, and awareness of the Docker multi-stage build context. Without this background, the message would appear to be about a trivial symlink issue; with it, the message reveals a deep understanding of the interaction between build systems, linkers, and container environments.

The output knowledge created by this message is the diagnosis itself: the root cause of the build failure is a missing libcuda.so.1 symlink in the CUDA devel image's stubs directory. This knowledge directly informs the fix — creating the symlink — and contributes to the broader understanding of how to containerize CUDA-accelerated Rust projects that perform runtime GPU probing during compilation.

The Broader Lesson

This message, brief as it is, illuminates a fundamental challenge in modern software engineering: the layering of build systems on top of each other. A Rust crate (neptune) runs a build script that compiles and executes a C/CUDA binary that links against a system library (libcuda), all within a Docker container that abstracts away the host's GPU drivers. Each layer introduces its own assumptions about library naming, linker behavior, and file system layout. When these assumptions don't align — when the CUDA toolkit's stubs directory provides libcuda.so but not libcuda.so.1 — the entire build collapses.

The assistant's ability to diagnose this issue without direct access to the failing environment (only through build logs) demonstrates the power of understanding the underlying systems. It's not just about knowing Docker or Rust or CUDA in isolation; it's about understanding how they interact at the boundaries — where the linker's NEEDED entry meets the filesystem's symlink, where the build script's runtime execution meets the container's library path, where the devel image's stubs meet the runtime linker's expectations.