The Missing Symlink: Diagnosing a CUDA Stub Resolution Failure in a Multi-Stage Docker Build

In the midst of constructing a Docker container to bundle the Curio Go binary, the cuzk Rust/CUDA daemon, and all their dependencies for Filecoin mainnet 32GiB proving, the assistant encountered a build failure that would require two rounds of diagnosis to fully resolve. The subject message — message index 579 in the conversation — captures the first diagnostic pass: a concise, technically dense reasoning chain that correctly identifies the surface problem but, as subsequent messages reveal, stops just short of the root cause. This message is a fascinating case study in how deep system knowledge enables rapid debugging, but also how assumptions about library naming conventions can lead to incomplete fixes in containerized build environments.

The Context: A High-Stakes Docker Build

The assistant had just completed writing a sophisticated multi-stage Dockerfile.cuzk ([msg 562]) designed to compile the entire Curio proving stack from source. The build was ambitious: Stage 1 used nvidia/cuda:13.0.2-devel-ubuntu24.04 as the builder image, installing Go 1.24.7, Rust 1.86.0, gcc-13, and a laundry list of system dependencies before running make clean deps with CUDA flags enabled and building both the curio binary and the cuzk-daemon Rust binary. Stage 2 used the leaner nvidia/cuda:13.0.2-runtime-ubuntu24.04 image to produce a minimal runtime container.

The first build attempt ([msg 574]) failed because the Filecoin FFI build script required jq, which wasn't installed in the builder stage. The assistant fixed that ([msg 576]) and relaunched ([msg 577]). Checking the build output ([msg 578]), the assistant saw the Rust compilation progressing through packages like neptune v11.0.0 and bellperson v0.26.0 — and then the build stalled. Message 579 is the assistant's response to that stall.

The Reasoning: Tracing a Linker-Level Failure

The subject message reads in full:

The neptune build script needs libcuda.so.1 at compile time (it probes GPU capabilities). The CUDA devel image has the toolkit but not the runtime driver stub. The nvidia/cuda devel images include a stub libcuda.so in /usr/local/cuda/lib64/stubs/ for exactly this purpose. I need to add that to LIBRARY_PATH (and optionally LD_LIBRARY_PATH) during the build.

This reasoning is remarkable for what it reveals about the assistant's mental model. The assistant has inferred, from the build failure output, that the neptune cryptographic library's build script is failing because it cannot find libcuda.so.1. This is not a typical compile-time dependency — most Rust crates don't link against the CUDA driver at build time. But neptune, a core dependency in the Filecoin proof pipeline, includes a build script that probes GPU capabilities to select optimal kernel parameters. That probe requires loading libcuda.so.1 via the dynamic linker, even though the final binary will be linked against the CUDA runtime library at runtime.

The assistant correctly reasons that the nvidia/cuda:13.0.2-devel-ubuntu24.04 image contains the CUDA toolkit (compiler, headers, libraries) but not the actual NVIDIA kernel-mode driver — you can't load the driver inside a Docker container during build. NVIDIA's solution is to provide stub libraries in /usr/local/cuda/lib64/stubs/: thin shared objects that export the same symbols as the real libcuda.so but contain no actual GPU functionality. These stubs are sufficient to satisfy the linker at build time.

The assistant's proposed fix — adding the stubs directory to LIBRARY_PATH (for the linker) and optionally LD_LIBRARY_PATH (for the dynamic linker at build script runtime) — is architecturally correct. The CUDA devel images are explicitly designed with this stubs directory for exactly this use case. The edit was applied, and the build was relaunched.

The Hidden Assumption: libcuda.so vs. libcuda.so.1

Here is where the message's reasoning, while sound in principle, reveals an incomplete diagnosis. The assistant assumed that adding the stubs directory to the library search path would be sufficient. But as messages 581 and 582 reveal, the stubs directory contains libcuda.so but lacks the libcuda.so.1 symlink. The neptune build script's binary, being dynamically linked, calls dlopen("libcuda.so.1", ...) — it looks for the versioned soname, not the bare libcuda.so. The Linux dynamic linker convention is that libcuda.so.1 is the runtime soname (the name embedded in the shared object's DT_SONAME field), while libcuda.so is the link-time name used with -lcuda. They are typically the same file (or a symlink), but the stubs directory only provides the link-time name.

This is a subtle but critical distinction. The assistant's fix was necessary but not sufficient. The root cause was not merely the absence of the stubs directory from the library path, but the absence of the libcuda.so.1 symlink within that directory. The assistant discovered this in message 581 by running ls -la /usr/local/cuda/lib64/stubs/ inside the devel image, and fixed it in message 582 by adding a ln -s command to the Dockerfile to create the missing symlink.

Input Knowledge Required

To understand and produce this message, the assistant drew on several domains of specialized knowledge:

Output Knowledge Created

The immediate output of this message was an edit to /tmp/czk/Dockerfile.cuzk that added the CUDA stubs directory to the library search path. While this edit alone did not resolve the build failure, it established the diagnostic direction that led to the complete fix in message 582. The message also created conceptual knowledge for anyone reading the conversation: it documented why a CUDA devel image's stubs directory exists, how neptune's build script uses the CUDA driver at compile time, and what the difference is between link-time and runtime library names.

The Thinking Process: A Window into Debugging Methodology

What makes this message particularly valuable is the visible thinking process. The assistant doesn't just apply a fix — it explains the causal chain:

  1. The neptune build script probes GPU capabilities → this requires loading libcuda.so.1
  2. The CUDA devel image has the toolkit but not the runtime driver → the real libcuda.so.1 isn't available
  3. NVIDIA provides stub libraries in the devel image → libcuda.so exists in stubs/
  4. The stubs directory needs to be on the library search path → add to LIBRARY_PATH and LD_LIBRARY_PATH This is textbook root-cause analysis: tracing from symptom (build failure) to mechanism (GPU probing in build script) to environmental cause (missing library in Docker image) to available remedy (stubs directory). The reasoning is linear, confident, and grounded in deep knowledge of the toolchain. The message also demonstrates the assistant's awareness of the difference between link-time and runtime resolution. The parenthetical "(and optionally LD_LIBRARY_PATH)" shows an understanding that the build script may resolve the library either through the linker (at compile time) or through dlopen (at runtime), and that both paths may need coverage.

Conclusion

Message 579 is a perfect snapshot of the debugging process in complex systems engineering: a correct diagnosis that leads to a necessary but incomplete fix, followed by further iteration that closes the gap. The assistant's reasoning about CUDA stub libraries, the neptune build script, and Linux dynamic linking conventions is technically precise and well-articulated. The message's only limitation is an understandable assumption about library naming — that libcuda.so and libcuda.so.1 would be interchangeable in the stubs directory. This assumption, rooted in how most Linux distributions package these libraries, turned out to be false for the specific NVIDIA devel image in use. The subsequent discovery and fix of the missing symlink ([msg 582]) completed the resolution, but message 579 stands as the critical first step: identifying what was missing and why, even if the exact how of the fix required one more iteration.