The Iterative Debugging Loop: A Docker Build's Journey Through CUDA Stubs and Missing Symlinks

Introduction

In the world of infrastructure engineering, few tasks are as deceptively complex as containerizing a heterogeneous software stack. When that stack spans Go, Rust, C++, CUDA, and Python build tools—all wired together by a labyrinthine Makefile system—the Docker build becomes less a simple packaging exercise and more an archaeological dig through layer after layer of dependency resolution. Message 580 of this opencode session captures a single, seemingly mundane moment in such a dig: the re-execution of a docker build command after a fix was applied. But this message is far from mundane. It represents the third attempt in a multi-hour debugging session to build a Docker image for the Curio Filecoin proving system with CUDA acceleration, and it embodies the iterative hypothesis-test-revise cycle that defines systems engineering at its most demanding.

The Scene: Containerizing a Proving Engine

To understand why message 580 exists, one must understand the project. The Curio project is a Filecoin storage mining implementation that includes cuzk, a custom CUDA-accelerated zero-knowledge proving engine. The proving pipeline involves multiple layers: a Go binary (curio) that orchestrates proving operations, a Rust/CUDA daemon (cuzk-daemon) that performs GPU-accelerated computation, and a C++ library (supraseal) that handles the low-level cryptographic operations. The build process is correspondingly complex, requiring CUDA 13 toolchain, Go 1.24, Rust 1.86, gcc-13, and a host of system libraries including hwloc, OpenSSL, NASM, and Python build tools.

The assistant had just completed writing a two-stage Dockerfile (Dockerfile.cuzk) and an entrypoint script (entrypoint.sh) designed to bundle all of this into a deployable container for mainnet 32GiB proving. The first stage (builder) used nvidia/cuda:13.0.2-devel-ubuntu24.04 to compile everything from source, while the second stage (runtime) used nvidia/cuda:13.0.2-runtime-ubuntu24.04 to produce a minimal image containing only the binaries and runtime libraries. The user had given the go-ahead to test the build on the current host ([msg 572]), which was confirmed to be CUDA-capable.

The First Two Attempts

The first build attempt ([msg 574]) failed early. The Filecoin FFI build script uses jq to parse JSON configuration files, but the CUDA devel image does not include jq by default. The assistant identified this, added jq to the apt-get install list in the Dockerfile, and re-ran the build ([msg 577]).

The second attempt progressed much further. The build successfully installed system dependencies, downloaded Go and Rust toolchains, verified the toolchain versions (rustc 1.86.0, cargo 1.86.0, go 1.24.7, nvcc 13.0.88), and began compiling the Rust/CUDA dependencies. The assistant checked the build output ([msg 578]) and saw that compilation was proceeding through the dependency tree: multihash-codetable, cid, ipld-core, bellpepper, bls-signatures, neptune, bellperson, and others. But then the build stalled.

The libcuda.so.1 Problem

The neptune build script, which is part of the bellperson cryptographic library, probes GPU capabilities at compile time. This probing requires loading libcuda.so.1 via the dynamic linker. The CUDA devel image includes the CUDA toolkit with stub libraries in /usr/local/cuda/lib64/stubs/, which are specifically designed for this purpose—they provide the CUDA runtime API symbols needed for compilation without requiring an actual GPU driver. However, the assistant discovered that simply having these stubs available was not enough.

In message 579, the assistant articulated the problem and its attempted fix:

The neptune build script needs libcuda.so.1 at compile time (it probes GPU capabilities). The CUDA devel image has the toolkit but not the runtime driver stub. The nvidia/cuda devel images include a stub libcuda.so in /usr/local/cuda/lib64/stubs/ for exactly this purpose. I need to add that to LIBRARY_PATH (and optionally LD_LIBRARY_PATH) during the build.

The assistant then edited the Dockerfile to add the CUDA stubs directory to both LIBRARY_PATH (for the linker at compile time) and LD_LIBRARY_PATH (for the dynamic linker at runtime). This seemed like the correct approach: the stubs directory contains a libcuda.so file that provides the necessary symbols, and pointing the build tools at it should resolve the missing library.

Message 580: The Third Attempt

Message 580 is the execution of that fix. The assistant runs:

docker build -f Dockerfile.cuzk -t curio-cuzk:latest . 2>&1

The output shown in the message is truncated—it captures only the initial stages of the Docker build:

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile.cuzk
#1 transferring dockerfile: 6.31kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/nvidia/cuda:13.0.2-runtime-ubuntu24.04
#2 DONE 0.4s

#3 [internal] load metadata for docker.io/nvidia/cuda:13.0.2-devel-ubuntu24.04
#3 DONE 0.4s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [runtime 1/9] FROM docker.io/nvidia/cuda:13.0.2-runtime-ubunt...

The build has begun. The Docker engine loads the build definition (the Dockerfile, now 6.31kB after the edit), pulls metadata for both base images, transfers the .dockerignore context, and starts executing the runtime stage. The message ends here because the build is still running—Docker builds of this complexity take many minutes, and the assistant captured the initial output to confirm the build started successfully.

On the surface, this message is just a build log. But its significance lies in what it represents: a hypothesis being tested. The assistant had a theory about why the build was failing (missing libcuda.so.1 at compile time) and a proposed solution (adding the CUDA stubs directory to the library path). Message 580 is the experiment designed to validate that theory.

The Assumption and Its Flaw

The assistant's assumption was that adding /usr/local/cuda/lib64/stubs to LIBRARY_PATH and LD_LIBRARY_PATH would be sufficient for the neptune build script to find and load libcuda.so.1. This assumption was reasonable—the CUDA devel images are specifically designed to support compilation of CUDA-dependent code, and the stubs directory is the canonical mechanism for providing CUDA runtime symbols without a physical GPU.

However, the assumption had a subtle flaw. The dynamic linker on Linux resolves shared library dependencies using the soname embedded in the library file. For the CUDA runtime, the soname is libcuda.so.1, meaning the dynamic linker looks for a file named exactly libcuda.so.1. The stubs directory in the CUDA devel image contains libcuda.so (a symlink or file used for compile-time linking with -lcuda), but it does not necessarily contain libcuda.so.1. These are different files serving different purposes: libcuda.so is for the static linker (-lcuda), while libcuda.so.1 is for the dynamic linker at runtime.

The assistant discovered this in the very next message ([msg 581]):

Still failing. The LD_LIBRARY_PATH approach isn't enough — the build script binary is linked at runtime and needs libcuda.so.1 specifically (not just libcuda.so). The stubs directory has libcuda.so but may lack the .so.1 symlink.

The assistant then inspected the stubs directory directly by running a container ([msg 581]) and confirmed: libcuda.so was present, but libcuda.so.1 was not. The fix required creating the missing symlink ([msg 582]), which was then added to the Dockerfile.

The Deeper Pattern: Iterative Debugging in Container Builds

Message 580 is a microcosm of a pattern that repeats throughout this session and across systems engineering more broadly: the iterative debug loop. Each cycle consists of:

Observation: The build fails at a specific point with a specific error.
Hypothesis: The assistant forms a theory about the root cause based on knowledge of the tools involved.
Fix: The assistant modifies the Dockerfile or build configuration.
Test: The assistant re-runs the build (message 580 is one such test).
Evaluation: The assistant checks the output to see if the fix worked.
Refinement: If the fix was incomplete, the assistant refines the hypothesis and tries again. What makes this particular cycle notable is the nature of the assumption. The assistant knew the correct mechanism (CUDA stubs directory) but missed a critical detail (the soname versioning convention). This is a common class of bug in systems engineering: knowing what to do but not getting the exact incantation right. The difference between libcuda.so and libcuda.so.1 is a single symlink, but that symlink determines whether the dynamic linker can resolve the dependency.

Input Knowledge Required

To understand message 580 fully, one needs knowledge of several domains:

Docker build mechanics: Understanding of multi-stage builds, layer caching, and how FROM, COPY, and RUN instructions interact.
CUDA toolkit structure: Knowledge that CUDA devel images include stub libraries for compile-time linking, and that these stubs live in /usr/local/cuda/lib64/stubs/.
Linux dynamic linking: Understanding of sonames, the difference between -l linker flags and LD_LIBRARY_PATH, and how the dynamic linker resolves *.so.* filenames.
The Curio/cuzk build system: Knowledge that the build uses make deps to compile filecoin-ffi and supraseal from source, that neptune/bellperson probe GPU capabilities at compile time, and that the build requires specific toolchain versions.
The session history: Understanding that this is the third build attempt, preceded by a missing jq fix and the initial Dockerfile creation.

Output Knowledge Created

Message 580 itself produces limited output—it's a build in progress. But the fact that the build was initiated, and the context of which fix was being tested, creates valuable knowledge:

It documents the assistant's hypothesis about the libcuda.so.1 issue.
It establishes a baseline for what happens when the fix is applied (the build starts but will later fail).
It creates a record of the iterative process, allowing a reader to trace the reasoning from problem identification through attempted solution to refinement. The full output of this build attempt is revealed in subsequent messages ([msg 581] and [msg 584]), where the assistant discovers that while the FFI build passed (the LIBRARY_PATH fix was partially effective), the supraseal build then failed due to a separate Python PEP 668 issue. This cascading failure pattern—where fixing one blocker reveals the next—is characteristic of complex Docker builds.

The Thinking Process

The assistant's reasoning is most visible in message 579, which immediately precedes the subject message. The assistant states:

The neptune build script needs libcuda.so.1 at compile time (it probes GPU capabilities). The CUDA devel image has the toolkit but not the runtime driver stub. The nvidia/cuda devel images include a stub libcuda.so in /usr/local/cuda/lib64/stubs/ for exactly this purpose.

This reveals several layers of reasoning:

Root cause identification: The assistant has traced the build failure to the neptune build script's GPU probing.
Mechanism knowledge: The assistant knows that CUDA devel images include stub libraries specifically for this use case.
Solution design: The assistant correctly identifies that adding the stubs directory to the library search path should resolve the issue. The flaw in the reasoning is subtle: the assistant assumed the stubs directory contained a libcuda.so.1 symlink (or that LD_LIBRARY_PATH would resolve libcuda.so as libcuda.so.1), when in fact only libcuda.so was present. This is the kind of detail that is easy to miss when working with a complex build system and unfamiliar base images.

Conclusion

Message 580 is a snapshot of engineering in motion. It captures the moment when a hypothesis meets reality, when a carefully considered fix is put to the test. The build output is unremarkable—just Docker's standard progress logging—but the context transforms it into a document of the debugging process. It shows that even with deep knowledge of the tools involved (CUDA stubs, library paths, dynamic linking), the exact implementation details of a base image can surprise even an experienced engineer.

The message also illustrates a fundamental truth about containerization: a Dockerfile is never truly finished. Each build attempt reveals new constraints, new edge cases, new dependencies that were not captured in the initial design. The Dockerfile for a project like Curio/cuzk is not a static document but a living artifact, shaped and reshaped by each iteration of the build-test-refine cycle. Message 580 is one such iteration—a single step in the long journey from "it doesn't build" to "it works."