The Quiet Diagnostic: How a Single ldd Command Shaped a Docker Build

In the midst of a complex multi-stage Docker build for a CUDA-accelerated Filecoin proving stack, a single bash command — seemingly trivial — captures the essence of systematic debugging. Message <msg id=616> is exactly one line of shell:

docker run --rm --entrypoint bash curio-cuzk:latest -c "ldd /usr/local/bin/cuzk 2>&1 | grep 'not found'"

And its output is even shorter:

libcuda.so.1 => not found

On its surface, this is barely a sentence. But in context, it is a pivotal diagnostic moment in a long debugging session that spanned multiple build blockers, linker errors, and missing runtime dependencies. This article unpacks why this message was written, what it reveals about the assistant's reasoning, and how a single ldd invocation fits into the broader effort of containerizing a production proving system.

The Context: A Docker Build Under Siege

The assistant had been constructing a Docker image (curio-cuzk:latest) intended to run the Curio Filecoin proving stack with CUDA 13 supraseal support. This was no ordinary Dockerfile — it involved compiling C++ FFI bindings, a Rust-based GPU proving engine called cuzk, and the Go-based Curio daemon, all within a multi-stage build on top of nvidia/cuda:13.0.2-devel-ubuntu24.04.

The build had already survived several crises. First, a pip conflict where the Debian-managed python3-pip package interfered with the virtual environment's pip upgrade, causing the supraseal build to fail. The assistant fixed this by removing python3-pip from the apt-get install list entirely, relying on ensurepip to bootstrap pip inside the venv. Next, a linker error struck: cannot find -lcudart_static.a during the Go link step. The assistant traced this to a missing LIBRARY_PATH entry and added /usr/local/cuda/lib64 to the environment variable. After these fixes, the build finally succeeded, producing three binaries: curio (163MB), cuzk (27MB), and sptool (210MB).

But success was short-lived. A smoke test in <msg id=614> revealed that curio crashed immediately at runtime:

curio: error while loading shared libraries: libconfig++.so.9: cannot open shared object file: No such file or directory

The Docker image had been built in a multi-stage setup where the runtime stage only copied the binaries and essential files from the build stage. The runtime stage was based on the same CUDA devel image, but it didn't include the development packages that provided shared libraries like libconfig++, libaio, libfuse3, and libarchive. The assistant ran ldd on curio in <msg id=615> and found exactly five missing libraries.

The Reasoning Behind Message 616

At this point, the assistant had identified curio's missing runtime dependencies. But the image contained two other binaries: cuzk and sptool. The assistant needed to know whether they had different missing libraries. Installing packages for curio's dependencies might not cover cuzk's needs, and discovering missing libraries one at a time after each rebuild would be painfully slow.

The decision to run ldd on cuzk specifically was a deliberate diagnostic choice. cuzk is the CUDA-based GPU proving engine — the most performance-critical component in the stack. It links against the CUDA runtime (libcuda.so) and potentially other GPU libraries. The assistant needed to know exactly what cuzk required at runtime so that the right packages could be installed in the Dockerfile's runtime stage before the next build iteration.

The command itself is carefully constructed:

What the Output Reveals

The result — libcuda.so.1 => not found — is both reassuring and informative. It tells the assistant that cuzk has only one missing runtime dependency, and that dependency is the CUDA driver library. This contrasts sharply with curio, which had five missing libraries spanning multiple packages (libconfig++, libaio, libfuse3, libarchive, plus libcuda.so.1).

The implication is significant: cuzk is a lean binary with minimal runtime dependencies beyond CUDA itself. Its Rust compilation statically links most of its dependencies, leaving only the CUDA runtime as a dynamic requirement. This is good engineering — it means the GPU proving engine will be robust against library version mismatches in deployment.

The missing libcuda.so.1 also tells the assistant something about the Docker image's runtime stage: it's based on the CUDA devel image, which includes CUDA development libraries and tools, but the runtime stage apparently doesn't include the CUDA runtime library symlink. This is a common pitfall with NVIDIA's CUDA Docker images — the devel image has everything for compilation, but the runtime libraries may need explicit installation or symlinking in the runtime stage.

Assumptions and Their Validity

The assistant made several assumptions in crafting this command:

That ldd would accurately report all missing libraries. This is generally true for dynamically linked ELF binaries on Linux, but ldd can miss libraries loaded via dlopen() at runtime. For cuzk, which is a Rust binary, most dependencies are statically linked, so ldd should give a complete picture of dynamic requirements.

That the binary path /usr/local/bin/cuzk was correct. This assumption was based on the Dockerfile's COPY commands in the build stage, which placed binaries in /usr/local/bin/. The assistant had verified the build output earlier and knew the binaries existed.

That cuzk's missing libraries would differ from curio's. This turned out to be correct — cuzk only needed libcuda.so.1, while curio needed five libraries. The assumption motivated the separate check, and the result validated it.

That fixing all missing libraries in one pass would be more efficient than iterating. This is a classic engineering trade-off: invest time in comprehensive diagnosis now to save rebuild cycles later. Given that each Docker build took many minutes (the build stage alone involved compiling C++ FFI, Rust, and Go code), the assistant's approach was sound.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with Docker's --entrypoint flag and how it overrides the default command; understanding of Linux's shared library resolution mechanism and the ldd tool; awareness that CUDA binaries depend on libcuda.so (the NVIDIA userspace driver library); and knowledge of the project's architecture — that cuzk is the GPU proving engine, curio is the main daemon, and both are deployed in the same container.

Output knowledge created by this message is precise and actionable: the cuzk binary has exactly one missing runtime dependency (libcuda.so.1), and no other shared libraries are unresolved. This tells the assistant that the runtime stage needs only the CUDA runtime library (or a symlink to it) to make cuzk functional. It also implies that cuzk's Rust compilation is doing its job of statically linking most dependencies, resulting in a portable binary.

The Broader Debugging Pattern

This message exemplifies a methodical debugging pattern that characterizes the entire session. The assistant works in cycles: build → test → diagnose → fix → rebuild. Each cycle narrows the problem space. First, the build itself failed (pip conflict, linker error). Then the build succeeded but runtime failed (missing shared libraries). Within the runtime failure, the assistant decomposed the problem by binary: check curio's libraries, check cuzk's libraries, check sptool's libraries.

This decomposition is important because different binaries have different dependency profiles. curio is a Go binary that links against system libraries for storage operations (FUSE, libarchive, libaio). cuzk is a Rust binary that links against CUDA. sptool likely has its own profile. By checking each separately, the assistant builds a complete dependency map before making changes to the Dockerfile.

The approach also reflects an understanding of Docker build caching. Each change to the Dockerfile invalidates some cache layers, triggering rebuilds. By diagnosing all missing libraries in one pass (running ldd on all binaries in a single container), the assistant can make a single comprehensive edit to the runtime stage's package installation, maximizing cache reuse.

Conclusion

Message <msg id=616> is a study in minimalism — one command, one line of output. But it is also a study in diagnostic discipline. The assistant could have assumed that fixing curio's missing libraries would suffice, or that cuzk would have the same dependencies. Instead, it verified empirically. This single ldd invocation transformed an assumption into knowledge: cuzk needs only libcuda.so.1 at runtime.

That knowledge directly informed the next Dockerfile edit — installing libconfig++, libaio, libfuse3, libarchive, and ensuring the CUDA runtime library symlink exists in the runtime stage. The build that followed would finally produce a working container, thanks in part to this quiet diagnostic moment.

In software engineering, the most valuable debugging steps are often the ones that produce negative results — confirming what is not broken. Here, the assistant learned that cuzk was not missing anything beyond the CUDA runtime, narrowing the remaining work to a focused set of package installations. It is a reminder that even a one-line command, when executed with intent and interpreted with context, can be the key that unlocks a successful build.