The Missing Libraries: Diagnosing Runtime Dependencies in a Multi-Stage Docker Build

In message 617 of a lengthy coding session to build a Docker container for the Curio/CuZK Filecoin proving stack, the assistant reaches a pivotal debugging moment. After successfully compiling all binaries and assembling the Docker image, a smoke test reveals that the container fails at runtime with shared library errors. This message captures the diagnostic step where the assistant systematically identifies the missing dependencies, categorizes them by severity, and prepares to patch the Dockerfile's runtime stage. It is a concise but revealing window into how real-world container builds often fail not at the compilation stage, but at the boundary between build and runtime environments.

The Context: A Successful Build, a Broken Container

The broader session (segment 5 of the conversation) revolves around constructing a production-ready Docker image for Filecoin's Curio proving system, which includes GPU-accelerated proof generation via CUDA and the CuZK proving engine. The assistant had already overcome several build blockers: a pip conflict with Debian's package manager that prevented the supraseal dependency from building, a missing libcudart_static.a linker error that required adding /usr/local/cuda/lib64 to the LIBRARY_PATH, and various other toolchain issues. By message 613, the build had finally succeeded, producing three binaries—curio (163MB), cuzk (27MB), and sptool (210MB)—and the image was tagged as curio-cuzk:latest.

The natural next step was a smoke test. In message 614, the assistant ran:

docker run --rm curio-cuzk:latest -c "curio --version && echo '---' && sptool --version && echo '---' && cuzk --help 2>&1 | head -5 && echo '---' && echo 'All binaries OK'"

The result was alarming. Instead of version strings, the output showed:

curio: error while loading shared libraries: libconfig++.so.9: cannot open shared object file: No such file or directory

The container built successfully, but the primary binary could not even start. This is a classic failure mode in multi-stage Docker builds: the build stage compiles against development headers and static libraries, but the runtime stage—typically based on a leaner image—may lack the shared libraries that the binaries dynamically link against. The build succeeds because the linker resolves symbols during compilation, but the loader fails at runtime because it cannot find the .so files.

The Diagnostic: Systematic Library Discovery

Message 617 opens with the assistant's immediate response to this failure. The first line reads:

libcuda.so.1 is expected — that comes from the nvidia driver at runtime. The others need to be installed in the runtime image.

This statement reveals an important piece of domain knowledge: in NVIDIA GPU containers, libcuda.so.1 is not packaged inside the container image itself. Instead, it is provided by the NVIDIA Container Toolkit at runtime, which mounts the host's CUDA driver library into the container. The assistant correctly recognizes that this missing library is a non-issue—it is a runtime injection, not a packaging error.

But the other missing libraries are genuine packaging gaps. To identify them precisely, the assistant runs two ldd commands. The first, executed against curio, reveals four missing libraries:

libconfig++.so.9 => not found
libaio.so.1t64 => not found
libfuse3.so.3 => not found
libarchive.so.13 => not found

The second, run against cuzk, shows only one:

libcuda.so.1 => not found

This confirms the assistant's expectation: cuzk (the GPU proving daemon) only needs the CUDA driver library, which will be injected at runtime. curio and sptool, however, have four additional dependencies that must be installed in the runtime image.

The assistant then reads the current Dockerfile to inspect the runtime stage. The relevant snippet shows a minimal set of runtime packages:

RUN apt-get update && apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        libhwloc15 \
        libnuma1 \
        libssl3

This is the current state of the runtime stage—a lean image based on nvidia/cuda:13.0.2-runtime-ubuntu24.04 with only basic utilities and a few libraries that were known to be needed. The four missing libraries (libconfig++, libaio, libfuse3, libarchive) are not present.

The Reasoning: Why These Libraries Are Needed

The assistant does not explicitly state why each library is required, but the context of the Curio project provides the answers:

libconfig++ (libconfig++.so.9): A C++ library for parsing configuration files. Curio uses configuration files to specify proving parameters, storage paths, and network settings. The library provides a structured way to read these configurations.
libaio (libaio.so.1t64): The Linux asynchronous I/O library. Filecoin's proving system involves heavy disk I/O for reading sector data and writing proofs. Async I/O is critical for performance, especially when dealing with the large parameter files (~100GB) that need to be fetched and cached.
libfuse3 (libfuse3.so.3): Filesystem in Userspace library, version 3. Curio may use FUSE for mounting remote storage or for interacting with the Filecoin network's storage subsystem.
libarchive (libarchive.so.13): A library for reading and writing various archive formats. The parameter files and proof data may be compressed or archived, requiring this library for extraction. These are not exotic dependencies—they are standard system libraries that any moderately complex application might need. But in a multi-stage Docker build, they are easy to overlook because the build stage (based on the CUDA devel image) includes development packages that pull in these libraries as dependencies, while the runtime stage (based on the CUDA runtime image) is much leaner and does not include them.

The Decision: A Targeted Fix

The assistant's plan is straightforward: add the missing packages to the apt-get install line in the runtime stage. The exact package names can be inferred from the library names:

libconfig++ → libconfig++9 or libconfig++-dev (runtime: libconfig++9v5 or similar)
libaio → libaio1t64 (the t64 suffix indicates the 64-bit time_t variant used in Ubuntu 24.04)
libfuse3 → libfuse3-3 or libfuse3
libarchive → libarchive13 The assistant does not execute the edit within this message—it reads the Dockerfile and prepares to make the change. The actual edit happens in subsequent messages (not shown in the target). But the decision framework is already established: identify all missing non-CUDA libraries, find their apt package names, and add them to the runtime stage. This approach reflects a key principle of Docker image construction: the runtime stage should be as minimal as possible while containing every shared library that the binaries actually need. Adding unnecessary packages bloats the image and increases the attack surface. The assistant's method—running ldd, filtering out the expected CUDA driver library, and installing only the missing packages—is the correct, surgical approach.

Assumptions and Their Validity

The message rests on several assumptions, all of which are sound:

libcuda.so.1 will be provided at runtime. This is correct for NVIDIA containerized environments. The NVIDIA Container Toolkit uses LD_PRELOAD or library path injection to make the host's CUDA driver available inside the container. As long as the container is run with --gpus all (or equivalent), libcuda.so.1 will be present.
The missing libraries can be installed via apt-get. This assumes the correct package names exist in Ubuntu 24.04's repositories. For standard libraries like libaio, libfuse3, libconfig++, and libarchive, this is a safe assumption—they are well-maintained packages in the Ubuntu ecosystem.
The runtime stage has network access for apt-get. The Dockerfile's runtime stage runs apt-get update and apt-get install, which require network access during the build. This is standard for Docker builds and is not expected to be a problem.
No other libraries are missing. The assistant only checked curio and cuzk. There is a third binary, sptool (210MB), which was not checked with ldd. It is possible that sptool has additional dependencies. However, since sptool is a CLI tool for managing proofs and parameters, it likely shares the same dependency set as curio. The assistant's focus on curio is reasonable—if the main binary works, the supporting tools are likely to work too.

Input Knowledge Required

To fully understand this message, the reader needs:

Understanding of multi-stage Docker builds: The distinction between the build stage (where compilation happens, with development headers and static libraries) and the runtime stage (where only the minimum needed to run the binaries is included). The failure mode—build succeeds but runtime fails due to missing shared libraries—is a classic symptom of incomplete runtime stages.
Knowledge of Linux shared library resolution: How ldd works, what "not found" means, and how the dynamic linker (ld.so) resolves libraries at runtime. The LD_LIBRARY_PATH environment variable and the system library cache (/etc/ld.so.cache) are relevant concepts.
Awareness of NVIDIA containerization: The NVIDIA Container Toolkit injects the CUDA driver library (libcuda.so.1) into containers at runtime. This is why the assistant dismisses that missing library as "expected." Without this knowledge, one might mistakenly try to install the CUDA driver inside the container, which would be incorrect and would create a larger, less portable image.
Familiarity with the Curio/CuZK project: Understanding that curio is the main daemon, cuzk is the GPU proving daemon, and sptool is a management CLI. The project involves Filecoin proof generation, which requires GPU acceleration and heavy I/O.

Output Knowledge Created

This message produces several valuable outputs:

A precise list of missing runtime libraries: libconfig++.so.9, libaio.so.1t64, libfuse3.so.3, libarchive.so.13. This is actionable data that directly informs the Dockerfile fix.
A categorization of the missing libraries: One (libcuda.so.1) is expected and requires no action; four others need to be installed. This prevents wasted effort trying to "fix" the CUDA library.
A snapshot of the current runtime stage: By reading and displaying the Dockerfile, the assistant establishes the baseline that needs to be modified. This makes the subsequent edit clear and auditable.
A validated diagnostic methodology: The assistant demonstrates a reproducible process for checking runtime dependencies: run the container, observe the error, use ldd to find missing libraries, filter out expected runtime-provided libraries, and install the remainder. This methodology can be applied to any Docker build.

The Thinking Process

The assistant's reasoning is visible in the structure of the message. It proceeds through a clear logical sequence:

Interpret the error: The smoke test failed because curio cannot find libconfig++.so.9. This is a runtime linking error, not a build error.
Broaden the diagnosis: Rather than fixing just libconfig++, run ldd on all binaries to find every missing library at once. This avoids a tedious cycle of fix-one-break-another.
Categorize the results: libcuda.so.1 is a known runtime injection; the other four are genuine gaps. This distinction is critical—it prevents over-patching.
Read the current state: Before editing, read the Dockerfile to understand what is already in the runtime stage. This ensures the fix is additive and does not duplicate or conflict with existing packages.
Prepare the fix: The message ends with the assistant ready to edit. The actual edit is deferred to the next tool call, but the decision is made: add the four missing packages to the runtime stage's apt-get install command. This sequence—observe, diagnose, categorize, inspect, plan—is a textbook debugging workflow. It minimizes wasted effort by gathering all relevant information before making changes, and it ensures that the fix addresses the root cause rather than just the first symptom.

Conclusion

Message 617 is a small but instructive moment in the Docker build process. It captures the transition from "the build succeeded" to "the container works" — a gap that is often wider than developers expect. The assistant's systematic approach to identifying missing runtime libraries, its correct interpretation of the CUDA driver dependency, and its methodical preparation of the fix all demonstrate the kind of disciplined debugging that separates a working container from a merely buildable one. The message is a reminder that in multi-stage Docker builds, the runtime stage is not an afterthought—it is a distinct environment that must be carefully curated to match the runtime needs of the compiled binaries.