From Symptom to System: The Diagnostic Pivot That Saved the Docker Build

Introduction

In the course of building a production Docker image for the Curio/CuZK Filecoin proving system, a single message marked the transition from chasing individual build failures to performing a comprehensive runtime audit. Message <msg id=615> is deceptively short — just a few lines of analysis and a single ldd command — but it represents a critical methodological shift. After successfully resolving two build-stage blockers (a Python pip conflict and a missing CUDA static library path), the assistant had every reason to believe the build was complete. Instead, a smoke test revealed a runtime library error, and rather than fixing that one library and moving on, the assistant paused to ask: what else is missing?

This article examines that diagnostic moment in depth: why it was necessary, what assumptions it challenged, and how it transformed the trajectory of the Docker build effort.

The Full Message

[assistant] Two issues: 1. The entrypoint runs and tries curio fetch-params but curio can't start due to missing libconfig++.so.9 in the runtime image 2. We need libconfig++ in the runtime stage

>

Let me check what other shared libs are needed: [bash] docker run --rm --entrypoint bash curio-cuzk:latest -c "ldd /usr/local/bin/curio 2>&1 | grep 'not found'" libconfig++.so.9 => not found libaio.so.1t64 => not found libfuse3.so.3 => not found libarchive.so.13 => not found libcuda.so.1 => not found

The Context: A Hard-Won Build

To understand why this message matters, one must appreciate what preceded it. The assistant had been iterating on a Dockerfile for the Curio project — a Filecoin storage mining system that integrates with the CuZK GPU proving engine and supraseal CUDA acceleration. The Docker build used a multi-stage approach: a devel stage based on nvidia/cuda:13.0.2-devel-ubuntu24.04 for compilation, and a runtime stage based on the corresponding CUDA runtime image for deployment.

The build had already survived two major crises. First, a Python pip conflict: the supraseal build script created a virtual environment and ran pip install --upgrade pip, but on Ubuntu 24.04 the system-installed python3-pip package caused pip to attempt uninstalling Debian-managed packages, triggering a failure. The assistant traced this through multiple rounds of investigation — checking whether python3-venv depended on python3-pip, verifying that ensurepip would bootstrap pip independently, and ultimately removing python3-pip from the apt-get install list entirely ([msg 602]).

Second, a linker error: the Go build step failed with cannot find -lcudart_static because the CUDA static library lived at /usr/local/cuda/lib64/ but the LIBRARY_PATH environment variable only included the stubs directory. The assistant located the library through a chain of symlink resolution and added the correct path ([msg 611]).

After these fixes, the build completed successfully. The assistant noted with satisfaction that all three binaries — curio (163MB), cuzk (27MB), and sptool (210MB) — were built and present in the image ([msg 613]). The build was declared complete.

The Smoke Test That Changed Everything

Message <msg id=614> was the smoke test. The assistant ran a simple command to verify the binaries worked:

docker run --rm curio-cuzk:latest -c "curio --version && echo '---' && sptool --version && echo '---' && cuzk --help 2>&1 | head -5 && echo '---' && echo 'All binaries OK'"

The result was immediate failure:

curio: error while loading shared libraries: libconfig++.so.9: cannot open shared object file: No such file or directory

This error reveals a classic multi-stage Docker build pitfall. The devel stage has all the development headers and libraries needed for compilation, but the runtime stage — which is what actually runs in production — is minimal. It contains only the binaries copied from the devel stage, without the shared libraries those binaries depend on at runtime. The curio binary was linked against libconfig++.so.9 during compilation, but that library wasn't installed in the runtime image.

The Diagnostic Turn

This is where message <msg id=615> becomes pivotal. The assistant could have taken the obvious path: add libconfig++ to the runtime stage, rebuild, and move on. That would have fixed the immediate symptom. But instead, the assistant asked a deeper question: what else is missing?

The reasoning is visible in the message itself. The assistant writes "Two issues" but then immediately pivots to "Let me check what other shared libs are needed." This is the critical insight — the assistant recognized that the libconfig++ error was just the tip of the iceberg. If one library was missing, others were almost certainly missing too. The ldd command was the right tool for this investigation because it enumerates all shared library dependencies and reports which ones are unresolvable.

The command used is worth examining:

docker run --rm --entrypoint bash curio-cuzk:latest -c "ldd /usr/local/bin/curio 2>&1 | grep 'not found'"

This runs ldd (the dynamic linker dependency tracer) on the curio binary inside the runtime image, filtering for lines containing "not found." The --entrypoint bash override is necessary because the image's default entrypoint would try to run curio itself. The result reveals not one but five missing libraries:

Assumptions and Their Corrections

This message reveals several assumptions that were implicitly held and then corrected:

Assumption 1: A successful build implies a runnable image. The assistant had been focused on getting the compilation to succeed — fixing build-time errors like pip conflicts and linker paths. The build succeeding felt like a milestone, but it only guaranteed that the binaries could be compiled, not that they could be executed in the target environment. The smoke test was designed to catch exactly this gap, and it did.

Assumption 2: The runtime image would have the necessary libraries. Multi-stage Docker builds are designed to produce small runtime images by copying only the compiled artifacts. But this only works if the binaries are statically linked or if the runtime image explicitly includes the shared libraries they need. The curio binary was dynamically linked against several system libraries that existed in the devel stage but not in the runtime stage.

Assumption 3: One missing library means one fix. The most natural response to "libconfig++.so.9 not found" is to install libconfig++ and move on. The assistant's decision to run ldd proactively — before making any edits — reflects an understanding that runtime dependency problems tend to cluster. If the build system didn't ensure one library was present, it likely didn't ensure others either.

Input Knowledge Required

To fully understand this message, one needs:

  1. Multi-stage Docker builds: The concept that a Dockerfile can have multiple FROM statements, with later stages (runtime) being minimal and earlier stages (devel) containing build toolchains. The runtime stage only has what is explicitly installed or copied into it.
  2. Dynamic linking and ldd: The Linux dynamic linker resolves shared library dependencies at runtime. The ldd command prints these dependencies and can show which ones are missing. The grep 'not found' filter is a standard diagnostic technique.
  3. The Curio/CuZK project architecture: Curio is a Filecoin storage mining system that integrates with CuZK (a GPU-accelerated proving engine) and supraseal (a CUDA-based seal operation implementation). It depends on SPDK for storage operations, which in turn depends on libaio, libfuse3, and libarchive.
  4. CUDA runtime vs. devel images: NVIDIA provides separate Docker images — the -devel images contain compilers, headers, and static libraries for building CUDA applications, while the runtime images contain only the shared libraries needed to execute them. The libcuda.so.1 dependency is expected to be provided by the host's NVIDIA driver at runtime, not by the container image itself.

Output Knowledge Created

This message produced several valuable outputs:

  1. A complete dependency inventory: The ldd output enumerated exactly which shared libraries were missing, turning a single symptom into a concrete list of five items to fix.
  2. A prioritized fix list: The libraries fall into two categories. Four (libconfig++, libaio, libfuse3, libarchive) are system packages that can be installed via apt-get in the runtime stage. One (libcuda.so.1) is a host-provided library that must be mounted from the NVIDIA driver at runtime — a fundamentally different fix.
  3. A diagnostic methodology: The assistant established a pattern of "check comprehensively before fixing" that would serve well throughout the remainder of the session. Rather than treating each error as an isolated incident, this approach treats errors as symptoms of underlying systemic gaps.
  4. Confirmation of the runtime stage structure: The fact that libcuda.so.1 was "not found" in the runtime image is actually correct behavior — the CUDA runtime image does not include the driver library, which must be provided by mounting the host's NVIDIA driver into the container. This confirmed that the runtime stage was correctly minimal and that no additional CUDA libraries needed to be installed.

The Thinking Process

The reasoning visible in this message follows a clear arc:

  1. Observation: The entrypoint runs and fails on libconfig++.so.9.
  2. Initial framing: "Two issues" — the entrypoint behavior and the missing library.
  3. Methodological pivot: Instead of fixing issue #2 immediately, the assistant asks "what else is missing?" This is the key insight — recognizing that the single error is likely not the only missing dependency.
  4. Tool selection: ldd is the right tool because it enumerates all dependencies, not just the first one the dynamic linker encounters.
  5. Execution: The command is carefully constructed — using --entrypoint bash to avoid triggering the default entrypoint, running ldd on the binary path, and filtering for "not found" to surface only the problems.
  6. Interpretation: The output reveals five missing libraries, which the assistant can categorize into "installable via apt" and "host-provided." This is not the thinking of someone who is surprised by the error. It is the thinking of someone who expected that a successful build might hide runtime problems and who had a diagnostic strategy ready. The smoke test was designed to catch this exact class of issue, and the ldd follow-up was designed to scope the fix comprehensively.

Significance

Message <msg id=615> is significant because it represents a transition from reactive debugging to proactive system verification. The earlier build fixes (pip, linker path) were reactive — each was triggered by a build failure. But this message is proactive: the build succeeded, yet the assistant chose to investigate further, uncovering a set of issues that would have caused cascading failures in production.

The message also demonstrates a principle that applies broadly in systems engineering: when you find one problem, look for others. The libconfig++ error was the first symptom the dynamic linker encountered, but it was not the only missing library. By running ldd, the assistant found all five at once, enabling a single coordinated fix rather than five separate iterations of "install library, rebuild, test, find next error."

This approach saved an enormous amount of time. Without it, the assistant would have fixed libconfig++, rebuilt, tested, discovered libaio, fixed that, rebuilt, tested, and so on through five cycles. Instead, one ldd command compressed all five discoveries into a single moment, and the next edit could install all four system libraries at once.

The message also reveals an understanding of the boundary between what the container should provide and what the host must provide. The assistant immediately recognized that libcuda.so.1 belongs to a different category — it is a host-provided library that must be mounted from the NVIDIA driver at container runtime, not something to install in the image. This distinction is crucial for correct container design.

Conclusion

Message <msg id=615> is a masterclass in diagnostic methodology. In a few short lines, the assistant demonstrates the difference between fixing symptoms and understanding system state. The ldd command is not complex — it is a standard Linux tool — but the decision to run it at this moment, before making any changes, is what separates a thorough engineer from a superficial one. The message transformed a single library error into a complete dependency audit, set up a clean fix for four system libraries, correctly identified the CUDA driver as a host-level concern, and established a pattern of comprehensive verification that would prevent countless future iterations.

In the broader arc of the session, this message is the moment the Docker build went from "it compiles" to "it will actually run." That distinction is everything in production systems.