The Smoke Test That Exposed Hidden Dependencies: A Docker Runtime Debugging Case Study

In the course of building a complex Docker image for the Curio/CuZK Filecoin proving stack, a single smoke-test command revealed a critical gap between build-time success and runtime viability. Message <msg id=614> captures the moment when the assistant, having just completed a multi-hour Docker build with three compiled binaries, ran a seemingly trivial verification command — and was confronted with a cascade of missing shared library errors. This message is a masterclass in the principle that "it compiles" is never the same as "it works," and it demonstrates the kind of iterative debugging that characterizes real-world infrastructure engineering.

The Message in Full

The message reads:

Let me run a basic smoke test — check that the binaries work and the entrypoint runs: `` docker run --rm curio-cuzk:latest -c "curio --version && echo '---' && sptool --version && echo '---' && cuzk --help 2>&1 | head -5 && echo '---' && echo 'All binaries OK'" Proving parameters not found in /var/tmp/filecoin-proof-parameters — fetching 32GiB params... This will download ~100GB and may take a while on first run. curio: error while loading shared libraries: libconfig++.so.9: cannot open shared object file: No such file or directory ``

The output is devastatingly succinct. The container's entrypoint kicked in (fetching proving parameters), but the curio binary itself failed immediately with a dynamic linker error. The smoke test did not even get to sptool or cuzk.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must understand what preceded it. The assistant had been working for hours — across multiple segments spanning dozens of messages — to build a Docker image for a specialized Filecoin proving system. The image needed to contain three binaries: curio (the main proving node), cuzk (the GPU-accelerated proving engine), and sptool (a supporting tool). The build had been plagued by blockers: a Python pip conflict caused by Debian's system package management, a missing libcudart_static.a linker error that required adjusting LIBRARY_PATH, and the general complexity of compiling the Filecoin FFI with CUDA supraseal support.

Message <msg id=613> immediately preceding the smoke test, declared victory: "The build succeeded! All three binaries are built." The assistant then updated a todo list, marking "Complete the Docker build successfully" as done and "Test the container (basic smoke test)" as in progress. This transition from "build succeeded" to "test the container" is the precise motivation for message <msg id=614>. The assistant needed to validate that the artifact of all this effort — the Docker image — actually functioned. A smoke test is the minimum viable verification: check that the binaries run, that they report their versions, that the entrypoint script doesn't crash.

The underlying assumption was that if the binaries compiled and linked successfully against their build-time dependencies, they would run in the runtime image. This assumption proved incorrect.

The Assumptions Embedded in the Smoke Test

The smoke test command itself reveals several assumptions the assistant was making:

First assumption: The runtime image has all necessary shared libraries. The Docker image uses a multi-stage build: a devel stage (based on nvidia/cuda:13.0.2-devel-ubuntu24.04) where compilation happens, and a runtime stage (based on nvidia/cuda:13.0.2-runtime-ubuntu24.04) that copies only the binaries. The assistant assumed that the runtime CUDA image, combined with the explicitly installed runtime packages (libhwloc15, libnuma1, libssl3, etc.), would cover all library dependencies. It did not.

Second assumption: The binaries would at least start. The command chain curio --version && ... && sptool --version && ... assumes each binary will successfully execute and exit with code zero. The && chaining means that if curio --version fails, the entire command stops — which is exactly what happened.

Third assumption: The entrypoint script would not interfere. The assistant ran the container without overriding the entrypoint, meaning the Dockerfile's ENTRYPOINT script executed first. That script (which handles parameter fetching) ran successfully — it detected missing parameters and began downloading. But then it tried to invoke curio, which crashed. The assistant likely assumed the entrypoint would gracefully handle missing binaries or libraries, or that the parameter check would not trigger on a simple version query.

Fourth assumption: The build-time linker path configuration was complete. The assistant had already fixed a libcudart_static.a linker error by adding /usr/local/cuda/lib64 to LIBRARY_PATH. This fixed the static link step, but the assistant may have implicitly assumed that dynamic linking at runtime would also be resolved — either by the runtime image's ld.so configuration or by LD_LIBRARY_PATH settings. Neither was configured.

The Mistake: Build-Time vs. Runtime Library Separation

The core mistake was a classic Docker multi-stage build pitfall: build-time dependencies are not automatically runtime dependencies. The devel stage has the full CUDA toolkit, development headers, static libraries, and build tools. The runtime stage is deliberately minimal — it only has the CUDA runtime libraries, not the development libraries. But the binaries, especially curio, were linked against shared libraries that exist only in the devel stage's apt installation.

Specifically, libconfig++.so.9 is the C++ configuration library. It was likely pulled in as a build dependency during compilation (perhaps by the Filecoin FFI or by Curio itself), but it was never explicitly installed in the runtime stage. The assistant had installed libconfig++ development packages in the devel stage for compilation, but the corresponding runtime library package (libconfig++9 or similar) was missing from the runtime stage's apt-get install list.

This is not a trivial oversight. The runtime stage's package list (visible in message <msg id=617> when the assistant reads the Dockerfile) includes libhwloc15, libnuma1, libssl3, and other libraries, but not libconfig++, libaio, libfuse3, or libarchive. The assistant had curated this list based on known dependencies, but had not done a systematic audit of what the compiled binaries actually need.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs:

Understanding of Docker multi-stage builds. The concept that a Docker image can have a "devel" stage for compilation and a separate "runtime" stage for execution, and that libraries available during compilation are not automatically present at runtime.
Knowledge of dynamic linking on Linux. The error "error while loading shared libraries" is produced by the dynamic linker (ld.so) when it cannot find a required .so file in its search path (LD_LIBRARY_PATH, /etc/ld.so.conf, or standard paths like /usr/lib and /lib).
Familiarity with the Filecoin/CuZK ecosystem. Understanding that curio is a node software that manages proving tasks, sptool is a sector-tool utility, and cuzk is the GPU proving engine. The fact that the entrypoint tried to fetch 32GiB parameters indicates this is a production system dealing with large Filecoin proof parameters (~100GB download).
Context from the preceding build fixes. The pip conflict resolution and the libcudart_static.a linker fix are necessary background — they show that the assistant was already in a debugging mindset, having just overcome two significant build blockers.
Knowledge of the ldd tool. The assistant would use ldd in the next message to discover all missing libraries, which is the standard tool for diagnosing dynamic linking failures.

Output Knowledge Created by This Message

This message created several critical pieces of knowledge:

First, it identified a specific missing library: libconfig++.so.9. This is actionable information — the assistant can now install the corresponding runtime package in the Dockerfile's runtime stage.

Second, it revealed that the entrypoint script runs before the binaries are tested. The output shows "Proving parameters not found... fetching 32GiB params..." before the crash. This means the entrypoint's parameter-fetch logic executed, which is both good (the entrypoint works) and potentially problematic (it triggered a large download that may be unnecessary for a smoke test). The assistant might need to override the entrypoint for future tests.

Third, it demonstrated that the smoke test methodology was insufficient. The && chaining meant that a single failure aborted the entire test. The assistant learned nothing about sptool or cuzk from this run. A more robust test might use ; separators or check exit codes individually.

Fourth, it validated that the build itself was sound. The binaries were compiled, linked, and present in the image at /usr/local/bin/. The failure was purely at the dynamic linking stage, not a compilation or packaging error. This is an important distinction — the assistant could rule out build-system issues and focus on runtime packaging.

Fifth, it exposed the need for a systematic dependency audit. The assistant would follow up in message <msg id=615> by running ldd on all binaries to find every missing library at once, rather than discovering them one by one. This is a more efficient approach that the smoke test failure motivated.

The Thinking Process: What the Assistant Was Reasoning

The assistant's thinking, visible in the structure of the command and the todo list update, reveals a methodical approach:

Confidence from build success. The assistant had just confirmed all three binaries were built. The natural next step is to verify they run. This is standard engineering practice — don't declare victory until you've tested the artifact.
Choosing the simplest possible test. Rather than writing a complex integration test or spinning up a proving environment, the assistant chose the minimal verification: --version and --help flags. These require no configuration, no network, no GPU, and should work in any environment. If they fail, something fundamental is wrong.
Using docker run without overriding the entrypoint. This was a deliberate choice — it tests the container as it would be used in production. The entrypoint is part of the container's interface, and testing it is important. However, it also introduced an unexpected variable (the parameter download) that complicated the output.
The todo list transition. The assistant updated the todo list from "Complete the Docker build successfully" (done) to "Test the container (basic smoke test)" (in progress). This shows a clear state machine: build → test → deploy. The smoke test is the gate between build and deployment.
Interpreting the error correctly. The assistant immediately recognized the "cannot open shared object file" error as a dynamic linker issue, not a binary corruption or missing file. In the next message, the assistant would run ldd to enumerate all missing libraries, demonstrating a systematic debugging approach.

Broader Implications for Infrastructure Engineering

This single message illustrates several enduring lessons for anyone building containerized applications:

The "works on my machine" fallacy applies to Docker build stages. Just because a binary compiles in the devel stage does not mean it runs in the runtime stage. The two stages have different library sets, and the linker resolves dependencies differently at build time (with -L flags and LIBRARY_PATH) than at runtime (with LD_LIBRARY_PATH and ld.so configuration).

Smoke tests should be designed to survive partial failure. The && chaining was brittle. A better approach would be to run each binary independently and collect all results, or use a script that reports individual pass/fail status. The assistant learned this and would adapt in subsequent tests.

Multi-stage Dockerfiles require explicit runtime dependency management. Unlike single-stage builds where everything is in one image, multi-stage builds force you to curate the runtime environment. This is a feature, not a bug — it produces smaller, more secure images — but it requires diligence. Every shared library linked by the binaries must be either statically linked, present in the runtime base image, or explicitly installed.

The entrypoint is part of the test surface. By not overriding the entrypoint, the assistant tested the full container interface. This was valuable — it revealed that the entrypoint's parameter-fetch logic triggers on every run, which might be undesirable for non-proving invocations like --version. A future improvement might be to make the entrypoint skip parameter fetching for informational commands.

Conclusion

Message <msg id=614> is a small moment in a long debugging session, but it encapsulates a universal truth of systems engineering: building is not the same as shipping. The Docker build succeeded, the binaries were compiled, the image was tagged — but the container could not run its primary binary. The smoke test, in its failure, provided precisely the information needed to fix the problem. It identified a specific missing library, validated the build process, and motivated a systematic dependency audit. In the next messages, the assistant would discover four more missing libraries (libaio, libfuse3, libarchive, libcuda) and fix them all, turning a failed smoke test into a complete, working Docker image.

The lesson is timeless: test your artifacts. A five-second smoke test saved hours of potential debugging in a production environment where this missing library would have manifested as a mysterious crash with an unhelpful error message. The assistant's methodical approach — build, test, diagnose, fix — is the pattern that separates robust infrastructure from fragile prototypes.