The Final Smoke Test: When "Expected" Failures Reveal Deeper Truths About Dockerized GPU Applications

Introduction

In the course of building a production-grade Docker container for the Curio/CuZK proving system, the assistant reached a pivotal moment of verification. Message 621 captures a seemingly mundane smoke test — checking which shared libraries are missing and attempting to run the compiled binaries. But beneath this routine check lies a rich tapestry of reasoning about dependency management, runtime expectations, and the boundary between build-time and run-time environments in GPU-accelerated containerized applications.

The message reads in full:

[assistant] Only libcuda.so.1 is missing, which is expected — that's provided by the NVIDIA driver at runtime via --gpus all. Let me verify the binaries can at least show their version/help: [bash] docker run --rm --entrypoint bash curio-cuzk:latest -c "curio --version 2>&1; echo '---'; sptool --version 2>&1; echo '---'; cuzk --help 2>&1 | head -10" curio: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory --- sptool: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory --- cuzk: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

The Context: A Long Road of Build Blockers

To understand why this message was written, one must appreciate the journey that preceded it. The assistant had been iterating through a cascade of build failures while constructing a Docker image for Curio, a Filecoin proving system that depends on CUDA-accelerated GPU computation. The build had already survived multiple crises:

• A pip conflict where the Debian-managed python3-pip package interfered with the venv-based pip upgrade in the supraseal build script, causing a RECORD file ownership error. The fix was to simply not install python3-pip at all, relying on ensurepip to bootstrap pip inside the virtual environment.
• A linker error where libcudart_static.a could not be found because /usr/local/cuda/lib64 was missing from the LIBRARY_PATH environment variable during the Go build step.
• Missing runtime libraries (libconfig++, libaio, libfuse3, libarchive) that caused curio to crash immediately after the build succeeded. Each of these had been diagnosed, fixed, and verified. The Docker build now completed successfully, producing three binaries (curio at 163MB, cuzk at 27MB, and sptool at 210MB). Message 621 represents the moment when the assistant turns from building to verifying — the crucial transition from "does it compile?" to "does it work?"

The Verification Strategy: What the Assistant Was Trying to Learn

The assistant's approach reveals a methodical testing philosophy. Rather than simply declaring victory after the build succeeded, it performed a layered verification:

1. Library dependency check: Using ldd to identify which shared libraries are missing from the runtime environment.
2. Functional smoke test: Attempting to run each binary with basic flags (--version, --help) to confirm they can at least initialize and respond to CLI commands. The ldd check (performed in the immediately preceding messages, not shown in message 621 itself but referenced by it) had revealed that only libcuda.so.1 was missing — all other runtime libraries (libconfig++, libaio, libfuse3, libarchive) had been successfully installed in the runtime stage. This was a significant achievement, representing the resolution of the earlier missing-library errors. The assistant then made a reasoned judgment: libcuda.so.1 is the NVIDIA CUDA driver library, which is provided by the host system's GPU driver and exposed to containers via the --gpus all Docker flag. It should not be bundled in the image itself. This was correct reasoning — bundling the CUDA driver library would create a fragile, host-dependent image that would break across different driver versions.

The Assumption That Collided with Reality

Here is where the message becomes particularly instructive. The assistant made a subtle but consequential assumption: that the binaries could perform basic CLI operations (printing version information or help text) without loading the CUDA library. The reasoning was that --version and --help are typically implemented as early-return operations that don't require full initialization of the application's dependencies.

This assumption is reasonable in many software contexts. Most well-designed CLI tools can print their version or help text without loading their heavy dependencies. However, in the case of these CUDA-accelerated binaries, the assumption proved incorrect. All three binaries — curio, sptool, and cuzk — immediately crashed with error while loading shared libraries: libcuda.so.1: cannot open shared object file before even reaching the version-printing code.

The failure mode is telling: the dynamic linker resolves all shared library dependencies at process startup, before main() is even called. If libcuda.so.1 cannot be found, the kernel's dynamic linker (ld-linux-x86-64.so.2) refuses to start the process at all. The binary never gets a chance to run its --version handler. This is a fundamental aspect of how ELF binaries work on Linux — lazy binding can defer some symbol resolution, but the library itself must be loadable.

The Thinking Process Revealed

The assistant's reasoning in this message demonstrates several cognitive layers:

Layer 1: Diagnostic confidence. The assistant correctly interprets the ldd output as a success condition — only libcuda.so.1 is missing, and that's expected. The phrase "Only libcuda.so.1 is missing, which is expected" shows that the assistant has internalized the architecture of NVIDIA containerization: the CUDA runtime library comes from the host driver, not the container image.

Layer 2: Verification instinct. Despite the expected nature of the missing library, the assistant doesn't stop at ldd. It pushes further: "Let me verify the binaries can at least show their version/help." This reveals a healthy skepticism about static analysis — ldd tells you about library dependencies, but it doesn't tell you whether the binary actually works. The assistant wants dynamic verification.

Layer 3: The hopeful assumption. The word "at least" in "at least show their version/help" is revealing. It suggests the assistant is setting a low bar for success, anticipating that full functionality won't work without CUDA, but hoping that basic CLI operations might. This is a reasonable heuristic from general software experience, but it collides with the specific behavior of dynamically-linked CUDA binaries.

Input Knowledge Required to Understand This Message

To fully grasp what is happening in message 621, a reader needs several pieces of background knowledge:

1. The ELF dynamic linking model: Understanding that shared library resolution happens at process startup, before any application code runs. This explains why --version fails even though it shouldn't logically need CUDA.
2. NVIDIA's container runtime architecture: Knowledge that libcuda.so.1 is provided by the NVIDIA driver on the host and exposed to containers via --gpus all (or --runtime=nvidia with the NVIDIA Container Toolkit). The CUDA devel and runtime Docker images include the CUDA toolkit libraries but not the kernel-mode driver library.
3. The build history: Awareness that the assistant had just resolved three significant build blockers (pip, linker, runtime libs) and was performing final validation. Without this context, the message reads as a simple failure report rather than a deliberate verification step.
4. The nature of the binaries: Understanding that curio, sptool, and cuzk are GPU-accelerated proving binaries that link against CUDA at compile time, making libcuda.so.1 a hard dependency even for trivial operations.

Output Knowledge Created by This Message

This message produces several valuable pieces of knowledge:

1. Confirmation that all non-CUDA runtime dependencies are resolved. The fact that only libcuda.so.1 shows as missing means that libconfig++, libaio, libfuse3, and libarchive are all properly installed in the runtime image. This is a significant validation of the earlier fixes.
2. Evidence that the binaries are correctly linked. The binaries crash with a clean "cannot open shared object file" error rather than a segfault or undefined symbol error, indicating that the build process produced valid ELF binaries with correct symbol resolution.
3. A constraint discovery: The assistant learns that these binaries require CUDA even for basic CLI operations. This is an important operational constraint — it means that any testing or debugging of the container must be done on a GPU-equipped host with the NVIDIA container runtime properly configured.
4. A verification protocol gap: The smoke test reveals that --version and --help cannot serve as pre-GPU sanity checks. Alternative verification strategies are needed, such as checking binary metadata with readelf or objdump, or running the container with --gpus all from the start.

Mistakes and Incorrect Assumptions

The primary incorrect assumption in this message is that --version and --help would work without CUDA. This assumption, while reasonable in many software contexts, fails for CUDA-linked binaries due to the dynamic linker's eager library resolution.

However, it is important to characterize this correctly: it is not a mistake in the sense of a wrong action, but rather an assumption that was tested and falsified. The assistant's methodology of testing the assumption is precisely correct — rather than assuming the binaries work and moving on, it verifies and discovers the constraint. In scientific terms, this is a well-designed experiment that produced a clear negative result.

A secondary observation: the assistant could have predicted this outcome by checking whether the binaries were linked with DT_NEEDED entries for libcuda.so.1 using readelf -d. The ldd output already showed this, but the assistant may have underestimated the strictness of the dynamic linker's enforcement. On Linux, ldd reports libraries as "not found" when they cannot be located, and the dynamic linker will refuse to start the binary under the same conditions.

Broader Significance

This message, while brief, encapsulates a fundamental tension in building containerized GPU applications: the boundary between what belongs in the image and what comes from the host. The assistant's correct identification of libcuda.so.1 as a host-provided library, combined with the failed attempt to run the binaries without it, illustrates the two-sided nature of GPU containerization:

• The image provides the application, its non-GPU dependencies, and the CUDA toolkit libraries (e.g., libcudart, libcublas).
• The host provides the kernel-mode driver and the CUDA runtime library (libcuda.so.1) through the NVIDIA Container Toolkit. This separation is by design — it allows the same container image to run across hosts with different GPU driver versions, as long as the driver is compatible with the CUDA toolkit version used during compilation. The assistant's verification confirms that this separation is correctly implemented: the image contains everything it should, and the only missing piece is the host-provided component.

Conclusion

Message 621 captures a moment of transition from build-time success to runtime verification. The assistant's methodical approach — checking dependencies, forming hypotheses, testing assumptions, and learning from results — exemplifies good engineering practice. The "failure" to run --version without CUDA is not a setback but a discovery: it reveals the strict dependency model of CUDA-linked binaries and confirms that the Docker image is correctly constructed for its intended deployment environment. The next logical step, which the assistant would go on to take, is to run the container with --gpus all on a GPU-equipped host, where libcuda.so.1 will be available and the true functional test can begin.