The Last Research Step: How a Docker Hub Search Unlocked a Multi-Stage CUDA 13 Container Build

The Message

The subject of this analysis is message index 552 in an opencode coding session, a brief but strategically pivotal message from the AI assistant:

[assistant] Good. Now let me check what CUDA 13 Docker base images exist: [exa_web_search_exa] {"numResults":5,"query":"nvidia/cuda:13.0 docker hub base image devel"}

On its surface, this is a trivial action: a web search for a Docker image tag. But in the context of the broader session — a multi-hour effort to containerize a deeply heterogeneous proof-of-spacetime proving stack for the Filecoin Curio project — this message represents the final research gate before committing to a complex multi-stage Dockerfile design. Understanding why this search was necessary, what assumptions it validated, and what it enabled downstream reveals much about the discipline required to build reproducible infrastructure for cutting-edge cryptographic proving systems.

Context: The Containerization Problem

To appreciate why a simple Docker Hub search matters, one must understand what was being built. The Curio project is a Filecoin storage mining implementation that includes a custom zero-knowledge proving engine called "cuzk." This engine performs GPU-accelerated SNARK proofs for several proof types: PoRep (Proof of Replication), WinningPoSt, WindowPoSt, and SnapDeals. The proving stack is a tower of heterogeneous technologies: Go (the main Curio binary), Rust (the cuzk daemon and bellperson library), C++ (the supraseal-c2 CUDA kernels), and raw CUDA C++ (the groth16 prover). Each layer has its own build system, compiler requirements, and runtime dependencies.

The session's goal was to produce a Docker container that bundles all of these components together with CUDA 13 support, targeting mainnet 32 GiB proving. The container would be published to Docker Hub for distribution. The user had explicitly chosen CUDA 13 as the base image (responding "cuda13 is a thing now" when the assistant asked about base image versions), and had decided that proving parameters should be fetched at runtime via an entrypoint script rather than baked into the image — a pragmatic choice given that the parameters alone are approximately 100 GB for 32 GiB sector sizes.

Why This Message Was Written

The assistant had spent the preceding messages methodically gathering information. It read the existing OpenCL-based Dockerfile at /tmp/czk/Dockerfile to understand the project's Docker patterns. It read the build makefiles (00-vars.mk, 10-deps.mk, 30-build.mk) to understand the build flags and dependency chain. It read the GitHub Actions dependency list to catalog every system package required. It read the supraseal build.sh to understand the CUDA compilation requirements, including GCC version constraints and architecture targets. It even checked the curio fetch-params command implementation to understand how parameter downloading works.

By message 551, the assistant had gathered enough information to begin writing the Dockerfile — except for one critical detail: the exact name of the CUDA 13 base image. The assistant knew that NVIDIA publishes CUDA Docker images under the nvidia/cuda repository on Docker Hub, with tags following a pattern like <cuda-version>-<flavor>-<os-version>. But CUDA 13 was relatively new at the time of this session (the user had described it as "a thing now"), and the assistant had not yet confirmed the exact tag format. Writing a Dockerfile that references a non-existent base image would produce a build failure, wasting time and obscuring the real errors.

This search was therefore a risk-reduction measure. Rather than guessing the tag (e.g., nvidia/cuda:13.0.0-devel-ubuntu24.04 or nvidia/cuda:13.0-devel-ubuntu24.04), the assistant chose to verify empirically. The query was carefully crafted: "nvidia/cuda:13.0 docker hub base image devel" with numResults:5. The inclusion of "devel" in the query is significant — the assistant needed the development image (which includes the full CUDA toolkit with nvcc, headers, and libraries) for the build stage, as opposed to the runtime or base images which only include the shared libraries needed for execution.

The Decision-Making Process

The assistant's decision to search rather than assume reflects a deeper architectural consideration. The multi-stage Dockerfile design had already been mentally planned: Stage 1 would use a CUDA 13 devel image as the foundation, install Go 1.24 and Rust 1.86.0 toolchains alongside gcc-13, build the filecoin-ffi CGO library, the supraseal CUDA library, the curio Go binary, and the cuzk Rust/CUDA daemon. Stage 2 would use a CUDA 13 runtime image (much smaller, without the build toolchain) and simply copy the compiled binaries from Stage 1.

This design depends critically on both images sharing the same CUDA version and Ubuntu base, because the runtime image must provide the same CUDA driver libraries that the binaries were linked against in the devel image. A mismatch would produce runtime linking errors. By confirming the exact tags for both devel and runtime variants, the assistant could ensure consistency across stages.

The search also implicitly validated an assumption about CUDA 13's availability. CUDA toolkit versions are tied to NVIDIA driver versions, and not all CUDA versions have Docker images published. If CUDA 13.0 had not yet been published to Docker Hub, the assistant would have needed to either use a different base image (perhaps building from a CUDA 13 installer script) or fall back to CUDA 12.8. The search confirmed the existence of the images, removing that uncertainty.

Assumptions Embedded in the Search

Several assumptions are baked into this seemingly simple query. First, the assistant assumed that the Docker Hub image naming convention for CUDA 13 follows the same pattern as earlier CUDA versions. This is a reasonable assumption — NVIDIA has maintained consistent naming for years — but it is not guaranteed. A new CUDA major version could have introduced a new tag scheme.

Second, the assistant assumed that the Ubuntu 24.04 variant would be the appropriate base. This assumption was informed by the existing Dockerfile, which used golang:1.24-trixie (Trixie is Debian 13, the testing branch) and rust:1.86.0-slim-bookworm (Bookworm is Debian 12). The project's existing build environment was Debian-based, and Ubuntu 24.04 (Noble) is derived from Debian testing, making it a natural choice for compatibility with the package versions expected by the build scripts.

Third, the assistant assumed that the search tool (exa_web_search_exa) would return sufficient information to identify the correct tags. The search result snippet shows the Docker Hub page title and description, but not the full tag list. The assistant would need to either click through to the tags page or make a follow-up query. Indeed, in the next message ([msg 553]), the assistant does exactly that, searching for the specific runtime tag.

Input Knowledge Required

To understand this message, one must be familiar with several domains. Docker multi-stage build patterns are essential: the distinction between devel, runtime, and base CUDA images, and why a build stage needs the full toolkit while the final image only needs the runtime libraries. The NVIDIA CUDA Docker image ecosystem must be understood: that nvidia/cuda is the official repository, that tags encode version-flavor-OS triples, and that the devel image can be 3-5 GB while the runtime image is under 1 GB.

One must also understand the project context: that Curio is a Filecoin implementation, that cuzk is a GPU proving engine, that the build requires specific GCC versions (gcc-13) and Rust versions (1.86.0), and that the proving parameters are too large to bake into the image. The assistant's earlier message ([msg 546]) provides this context in detail, including the complete list of build dependencies and the multi-stage design plan.

Output Knowledge Created

The search produced a concrete result: confirmation that nvidia/cuda:13.0.2-devel-ubuntu24.04 and nvidia/cuda:13.0.2-runtime-ubuntu24.04 exist on Docker Hub. This is the output knowledge that directly enables the next action. In the following message ([msg 554]), the assistant states "Good. nvidia/cuda:13.0.2-devel-ubuntu24.04 and nvidia/cuda:13.0.2-runtime-ubuntu24.04 exist. Now I have everything I need. Let me write both files."

The specific version 13.0.2 is notable — it indicates that CUDA 13.0 had already received at least two patch releases by the time of this session. The Ubuntu 24.04 variant confirms the OS choice. The existence of both devel and runtime images with matching version numbers validates the multi-stage design.

The Thinking Process Visible in the Message

The message begins with "Good." — a discourse marker indicating that the assistant has completed its previous line of inquiry and is transitioning to the next step. The preceding message ([msg 551]) involved checking the directory structure of /tmp/czk/docker/ and reading the fetch-params command implementation. The assistant had confirmed that the docker/curio/ directory exists with an existing Dockerfile and entrypoint, and that the fetch-params command accepts a sector size argument.

The phrase "Now let me check what CUDA 13 Docker base images exist" reveals the assistant's mental model: it has a checklist of information items needed before writing the Dockerfile, and this is the last unchecked item. The use of "check" rather than "find" or "search" suggests verification of an expected value rather than open-ended discovery. The assistant likely expected the images to exist and was confirming the exact tag strings.

The tool call itself — exa_web_search_exa — is a web search tool that returns summarized results from the web. The assistant configured it with numResults:5 to get a manageable set of results, and the query explicitly includes "docker hub" and "base image devel" to narrow the search to the specific image flavor needed for the build stage.

Significance in the Larger Narrative

This message, while brief, sits at the inflection point between research and implementation in the session's narrative arc. The assistant had spent approximately 10 messages reading reference files, understanding build systems, and gathering requirements. This search was the final research action. Immediately after confirming the image tags, the assistant creates the directory structure (mkdir -p /tmp/czk/docker/cuzk), writes the Dockerfile and entrypoint script, and initiates the build.

The build would later encounter environment-specific blockers — a missing jq dependency, a missing libcuda.so.1 symlink, Python PEP 668 restrictions — but the base image choice was validated and never caused issues. The search thus served its purpose: eliminating uncertainty about the foundation of the entire container.

In the broader context of the session's themes — implementing PCE extraction, fixing GPU routing, debugging WindowPoSt crashes, and containerizing the proving stack — this message represents the quiet, unglamorous work of infrastructure engineering: verifying assumptions before committing to a design. It is the kind of step that experienced engineers learn to take after being burned by incorrect assumptions about base image availability, version numbering schemes, or OS compatibility. The assistant's decision to search rather than guess reflects a disciplined approach to system building that values verification over speed.