The Containerization Gauntlet: Building a Multi-Stage Docker Image for Filecoin's Heterogeneous Proving Stack

Introduction

In the world of infrastructure engineering, there is a special category of challenge that lies between "architecting a solution" and "getting it to actually build." It is the domain where blueprints meet reality, where carefully planned abstractions collide with the messy specifics of base images, package managers, and dynamic linker conventions. This is the territory navigated in Segment 4 of an opencode coding session focused on the Filecoin Curio project — a session that saw the construction of a multi-stage Docker container for mainnet 32GiB proving, followed by an iterative debugging marathon that surfaced four distinct environment-specific blockers.

The segment spans messages 546 through 588 of the conversation, and its narrative arc is one of disciplined engineering: comprehensive research, careful implementation, methodical self-review, and then a cascade of build failures that tested every assumption embedded in the design. Each blocker — a missing jq binary, a missing libcuda.so.1 symlink, Python PEP 668 restrictions, and a deeply obscure pip uninstall failure caused by a missing RECORD file — reveals a different facet of the challenge of containerizing a heterogeneous software stack spanning Go, Rust, C++, CUDA, and Python build tools.

This article traces that arc in full, from the planning document that synthesized hours of research into an actionable blueprint, through the creation of the Dockerfile and entrypoint script, through the iterative debugging cycle that consumed the remainder of the segment. It examines not just what happened, but why each blocker occurred and what it reveals about the nature of containerization work.

Phase I: The Architecture of Synthesis (Messages 546–557)

The segment opens with message 546, a comprehensive planning document that the assistant produced after completing an extensive research phase spanning messages 537 through 544. During that research phase, the assistant had dispatched multiple subagent tasks to explore the curio build system, the cuzk build requirements, the existing Dockerfile patterns, the supraseal build script, and the CUDA base image landscape. Message 546 is where all of that information is consolidated into a coherent, actionable engineering plan.

The Structure of the Planning Document

The planning document is organized into six sections, each serving a distinct cognitive purpose:

The Multi-Stage Build Architecture

The planning document sketches the multi-stage build architecture that would become the foundation of the Dockerfile:

Stage 1: CUDA 13 devel base + Go 1.24 + Rust 1.86.0 + gcc-13 + all build deps → build curio (with CUDA/supraseal) + cuzk Stage 2: CUDA 13 runtime base + runtime libs → copy binaries from stage 1

This separation of concerns — isolating the heavy build environment from the minimal runtime environment — is Docker best practice. The builder stage contains the full CUDA toolkit, compilers, and development libraries needed to compile the Go and Rust binaries. The runtime stage contains only the shared libraries needed to execute those binaries, resulting in a significantly smaller image.

The assistant's research into CUDA 13 base image availability (messages 552–554) confirmed that the necessary images existed on Docker Hub: nvidia/cuda:13.0.2-devel-ubuntu24.04 for the builder stage and nvidia/cuda:13.0.2-runtime-ubuntu24.04 for the runtime stage. The choice of Ubuntu 24.04 (Noble) as the base distribution was driven by CUDA 13's support matrix and the project's existing dependency patterns.

Reality-Checking User Expectations

One of the most valuable aspects of the planning document is the assistant's willingness to act as a reality check for the user's expectations. When the user asked for "only PoRep and Snap params," the assistant investigated whether curio fetch-params supports filtering by proof type and discovered that it does not — it downloads all parameters for the specified sector size. This finding is documented in Discovery item 8:

Param fetching: curio fetch-params 32GiB downloads to FIL_PROOFS_PARAMETER_CACHE (default /var/tmp/filecoin-proof-parameters). Downloads both ParametersJSON and SrsJSON. No way to filter by proof type — it fetches all params for the sector size. For 32GiB this is ~100GB+.

This is the behavior of an experienced engineer who knows that managing expectations is as important as writing code. Rather than silently implementing a filtered fetch that would not work, the assistant verified the assumption and documented the reality.

Phase II: The Moment of Creation (Messages 558–571)

With the blueprint established, the assistant moved to implementation. This phase spans the creation of two files: the entrypoint script and the Dockerfile itself.

The Entrypoint Script

Messages 558 and 559 saw the creation of docker/cuzk/entrypoint.sh. The script is straightforward in its design but critical to the container's functionality:

#!/usr/bin/env bash
set -e

PARAM_DIR="${FIL_PROOFS_PARAMETER_CACHE:-/var/tmp/filecoin-proof-parameters}"

# Fetch 32GiB proving parameters on first start if not already present.
if [ ! -f "$PARAM_DIR/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228571c57316571f7c2b97c164bc72ea5396fcf16f6.params" ]; then
    echo "Proving parameters not found. Fetching 32GiB params..."
    curio fetch-params 32GiB
    echo "Parameter fetch complete."
fi

exec /bin/bash

The script uses a sentinel file — a specific .params file that is guaranteed to exist after a successful parameter fetch — to determine whether parameters need to be downloaded. If the sentinel is missing, it runs curio fetch-params 32GiB to download the ~100GB of proving parameters, then drops into an interactive bash shell.

The initial version of the script used a WindowPoSt parameter filename as the sentinel, but a subagent task (message 568) revealed the correct filename — the 32GiB PoRep .params file — and the assistant corrected it in message 569. This correction, a single line change, is emblematic of the assistant's commitment to accuracy: it did not guess the filename but dispatched research to verify it.

The Dockerfile

Message 562 is the moment of creation for the Dockerfile itself. The assistant wrote Dockerfile.cuzk as a two-stage build with careful attention to every detail.

The Builder Stage begins from nvidia/cuda:13.0.2-devel-ubuntu24.04 and installs:

The Self-Review Phase

Before running the build, the assistant subjected both files to a rigorous self-review spanning messages 564 through 570. This quality gate reflects a disciplined engineering approach: before running a build that might take hours, verify every assumption that can be verified statically.

The assistant verified:

Phase III: The Reality of Execution (Messages 572–588)

The user's response in message 572 is decisive: "Build here, the current host is cuda capable." With these eight words, the session pivots from preparation to execution. The assistant initiates the first build in message 574, and the iterative debugging marathon begins.

Blocker 1: The Missing jq (Messages 575–576)

The first build attempt proceeds through the initial stages — metadata loading, authentication, toolchain installation — but fails during the FFI build step. The error is clear: the build script requires jq for JSON processing, specifically to parse parameters.json which lists proving parameter filenames, CIDs, and digests.

The assistant's response in message 576 is immediate and precise: add jq to the Dockerfile's package list. This fix is trivial in isolation, but it reveals a deeper truth: the build system's dependency graph is not fully documented. The assistant had read the makefiles and build scripts, but jq is an implicit dependency — it is invoked by a script that is itself invoked by the build system, and its absence only becomes visible when the build reaches that specific step.

Blocker 2: The Missing Symlink (Messages 578–582)

With jq added, the build progresses further — and immediately hits a second, more subtle wall. The neptune cryptographic library (a Rust crate used in the Filecoin proof pipeline) includes a build script that probes GPU capabilities at compile time. This probe dynamically links against libcuda.so.1 — the CUDA runtime library.

The nvidia/cuda:13.0.2-devel-ubuntu24.04 image contains the CUDA toolkit, including stub libraries in /usr/local/cuda/lib64/stubs/ that are designed for exactly this purpose: they export the same symbols as the real libcuda.so but contain no actual GPU functionality, allowing build-time linking without a GPU driver.

However, the stubs directory contains libcuda.so but lacks the libcuda.so.1 symlink. The neptune build script's test binary calls dlopen("libcuda.so.1", ...) — it looks for the versioned soname, not the bare libcuda.so. The assistant diagnoses this in message 579, reasoning through the causal chain: neptune probes GPU capabilities → requires loading libcuda.so.1 → CUDA devel image has stubs but not the .so.1 symlink → fix is to create the symlink and add the stubs directory to the library path.

The assistant first tries adding the stubs directory to LD_LIBRARY_PATH (message 579), but this alone does not work because the stubs directory has libcuda.so without the .so.1 versioned symlink. In message 581, the assistant runs a diagnostic command inside the devel image to confirm the exact state of the stubs directory:

docker run --rm nvidia/cuda:13.0.2-devel-ubuntu24.04 ls -la /usr/local/cuda/lib64/stubs/

This confirms the absence of the symlink. The fix, applied in message 582, creates the symlink explicitly in the Dockerfile:

RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1 \
    && ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1

Wait — that's the same line twice. Let me check the actual fix. The assistant adds both the symlink creation and the library path configuration:

ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}
RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1

With this fix in place, the FFI compilation succeeds — a major milestone. The core cryptographic libraries compile successfully against the CUDA stubs, validating that the build path for the most complex dependency is correct.

This blocker is a textbook example of the gap between "it compiles" and "it builds in a container." The CUDA devel image is designed for development workstations where the NVIDIA driver is installed and the .so.1 symlink is provided by the driver package. In a container build, the driver is absent, and the symlink must be created manually. The assistant's deep understanding of Linux dynamic linking conventions — the distinction between link-time names (libcuda.so) and runtime sonames (libcuda.so.1) — was essential to diagnosing this issue.

Blocker 3: PEP 668 (Messages 584–585)

With the symlink fix in place, the build progresses further — and immediately hits a third wall. Message 584 reveals the next failure: the supraseal dependency setup fails because pip install is blocked by PEP 668.

PEP 668 is a Python enhancement proposal implemented in Ubuntu 24.04 that prevents pip from modifying the system Python environment outside a virtual environment. When a pip command attempts to install or modify packages in the system Python, it receives an error message like:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python
installation or OS distribution provider. You can override this, at the
risk of breaking your Python installation or OS, by passing
--break-system-packages.
hint: See PEP 668 for the detailed specification.

The supraseal build.sh script creates a virtual environment for its own pip operations, but the SPDK pkgdep.sh script apparently attempts a pip operation outside that venv. The assistant's response in message 585 is to set PIP_BREAK_SYSTEM_PACKAGES=1 as a Dockerfile environment variable:

ENV PIP_BREAK_SYSTEM_PACKAGES=1

This is the standard workaround for PEP 668 in containerized builds. In a disposable Docker image, the concerns that motivated PEP 668 — protecting a user's system Python from accidental breakage — are largely irrelevant. The fix is pragmatic and correct for the context.

Blocker 4: The RECORD File That Wasn't (Messages 587–588)

The PEP 668 fix allows the build to progress further — but not to completion. Message 587 reveals the fourth and most obscure blocker. The SPDK pkgdep.sh script attempts to uninstall and reinstall pip itself, and this fails because the base image's pip was installed via Debian's package manager (apt), which does not create the RECORD file that pip uninstall expects.

The error message is precise:

ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.

This is the most esoteric blocker of the four. The pkgdep.sh script from the SPDK (Storage Performance Development Kit) project attempts to ensure a specific version of pip is installed by uninstalling the existing version and reinstalling. But when pip was installed via Debian's package manager, the installation metadata is stored in Debian's package database, not in pip's internal RECORD file format. When pip uninstall tries to find the RECORD file to determine what files belong to the pip package, it finds nothing and aborts.

This blocker is where the segment ends. The build is stalled at the supraseal SPDK dependency setup, with the FFI compilation — the core of the build — already validated as successful. The assistant has not yet resolved this final blocker within the segment's boundaries. The user's response in message 588 is empty, indicating the conversation continues beyond the segment.

Analysis: What This Segment Reveals About Containerization

The arc of this segment — from comprehensive planning through careful implementation to iterative debugging — reveals several fundamental truths about containerizing complex software stacks.

Containerization Shifts Integration Problems to Build Time

In a traditional deployment environment, the integration of Go, Rust, C++, CUDA, and Python components happens gradually, on a running system where each component can be debugged in isolation. A missing library can be installed with apt, a symlink can be created manually, a Python restriction can be bypassed with an environment variable. In a Docker build, all of these components must compile and link together in a single, non-interactive process. Every assumption that a build script makes about its environment — that jq is installed, that libcuda.so.1 exists, that pip can modify system packages, that pip's RECORD file exists — becomes a potential failure point.

Debugging Is Inherently Layered

Each fix in this segment reveals the next problem. The assistant fixed jq, which revealed the libcuda.so.1 issue. Fixing that revealed the PEP 668 issue. Fixing PEP 668 revealed the RECORD file issue. This layering is not a sign of poor planning; it is the natural consequence of building a system whose failure modes are nested. The first blocker is a surface-level dependency (a missing binary). The second requires understanding of Linux dynamic linking conventions (the distinction between libcuda.so and libcuda.so.1). The third requires knowledge of Python packaging policy (PEP 668 and its implementation in Ubuntu 24.04). The fourth requires understanding of the interaction between Debian's package manager and pip's self-modification logic. Each successive blocker demands deeper system knowledge.

The Research Phase Is the Foundation

The reason the assistant could diagnose the libcuda.so.1 issue so quickly is that it already knew, from its earlier research, that the neptune build script probes GPU capabilities at compile time. The reason it knew to set PIP_BREAK_SYSTEM_PACKAGES=1 rather than some other fix is that it understood PEP 668 and its implementation in Ubuntu 24.04. The research phase (messages 546–557) was not busywork; it was the foundation upon which all subsequent debugging rested.

The Assistant's Methodology Is a Case Study in Effective Debugging

The pattern is consistent across all four blockers:

  1. Inspect the build output to identify the error.
  2. Research the root cause by examining the base image or build scripts.
  3. Apply a targeted fix to the Dockerfile.
  4. Re-run the build. The assistant never resorts to guesswork or shotgun debugging. Each fix is grounded in a causal understanding of why the failure occurred. When the libcuda.so.1 symlink is missing, the assistant does not blindly add every possible symlink; it runs ls -la inside the devel image to confirm the exact state of the stubs directory. When PEP 668 blocks pip, the assistant does not install Python packages via apt as an alternative; it sets the specific environment variable that bypasses the restriction.

Conclusion

Segment 4 of this opencode session is a microcosm of the challenge of modern infrastructure engineering. It begins with the clean abstraction of a planning document — a blueprint that organizes knowledge, captures decisions, and sequences work. It proceeds through the satisfying act of creation — writing a Dockerfile and entrypoint script that embody the blueprint's design. And it culminates in the messy, iterative reality of debugging — a cascade of environment-specific blockers that test every assumption the blueprint encoded.

The Docker container for Curio/cuzk mainnet proving is not yet built at the end of this segment. The SPDK RECORD file issue remains unresolved. But the foundation is solid: the FFI compilation succeeds, the multi-stage architecture is validated, and four distinct classes of build failure have been identified and addressed. The assistant's systematic methodology — research, plan, implement, test, diagnose, fix, iterate — has proven itself against the unforgiving reality of a heterogeneous software stack. The remaining blocker is just the next layer of the onion, waiting to be peeled.

The lessons of this segment extend far beyond the specific context of Filecoin and Curio. Any engineer who has attempted to containerize a complex software stack will recognize the pattern: the planning document that captures everything you know, the Dockerfile that encodes your best understanding of the build process, and the cascade of failures that reveals what you didn't know. Each blocker is a tuition payment in the currency of system knowledge, and the final working container is the diploma.