From Blueprint to Blocker: The Iterative Journey of Containerizing a Heterogeneous Proving Stack

Introduction

In the sprawling narrative of an opencode coding session focused on the Filecoin Curio project, a single chunk of work — spanning messages 546 through 588 — captures one of the most instructive arcs in modern infrastructure engineering: the journey from architectural planning through implementation to the gritty, iterative reality of debugging a complex Docker build. This chunk tells the story of how an AI assistant, guided by a user's directive to "prepare a Docker container with curio, cuzk, all curio dependencies, and fetching 32G PoRep and Snap proving params," navigated the treacherous waters of containerizing a heterogeneous software stack spanning Go, Rust, C++, CUDA, and Python build tools.

The narrative arc of this chunk is deceptively simple. It begins with a comprehensive planning document (message 546) that synthesizes hours of research into an actionable blueprint. It proceeds through the careful construction of a multi-stage Dockerfile and an entrypoint script (messages 558–571). It culminates in a series of build attempts (messages 574–588) that reveal one after another environment-specific blocker: a missing jq binary, a missing libcuda.so.1 symlink in the CUDA toolkit stubs directory, Python PEP 668 restrictions on pip installations, and finally a deeply obscure pip uninstall failure caused by a missing RECORD file in the base image's Debian-packaged Python environment. Each blocker is a microcosm of the challenge of containerizing software that was never designed to be containerized — software whose build scripts make implicit assumptions about their environment that only surface when that environment is stripped down to a minimal Docker base image.

This article synthesizes the entire chunk, tracing the thread from research to implementation to debugging, and extracting the lessons embedded in each phase of the work.

Phase I: The Architecture of Synthesis (Messages 546–557)

The chunk opens with message 546, a remarkable planning document that the assistant produced after completing an extensive research phase. This message is not merely a status update; it is a complete knowledge consolidation exercise that transforms scattered research findings into an actionable engineering plan. The assistant had spent messages 537 through 544 spawning subagent tasks to explore the curio build system, the cuzk build requirements, the existing Dockerfile patterns, the supraseal build script, and the CUDA base image landscape. Message 546 is where all that information is integrated into a coherent whole.

The planning document is structured into six sections: Goal, Instructions, Discoveries, Accomplished, Remaining, and Relevant Files. Each section serves a distinct cognitive purpose. The Goal section re-establishes shared context between user and assistant, confirming that the user's decisions about mainnet tags, CUDA 13, and runtime parameter fetching have been absorbed. The Instructions section captures concrete operational details — remote host addresses, exact build commands, deployment procedures — that would otherwise be scattered across multiple tool outputs. The Discoveries section is the heart of the message: eleven numbered items representing the assistant's synthesized understanding of the system, ranging from the GPU index routing fix (relevant because older builds would have the GPU routing bug) to the reality check that curio fetch-params 32GiB downloads all parameters for the sector size with no way to filter by proof type.

The planning document also sketches the multi-stage build architecture that would become the foundation of the Dockerfile: "Stage 1: CUDA 13 devel base + Go 1.24 + Rust 1.86.0 + gcc-13 + all build deps → build curio (with CUDA/supraseal) + cuzk. Stage 2: CUDA 13 runtime base + runtime libs → copy binaries from stage 1." This separation of concerns — isolating the heavy build environment from the minimal runtime environment — is Docker best practice, and the assistant's research into CUDA 13 base image availability (messages 552–554) confirmed that the necessary images existed.

What makes this planning phase particularly noteworthy is the assistant's willingness to act as a reality check for the user's expectations. When the user asked for "only PoRep and Snap params," the assistant investigated whether curio fetch-params supports filtering and reported that it does not. This is the behavior of an experienced engineer who knows that managing expectations is as important as writing code.

Phase II: The Moment of Creation (Messages 558–571)

With the blueprint established, the assistant moved to implementation. Messages 558 and 559 saw the creation of docker/cuzk/entrypoint.sh — a script that would check for existing proving parameters at runtime and, if missing, run curio fetch-params 32GiB to download the ~100GB of parameters before dropping into an interactive bash shell. The assistant initially used a WindowPoSt parameter filename as the sentinel for whether parameters had been downloaded, but a subagent task (message 568) revealed the correct filename — the 32GiB PoRep .params file — and the assistant corrected it (message 569). This correction, a single line change, is emblematic of the assistant's commitment to accuracy: it did not guess the filename but dispatched research to verify it.

Message 562 is the moment of creation for the Dockerfile itself. The assistant wrote Dockerfile.cuzk as a two-stage build. The builder stage (nvidia/cuda:13.0.2-devel-ubuntu24.04) installs Go 1.24.7, Rust 1.86.0, gcc-13, and a comprehensive list of system dependencies, then compiles the curio binary (with CUDA/supraseal flags) and the cuzk-daemon Rust binary. The runtime stage (nvidia/cuda:13.0.2-runtime-ubuntu24.04) is minimal, containing only the runtime libraries needed to execute the compiled binaries. The assistant's choice of Go 1.24.7 and Rust 1.86.0 was not arbitrary — these versions were extracted from the project's go.mod and rust-toolchain.toml files during the research phase.

The assistant then subjected the Dockerfile to a rigorous self-review (messages 564–569), verifying each build dependency against Ubuntu 24.04 package names, checking the correctness of the sentinel file, and confirming that the runtime library packages existed. This quality gate reflects a disciplined engineering approach: before running a build that might take hours, verify every assumption that can be verified statically.

Message 571 is the summary that marks the transition from implementation to validation. The assistant presents the completed files, explains the design rationale, and explicitly invites the user to review before testing: "Want me to test the build on one of the remote hosts, or is there anything you'd like changed first?" This handoff is crucial — the assistant cannot independently verify that the Dockerfile will build successfully in the target environment, and it recognizes that the user's review is the next necessary step.

Phase III: The Reality of Execution (Messages 572–588)

The user's response, delivered in message 572, is decisive: "Build here, the current host is cuda capable." With these eight words, the session pivots from preparation to execution. The assistant initiates the first build (message 574) with docker build -f Dockerfile.cuzk -t curio-cuzk:latest . 2>&1. The build begins, metadata is loaded, authentication succeeds — and then the first blocker appears.

Blocker 1: The Missing jq

The FFI build script requires jq for JSON processing — specifically, to parse parameters.json which lists proving parameter filenames, CIDs, and digests. The assistant had not included jq in the Dockerfile's package list. In message 576, the assistant recognizes the gap and adds it. This fix is trivial in isolation, but it reveals a deeper truth: the build system's dependency graph is not fully documented. The assistant had read the makefiles and build scripts, but jq is an implicit dependency — it is invoked by a script that is itself invoked by the build system, and its absence only becomes visible when the build reaches that specific step.

Blocker 2: The Missing Symlink

The second blocker is more subtle. The neptune cryptographic library (a Rust crate used in the Filecoin proof pipeline) includes a build script that probes GPU capabilities at compile time. This probe dynamically links against libcuda.so.1 — the CUDA runtime library. The nvidia/cuda:13.0.2-devel-ubuntu24.04 image contains the CUDA toolkit, including stub libraries in /usr/local/cuda/lib64/stubs/ that are designed for exactly this purpose: they export the same symbols as the real libcuda.so but contain no actual GPU functionality, allowing build-time linking without a GPU driver.

However, the stubs directory contains libcuda.so but lacks the libcuda.so.1 symlink. The neptune build script's test binary calls dlopen("libcuda.so.1", ...) — it looks for the versioned soname, not the bare libcuda.so. The assistant diagnoses this in message 579, reasoning through the causal chain: neptune probes GPU capabilities → requires loading libcuda.so.1 → CUDA devel image has stubs but not the .so.1 symlink → fix is to create the symlink and add the stubs directory to the library path. The fix is applied in message 582, and the FFI compilation succeeds — a major milestone.

This blocker is a textbook example of the gap between "it compiles" and "it builds in a container." The CUDA devel image is designed for development workstations where the NVIDIA driver is installed and the .so.1 symlink is provided by the driver package. In a container build, the driver is absent, and the symlink must be created manually. The assistant's deep understanding of Linux dynamic linking conventions — the distinction between link-time names (libcuda.so) and runtime sonames (libcuda.so.1) — was essential to diagnosing this issue.

Blocker 3: PEP 668

With the symlink fix in place, the build progresses further — and immediately hits a third wall. The supraseal dependency setup fails because pip install is blocked by PEP 668, a Python enhancement proposal implemented in Ubuntu 24.04 that prevents pip from modifying the system Python environment outside a virtual environment. The supraseal build.sh script creates a venv for its own pip operations, but the SPDK pkgdep.sh script apparently attempts a pip operation outside that venv.

The assistant's response (message 585) is to set PIP_BREAK_SYSTEM_PACKAGES=1 as a Dockerfile environment variable. This is the standard workaround for PEP 668 in containerized builds — in a disposable Docker image, the concerns that motivated PEP 668 (protecting a user's system Python from accidental breakage) are largely irrelevant. The fix is pragmatic and correct for the context.

Blocker 4: The RECORD File That Wasn't

The PEP 668 fix allows the build to progress further — but not to completion. Message 587 reveals the next failure: the SPDK pkgdep.sh script attempts to uninstall and reinstall pip itself, and this fails because the base image's pip was installed via Debian's package manager (apt), which does not create the RECORD file that pip uninstall expects. The error message is precise: "ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian."

This is the most obscure blocker of the four, and it is where the chunk ends. The build is stalled at the supraseal SPDK dependency setup, with the FFI compilation — the core of the build — already validated as successful. The assistant has not yet resolved this final blocker within the chunk's boundaries.

The Deeper Narrative: What This Chunk Reveals

The arc of this chunk — from comprehensive planning through careful implementation to iterative debugging — reveals several fundamental truths about containerizing complex software stacks.

First, containerization does not eliminate integration problems; it shifts them from runtime to build time. In a traditional deployment, the integration of Go, Rust, C++, CUDA, and Python components happens gradually, on a running system where each component can be debugged in isolation. In a Docker build, all of these components must compile and link together in a single, non-interactive process. Every assumption that a build script makes about its environment — that jq is installed, that libcuda.so.1 exists, that pip can install system packages, that pip's RECORD file exists — becomes a potential failure point.

Second, the debugging process is inherently layered. Each fix reveals the next problem. The assistant fixed jq, which revealed the libcuda.so.1 issue. Fixing that revealed the PEP 668 issue. Fixing PEP 668 revealed the RECORD file issue. This layering is not a sign of poor planning; it is the natural consequence of building a system whose failure modes are nested. The first blocker is a surface-level dependency, the second requires understanding of dynamic linking conventions, the third requires knowledge of Python packaging policy, and the fourth requires understanding of the interaction between Debian's package manager and pip's self-modification logic. Each successive blocker demands deeper system knowledge.

Third, the assistant's methodology is a case study in effective debugging. The pattern is consistent across all four blockers: inspect the build output to identify the error, research the root cause by examining the base image or build scripts, apply a targeted fix to the Dockerfile, and re-run the build. The assistant never resorts to guesswork or shotgun debugging — each fix is grounded in a causal understanding of why the failure occurred. When the libcuda.so.1 symlink is missing, the assistant does not blindly add every possible symlink; it runs ls -la inside the devel image to confirm the exact state of the stubs directory. When PEP 668 blocks pip, the assistant does not install Python packages via apt as an alternative; it sets the specific environment variable that bypasses the restriction.

Fourth, the chunk demonstrates the importance of the research phase. The reason the assistant could diagnose the libcuda.so.1 issue so quickly is that it already knew, from its earlier research, that the neptune build script probes GPU capabilities at compile time. The reason it knew to set PIP_BREAK_SYSTEM_PACKAGES=1 rather than some other fix is that it understood PEP 668 and its implementation in Ubuntu 24.04. The research phase (messages 546–557) was not busywork; it was the foundation upon which all subsequent debugging rested.

Conclusion

The chunk spanning messages 546 through 588 is a microcosm of the challenge of modern infrastructure engineering. It begins with the clean abstraction of a planning document — a blueprint that organizes knowledge, captures decisions, and sequences work. It proceeds through the satisfying act of creation — writing a Dockerfile and entrypoint script that embody the blueprint's design. And it culminates in the messy, iterative reality of debugging — a cascade of environment-specific blockers that test every assumption the blueprint encoded.

The Docker container for Curio/cuzk mainnet proving is not yet built at the end of this chunk. The SPDK RECORD file issue remains unresolved. But the foundation is solid: the FFI compilation succeeds, the multi-stage architecture is validated, and four distinct classes of build failure have been identified and addressed. The assistant's systematic methodology — research, plan, implement, test, diagnose, fix, iterate — has proven itself against the unforgiving reality of a heterogeneous software stack. The remaining blocker is just the next layer of the onion, waiting to be peeled.