Building the CuZK Docker Container: From Parameter Discovery to Production Deployment

Introduction

In the previous segment, the assistant had resolved a subtle crash in the WindowPoSt proof path by aligning the is_extensible() behavior of RecordingCS with WitnessCS, enabling Pre-Compiled Constraint Evaluator (PCE) extraction across all proof types. With those core engine fixes in place, the focus shifted to a practical, infrastructure-oriented challenge: packaging the CuZK proving engine and its Curio Go binary into a production-ready Docker container for mainnet 32GiB proving. This chunk chronicles that journey, from researching the existing build system to writing a multi-stage Dockerfile, debugging environment-specific blockers, and—crucially—discovering the exact proving parameter filenames needed for the container's entrypoint script.

What makes this chunk particularly interesting is the interplay between two parallel threads of work. On one track, the assistant designed and iterated on a Docker build for a heterogeneous stack combining Go, Rust, C++, CUDA, and Python build tools. On another track, a subagent was dispatched to search the codebase for v28 proving parameter filenames—a seemingly mundane task that unexpectedly uncovered a bug in the existing Docker entrypoint's sentinel check. Together, these threads illustrate the full lifecycle of infrastructure engineering: from architectural design to implementation to operational validation.

The Docker Container Challenge

The CuZK proving engine is not a simple application. It combines a Go binary (Curio, the Filecoin node) with a Rust/CUDA daemon (cuzk) that performs GPU-accelerated zero-knowledge proving. The build chain involves multiple languages, each with its own toolchain: Go modules, Rust with Cargo, CUDA toolkit for GPU kernels, C++ for the bellperson FFI layer, and Python for the supraseal SPDK dependency setup. Containerizing such a stack requires careful orchestration of build stages, base images, and dependency management.

The assistant began by researching the existing OpenCL-based Dockerfile in the repository, studying how the project had previously handled multi-language builds. The goal was to adapt this pattern for CUDA 13, using NVIDIA's official CUDA devel and runtime base images. The design that emerged was a multi-stage Dockerfile (Dockerfile.cuzk) with three distinct phases:

  1. Build stage: Based on nvidia/cuda:13.0-devel-ubuntu22.04, this stage would compile the Go binary, the Rust/CUDA daemon, and all FFI bindings. It needed the full CUDA toolkit, Rust toolchain, Go compiler, and various system dependencies.
  2. Runtime stage: Based on nvidia/cuda:13.0-runtime-ubuntu22.04, this stage would contain only the compiled binaries and runtime dependencies, keeping the image size manageable.
  3. Entrypoint: A Bash script (entrypoint.sh) that would handle first-run initialization, including downloading proving parameters and setting up the environment. The assistant also wrote the docker/cuzk/entrypoint.sh script, which implemented a sentinel-based parameter download pattern: check for the presence of a specific parameter file on disk; if missing, trigger curio fetch-params to download the full set of v28 proving parameters.

The Parameter Reconnaissance Mission

To write a correct sentinel check, the assistant needed to know the exact filenames of the v28 proving parameters for 32GiB sectors. Rather than guessing or hard-coding placeholder values, the assistant spawned a subagent with a focused mission: search the codebase for parameter filenames across three specific loci of truth—JSON manifest files, Go source code, and the filecoin-ffi directory.

The subagent's investigation, documented across messages 0 through 5 of the subagent session, was a masterclass in systematic codebase exploration [1][4][5][6]. It began with broad searches using glob and grep to locate relevant files, then progressively narrowed its focus by reading the discovered files in sequence.

The Canonical Source: parameters.json

The first and most important discovery was /tmp/czk/extern/filecoin-ffi/parameters.json, the canonical manifest of all v28 proving parameter filenames. Each entry in this JSON file includes the filename, a content identifier (CID), a digest, and a sector_size field. By filtering for entries with sector_size equal to 34359738368 (32 GiB in bytes), the subagent extracted a clean mapping of five parameter files:

| Proof Type | Filename | |---|---| | PoRep 32GiB | v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params | | Window PoSt 32GiB | v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params | | Winning PoSt 32GiB | v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.params | | SnapDeals 32GiB | v28-empty-sector-update-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.params | | SnapDeals (poseidon) 32GiB | v28-empty-sector-update-poseidon-merkletree-poseidon_hasher-8-8-0-3b7f44a9362e3985369454947bc94022e118211e49fd672d52bec1cbfd599d18.params |

These filenames are not arbitrary. As the subagent learned from comments in the Rust source code, they are deterministic: they encode the tree shape, hasher algorithm, and a SHA-256 hash of the public parameters identifier string. The hash is stable across builds because it depends only on the circuit structure—node count, layers, challenges, degree, and expansion degree. This stability is what makes the filenames reliable sentinel candidates.

Cross-Referencing Across Sources

The subagent did not stop at the JSON file. It cross-referenced the filenames against two other independent sources:

  1. /tmp/czk/extern/cuzk/cuzk-core/src/srs_manager.rs: The Rust function circuit_id_to_param_filename() provides a clean mapping from CircuitId enum variants to the exact parameter filenames. This confirmed the same filenames as the JSON. Comments in this file also noted file sizes: Window PoSt 32G at approximately 57 GiB, Winning PoSt 32G at approximately 184 MiB, and SnapDeals 32G at approximately 33 GiB.
  2. /tmp/czk/lib/fastparamfetch/paramfetch.go: The Go-side fetch logic reads parameters.json directly, filters .params files by sector_size matching the requested storageSize, and downloads any missing files. This confirmed that the JSON is the authoritative source for the fetch process. The subagent also consulted the design document /tmp/czk/cuzk-phase2-design.md, which duplicated the mapping from srs_manager.rs, providing a third point of confirmation.

The Bug That Shouldn't Have Been

The most consequential finding of the subagent's investigation was not the list of correct filenames but the discovery of an incorrect one. When the subagent examined the existing sentinel check in /tmp/czk/docker/cuzk/entrypoint.sh, it found:

if [ ! -f "$PARAM_DIR/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228571c57316571f7c2b97c164bc72ea5396fcf16f6.params" ]; then

The subagent's analysis was precise and damning [3]: this filename does not exist in parameters.json. The hash suffix ...571c57316571f7c2b97c164bc72ea5396fcf16f6 does not match any entry in the canonical manifest. The closest match is the 2048-byte sector size variant with hash ...0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0—a completely different parameter file for a negligible sector size.

Two problems compound in this sentinel:

  1. Wrong tree shape: The filename contains 8-0-0 instead of the correct 8-8-0. The tree shape 8-0-0 corresponds to a minimal tree structure (likely for a tiny 2048-byte sector), while 8-8-0 is the correct shape for 32GiB sectors.
  2. Wrong hash: The SHA-256 hash suffix is entirely different from any 32GiB entry in parameters.json. Since the hash is derived from the circuit's public parameters identifier, a different hash means a different circuit—one that will never be downloaded by the standard fetch process. The consequence is stark: the sentinel check will never find a matching file on disk, even after all 32GiB proving parameters have been successfully downloaded. Every container restart will trigger a redundant curio fetch-params call, wasting bandwidth, time, and storage I/O. For a parameter set that includes a 57 GiB Window PoSt file, this is not a trivial overhead. The subagent recommended replacing the broken sentinel with the PoRep 32GiB parameter file, which is confirmed in all three independent sources. The PoRep parameter is the best sentinel because it is required for all sealing operations, is the largest file among the 32GiB params, and is downloaded last (or among the last) by the fetch process, so its presence strongly indicates that the full download is complete.

Building Through Adversity

While the subagent was hunting for parameter filenames, the main session pressed forward with the Docker build. The initial build attempt revealed a series of environment-specific blockers, each requiring iterative debugging against the quirks of the CUDA 13 base image.

The Missing jq

The first blocker was a missing jq dependency. The build scripts used jq for JSON processing—likely for parsing parameters.json or configuration files—but the CUDA base image did not include it. The fix was straightforward: add jq to the list of packages installed in the build stage.

The Phantom libcuda.so.1

A more subtle issue emerged during the bellperson build. The bellperson build script attempts to detect the presence of a GPU by probing for libcuda.so.1. In the CUDA devel image, the CUDA toolkit stubs directory contains the CUDA libraries, but the symlink libcuda.so.1 was missing. Without this symlink, the build script assumed no GPU was available and fell back to a CPU-only configuration, which would defeat the purpose of a GPU-accelerated proving container.

The assistant traced the issue to the CUDA toolkit stubs directory and added the missing symlink, allowing the GPU detection logic to succeed. This fix was critical: without it, the bellperson FFI would compile without CUDA support, and the entire CuZK proving engine would be non-functional.

Python PEP 668 and the Pip Rebellion

The next blocker came from Python's PEP 668, which restricts pip install operations when the system Python environment is managed by the system package manager. The CUDA base image ships with a system-managed Python, and pip refuses to install packages into it without explicit permission. The assistant resolved this by setting PIP_BREAK_SYSTEM_PACKAGES=1, overriding the restriction and allowing the pip-based dependencies to install.

The SPDK Stalemate

Despite these successes, the build ultimately stalled during the supraseal SPDK (Storage Performance Development Kit) dependency setup. The SPDK build script attempted a pip uninstall operation that failed because of a missing RECORD file in the base image's Python environment. This is a known issue with certain Python package installations where the package metadata is incomplete. The assistant was unable to resolve this within the session, leaving the SPDK step as an unresolved blocker.

Themes and Lessons

This chunk illustrates several important themes in infrastructure engineering for complex systems.

The Heterogeneous Stack Tax

Containerizing a project that spans Go, Rust, C++, CUDA, and Python is exponentially harder than containerizing a single-language application. Each language brings its own toolchain, dependency management system, and build-time quirks. The CUDA base image, while providing the GPU toolkit, introduces its own idiosyncrasies (missing symlinks, Python environment restrictions) that must be discovered and worked around. The "stack tax" is real: every additional language in the build chain multiplies the surface area for environment-specific bugs.

The Value of Systematic Codebase Exploration

The subagent's parameter filename investigation demonstrates the power of systematic, multi-source cross-referencing. Rather than accepting the first filename found, the subagent verified each filename against three independent sources (JSON, Rust, Go), building a chain of evidence that increased confidence in the results. This approach also enabled the discovery of the sentinel bug—a finding that would have been missed by a superficial search.

The Sentinel Pattern and Its Fragility

The sentinel-based parameter download pattern is a common and sensible approach for Docker containers that need to download large assets on first run. However, the pattern is fragile when the sentinel filename is derived from deterministic circuit parameters. A single character difference in the hash or tree shape renders the sentinel permanently non-matching. The bug in the existing entrypoint script is a cautionary tale: the sentinel filename was likely copied from a different context (perhaps a test or a different sector size) without verification against the canonical source.

Operational Awareness in Engineering

The assistant's decision to spawn a subagent for parameter filename discovery, rather than hard-coding a placeholder or guessing, reflects a production-minded approach. The assistant recognized that the Docker container would be deployed in production, where incorrect parameter filenames would cause real operational issues (redundant downloads, startup delays, potential proving failures). By investing the time to find the correct filenames—and in the process, discovering a bug—the assistant demonstrated that operational awareness is not separate from engineering but integral to it.

Conclusion

This chunk captures a pivotal transition in the CuZK proving engine project: from fixing core proof engine bugs to packaging the system for production deployment. The Docker container work required deep knowledge of the build system, the CUDA toolchain, and the proving parameter ecosystem. The parameter filename investigation, while seemingly a minor sub-task, uncovered a genuine bug in the existing entrypoint script and produced a definitive, cross-referenced mapping of v28 32GiB parameter filenames.

The story is not yet complete—the SPDK dependency remains unresolved, and the Docker build cannot finish until that blocker is addressed. But the foundation is laid: the multi-stage Dockerfile design is sound, the entrypoint script has been corrected (once the sentinel filename is updated), and the core FFI compilation path has been validated. The remaining work is a matter of resolving the Python environment issue in the SPDK setup, a well-understood class of problem with known workarounds.

In the broader arc of the session, this chunk represents the shift from "does it compile?" to "does it deploy?"—a critical transition in any engineering project. The assistant's methodical approach to both the build infrastructure and the parameter discovery demonstrates that production readiness is not an afterthought but a parallel track that runs alongside core development.## References

[1] "The Sentinel Hunt: Searching for Proving Parameter Filenames in a Zero-Knowledge Proof Codebase" — Article analyzing the subagent's initial search strategy and the user's structured request for parameter filenames across JSON, Go, and FFI sources.

[2] "The Art of Thoroughness: Reading paramfetch.go in a Filecoin Parameter Hunt" — Article examining the assistant's methodical reading of the Go-side parameter fetch logic and the reasoning behind tracing the full data flow.

[3] "The Sentinel That Never Matches: Uncovering a Bug in Filecoin Proving Parameter Discovery" — Article detailing the discovery of the incorrect sentinel filename in the Docker entrypoint and its implications for operational reliability.

[4] "The First Move: Systematic Codebase Exploration for Proving Parameter Filenames" — Article covering the initial glob and grep searches that located the key files for parameter filename discovery.

[5] "The Sentinel Hunt: Deep-Diving Into Filecoin Proving Parameter Filenames" — Article analyzing the intermediate search steps and the discovery of the Rust circuit-to-filename mapping in srs_manager.rs.

[6] "The Sentinel Hunt: Pinpointing a Proving Parameter Filename Across a Multi-Language Codebase" — Article covering the targeted grep searches that revealed references across Go test files and the paramfetch library.