The Upstreaming of a GPU Proving Engine: From Working Prototype to Production-Ready Integration

Introduction

In any substantial software engineering effort, there comes a moment when the focus shifts from building to shipping. The architecture is proven, the benchmarks are promising, and the code works. But a working prototype is not a production deployment — there remains a gap between "it works on my machine" and "it works on anyone's machine." Bridging that gap requires a distinct phase of work: upstreaming, where the code is packaged, documented, and integrated into the build system so that others can build, deploy, and maintain it.

This article examines the upstreaming phase of the cuzk proving engine integration into the Curio Filecoin storage provider, as documented in segment 34 of an opencode coding session. Over the course of dozens of messages, the team transitioned from implementation to integration, making critical decisions about dependency management, build system design, CI compatibility, and documentation strategy. The result was a self-contained, reproducible build that any storage provider could clone and build with nothing more than git clone && make curio cuzk.

The Strategic Decision: Vendoring Over Upstream Coordination

The upstreaming effort began with a fundamental question: how should the forked Rust crates — bellperson, bellpepper-core, and supraseal-c2 — be managed? These crates contained custom patches for the split GPU proving API, memory backpressure mechanisms, and mutex-based synchronization that enabled the pipelined proving architecture. They were not yet accepted upstream, and waiting for upstream maintainers to review and merge the changes would take weeks or months.

The assistant laid out two options in [msg 3499]. Option A was the "clean" approach: push the forked crates to branches on the official GitHub repositories (e.g., filecoin-project/bellperson branch feat/cuzk-async) and reference them via Cargo's [patch.crates-io] mechanism. This kept the Curio repository lean and followed standard Rust pre-upstream practices, but it required external coordination and introduced dependency on external repositories. Option B was more radical: vendor the complete crate source trees directly inside the Curio repository under extern/, trading repository bloat for zero external coordination and fully reproducible builds.

The user's response in [msg 3500] was decisive: "Do option B, removed some testing blobs so the overhead should be pretty small." This choice reflected a clear priority hierarchy — speed to production over repository tidiness, self-containment over upstream coordination. The user had already mitigated the primary objection (repository bloat) by cleaning out test artifacts, making the trade-off calculus straightforward.

This decision cascaded through everything that followed. It meant the Curio repository would now contain complete copies of three Rust crates alongside the cuzk workspace itself. It meant the git tracking state of extern/bellperson/, extern/bellpepper-core/, and extern/supraseal-c2/ needed to be audited — which the assistant did across messages [msg 3502] through [msg 3510], discovering that only a subset of source files had been originally committed while essential files like Cargo.toml, build.rs, and license files were untracked.

The Audit: Discovering What Was Missing

The assistant's first step after the vendoring decision was not to write code, but to establish ground truth. In [msg 3502], it ran git ls-tree -r HEAD --name-only on each vendored directory to see what was already tracked. The results were revealing:

Build System Integration: The Makefile Targets

With the vendoring strategy settled, the assistant turned to the Makefile. The design goals were clear from the plan laid out in [msg 3499]:

  1. Opt-in, not mandatory: The cuzk daemon requires CUDA (nvcc), which is not available in Curio's standard CI environment. The build target must gracefully skip on machines without NVIDIA GPUs.
  2. Self-contained: The build should work from a clean checkout with no external dependencies beyond cargo and nvcc.
  3. Standard lifecycle: The daemon should be buildable, installable, uninstallable, and cleanable using the same patterns as every other Curio binary. The assistant read the Makefile in [msg 3512] to find the right insertion point, studying the curio-native target and the existing conventions. In [msg 3513], it announced its readiness: "Now I have a clear picture. I'll add the cuzk target after the pdptool target (around line 241)." The implementation proceeded in three sequential edits. The first edit ([msg 3514]) added the core cuzk target with dual pre-flight checks for cargo and nvcc, each with descriptive error messages directing the user to install the missing dependency. The target ran cargo build --release --bin cuzk-daemon inside extern/cuzk and copied the resulting binary to ./cuzk at the repository root. The second edit ([msg 3515]) added install-cuzk and uninstall-cuzk targets, following the established pattern of install-curio and install-sptool. These targets placed the binary in /usr/local/bin and were deliberately excluded from the default install: target — storage providers opt in explicitly. The third edit ([msg 3516]) added cuzk to the clean target, ensuring that make clean removes the binary and runs cargo clean in the vendored workspace. This attention to the clean target reflected an understanding that production software is not just about making things work, but about making them work cleanly over time.

The CI Correction: A Moment of Self-Discovery

One of the most instructive moments in the upstreaming effort was the assistant's self-correction regarding CI compatibility. After adding the Makefile targets, the assistant ran make -n buildall in [msg 3523] and found that cuzk was referenced four times — it had been added to the BINS variable, and buildall iterates over $(BINS).

The realization hit in [msg 3524]: "Wait — buildall resolves $(BINS) which includes cuzk. That means make buildall on CI (no CUDA) would fail."

This was a classic build system error. The assistant had correctly identified the need for conditional CUDA support in the design phase, but the implementation lost that nuance. The cuzk target itself had pre-flight checks for nvcc, but those checks would fail with an error — they wouldn't skip gracefully. On CI, this would cause the entire buildall pipeline to fail.

The assistant's response was immediate and methodical. It checked what CI actually runs by grepping the CI workflow file ([msg 3524]), confirming that CI uses make build and make deps, not buildall. But it didn't stop there. In [msg 3525], it reasoned: "But buildall is a convenience target that SPs or developers might use. I should not add cuzk to BINS unconditionally." The assistant then removed cuzk from BINS entirely, keeping it as a standalone target with the key insight: "it has its own build chain (Rust) not Go."

This correction reflects a mature understanding of build system design. The BINS variable in Curio's Makefile is designed for Go binaries that share a common toolchain. cuzk is architecturally different — a Rust project with CUDA kernel code, requiring both the Rust toolchain and the NVIDIA CUDA toolkit. It belongs in a different category of build artifact, and the build system should reflect that distinction.

Verification: Trust but Validate

The assistant did not stop at writing Makefile rules. It systematically verified every aspect of the integration:

  1. Build verification ([msg 3518]): The assistant ran make cuzk and confirmed the build succeeded. The output showed two warnings about JobTracker visibility in the Rust code — non-fatal code quality issues that the assistant judged acceptable for a pre-release integration.
  2. Binary verification ([msg 3519]): The assistant ran ./cuzk --help and confirmed the binary produced the expected output: "cuzk proving engine daemon" with options for --config, --listen, and --log-level. The help text revealed design decisions about the daemon's interface — default config path at /data/zk/cuzk.toml, support for both TCP and Unix sockets, and production-grade logging.
  3. Go vet verification ([msg 3520]): The assistant ran go vet on all modified Go packages (lib/cuzk/, deps/config/, tasks/proofshare/, tasks/seal/, tasks/snap/, cmd/curio/tasks/, lib/ffi/), confirming no new warnings were introduced.
  4. Clean build verification ([msg 3548]): The assistant ran cargo clean (removing 17,616 files totaling 12.5 GiB of cached build artifacts) and rebuilt from scratch. The build completed in 1 minute and 51 seconds, proving that the vendored crates were complete and the build system worked from a clean state.
  5. Buildall verification ([msg 3523]): The assistant ran make -n buildall | grep -c "cuzk" to confirm that after the correction, buildall no longer attempted to build cuzk. This verification ritual reflects a testing discipline that prioritizes regression prevention. The assistant never assumed that a successful build implied everything was correct — it systematically checked each integration point.

Documentation: The Final Piece

With the code staged and verified, the assistant turned to documentation. The user's instruction in [msg 3533] was clear: "Add docs to documentation/ (Experimental section of the gitbook)."

The assistant began with reconnaissance. In [msg 3553], it read the documentation/en directory to understand the GitBook structure. It confirmed that experimental-features/ existed, that SUMMARY.md controlled the table of contents, and that existing docs like supraseal.md and Snark-Market.md provided style templates.

Over the next several messages, the assistant studied the existing documentation conventions. It read the experimental features README ([msg 3554]), which carried an important warning: "It is not recommended to run experimental features in production." It read gpu-over-provisioning.md and Snark-Market.md ([msg 3557]) to understand the YAML frontmatter format, warning banners, and section structure. It even read the example config file ([msg 3558]) to ensure accurate parameter documentation.

In [msg 3559], the assistant wrote cuzk-proving-daemon.md — a 253-line document covering requirements, building, daemon configuration, Curio integration, deployment patterns, monitoring, and troubleshooting. The document was placed in the experimental-features/ directory, following the established convention. The assistant then updated SUMMARY.md and experimental-features/README.md to include the new page.

The documentation was written last, after all code changes were verified. This ordering reflects a crucial assumption: documentation is only trustworthy when it describes a verified reality. Writing the docs earlier would have risked inaccuracies as the build system or integration points changed during staging.

The Commit: A Permanent Record

The final step was the commit itself. In [msg 3576], the assistant created commit 3c53695c on the feat/cuzk branch with a comprehensive message. The commit message was structured with deliberate sections:

The PR Description: Synthesis and Communication

After the commit, the user asked the assistant to prepare a PR description ([msg 3580]): "read cuzk..md and c2-..md and put together a PR description saying what was changed and how pipelining / memory management / cpu locking works."

The assistant selected three documents to read: cuzk-project.md (the architecture document), c2-improvement-background.md (the problem statement), and c2-optimization-proposal-1.md (the core technique of Sequential Partition Synthesis). This triad was carefully constructed: the what, the why, and the how. Together they provided the narrative arc that a compelling PR description requires.

This moment — the shift from building to explaining — is the final transition in any complex integration. The 37 files are committed. The branch is pushed. Now the assistant must step back and tell the story to reviewers who have not lived through the 34 segments of development. The PR description is the capstone document that justifies the complexity, explains the trade-offs, and provides enough mental model for someone to understand the code without having experienced the journey.

Conclusion

The upstreaming of the cuzk proving engine into the Curio repository is a case study in production integration engineering. It demonstrates that the last 10% of the work — the build system plumbing, the git tracking audit, the CI compatibility checks, the documentation — is often the most consequential. A working prototype is not a shipping product; the gap between them is filled by the kind of methodical, verification-driven work documented in this session.

The key decisions that shaped the integration were:

  1. Vendoring over upstream coordination: A deliberate trade-off that prioritized self-contained reproducibility over repository tidiness, justified by the user's proactive cleanup of test artifacts.
  2. Opt-in build system design: The cuzk daemon is excluded from default build targets, ensuring that CI pipelines without CUDA remain unaffected while storage providers with GPU hardware can opt in explicitly.
  3. Complete crate tracking over diff-only tracking: The discovery that vendored crates must be tracked in their entirety, not just the modified source files, prevented a build failure that would have been difficult to diagnose.
  4. Documentation as the final verification gate: Documentation was written last, after all code changes were verified, ensuring it accurately described the committed state. The result is a commit that any developer can clone and build with git clone && make curio cuzk, with clear error messages if prerequisites are missing, and with documentation that guides storage providers through deployment. This is the essence of upstreaming: not just making code work, but making it work for others, on their machines, with their constraints.