The Final Mile: Upstreaming a GPU Proving Engine into Production

Introduction

There is a well-known adage in software engineering: the last 10% of the work takes 90% of the time. But this framing is misleading — it suggests that the final phase is merely a tedious extension of the same building activity. In reality, the transition from a working prototype to a production-ready integration is a qualitatively different kind of work. It requires shifting from a mindset of exploration and optimization to one of packaging, documentation, and reproducibility. The code must not only work on the developer's machine; it must work on anyone's machine, from a clean checkout, with clear error messages when prerequisites are missing, and with documentation that guides a system administrator through deployment.

This article examines that transition as it unfolded in segment 34 of an opencode coding session, where the cuzk pipelined SNARK proving engine — the product of 33 prior segments of architecture design, implementation, optimization, and debugging — was finally upstreamed into the Curio Filecoin storage provider repository. Over the course of dozens of messages, the team made strategic decisions about dependency management, audited git tracking state, extended the build system, verified every integration point, wrote documentation, and committed the result. The session is a masterclass in production integration engineering.

The Strategic Fork: Vendoring vs. Upstream Coordination

The upstreaming effort began with a fundamental architectural question. The cuzk proving engine depended on custom patches to three Rust crates — bellperson, bellpepper-core, and supraseal-c2 — that had been modified to support the split GPU proving API, memory backpressure mechanisms, and mutex-based synchronization that enabled the pipelined architecture. These patches were not yet accepted upstream, and waiting for upstream maintainers to review and merge them would take weeks or months.

In [msg 3499], the assistant laid out two options. Option A was the "clean" approach: push the forked crates to branches on the official GitHub repositories and reference them via Cargo's [patch.crates-io] mechanism. This kept the Curio repository lean and followed standard Rust pre-upstream practices, but it required external coordination and introduced dependency on external repositories. Option B was more radical: vendor the complete crate source trees directly inside the Curio repository under extern/, trading repository bloat for zero external coordination and fully reproducible builds.

The user's response in [msg 3500] was immediate and decisive: "Do option B, removed some testing blobs so the overhead should be pretty small." This choice reflected a clear priority hierarchy — speed to production over repository tidiness, self-containment over upstream coordination. The user had already mitigated the primary objection (repository bloat) by cleaning out test artifacts, making the trade-off straightforward.

This decision cascaded through everything that followed. It meant the Curio repository would now contain complete copies of three Rust crates alongside the cuzk workspace itself. It meant the git tracking state of extern/bellperson/, extern/bellpepper-core/, and extern/supraseal-c2/ needed to be audited. And it meant that the build system needed to be designed around the assumption that all dependencies were local — no network fetches, no GitHub availability concerns, nothing but git clone and make.

The Audit: Discovering What Was Missing

The assistant's first step after the vendoring decision was not to write code, but to establish ground truth. In [msg 3502], it ran git ls-tree -r HEAD --name-only on each vendored directory to see what was already tracked. The results were revealing:

Build System Design: The Makefile Targets

With the vendoring strategy settled, the assistant turned to the Makefile. The design goals were clear from the plan laid out in [msg 3499]:

  1. Opt-in, not mandatory: The cuzk daemon requires CUDA (nvcc), which is not available in Curio's standard CI environment. The build target must gracefully skip on machines without NVIDIA GPUs.
  2. Self-contained: The build should work from a clean checkout with no external dependencies beyond cargo and nvcc.
  3. Standard lifecycle: The daemon should be buildable, installable, uninstallable, and cleanable using the same patterns as every other Curio binary. The assistant read the Makefile in [msg 3512] to find the right insertion point, studying the curio-native target and the existing conventions. In [msg 3513], it announced its readiness: "Now I have a clear picture. I'll add the cuzk target after the pdptool target (around line 241)." The implementation proceeded in three sequential edits. The first edit ([msg 3514]) added the core cuzk target with dual pre-flight checks for cargo and nvcc, each with descriptive error messages directing the user to install the missing dependency. The target ran cargo build --release --bin cuzk-daemon inside extern/cuzk and copied the resulting binary to ./cuzk at the repository root. The second edit ([msg 3515]) added install-cuzk and uninstall-cuzk targets, following the established pattern of install-curio and install-sptool. These targets placed the binary in /usr/local/bin and were deliberately excluded from the default install: target — storage providers opt in explicitly. The third edit ([msg 3516]) added cuzk to the clean target, ensuring that make clean removes the binary and runs cargo clean in the vendored workspace. This attention to the clean target reflected an understanding that production software is not just about making things work, but about making them work cleanly over time — a principle that is easy to overlook in the rush to ship.

The CI Correction: A Moment of Self-Discovery

One of the most instructive moments in the upstreaming effort was the assistant's self-correction regarding CI compatibility. After adding the Makefile targets, the assistant ran make -n buildall in [msg 3523] and found that cuzk was referenced four times — it had been added to the BINS variable, and buildall iterates over $(BINS).

The realization hit in [msg 3524]: "Wait — buildall resolves $(BINS) which includes cuzk. That means make buildall on CI (no CUDA) would fail."

This was a classic build system error. The assistant had correctly identified the need for conditional CUDA support in the design phase, but the implementation lost that nuance. The cuzk target itself had pre-flight checks for nvcc, but those checks would fail with an error — they wouldn't skip gracefully. On CI, this would cause the entire buildall pipeline to fail.

The assistant's response was immediate and methodical. It checked what CI actually runs by grepping the CI workflow file ([msg 3524]), confirming that CI uses make build and make deps, not buildall. But it didn't stop there. In [msg 3525], it reasoned: "But buildall is a convenience target that SPs or developers might use. I should not add cuzk to BINS unconditionally." The assistant then removed cuzk from BINS entirely, keeping it as a standalone target with the key insight: "it has its own build chain (Rust) not Go."

This correction reflects a mature understanding of build system design. The BINS variable in Curio's Makefile is designed for Go binaries that share a common toolchain. cuzk is architecturally different — a Rust project with CUDA kernel code, requiring both the Rust toolchain and the NVIDIA CUDA toolkit. It belongs in a different category of build artifact, and the build system should reflect that distinction. The assistant's willingness to catch its own mistake and correct it transparently — rather than pretending the first approach was correct — is a hallmark of rigorous engineering.

Verification: Trust but Validate

The assistant did not stop at writing Makefile rules. It systematically verified every aspect of the integration:

  1. Build verification ([msg 3518]): The assistant ran make cuzk and confirmed the build succeeded. The output showed two warnings about JobTracker visibility in the Rust code — non-fatal code quality issues that the assistant judged acceptable for a pre-release integration.
  2. Binary verification ([msg 3519]): The assistant ran ./cuzk --help and confirmed the binary produced the expected output: "cuzk proving engine daemon" with options for --config, --listen, and --log-level. The help text revealed design decisions about the daemon's interface — default config path at /data/zk/cuzk.toml, support for both TCP and Unix sockets, and production-grade logging.
  3. Go vet verification ([msg 3520]): The assistant ran go vet on all modified Go packages (lib/cuzk/, deps/config/, tasks/proofshare/, tasks/seal/, tasks/snap/, cmd/curio/tasks/, lib/ffi/), confirming no new warnings were introduced.
  4. Clean build verification ([msg 3548]): The assistant ran cargo clean (removing 17,616 files totaling 12.5 GiB of cached build artifacts) and rebuilt from scratch. The build completed in 1 minute and 51 seconds, proving that the vendored crates were complete and the build system worked from a clean state.
  5. Buildall verification ([msg 3523]): The assistant ran make -n buildall | grep -c "cuzk" to confirm that after the correction, buildall no longer attempted to build cuzk. This verification ritual reflects a testing discipline that prioritizes regression prevention. The assistant never assumed that a successful build implied everything was correct — it systematically checked each integration point. The clean build test was particularly important: it proved that the vendored crates were complete and that no cached artifacts were masking missing dependencies. In production integration, the clean build is the ultimate truth-teller.

Documentation: The Final Piece

With the code staged and verified, the assistant turned to documentation. The user's instruction in [msg 3533] was clear: "Add docs to documentation/ (Experimental section of the gitbook)."

The assistant began with reconnaissance. In [msg 3553], it read the documentation/en directory to understand the GitBook structure. It confirmed that experimental-features/ existed, that SUMMARY.md controlled the table of contents, and that existing docs like supraseal.md and Snark-Market.md provided style templates.

Over the next several messages, the assistant studied the existing documentation conventions. It read the experimental features README ([msg 3554]), which carried an important warning: "It is not recommended to run experimental features in production." It read gpu-over-provisioning.md and Snark-Market.md ([msg 3557]) to understand the YAML frontmatter format, warning banners, and section structure. It even read the example config file ([msg 3558]) to ensure accurate parameter documentation.

In [msg 3559], the assistant wrote cuzk-proving-daemon.md — a 253-line document covering requirements, building, daemon configuration, Curio integration, deployment patterns, monitoring, and troubleshooting. The document was placed in the experimental-features/ directory, following the established convention. The assistant then updated SUMMARY.md and experimental-features/README.md to include the new page.

The documentation was written last, after all code changes were verified. This ordering reflects a crucial assumption: documentation is only trustworthy when it describes a verified reality. Writing the docs earlier would have risked inaccuracies as the build system or integration points changed during staging. By waiting until the code was committed and verified, the assistant ensured that every command, every config option, and every deployment step in the documentation corresponded to the actual committed state.

The Commit: A Permanent Record

The final step was the commit itself. In [msg 3576], the assistant created commit 3c53695c on the feat/cuzk branch with a comprehensive message. The commit message was structured with deliberate sections:

The PR Description: Synthesis and Communication

After the commit, the user asked the assistant to prepare a PR description ([msg 3580]): "read cuzk..md and c2-..md and put together a PR description saying what was changed and how pipelining / memory management / cpu locking works."

The assistant selected three documents to read: cuzk-project.md (the architecture document), c2-improvement-background.md (the problem statement), and c2-optimization-proposal-1.md (the core technique of Sequential Partition Synthesis). This triad was carefully constructed: the what, the why, and the how. Together they provided the narrative arc that a compelling PR description requires.

This moment — the shift from building to explaining — is the final transition in any complex integration. The 37 files are committed. The branch is pushed. Now the assistant must step back and tell the story to reviewers who have not lived through the 34 segments of development. The PR description is the capstone document that justifies the complexity, explains the trade-offs, and provides enough mental model for someone to understand the code without having experienced the journey.

Themes and Lessons

Several overarching themes emerge from this upstreaming session:

Pragmatism over purity. The decision to vendor Rust crates rather than push branches to upstream repositories was a deliberate trade-off. It prioritized speed to production and reproducibility over repository tidiness. This is a recurring pattern in production integration: the cleanest architectural solution is often not the best operational solution.

Verification as a discipline. The assistant's systematic verification — build, binary, Go vet, clean build, buildall — was not a checklist to be completed but a discipline to be practiced. Each verification step caught a different class of potential failure: build failures, runtime errors, Go type mismatches, missing dependencies, and CI regressions.

Documentation as the last verification gate. Writing documentation after all code changes were verified ensured that the documentation accurately described the committed state. This ordering is counterintuitive — many teams write documentation first — but it produces more reliable results.

Self-correction as a strength. The assistant's discovery that it had inadvertently added cuzk to BINS (and thus to buildall) was not a failure; it was a demonstration of the verification process working. The willingness to catch, acknowledge, and fix the mistake transparently is a hallmark of rigorous engineering.

Conclusion

The upstreaming of the cuzk proving engine into the Curio repository is a case study in production integration engineering. It demonstrates that the last 10% of the work — the build system plumbing, the git tracking audit, the CI compatibility checks, the documentation — is often the most consequential. A working prototype is not a shipping product; the gap between them is filled by the kind of methodical, verification-driven work documented in this session.

The result is a commit that any developer can clone and build with git clone && make curio cuzk, with clear error messages if prerequisites are missing, and with documentation that guides storage providers through deployment. This is the essence of upstreaming: not just making code work, but making it work for others, on their machines, with their constraints.

The cuzk proving engine had already been optimized across 33 segments of development — its pipelined architecture, memory management, GPU utilization, and CPU locking had been refined through countless iterations. But none of that optimization matters if the code cannot be built and deployed by someone who wasn't in the room when it was written. Segment 34 closed that gap, transforming a brilliant prototype into a shippable product.