From Benchmark to Integration: The cuzk–Curio Proving Pipeline Comes Together

Introduction

The opencode session captured in Segment 33, Chunk 0 represents a pivotal transition in the development of the cuzk pipelined SNARK proving engine for Filecoin's Curio storage mining platform. This chunk documents two distinct but deeply connected efforts: the finalization of Phase 12 memory optimization and benchmarking, followed by the architectural planning and initial implementation of the cuzk daemon's integration into Curio's task orchestrator. Together, these efforts mark the shift from building a standalone proving engine to wiring it into a production-grade storage mining system.

The work spans nearly 90 messages (indexes 3330 through 3418), covering everything from formal documentation of memory scaling formulas to the creation of Go gRPC client stubs, from RAM tier recommendations to the careful splitting of vanilla proof generation from SNARK computation. This article synthesizes the major themes, architectural decisions, and engineering practices that emerge across this chunk.

Phase 12 Finalization: From Raw Data to Actionable Knowledge

The chunk opens with the culmination of Phase 12 of the cuzk proving engine — a phase that introduced a split GPU proving API and memory backpressure mechanisms. The assistant had spent considerable effort characterizing the memory scaling behavior of the proving pipeline, and the results were now being formalized into documentation.

The Low-Memory Benchmark Sweep

The assistant conducted a comprehensive benchmark sweep across nine configurations, varying partition_workers (pw) from 1 to 12 and gpu_workers_per_device (gw) between 1 and 2. The goal was to establish minimum RAM requirements and throughput characteristics for systems of varying sizes. The resulting data was distilled into a linear memory scaling formula:

Peak RSS ≈ 69 GiB baseline + (pw × ~20 GiB)

The 69 GiB baseline was decomposed into its components: 44 GiB for the SRS (Structured Reference String) and 25.7 GiB for the PCE (Proving Circuit Evaluation). This decomposition is practically important because it tells operators which costs are fixed and unavoidable, versus which costs scale with parallelism and can be tuned.

The benchmark also revealed a subtle finding about GPU worker configuration: at pw ≤ 7, the system is synthesis-bound (the GPU is starved for work), meaning gw=2 provides no throughput benefit over gw=1. In fact, gw=1 yields slightly better average prove times due to less CPU contention on the post-processing path. At pw ≥ 10, gw=2 provides a ~6% throughput gain by eliminating GPU idle gaps.

RAM Tier Recommendations

The benchmark data was translated into concrete, actionable RAM tier recommendations:

Documentation and Commit

The findings were documented in cuzk-project.md, the example configuration file (cuzk.example.toml) was updated with optimal defaults and RAM-tier guidelines, and all changes were committed as 9bb657e5. The assistant produced a structured handover summary ([msg 3330]) that captured the goal, instructions, discoveries, accomplished items, and relevant files — a template for how to crystallize experimental work into permanent knowledge.

The Integration Architecture: A User Message That Defined Everything

The turning point in the chunk comes at message [msg 3331], where the user delivers a concise but architecturally decisive instruction:

"Plan cuzk integration with curio - cuzk daemon probably let's keep independent for now, we want to wire into porep C2, snark PRU and PSProve; Probably needs a way to tell curio scheduler to not worry about task resources and delegate to CanAccept where we backpressure on pipeline capacity/state"

In fewer than 100 words, the user establishes five critical design decisions:

  1. Keep the daemon independent — cuzk runs as a separate process, not embedded in Curio. Communication happens over gRPC.
  2. Wire three task types — PoRep C2 (seal commit phase 2), SnapDeals ProveReplicaUpdate, and PSProve (proofshare).
  3. Zero local resource costs — When cuzk is enabled, TypeDetails() should return zero GPU and RAM costs, telling Curio's scheduler not to account for local resources.
  4. Backpressure via CanAccept — Instead of local resource accounting, CanAccept() should query the cuzk daemon's queue state to determine how many tasks can be accepted.
  5. Split vanilla proof generation from SNARK proving — The vanilla proof (CPU-bound, needs sector data) stays local; the SNARK computation (GPU-bound) goes to cuzk. This message is a masterclass in concise architectural direction. It provides clear guidance without being prescriptive about implementation details, trusting the assistant to fill in the specifics through codebase exploration.

The Research Phase: Mapping Curio's Task System

The assistant's response to the user's directive was not to immediately write code, but to explore. Messages [msg 3332] through [msg 3389] document an extensive research phase in which the assistant systematically mapped Curio's task system.

Finding the Task Files

The assistant used find, grep, and sed commands to locate the relevant files. The initial hypothesis — that PoRep C2 proving would be in a file called task_commit2.go — proved incorrect. The actual C2 logic was embedded within task_submit_commit.go. Similarly, the SnapDeals ProveReplicaUpdate task was not in a file named with the "pru" convention but in tasks/snap/task_prove.go.

These negative results (files that don't exist) are as informative as positive ones. They force the assistant to update its mental model of the codebase and broaden its search.

Understanding the Harmony Task Framework

The assistant investigated the Harmony task framework, discovering the key interfaces:

Reading the PoRep Task Implementation

The assistant read the PoRep task's Do() method to understand the existing proof generation flow. The critical insight emerged in [msg 3413]: PoRepSnark does two things — it generates a vanilla proof via GeneratePoRepVanillaProof (CPU, data-local) and then runs the SNARK computation via SealCommitPhase2 (GPU). For cuzk integration, only the SNARK step needs to be offloaded; the vanilla proof generation must remain local because it requires access to sector data.

This decomposition is the foundational insight that shapes the entire integration architecture.

Building the Integration Infrastructure

With the research complete, the assistant began constructing the integration layer piece by piece.

Configuration: Adding CuzkConfig

The first step was adding a CuzkConfig section to Curio's configuration (deps/config/types.go). The configuration includes fields for the daemon address, per-task-type enable flags (PoRep, SnapDeals, WindowPoSt, WinningPoSt), and a MaxPending setting for backpressure control. This design provides a kill switch (empty address disables the integration), supports progressive rollout (enable one task type at a time), and configures the backpressure threshold.

Protobuf Code Generation

The assistant generated Go gRPC client stubs from the existing protobuf definitions in extern/cuzk/cuzk-proto/proto/cuzk/v1/proving.proto. This process was not straightforward — it involved multiple attempts with different protoc invocations, debugging a nested directory structure that the --go_opt=paths=source_relative flag created, and verifying that the generated files compiled correctly ([msg 3397] through [msg 3406]).

The protobuf service definition includes RPCs for SubmitProof, AwaitProof, Prove (combined submit+await), GetStatus, CancelProof, PreloadSRS, EvictSRS, and GetMetrics. The GetStatus RPC, which returns queue status per proof kind, is the critical endpoint for the backpressure design.

The gRPC Client

A new package lib/cuzk/ was created to house the generated protobuf stubs and a Go client wrapper. The client provides high-level methods like Prove() and GetStatus() that task handlers can call directly, abstracting away the gRPC connection management.

The Integration File: cuzk_funcs.go

The most architecturally significant file created in this chunk is lib/ffi/cuzk_funcs.go ([msg 3416]). This file defines new methods on the existing SealCalls struct — the central Go type that wraps Filecoin's FFI calls. The new methods implement the split between local vanilla proof generation and remote SNARK proving:

Build Verification and Diagnostic Filtering

After writing cuzk_funcs.go, the assistant ran go build ./lib/ffi/ to verify compilation. The build failed — but not because of the new code. The errors were all in extern/filecoin-ffi/cgo/fvm.go, a CGO binding file that requires FVM (Filecoin Virtual Machine) C headers that were not installed on the development machine.

The assistant's response ([msg 3418]) is a masterclass in diagnostic filtering:

"CGO/FVM errors are expected on this machine (no FVM headers). Let me use build tags to verify just the Go source."

The assistant ran go vet ./lib/ffi/ with output filtered through grep -v to remove the known noise patterns (cgo, extern/filecoin-ffi, could not determine). This surgical approach separated signal from noise, confirming that the new Go code had no errors while acknowledging the environmental limitation.

This moment exemplifies a critical engineering skill: knowing which build failures to fix and which to work around. The assistant correctly identified that the FVM errors were pre-existing, environmental, and unrelated to the integration work. Rather than spending time installing C headers for a dependency they didn't need to modify, the assistant pivoted to a targeted verification strategy and proceeded.

The Integration Pattern: Consistent Across Task Types

The integration follows a consistent pattern for each task type:

  1. Add a cuzkClient field to the task struct.
  2. Modify the constructor to accept a *cuzk.Client parameter.
  3. Modify TypeDetails() to return zero GPU/RAM costs when cuzk is enabled.
  4. Modify CanAccept() to query the daemon's queue state for backpressure.
  5. Modify Do() to call the cuzk client methods instead of the local FFI function. This consistency is architecturally important. It means that once the pattern is established for one task type (e.g., PoRep), extending it to others (SnapDeals, PSProve, and eventually WindowPoSt and WinningPoSt) is straightforward. It also means that the integration can be toggled on and off per task type via configuration, supporting progressive rollout.

The Split Between Vanilla and SNARK Proving

A subtle but critical aspect of the integration is the split between vanilla proof generation and SNARK proving. The vanilla proof (C1 for PoRep, PRU1 for SnapDeals) is a CPU-bound computation that requires access to sector data stored on the local machine's filesystem. The SNARK proof (C2) is a GPU-bound computation that takes the vanilla proof as input and produces a compact Groth16 proof.

The cuzk daemon only handles the SNARK step. The vanilla proof generation remains local because:

  1. Data locality: The vanilla proof generation needs access to sector data (sealed files, cache files) that may be terabytes in size. Sending this data over gRPC would be impractical.
  2. CPU-bound: Vanilla proof generation doesn't benefit from GPU acceleration. Running it locally doesn't compete with the daemon's GPU work.
  3. Network efficiency: The vanilla proof output (~50 MB for PoRep) is much smaller than the sector data (32 GiB). Sending it over gRPC is feasible. This split enables a new deployment architecture where storage-heavy machines (with large disk arrays) can be separate from computation-heavy machines (with multiple GPUs). The storage machine generates vanilla proofs locally and ships them to a GPU cluster for SNARK proving.

Backpressure Design: A Feedback Control System

The backpressure design is one of the most elegant aspects of the integration. Rather than having Curio's scheduler guess at the daemon's capacity based on local resource counters (which would be meaningless when the daemon is on a different machine), the design uses a feedback loop:

  1. Curio's scheduler proposes tasks to CanAccept().
  2. CanAccept() queries the cuzk daemon's GetStatus() RPC.
  3. The daemon returns queue state (pending and in-progress counts per proof type).
  4. CanAccept() computes how many tasks it can accept based on MaxPending minus current pending.
  5. Curio assigns the accepted tasks to Do().
  6. Do() calls cuzkClient.Prove() to submit the proof and wait for the result. This is a pull-based backpressure system: Curio pulls capacity information from the daemon before submitting work. It prevents the daemon from being overwhelmed while keeping Curio's scheduler informed about actual capacity.

The Broader Significance

The work in this chunk represents a transition from engine development to production integration. Before this chunk, cuzk was a standalone proving engine — powerful but isolated. After this chunk, it becomes part of Curio's production workflow, with a clean integration architecture, a gRPC client library, and modifications to three task types.

Several engineering principles emerge from this work:

Delegation, not duplication: Rather than duplicating resource management logic in both systems, the design delegates GPU scheduling to cuzk and task lifecycle to Curio. Each system focuses on its core competency.

Clean boundaries: The boundary between Curio and cuzk is defined by the gRPC protocol and the CanAccept() backpressure mechanism. Each system has clear responsibilities.

Progressive scope: Starting with three task types (PoRep, SnapDeals, PSProve) allows the team to validate the integration pattern before expanding to other proof types.

Diagnostic discipline: The assistant's response to the CGO build failure — filtering noise, recognizing environmental limitations, pivoting to targeted verification — demonstrates the importance of diagnostic filtering in complex systems.

Conclusion

Segment 33, Chunk 0 captures a remarkable arc of software engineering: from low-memory benchmarking and RAM tier recommendations, through architectural planning and codebase exploration, to the creation of a gRPC client library and the wiring of a remote proving daemon into a production task orchestrator. The work demonstrates the value of methodical research before implementation, the power of concise architectural direction, and the importance of diagnostic filtering when navigating complex build environments.

The resulting integration architecture — with its clean separation of concerns, feedback-based backpressure, and consistent per-task-type pattern — provides a solid foundation for the next phase of work: completing the wiring of all four task types, adding WindowPoSt and WinningPoSt support, and production hardening. The cuzk proving engine has taken its place as a production component of the Curio storage mining platform.