From Benchmarks to Production: The cuzk–Curio Integration Journey

Introduction

The development of the cuzk pipelined SNARK proving engine for Filecoin's Curio storage mining platform has been a multi-phase journey spanning optimization of GPU kernels, CPU synthesis, memory management, and pipeline architecture. Segment 33 of this opencode session captures a pivotal transition: the finalization of Phase 12's memory optimization and benchmarking work, followed by the architectural planning and implementation of the cuzk daemon's integration into Curio's production task orchestrator. This segment marks the shift from building a standalone proving engine to wiring it into a live storage mining system, bridging the gap between experimental optimization and production deployment.

The work spans two deeply connected efforts. The first is the culmination of Phase 12—documenting the split GPU proving API, conducting a systematic low-memory benchmark sweep across nine configurations, distilling the results into RAM tier recommendations for operators, and updating the example configuration with optimal defaults. The second is the creation of a Go gRPC client library for the cuzk daemon and the modification of three Curio task types—PoRep C2, SnapDeals ProveReplicaUpdate, and proofshare (PSProve)—to delegate their GPU-intensive SNARK computation to the daemon while preserving backward compatibility, scheduler semantics, and resource accounting integrity.

Together, these efforts represent the transformation of raw experimental data into actionable deployment guidance and the translation of an architectural vision into production code. This article synthesizes the major themes, engineering practices, and architectural decisions that emerge across this segment.

Phase 12 Finalization: From Experimental Data to Actionable Knowledge

The Low-Memory Benchmark Sweep

The segment opens with the culmination of Phase 12's memory characterization work. The assistant had introduced a split GPU proving API and memory backpressure mechanisms in earlier phases, and the results of extensive benchmarking were now being formalized into documentation. A comprehensive benchmark sweep was conducted across nine configurations, varying partition_workers (pw) from 1 to 12 and gpu_workers_per_device (gw) between 1 and 2. The goal was to establish minimum RAM requirements and throughput characteristics for systems of varying sizes, providing operators with concrete deployment guidance.

The resulting data was distilled into a linear memory scaling formula:

Peak RSS ≈ 69 GiB baseline + (pw × ~20 GiB)

This formula is deceptively simple but practically powerful. The 69 GiB baseline was decomposed into its components: 44 GiB for the SRS (Structured Reference String) and 25.7 GiB for the PCE (Proving Circuit Evaluation). This decomposition matters because it tells operators which costs are fixed and unavoidable versus which costs scale with parallelism and can be tuned. The SRS and PCE are one-time allocations that do not grow with the number of concurrent partitions, while the per-partition cost of ~20 GiB represents the working memory for each partition's synthesis and GPU proving.

RAM Tier Recommendations

The benchmark data was translated into concrete, actionable RAM tier recommendations that bridge the gap between the engineering lab and production operations:

Documentation and Commit

The findings were documented in cuzk-project.md, the example configuration file (cuzk.example.toml) was updated with optimal defaults and RAM-tier guidelines, and all changes were committed as 9bb657e5. The assistant produced a structured handover summary that captured the goal, instructions, discoveries, accomplished items, and relevant files—a template for how to crystallize experimental work into permanent knowledge. This documentation discipline ensures that the months of optimization work are not lost but become accessible to future operators and developers.

The Integration Architecture: A User Message That Defined Everything

The turning point in the segment comes when the user delivers a concise but architecturally decisive instruction ([msg 3331]):

"Plan cuzk integration with curio - cuzk daemon probably let's keep independent for now, we want to wire into porep C2, snark PRU and PSProve; Probably needs a way to tell curio scheduler to not worry about task resources and delegate to CanAccept where we backpressure on pipeline capacity/state"

In fewer than 100 words, the user establishes five critical design decisions:

  1. Keep the daemon independent — cuzk runs as a separate process, not embedded in Curio. Communication happens over gRPC.
  2. Wire three task types — PoRep C2 (seal commit phase 2), SnapDeals ProveReplicaUpdate, and PSProve (proofshare).
  3. Zero local resource costs — When cuzk is enabled, TypeDetails() should return zero GPU and RAM costs, telling Curio's scheduler not to account for local resources.
  4. Backpressure via CanAccept — Instead of local resource accounting, CanAccept() should query the cuzk daemon's queue state to determine how many tasks can be accepted.
  5. Split vanilla proof generation from SNARK proving — The vanilla proof (CPU-bound, needs sector data) stays local; the SNARK computation (GPU-bound) goes to cuzk. This message is a masterclass in concise architectural direction. It provides clear guidance without being prescriptive about implementation details, trusting the assistant to fill in the specifics through codebase exploration. The assistant's response was not to immediately write code but to systematically map Curio's task system—an approach that paid dividends throughout the integration.

The Research Phase: Mapping Curio's Task System

The assistant's response to the user's directive was methodical exploration. Messages document an extensive research phase in which the assistant systematically mapped Curio's task system using find, grep, and sed commands to locate the relevant files.

Finding the Task Files

The initial hypothesis—that PoRep C2 proving would be in a file called task_commit2.go—proved incorrect. The actual C2 logic was embedded within task_submit_commit.go. Similarly, the SnapDeals ProveReplicaUpdate task was not in a file named with the "pru" convention but in tasks/snap/task_prove.go. These negative results (files that don't exist) are as informative as positive ones—they force the assistant to update its mental model of the codebase and broaden its search.

Understanding the Harmony Task Framework

The assistant investigated the Harmony task framework, discovering the key interfaces:

The Critical Insight: Vanilla vs. SNARK Split

Reading the PoRep task's Do() method revealed the critical insight: PoRepSnark does two things—it generates a vanilla proof via GeneratePoRepVanillaProof (CPU, data-local) and then runs the SNARK computation via SealCommitPhase2 (GPU). For cuzk integration, only the SNARK step needs to be offloaded; the vanilla proof generation must remain local because it requires access to sector data stored on the local machine's filesystem.

This decomposition is the foundational insight that shapes the entire integration architecture. The vanilla proof (C1 for PoRep, PRU1 for SnapDeals) is a CPU-bound computation that requires access to sector data (sealed files, cache files) that may be terabytes in size. Sending this data over gRPC would be impractical. The SNARK proof (C2) is a GPU-bound computation that takes the vanilla proof as input and produces a compact Groth16 proof. The vanilla proof output (~50 MB for PoRep) is much smaller than the sector data (32 GiB), making it feasible to send over gRPC.

This split enables a new deployment architecture where storage-heavy machines (with large disk arrays) can be separate from computation-heavy machines (with multiple GPUs). The storage machine generates vanilla proofs locally and ships them to a GPU cluster for SNARK proving.

Building the Integration Infrastructure

Configuration: Adding CuzkConfig

The first step was adding a CuzkConfig section to Curio's configuration (deps/config/types.go). The configuration includes fields for the daemon address, per-task-type enable flags (PoRep, SnapDeals, WindowPoSt, WinningPoSt), and a MaxPending setting for backpressure control. This design provides a kill switch (empty address disables the integration), supports progressive rollout (enable one task type at a time), and configures the backpressure threshold.

Protobuf Code Generation

The assistant generated Go gRPC client stubs from the existing protobuf definitions in extern/cuzk/cuzk-proto/proto/cuzk/v1/proving.proto. This process was not straightforward—it involved multiple attempts with different protoc invocations, debugging a nested directory structure that the --go_opt=paths=source_relative flag created, and verifying that the generated files compiled correctly.

The protobuf service definition includes RPCs for SubmitProof, AwaitProof, Prove (combined submit+await), GetStatus, CancelProof, PreloadSRS, EvictSRS, and GetMetrics. The GetStatus RPC, which returns queue status per proof kind, is the critical endpoint for the backpressure design.

The gRPC Client

A new package lib/cuzk/ was created to house the generated protobuf stubs and a Go client wrapper. The client provides high-level methods like Prove() and GetStatus() that task handlers can call directly, abstracting away the gRPC connection management.

The Integration File: cuzk_funcs.go

The most architecturally significant file created is lib/ffi/cuzk_funcs.go. This file defines new methods on the existing SealCalls struct—the central Go type that wraps Filecoin's FFI calls. The new methods implement the split between local vanilla proof generation and remote SNARK proving:

The Integration Pattern: Consistent Across Task Types

The integration follows a consistent four-point pattern for each task type:

  1. Add a cuzkClient *cuzk.Client field to the task struct, following Go's dependency injection pattern.
  2. Zero out GPU/RAM requirements in TypeDetails() when cuzk is enabled, effectively telling Curio's harmony scheduler that the task consumes no local resources.
  3. Use daemon backpressure in CanAccept() by querying the cuzk daemon's queue depth instead of checking local GPU availability.
  4. Call a cuzk-aware SNARK function in Do() that splits proof generation into local vanilla proof creation and remote SNARK computation. This pattern represents a fundamental shift in Curio's resource accounting model. Normally, each task type declares its GPU and RAM requirements in TypeDetails(), and the harmony scheduler uses those to decide whether a task can run on the local machine. When cuzk is enabled, those requirements are zeroed—the task becomes "weightless" from the scheduler's perspective. The actual resource management moves to the daemon's queue, queried via CanAccept(). This two-level scheduling (Curio's local scheduler + daemon's queue) is a deliberate design for heterogeneous deployments where GPU resources are remote.

PoRep: The First Integration

The PoRep (Proof of Replication) seal task was the simplest and therefore the first target. The assistant read the existing task_porep.go file to understand its structure, then proceeded through the four-point plan methodically.

The first edit added the cuzk import and the cuzkClient field to the PoRepTask struct. This triggered an LSP error—the import was unused—which was resolved in the next edit by updating the constructor to store the client. The critical edit came next: modifying the Do() method to conditionally call PoRepSnarkCuzk when the client is non-nil, falling back to the existing PoRepSnark otherwise.

The CanAccept() adaptation was particularly elegant. Instead of simply rejecting tasks when cuzk is enabled (as the existing enableRemoteProofs mechanism did), the assistant made it query the daemon's queue depth. This transforms CanAccept() from a binary gate into a continuous backpressure valve: when the daemon is busy, it returns fewer tasks, and Curio's scheduler naturally distributes work accordingly. The TypeDetails() edit zeroed out GPU and RAM requirements, completing the pattern.

Each edit was followed by verification. After the PoRep integration was complete, the assistant ran go vet ./tasks/seal/, confirming that only pre-existing CGO errors appeared. The pattern was validated.

SnapDeals: Applying the Template

With the PoRep integration serving as a proven template, the SnapDeals prove task followed the same pattern with minimal adaptation. The assistant recognized that the SnapDeals task had an inverted enableRemoteProofs logic compared to PoRep—it used cfg.Subsystems.EnableRemoteProofs (when true, meaning "don't do locally") whereas PoRep used cfg.Subsystems.EnablePoRepProof (when false, meaning "don't do locally"). This discrepancy was neatly sidestepped by the cuzk integration: the cuzkClient != nil check overrides both legacy mechanisms uniformly.

The edits to tasks/snap/task_prove.go followed the same sequence: add the import and field, update the constructor, modify Do() to call SnapProveCuzk, adapt CanAccept() for backpressure, and zero out TypeDetails(). Each edit was verified with go vet, confirming the pattern's correctness.

The SnapDeals integration was notable for its smoothness. The assistant had learned from the PoRep experience and applied the pattern with confidence. The only hiccup was the "imported and not used" LSP error, which was resolved in the very next edit—a pattern that would recur in the proofshare integration.

Proofshare: The Cascade of Signatures

The proofshare (PSProve) task was the most complex of the three, and it was here that the simple template met its match. Unlike PoRep and SnapDeals, which called their SNARK functions directly from Do(), the proofshare task used a package-level helper function hierarchy: Do() called computeProof, which dispatched to computePoRep or computeSnap depending on the proof type. These helper functions directly invoked ffiselect.FFISelect.SealCommitPhase2 and ffiselect.FFISelect.GenerateUpdateProofWithVanilla—the actual GPU compute calls.

The assistant recognized this architectural difference in the planning phase: "For PSProve, the computeProof function is a package-level function that directly calls ffiselect.FFISelect.SealCommitPhase2 and ffiselect.FFISelect.GenerateUpdateProofWithVanilla. To integrate cuzk here, I need to pass the client through and create cuzk-aware variants of those compute functions."

The integration began with the familiar pattern: add the import, add the field, and update the constructor. Then came the critical edit: updating Do() to pass the *cuzk.Client to computeProof. This triggered the first LSP error:

ERROR [184:51] too many arguments in call to computeProof
    have (context.Context, harmonytask.TaskID, ProofData, *cuzk.Client)
    want (context.Context, harmonytask.TaskID, ProofData)

This error was the first domino in a cascade. The assistant updated computeProof's signature, which then revealed that computePoRep needed updating. Fixing computePoRep revealed that computeSnap needed updating. Fixing computeSnap finally resolved the cascade—but the assistant then discovered that the internal logic of computePoRep and computeSnap needed to actually use the client to delegate SNARK computation.

This cascade is a classic example of "shotgun surgery" refactoring—a change that requires modifying multiple scattered functions. The assistant's incremental, error-driven approach was both pragmatic and revealing. Rather than attempting to predict every signature change in advance, the assistant made one edit, observed the LSP error, and fixed the next function in the chain. This workflow mirrors how experienced human developers work when refactoring deep call chains: trust the type system to guide the sequence of changes.

The cascade also revealed a key insight about the proofshare task's architecture. The helper function hierarchy existed because proofshare handles multiple proof types (PoRep and SnapDeals) through a unified dispatch mechanism. This abstraction, while elegant, created a deeper call chain that required more invasive refactoring than the flat Do() → SNARK function pattern used by the other tasks.

Build Verification and Diagnostic Filtering

After writing cuzk_funcs.go, the assistant ran go build ./lib/ffi/ to verify compilation. The build failed—but not because of the new code. The errors were all in extern/filecoin-ffi/cgo/fvm.go, a CGO binding file that requires FVM (Filecoin Virtual Machine) C headers that were not installed on the development machine.

The assistant's response is a masterclass in diagnostic filtering. Rather than installing the missing headers or ignoring the errors, the assistant iterated on the verification strategy. The first attempt filtered output with grep -v to exclude known noise patterns. The second, more principled approach used the nofvm build tag: go vet -tags nofvm ./lib/ffi/. This instructed the Go toolchain to skip FVM-dependent code paths entirely, performing static analysis only on the relevant code. The result was clean—no Go-level errors—confirming that the new bridge code was sound and that the CGO issues were pre-existing environmental artifacts.

This moment exemplifies a critical engineering skill: knowing which build failures to fix and which to work around. The assistant correctly identified that the FVM errors were pre-existing, environmental, and unrelated to the integration work. Rather than spending time installing C headers for a dependency they didn't need to modify, the assistant pivoted to a targeted verification strategy and proceeded.

Backpressure Design: A Feedback Control System

The backpressure design is one of the most elegant aspects of the integration. Rather than having Curio's scheduler guess at the daemon's capacity based on local resource counters (which would be meaningless when the daemon is on a different machine), the design uses a feedback loop:

  1. Curio's scheduler proposes tasks to CanAccept().
  2. CanAccept() queries the cuzk daemon's GetStatus() RPC.
  3. The daemon returns queue state (pending and in-progress counts per proof type).
  4. CanAccept() computes how many tasks it can accept based on MaxPending minus current pending.
  5. Curio assigns the accepted tasks to Do().
  6. Do() calls cuzkClient.Prove() to submit the proof and wait for the result. This is a pull-based backpressure system: Curio pulls capacity information from the daemon before submitting work. It prevents the daemon from being overwhelmed while keeping Curio's scheduler informed about actual capacity. The MaxPending configuration parameter acts as a governor, limiting how many tasks Curio will queue on the daemon at any given time.

The Broader Significance

The work in this segment represents a transition from engine development to production integration. Before this segment, cuzk was a standalone proving engine—powerful but isolated. After this segment, it becomes part of Curio's production workflow, with a clean integration architecture, a gRPC client library, and modifications to three task types.

Several engineering principles emerge from this work:

Delegation, not duplication: Rather than duplicating resource management logic in both systems, the design delegates GPU scheduling to cuzk and task lifecycle to Curio. Each system focuses on its core competency.

Clean boundaries: The boundary between Curio and cuzk is defined by the gRPC protocol and the CanAccept() backpressure mechanism. Each system has clear responsibilities.

Progressive scope: Starting with three task types (PoRep, SnapDeals, PSProve) allows the team to validate the integration pattern before expanding to other proof types like WindowPoSt and WinningPoSt.

Diagnostic discipline: The assistant's response to the CGO build failure—filtering noise, recognizing environmental limitations, pivoting to targeted verification—demonstrates the importance of diagnostic filtering in complex systems.

Incremental integration with verification gates: The assistant's workflow—plan, read, edit, verify, proceed—reflects a disciplined engineering approach. Each task type was integrated independently, with verification steps (LSP diagnostics, go vet) serving as gates between phases.

Conclusion

Segment 33 captures a remarkable arc of software engineering: from low-memory benchmarking and RAM tier recommendations, through architectural planning and codebase exploration, to the creation of a gRPC client library and the wiring of a remote proving daemon into a production task orchestrator. The work demonstrates the value of methodical research before implementation, the power of concise architectural direction, and the importance of diagnostic filtering when navigating complex build environments.

The resulting integration architecture—with its clean separation of concerns, feedback-based backpressure, and consistent per-task-type pattern—provides a solid foundation for the next phase of work: completing the wiring of all four task types, adding WindowPoSt and WinningPoSt support, and production hardening. The cuzk proving engine has taken its place as a production component of the Curio storage mining platform.

For Filecoin storage providers, this integration means they can separate storage capacity from proving capacity. Cheap, high-density machines can handle sector storage and vanilla proof generation, while a dedicated pool of cuzk daemons—running on machines with ample GPU and memory—handles the expensive SNARK computation. This decoupling has the potential to reduce hardware costs, improve GPU utilization, and enable more flexible deployment topologies. The code changes in this segment are the bridge that makes that vision a reality.