Wiring Remote GPU Proving into Curio: A Systematic Integration Journey

Introduction

In the landscape of Filecoin storage proving, few operations are as computationally demanding as Groth16 proof generation. The cuzk proving daemon—a persistent GPU service developed over a dozen optimization phases—was built to offload these memory-intensive SNARK computations from Curio's local task workers to a remote proving cluster. But a daemon without clients is merely a technical curiosity. The real engineering challenge lay in bridging this new service with Curio's existing task orchestrator, adapting three distinct proof-related task types to delegate their GPU work to the daemon while preserving backward compatibility, scheduler semantics, and resource accounting integrity.

This article synthesizes the integration work captured in a single chunk of an opencode coding session, where the assistant systematically wired the cuzk remote proving daemon into Curio's PoRep (seal), SnapDeals prove, and proofshare (PSProve) tasks. The work followed a consistent architectural pattern across all three task types, but each presented unique challenges—from the straightforward conditional branching in PoRep to the cascading signature refactoring required by proofshare's deeper call chain. Understanding this integration reveals not only how remote GPU proving was achieved, but also the iterative, error-driven methodology that characterized the implementation.

The Architectural Pattern: Four Points of Integration

Before any code was written, the assistant articulated a clear four-point plan for integrating cuzk into each task type ([msg 3420]). This plan became the template for all subsequent work:

  1. Add a cuzkClient *cuzk.Client field to the task struct, following Go's dependency injection pattern.
  2. Zero out GPU/RAM requirements in TypeDetails() when cuzk is enabled, effectively telling Curio's harmony scheduler that the task consumes no local resources.
  3. Use daemon backpressure in CanAccept() by querying the cuzk daemon's queue depth instead of checking local GPU availability.
  4. Call a cuzk-aware SNARK function in Do() that splits proof generation into local vanilla proof creation and remote SNACK computation. This pattern represents a fundamental shift in Curio's resource accounting model. Normally, each task type declares its GPU and RAM requirements in TypeDetails(), and the harmony scheduler uses those to decide whether a task can run on the local machine. When cuzk is enabled, those requirements are zeroed—the task becomes "weightless" from the scheduler's perspective. The actual resource management moves to the daemon's queue, queried via CanAccept(). This two-level scheduling (Curio's local scheduler + daemon's queue) is a deliberate design for heterogeneous deployments where GPU resources are remote. The architectural insight behind the split is equally important. In Filecoin's proof pipeline, proof generation has two distinct phases. The "vanilla proof" phase (GeneratePoRepVanillaProof) requires access to the sector data on disk—it reads sealed and unsealed sector files, computes Merkle trees, and produces an intermediate representation. This phase is CPU-bound and storage-bound, and it must run on the machine that physically holds the sector. The second phase, "SNARK computation" (SealCommitPhase2), takes that intermediate representation and produces the final Groth16 proof. This phase is GPU-bound and memory-intensive, consuming up to ~200 GiB of RAM. The cuzk daemon was built specifically to handle this second phase remotely, pooling GPU resources across multiple miner machines. The integration preserves this split: vanilla proof generation stays local, while only the SNARK computation is offloaded.

Verification as a Prerequisite

Before any task integration could begin, the assistant needed to verify that the new bridge code—the PoRepSnarkCuzk and SnapProveCuzk methods written in lib/ffi/cuzk_funcs.go—compiled correctly. This seemingly mundane step became a study in pragmatic verification ([msg 3419]). The direct approach, go build ./lib/ffi/, failed due to CGO (C-Go binding) errors from the Filecoin FFI library's FVM integration, which required C headers not installed on the development machine.

Rather than installing the missing headers or ignoring the errors, the assistant iterated on the verification strategy. The first attempt filtered output with grep -v to exclude known noise patterns. The second, more principled approach used the nofvm build tag: go vet -tags nofvm ./lib/ffi/. This instructed the Go toolchain to skip FVM-dependent code paths entirely, performing static analysis only on the relevant code. The result was clean—no Go-level errors—confirming that the new bridge code was sound and that the CGO issues were pre-existing environmental artifacts. This verification checkpoint was the green light that enabled the entire integration effort to proceed.

PoRep: The First Integration

The PoRep (Proof of Replication) seal task was the simplest and therefore the first target. The assistant read the existing task_porep.go file ([msg 3421]) to understand its structure, then proceeded through the four-point plan methodically.

The first edit added the cuzk import and the cuzkClient field to the PoRepTask struct ([msg 3422]). This triggered an LSP error—the import was unused—which was resolved in the next edit ([msg 3423]) by updating the constructor to store the client. The critical edit came next ([msg 3424]): modifying the Do() method to conditionally call PoRepSnarkCuzk when the client is non-nil, falling back to the existing PoRepSnark otherwise.

The CanAccept() adaptation ([msg 3425]) was particularly elegant. Instead of simply rejecting tasks when cuzk is enabled (as the existing enableRemoteProofs mechanism did), the assistant made it query the daemon's queue depth. This transforms CanAccept() from a binary gate into a continuous backpressure valve: when the daemon is busy, it returns fewer tasks, and Curio's scheduler naturally distributes work accordingly. The TypeDetails() edit ([msg 3426]) zeroed out GPU and RAM requirements, completing the pattern.

Each edit was followed by verification. After the PoRep integration was complete, the assistant ran go vet ./tasks/seal/ ([msg 3427]), confirming that only pre-existing CGO errors appeared. The pattern was validated.

SnapDeals: Applying the Template

With the PoRep integration serving as a proven template, the SnapDeals prove task followed the same pattern with minimal adaptation. The assistant recognized that the SnapDeals task had an inverted enableRemoteProofs logic compared to PoRep—it used cfg.Subsystems.EnableRemoteProofs (when true, meaning "don't do locally") whereas PoRep used cfg.Subsystems.EnablePoRepProof (when false, meaning "don't do locally"). This discrepancy, discovered during earlier research ([msg 3382]), was neatly sidestepped by the cuzk integration: the cuzkClient != nil check overrides both legacy mechanisms uniformly.

The edits to tasks/snap/task_prove.go followed the same sequence: add the import and field ([msg 3430]), update the constructor ([msg 3431]), modify Do() to call SnapProveCuzk ([msg 3432]), adapt CanAccept() for backpressure ([msg 3433]), and zero out TypeDetails() ([msg 3434]). Each edit was verified with go vet ([msg 3435]), confirming the pattern's correctness.

The SnapDeals integration was notable for its smoothness. The assistant had learned from the PoRep experience and applied the pattern with confidence. The only hiccup was the "imported and not used" LSP error, which was resolved in the very next edit—a pattern that would recur in the proofshare integration.

Proofshare: The Cascade of Signatures

The proofshare (PSProve) task was the most complex of the three, and it was here that the simple template met its match. Unlike PoRep and SnapDeals, which called their SNARK functions directly from Do(), the proofshare task used a package-level helper function hierarchy: Do() called computeProof, which dispatched to computePoRep or computeSnap depending on the proof type. These helper functions directly invoked ffiselect.FFISelect.SealCommitPhase2 and ffiselect.FFISelect.GenerateUpdateProofWithVanilla—the actual GPU compute calls.

The assistant recognized this architectural difference in the planning phase ([msg 3438]): "For PSProve, the computeProof function is a package-level function that directly calls ffiselect.FFISelect.SealCommitPhase2 and ffiselect.FFISelect.GenerateUpdateProofWithVanilla. To integrate cuzk here, I need to pass the client through and create cuzk-aware variants of those compute functions."

The integration began with the familiar pattern: add the import ([msg 3438]), add the field and update the constructor ([msg 3439]). Then came the critical edit: updating Do() to pass the *cuzk.Client to computeProof ([msg 3440]). This triggered the first LSP error:

ERROR [184:51] too many arguments in call to computeProof
    have (context.Context, harmonytask.TaskID, ProofData, *cuzk.Client)
    want (context.Context, harmonytask.TaskID, ProofData)

This error was the first domino in a cascade. The assistant updated computeProof's signature ([msg 3441]), which then revealed that computePoRep needed updating. Fixing computePoRep ([msg 3442]) revealed that computeSnap needed updating. Fixing computeSnap ([msg 3443]) finally resolved the cascade—but the assistant then discovered that the internal logic of computePoRep and computeSnap needed to actually use the client to delegate SNARK computation ([msg 3444]).

This cascade is a classic example of "shotgun surgery" refactoring—a change that requires modifying multiple scattered functions. The assistant's incremental, error-driven approach was both pragmatic and revealing. Rather than attempting to predict every signature change in advance, the assistant made one edit, observed the LSP error, and fixed the next function in the chain. This workflow mirrors how experienced human developers work when refactoring deep call chains: trust the type system to guide the sequence of changes.

The cascade also revealed a key insight about the proofshare task's architecture. The helper function hierarchy existed because proofshare handles multiple proof types (PoRep and SnapDeals) through a unified dispatch mechanism. This abstraction, while elegant, created a deeper call chain that required more invasive refactoring than the flat Do() → SNARK function pattern used by the other tasks.

The Silence Between Actions

After the final proofshare edit, the session reached a natural pause point. The user sent an empty message ([msg 3445])—no text, no instruction, no question. In the context of the session, this silence was meaningful. The assistant interpreted it not as a command to stop, but as an implicit request for status ([msg 3446]), producing a comprehensive summary document that captured the goal, accomplishments, discoveries, and remaining work.

This moment highlights the collaborative dynamics of the session. The assistant had been working autonomously through a complex integration, and the empty message signaled that the user wanted to review progress before deciding on next steps. The assistant's response—a structured markdown document rather than continued coding—demonstrated an understanding that communication is sometimes more valuable than action. The summary served as both a status report and a design document, capturing the architectural decisions, the verification strategy, and the open questions about WindowPoSt and WinningPoSt tasks.

Themes and Architectural Insights

Several themes emerge from this integration effort:

Remote GPU offloading as a scheduling inversion. The most architecturally significant decision was the inversion of Curio's resource accounting. By zeroing local GPU and RAM requirements in TypeDetails() and relying on daemon-side backpressure in CanAccept(), the integration effectively tells Curio's scheduler: "Don't worry about local resources; ask the daemon if it has capacity." This two-level scheduling model is a clean separation of concerns—Curio manages task lifecycle and persistence, while the daemon manages GPU allocation and memory pressure.

The split-pipeline design. The decision to keep vanilla proof generation local while offloading only SNARK computation was driven by data locality. Vanilla proofs require access to sealed sector data on disk, which cannot be efficiently moved to a remote service. The SNARK computation, by contrast, operates on compact intermediate representations (a few kilobytes) and is purely GPU-bound. This split minimizes data transfer while maximizing the benefit of centralized GPU resources.

Incremental integration with verification gates. The assistant's workflow—plan, read, edit, verify, proceed—reflects a disciplined engineering approach. Each task type was integrated independently, with verification steps (LSP diagnostics, go vet) serving as gates between phases. When the proofshare task required a more complex cascade of signature changes, the assistant maintained the same incremental discipline, letting the type system guide each subsequent edit.

Pattern reusability and its limits. The four-point integration pattern proved directly reusable for PoRep and SnapDeals, but the proofshare task required adaptation due to its deeper call chain. The assistant's recognition of this difference—and its willingness to work through the cascade rather than forcing the flat pattern—demonstrates the importance of understanding architectural variation beneath surface-level similarity.

Conclusion

The integration of the cuzk remote GPU proving daemon into Curio's task orchestrator was a systematic effort spanning three task types, each with its own architectural nuances. The work followed a consistent pattern—add client field, zero local resources, query daemon for backpressure, delegate SNARK computation—but required adaptation for the proofshare task's deeper call chain. The assistant's incremental, error-driven methodology, combined with careful verification at each step, produced a robust integration that fundamentally rearchitects how Filecoin proof generation is distributed across a cluster.

For Filecoin storage providers, this integration means they can separate storage capacity from proving capacity. Cheap, high-density machines can handle sector storage and vanilla proof generation, while a dedicated pool of cuzk daemons—running on machines with ample GPU and memory—handles the expensive SNARK computation. This decoupling has the potential to reduce hardware costs, improve GPU utilization, and enable more flexible deployment topologies. The code changes in this chunk are the bridge that makes that vision a reality.