The Final Stitch: Wiring a GPU Proving Daemon into Curio's Task Orchestrator

Introduction

In large-scale distributed systems, the most architecturally significant changes often culminate in deceptively simple summary messages. Message [msg 3482] is precisely such a message: a concise status report from an AI assistant working on the Filecoin Curio project, declaring that four remaining TODO items are complete and that the integration of a custom GPU-resident SNARK proving engine — called "cuzk" — into Curio's task orchestration layer is now "functionally complete."

But behind this brief summary lies a complex engineering journey spanning dozens of prior messages, hundreds of code edits, and a fundamental rethinking of how Filecoin storage providers generate the Groth16 zero-knowledge proofs required for Proof-of-Replication (PoRep) and SnapDeals. This message represents the moment when a multi-phase optimization project — one that began with deep-dive analysis of ~200 GiB memory footprints and CUDA kernel hotpaths — finally reaches the integration finish line, connecting a purpose-built proving daemon to Curio's existing task scheduling infrastructure.

The Broader Context: Why cuzk Exists

To understand this message, one must first understand the problem it solves. Filecoin storage providers must periodically generate SNARK proofs to convince the network that they are still storing the data they committed to store. These proofs — Groth16 proofs over the BLS12-381 curve — are computationally expensive, requiring GPU-accelerated number-theoretic transforms (NTTs) and multi-scalar multiplications (MSMs) that can consume up to 200 GiB of peak memory per proof. The existing approach, implemented in the supraseal-c2 library, was monolithic: each proof request would load the Structured Reference String (SRS) into GPU memory, perform all computation synchronously, and then tear down the GPU state, incurring massive overhead from repeated SRS loading and GPU initialization.

The cuzk project (short for "CUDA SNARK") was conceived as a persistent, pipelined proving engine — analogous to how vLLM or TensorRT serve inference models in the AI world. Instead of starting from scratch for each proof, cuzk keeps the GPU warm, maintains SRS residency, and pipelines proof requests through a multi-stage architecture: CPU-based circuit synthesis feeds into GPU-based NTT/MSM computation, with careful memory management to prevent out-of-memory conditions. The project progressed through 13 phases (Phase 0 through Phase 12), each addressing a specific bottleneck: SRS swapping, pipelined synthesis, cross-sector batching, pre-compiled constraint evaluation, split GPU proving APIs, and memory backpressure mechanisms.

What This Message Actually Says

The subject message is a summary written after the assistant completed four remaining integration tasks. It is structured as a status report with three sections: a list of completed items, a brief description of each, and an "Integration Status" conclusion. The assistant writes:

All four remaining TODO items are now complete. Here's a summary of what was done in this session:

The four completed items are:

  1. PSProve TypeDetails/CanAccept — Updates to the proof-share task type to report zero GPU requirements and delegate capacity checks to the cuzk daemon when enabled.
  2. Wired cuzk client in cmd/curio/tasks/tasks.go — Initialization of the cuzk gRPC client at task startup and passing it to all three proof-related task constructors.
  3. Verified all packages with go vet — Confirmation that the modified Go code compiles and passes static analysis.
  4. Updated cuzk-project.md — Documentation of Phase 13 (the Curio integration phase) in the project's comprehensive design document. The message concludes with the critical integration status statement:
The Curio-side integration is now functionally complete. All three task types (PoRep C2, SnapDeals Prove, PSProve) are wired to use the cuzk daemon when Cuzk.Address is set in config. When empty (default), behavior is unchanged from before.

This last sentence is architecturally significant: it establishes a graceful fallback path. If no cuzk daemon is configured, Curio falls back to the original inline proof generation. This means the integration is non-disruptive — storage providers can adopt cuzk incrementally without breaking existing workflows.

Deep Dive: The Four Tasks

Task 1: PSProve TypeDetails/CanAccept with Cuzk Backpressure

The first task involved modifying the PSProve task type — a task handler for proof-share operations that handles both PoRep and SnapDeals proofs. Two methods needed updating:

TypeDetails() reports the resource requirements of a task to the Curio scheduler. When cuzk is enabled, the assistant zeroes the GPU requirement and sets RAM to 1 GiB (a nominal value). This is a critical signaling mechanism: since cuzk runs as an external daemon with its own GPU management, the Curio scheduler should not reserve GPU resources locally. By reporting zero GPU requirements, the scheduler will avoid over-committing GPU capacity and will not attempt to serialize proof tasks based on local GPU availability.

CanAccept() is the admission control method that determines whether a task can be accepted for execution. The assistant added a cuzk backpressure check: when the cuzk client is enabled, the method calls HasCapacity() on the daemon's queue status rather than checking local parameter availability. This delegates capacity decisions to the daemon, which has real-time visibility into its GPU queue depth, memory pressure, and processing throughput. The existing params-ready check is preserved in the else branch for non-cuzk mode.

The assistant also fixed a variable scoping bug (err → := declaration) that arose from restructuring the conditional logic — a small but necessary correction that prevented a compilation error.

Task 2: Wiring the cuzk Client in tasks.go

The second task was the most structurally significant: integrating the cuzk client initialization into Curio's main task registration function, addSealingTasks. This function is called at startup to register all task handlers with the harmonytask engine.

The assistant added an import for the lib/cuzk package, then created a cuzk.Client instance from configuration parameters (Cuzk.Address, MaxPending, ProveTimeout). This client was then passed as a new argument to three constructor functions:

Task 3: Verification with go vet

After making the code changes, the assistant ran go vet on all affected packages: lib/cuzk, tasks/proofshare, tasks/seal, tasks/snap, deps/config, cmd/curio/tasks, and lib/ffi. The output showed only pre-existing errors from the extern/filecoin-ffi/cgo/fvm.go file — a known issue with CGO bindings for the Filecoin Virtual Machine that is unrelated to the cuzk changes. All cuzk-related packages passed cleanly.

This verification step is crucial in a production codebase like Curio, which manages real Filecoin storage provider operations. A compilation error or vet warning could prevent the entire binary from building, affecting live proving operations.

Task 4: Documentation Update

The final task was updating cuzk-project.md — a 1,984-line design document that has grown alongside the project through all 13 phases. The assistant added:

Reasoning and Decision-Making

The subject message is a summary, so it does not contain the assistant's reasoning in real time. However, the reasoning is visible in the preceding messages ([msg 3453] through [msg 3481]), which show the assistant working through the TODO list step by step.

Several key decisions are implicit in the work:

Decision 1: Delegate capacity checks to the daemon. Rather than implementing a separate capacity estimation algorithm in Curio (e.g., tracking GPU memory usage locally), the assistant chose to delegate entirely to the cuzk daemon's HasCapacity() method. This is the right architectural choice because the daemon has perfect information about its internal state — queue depth, GPU utilization, memory pressure — while Curio would only have stale or approximate information. This follows the principle of "tell, don't ask": Curio tells the daemon what it needs, and the daemon decides whether it can handle it.

Decision 2: Zero GPU requirements in TypeDetails. This is a subtle but important design choice. If the assistant had left the GPU requirement at its default value, the Curio scheduler might have attempted to serialize proof tasks based on GPU availability, even though the actual GPU work is happening in the external daemon. By zeroing the GPU requirement, the assistant ensures that Curio's scheduler treats proof tasks as CPU-only from its perspective, allowing multiple proof requests to be dispatched concurrently to the daemon.

Decision 3: Graceful fallback via empty config. The assistant ensured that when Cuzk.Address is empty (the default), behavior is completely unchanged. This is essential for a production rollout: storage providers can upgrade to a version of Curio that includes the cuzk integration without immediately configuring a daemon. They can set up the daemon later, restart Curio with the address configured, and the integration activates automatically.

Decision 4: Pass client to constructors rather than using global state. The cuzk client is created once at startup and passed explicitly to each task constructor. This avoids global mutable state, makes the dependency explicit, and allows different task types to potentially use different cuzk configurations in the future.

Assumptions Made

The assistant made several assumptions during this work:

  1. The cuzk daemon is always reachable when configured. The code does not implement retry logic or health-check polling at the Curio level. If the daemon crashes or becomes unreachable, proof tasks will likely fail with connection errors. The assumption is that the daemon's own resilience mechanisms (or external monitoring) will handle this.
  2. HasCapacity() is a sufficient backpressure signal. The assistant assumes that the daemon's capacity check accurately reflects its ability to accept new work. If the daemon's queue management is flawed (e.g., it accepts work that later causes OOM), Curio has no defense.
  3. The three task types (PoRep, SnapDeals, PSProve) can all use the same client configuration. The assistant passes the same cuzkClient instance to all three constructors. This assumes that the daemon's queue capacity and timeout settings are appropriate for all proof types, even though they may have different computational profiles.
  4. Zero GPU requirement is safe. By reporting zero GPU requirements, the assistant assumes that no other part of Curio's scheduler will misinterpret this value. If some scheduling logic uses GPU requirements for prioritization or fairness, zero could cause unexpected behavior.
  5. The pre-existing CGO/FVM vet errors are harmless. The assistant correctly identifies these as pre-existing and unrelated, but this is an assumption that could be wrong if a future Go version or build environment treats these errors more strictly.

Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs:

Knowledge Created by This Message

This message produces several important pieces of knowledge:

  1. Integration completion signal. The most important output is the confirmation that the Curio-side integration is functionally complete. This is a milestone that unblocks deployment and testing.
  2. Graceful fallback guarantee. The message establishes that existing behavior is preserved when cuzk is not configured. This is critical knowledge for operators considering adoption.
  3. Three task types are covered. PoRep C2, SnapDeals Prove, and PSProve — all proof-generating paths in Curio — are wired to use cuzk. No proof path is left behind.
  4. Documentation is updated. The cuzk-project.md file now contains Phase 13 documentation, making the integration visible to anyone reading the design document.
  5. Code quality is verified. The go vet results confirm that the changes compile cleanly and do not introduce new issues.

The Thinking Process

While the subject message itself is a summary, the thinking process that led to it is visible in the preceding messages. The assistant worked through the TODO list methodically:

In [msg 3453], the assistant laid out the four tasks with priorities and statuses. This structured approach — listing tasks, then working through them one by one — reflects a disciplined engineering workflow.

In [msg 3454], the assistant started with the PSProve TypeDetails() change, applying the edit and immediately checking for LSP errors. When errors appeared in [msg 3455], the assistant read the full file to understand the scoping issue, then fixed it with a targeted edit in [msg 3457]. This iterative edit-fix cycle is characteristic of careful development.

In [msg 3459], the assistant planned the tasks.go changes by reading the file to find exact constructor call sites and line numbers before making any edits. This "read first, edit second" approach minimizes mistakes.

In [msg 3465], the assistant ran go vet on multiple packages, carefully distinguishing between pre-existing errors and new ones. The assistant noted that the CGO/FVM errors were "not related to our changes" — a correct diagnosis that prevented unnecessary debugging.

In [msg 3468] through [msg 3480], the assistant updated the documentation by reading the existing file, finding the right insertion points, and making targeted edits. The assistant even counted lines (1,984) and used grep to find relevant sections.

Throughout this process, the assistant used a TODO tracking tool to mark items as "in_progress" and "completed," maintaining situational awareness across a multi-step workflow.

Conclusion

Message [msg 3482] is the final stitch in a long integration tapestry. It represents the moment when a complex, multi-phase optimization project — one that began with deep analysis of GPU memory bottlenecks and CUDA kernel characteristics — finally connects to the production task orchestrator that Filecoin storage providers actually run. The message is concise, but the work behind it spans dozens of files, hundreds of lines of code, and careful architectural decisions about capacity delegation, graceful fallback, and resource reporting.

For the cuzk project, this message marks the transition from development to integration. The proving engine exists, the benchmarks are done, the documentation is written, and now — with these four final tasks — Curio knows how to talk to it. The next phase will be deployment, testing, and real-world validation on storage provider hardware. But for this moment, the integration is complete, and the message captures that milestone with clarity and precision.