Architecting the Cuzk-Curio Integration: A Study in Distributed Proving System Design

Introduction

In the evolution of the SUPRASEAL_C2 Groth16 proof generation pipeline, a critical architectural transition occurs when the system moves from a monolithic, locally-bound proving engine to a distributed architecture where proof computation is offloaded to a remote daemon. Message [msg 3361] captures this pivotal moment: the assistant is deep in the design phase of integrating the cuzk proving daemon into Curio's task orchestrator, reasoning through configuration structure, proof routing, backpressure semantics, and resource accounting. This message is not about writing code—it is about making architectural decisions that will shape how Filecoin storage providers deploy GPU-accelerated proof generation across heterogeneous hardware.

The message sits at the boundary between two major phases of the project. The preceding segments (28–32) focused on optimizing the cuzk proving engine itself—implementing the split GPU proving API, diagnosing DDR5 memory bandwidth contention, adding memory backpressure, and benchmarking to establish RAM-tier recommendations. With Phase 12 finalized and committed as 9bb657e5, the assistant now pivots to the integration problem: how does cuzk—a standalone daemon with its own gRPC API, queue management, and resource accounting—plug into Curio's existing harmony task scheduler? This is a systems integration problem of considerable subtlety, and the assistant's reasoning in this message reveals the design tensions that arise when two independently-developed systems must cooperate.

The Message in Full

The assistant begins by surfacing a discrepancy in the project's documentation:

Investigating cuzk Client

Okay, so I dug into this cuzk thing and found some potential discrepancies. The notes point towards a gRPC client in lib/cuzk/client.go, but that file doesn't seem to exist. I ran a find command and, instead, the trail leads to ./extern/cuzk and a nested path within that. This warrants further investigation into the extern directory to clarify the location and status of the client code.

This opening reveals a critical moment of discovery. The assistant had been working from documentation or notes that referenced a gRPC client at lib/cuzk/client.go, but empirical investigation via find showed that no such file exists. Instead, the cuzk code lives under ./extern/cuzk—a directory structure that suggests it is an external dependency, possibly vendored or cloned. This discrepancy between documentation and reality is the catalyst for the entire reasoning chain that follows.

The assistant then pivots from investigation to planning:

Planning Integration with Cuzk

I've shifted focus to planning the cuzk integration with curio, as the implementation is out of scope for now. I'm now drafting the integration plan for the user, starting with updates to the configuration file. My plan involves introducing a Cuzk config section in deps/config/types.go that includes an enable flag and the gRPC server address.

This is a deliberate scoping decision. The assistant recognizes that implementation is premature—the architecture must be settled first. The plan begins at the configuration layer, which is the natural entry point: Curio's configuration file (deps/config/types.go) needs a new CuzkConfig section. The minimal viable config requires just two fields: an Enabled boolean and the daemon's gRPC address.

The assistant then drills into proof routing:

Defining Proof Routing Logic

I'm now determining how to route specific proofs to cuzk. My current approach involves an EnabledProofs configuration option. By default, I'm thinking of enabling all proof types when Cuzk.Enabled is set to true. I am drafting deps/config/types.go to include this new configuration. I plan to extend the configuration to support PoRep C2, Snap, and WindowPost proof types as specific examples.

This reveals a design decision with significant operational implications. The assistant considers an EnabledProofs list that would allow operators to selectively route certain proof types to the remote daemon while keeping others local. The default behavior—all proofs enabled—is the simplest path, but the list-based approach provides future flexibility for heterogeneous deployments where some proof types might benefit more from offloading than others.

The most architecturally significant reasoning appears in the next two sections:

Revising Acceptance Logic

I'm now revising the CanAccept logic within tasks. Instead of the usual RAM/GPU checks, which Curio handles, I need to bypass those resource constraints if Cuzk.Enabled is true. Cuzk will manage its own scheduling, SRS memory, and queue backpressure. I need to update task-specific CanAccept methods, like PoRepTask.CanAccept, to reflect this change.

Refining Task Acceptance

I'm now refining the CanAccept logic in conjunction with the cuzk daemon. I've realized CanAccept returns accepted TaskIDs, not just a boolean. Now, I'm integrating calls to cuzkClient.GetStatus(ctx) to check the queue capacity of specific proof types. The queue's pending size relative to a configurable limit will determine whether a task is accepted. This will avoid overwhelming the cuzk daemon. I also have to ensure that Curio's task engine doesn't deduct local resources if cuzk handles it.

These paragraphs encode the core architectural insight of the entire integration. Curio's harmony scheduler uses CanAccept() as a gate: each task type implements this method to declare whether it has sufficient local resources (GPU, RAM) to execute a task. When cuzk is enabled, this logic must be inverted. Instead of checking local GPU availability, the task should check the cuzk daemon's queue depth. And instead of reporting local resource costs, TypeDetails() should zero them out, so Curio's scheduler treats cuzk-enabled tasks as essentially free in terms of local resources.

The message concludes with a concrete code exploration:

grep -A 10 "TypeDetails() harmonytask.TaskTypeDetails" ./tasks/seal/task_porep.go
func (p *PoRepTask) TypeDetails() harmonytask.TaskTypeDetails {
    gpu := 1.0
    mem := uint64(128 << 30) // for GPU sealing. 160 for CPU sealing, which we pretend nobody uses.
    if IsDevnet {
        gpu = 0
        mem = 1 << 30
    }
    res := harmonytask.TaskTypeDetails{
        Max:  taskhelp.Max(p.max),
        Name: "PoRep",
        Cost: resources.Resources{

This grep output shows the current state of TypeDetails() for the PoRep task. It declares 1 GPU and 128 GiB of RAM as the local resource cost. The assistant is examining this code to understand exactly what needs to change: when cuzk is enabled, these resource declarations must be zeroed, and the CanAccept logic must query the daemon instead of checking local availability.

The Reasoning Process: From Investigation to Architecture: From Investigation to Architecture

The assistant's reasoning in [msg 3361] follows a clear trajectory: discover, scope, plan, refine, validate. This is a textbook example of how an AI assistant navigates ambiguity in a large codebase.

The first section ("Investigating cuzk Client") is a discovery phase. The assistant finds a discrepancy between documentation and reality. The notes referenced lib/cuzk/client.go, but the actual code lives in ./extern/cuzk. This is not a trivial detail—it changes the integration strategy. If the gRPC client already existed at lib/cuzk/client.go, the integration would be a matter of importing and calling it. But since it doesn't exist, the assistant must create it, which means understanding the protobuf definitions in ./extern/cuzk/cuzk-proto/proto/cuzk and generating Go stubs. This discovery shapes the entire implementation plan that follows.

The second section ("Planning Integration with Cuzk") is a scoping decision. The assistant explicitly states "implementation is out of scope for now." This is a crucial metacognitive move: the assistant recognizes that it is in a planning phase and should not prematurely jump to coding. Instead, it focuses on the architectural skeleton—the configuration file. This prioritization reflects an understanding that configuration is the contract between systems; once the config structure is right, the code that reads it falls into place naturally.

The third section ("Defining Proof Routing Logic") introduces the EnabledProofs concept. This is where the assistant makes a design choice that balances simplicity against flexibility. The default "all proofs enabled" is the simplest path, but the list-based approach anticipates future needs. This kind of forward-thinking design is characteristic of experienced systems architects.

The fourth and fifth sections ("Revising Acceptance Logic" and "Refining Task Acceptance") contain the deepest architectural reasoning. The assistant works through the implications of the CanAccept interface. It discovers that CanAccept returns a list of accepted TaskIDs, not a boolean—a subtle but important detail that means the method can selectively accept or reject individual tasks from a batch. The assistant then designs the integration with cuzkClient.GetStatus(ctx) to query queue capacity, introducing a configurable limit to prevent overwhelming the daemon.

The final bash command is a validation step. The assistant checks the actual TypeDetails() implementation in task_porep.go to confirm its understanding of the current resource accounting. This grounds the architectural reasoning in concrete code, ensuring that the planned changes are feasible.

Assumptions and Their Implications

The assistant makes several assumptions in this message, each with downstream consequences:

Assumption 1: The cuzk daemon exposes a GetStatus RPC that returns queue capacity by proof type. This is a critical assumption because the entire backpressure mechanism depends on it. If the daemon does not expose such an endpoint, the assistant will need to either add it to the protobuf definitions or design a different backpressure strategy (e.g., relying on gRPC connection errors or timeouts). The assumption is reasonable—any well-designed daemon should expose its load—but it is unverified at this point.

Assumption 2: Curio's task engine does not deduct local resources when TypeDetails() reports zero costs. The assistant plans to zero out GPU and RAM costs when cuzk is enabled, but this assumes that Curio's scheduler handles zero-cost tasks correctly. If the scheduler has edge cases for zero-resource tasks (e.g., assuming they are idle tasks or skipping them in priority calculations), this could cause unexpected behavior. The assistant is relying on the harmony task engine's documented behavior.

Assumption 3: The EnabledProofs configuration option is sufficient for routing. The assistant assumes that a static list of proof types in the config file is the right abstraction. An alternative would be dynamic routing based on daemon load, proof complexity, or sector priority. The static list is simpler but less adaptive.

Assumption 4: The cuzk daemon is always reachable when enabled. The assistant does not discuss failure modes—what happens if the daemon is down, if the gRPC connection times out, or if the daemon's queue is full. The CanAccept method will reject tasks when the daemon is unreachable, but the task lifecycle implications (retry, backoff, fallback to local proving) are not addressed in this message.

Input Knowledge Required

To fully understand [msg 3361], the reader needs knowledge spanning several domains:

Curio's harmony task scheduler: Understanding the CanAccept interface, TypeDetails(), and how Curio manages task lifecycle, resource accounting, and scheduling. The assistant has been exploring these concepts across multiple previous messages ([msg 3332] through [msg 3360]).
The cuzk proving daemon: Knowledge that cuzk is a standalone gRPC service for Groth16 proof generation, that it manages its own GPU resources and SRS memory, and that it exposes a protobuf API. The assistant discovered the daemon's location at ./extern/cuzk in earlier messages.
Filecoin proof types: Understanding the distinction between PoRep C2 (Proof-of-Replication, Commit 2), SnapDeals ProveReplicaUpdate, WindowPoSt, and WinningPoSt. Each has different computational profiles and resource requirements.
Go configuration patterns: Familiarity with how Curio structures its configuration in deps/config/types.go, the use of TOML serialization, and the pattern of embedding config structs within CurioConfig.
The Phase 12 context: Understanding that the cuzk engine has just been optimized with split GPU proving API and memory backpressure, and that the integration into Curio is the natural next step after engine optimization.

Output Knowledge Created

This message produces several forms of output knowledge:

An architectural plan: The assistant creates a high-level design for the integration, covering configuration, proof routing, backpressure, and resource accounting. This plan will be implemented in subsequent messages.
A discovered discrepancy: The assistant surfaces that lib/cuzk/client.go does not exist, correcting the project's documentation. This is valuable knowledge for anyone working on the integration.
A design decision about CanAccept: The insight that CanAccept must query the daemon's queue status rather than checking local resources is a non-obvious architectural choice that shapes the entire integration.
A concrete code reference: The grep output of TypeDetails() provides a snapshot of the current resource accounting that must be modified.

Mistakes and Incorrect Assumptions

While the reasoning in [msg 3361] is generally sound, there are potential issues worth examining:

The EnabledProofs approach may be premature abstraction. The assistant introduces a list-based proof routing mechanism before confirming that the cuzk daemon actually supports per-proof-type queue management. If the daemon exposes a single queue for all proof types, the EnabledProofs list becomes a client-side filter that doesn't align with the daemon's actual capacity reporting. A simpler approach—just enabling or disabling the daemon globally—might be more appropriate until the daemon's API is fully understood.

The backpressure design assumes synchronous queue checking. The assistant plans to call cuzkClient.GetStatus(ctx) inside CanAccept(), but this introduces a synchronous gRPC call into the task scheduling path. If the daemon is slow to respond, this could delay task scheduling across all tasks. An alternative would be to cache the daemon's status and update it asynchronously, but the assistant does not discuss caching.

The resource zeroing approach may interact poorly with Curio's scheduling heuristics. Zeroing GPU and RAM costs in TypeDetails() means Curio will treat cuzk-enabled tasks as having no resource footprint. This could lead to over-scheduling of other local tasks, since the scheduler thinks it has more free resources than it actually does (the GPU is still in use, just by the daemon rather than by Curio directly). The assistant assumes this is safe because the daemon manages its own resources, but the scheduler's global view of resource availability becomes inaccurate.

The Broader Context: Why This Integration Matters

The integration of cuzk into Curio represents a fundamental shift in how Filecoin storage providers deploy proof generation infrastructure. In the traditional model, each Curio node runs proof generation locally, consuming its own GPU and RAM. This ties proof capacity to the number of nodes and makes it difficult to right-size hardware: a node with one GPU can only generate one proof at a time, regardless of how many sectors it manages.

The cuzk daemon decouples proof generation from task orchestration. A single powerful machine with multiple GPUs and abundant RAM can serve as a centralized proving server, accepting work from multiple Curio nodes. This enables heterogeneous deployments where cheap orchestration nodes handle task management while expensive GPU nodes handle proof generation. It also enables better resource utilization: instead of each node having idle GPU cycles between proofs, the daemon can queue work and keep GPUs saturated.

The design decisions in [msg 3361] directly enable this vision. The CanAccept backpressure mechanism ensures that the daemon is not overwhelmed, the TypeDetails zeroing ensures that Curio's scheduler doesn't double-count resources, and the EnabledProofs routing allows operators to gradually migrate proof types to the remote daemon. These are not just implementation details—they are the architectural foundation for a distributed proving system.

Conclusion

Message [msg 3361] is a masterclass in architectural reasoning under uncertainty. The assistant navigates a complex codebase, surfaces discrepancies between documentation and reality, makes deliberate scoping decisions, and designs a clean integration pattern that respects the boundaries between two independently-developed systems. The reasoning process—discover, scope, plan, refine, validate—is visible in every paragraph, from the initial find command that revealed the missing client file to the final grep that validated the resource accounting code.

The message also reveals the assistant's strengths and limitations. Its strengths lie in systematic exploration, architectural thinking, and the ability to hold multiple design constraints in mind simultaneously. Its limitations are visible in the assumptions it makes without verification—the daemon's API capabilities, the scheduler's behavior with zero-cost tasks, the failure modes of synchronous gRPC calls. These are not flaws but rather the natural boundaries of a planning phase: some questions can only be answered by implementation and testing.

For anyone studying how to integrate a standalone daemon into an existing task scheduler, [msg 3361] offers a valuable case study. It demonstrates that the hardest part of integration is not writing the code but designing the resource accounting and backpressure semantics—the invisible contracts that ensure two systems cooperate without conflicting. The code will follow; the architecture must come first.