The Backpressure Edit: Wiring Remote GPU Proving into Curio's Scheduler

In the sprawling codebase of Curio—Filecoin's storage-node orchestrator—a single edit to a file called task_porep.go represents a pivotal architectural decision. The message at index 3425 in this opencode session is deceptively brief:

Now update CanAccept() to use cuzk backpressure: [edit] /home/theuser/curio/tasks/seal/task_porep.go Edit applied successfully.

Three lines of conversation text, one file edit. Yet this message sits at the intersection of distributed systems design, GPU resource management, and the careful adaptation of an existing task scheduler to support a new remote-proving architecture. Understanding why this particular edit matters requires tracing the threads that lead to it—and recognizing the assumptions and design philosophy embedded in this seemingly minor change.

The Architecture Behind the Edit

The context for this message is the integration of cuzk, a remote GPU proving daemon, into Curio's task orchestrator. Curio manages Filecoin storage providers' operations: sealing sectors (PoRep), proving sector updates (SnapDeals), and generating proofs for WindowPoSt and WinningPoSt. These operations are orchestrated by the "harmony" task framework, a scheduler that assigns work to workers based on resource availability—particularly GPU and RAM capacity.

The cuzk daemon, developed in earlier phases of this project, is a high-performance Groth16 proof generator that runs as a separate process, communicating via gRPC. It can generate proofs more efficiently than Curio's built-in GPU pipeline, but it introduces a new architectural challenge: how should Curio's scheduler interact with an external proving service? The daemon has its own queue, its own GPU management, and its own capacity limits. Curio's scheduler, which tracks local GPU and RAM resources meticulously, needs to be told not to count resources for tasks that will be proven remotely—and to check whether the remote daemon has capacity before accepting new work.

This is where CanAccept() comes in.

The CanAccept Method: Curio's Gatekeeper

In Curio's harmony framework, every task type implements a CanAccept() method. The scheduler calls this method to determine whether a task is ready to be executed. It returns a list of task IDs that can be accepted. If a task isn't ready—because its parameters haven't been fetched yet, or because local resources are insufficient—CanAccept() returns an empty slice, and the scheduler moves on.

The existing PoRep task already had a CanAccept() implementation with a curious inversion. The field was confusingly named enableRemoteProofs, but it actually corresponded to the config value EnablePoRepProof—meaning "enable local proof generation." When this flag was false, CanAccept() returned an empty slice with the comment "remote proofs enabled but not local prove—we still need the task for poller." This was the original pattern for delegating proofs to an external system: the task still exists in the database (for polling purposes), but the local scheduler never accepts it for execution.

The cuzk integration needed to extend this pattern. Instead of simply rejecting all tasks when proofs are remote, the new CanAccept() needed to actively query the cuzk daemon's queue depth and use that information for backpressure. If the daemon's queue is full, the scheduler should stop accepting tasks, allowing the daemon to drain its backlog before receiving more work. This is a classic backpressure pattern, applied at the scheduler level rather than at the RPC call level.

Why Backpressure Matters

The decision to implement backpressure in CanAccept() rather than in the gRPC client or the task's Do() method reflects a deliberate architectural choice. Consider the alternatives:

  1. Reactive backpressure: Let the task proceed to Do(), make the gRPC call, and handle "queue full" errors with retries. This wastes scheduler resources—the task is dispatched, a worker is allocated, and only then does it discover the daemon is busy.
  2. Push-based queue management: Have the daemon push its capacity information to Curio periodically. This adds complexity, requires a separate control channel, and introduces staleness issues.
  3. Pull-based backpressure via CanAccept (the chosen approach): Before the scheduler even dispatches a task, it queries the daemon's queue status. If the daemon is busy, the task simply isn't accepted, and the scheduler moves on to other work. No wasted dispatch, no error handling, no stale state. This pull-based approach integrates naturally with Curio's existing scheduling loop. The CanAccept() method is already called frequently by the scheduler; adding a lightweight gRPC call to check queue depth is a minimal extension. The daemon's queue acts as a natural rate limiter: when it's full, Curio stops sending work; when it drains, Curio resumes. This creates a self-regulating system where the proving pipeline's throughput is governed by the daemon's actual processing capacity rather than by Curio's estimates.

The Assumptions Embedded in the Edit

This edit rests on several assumptions, some explicit and some implicit:

The daemon has a queryable queue. The cuzk daemon must expose an RPC endpoint (or the gRPC client must be able to infer queue depth from available metrics) that CanAccept() can call synchronously. This assumes the daemon's queue state is cheap to query—that it doesn't require a database scan or expensive computation.

Backpressure at the scheduler level is sufficient. The assumption is that if CanAccept() gates task dispatch based on daemon capacity, no additional backpressure is needed at the RPC call level. This holds if the daemon's queue depth is a reliable indicator of its ability to accept new work, and if there are no race conditions where multiple tasks are accepted simultaneously but the daemon's queue fills between checks.

The scheduler's polling frequency is appropriate. If CanAccept() is called too infrequently, the daemon may sit idle while tasks wait in Curio's database. If called too frequently, the gRPC call becomes a bottleneck. The existing harmony framework's polling cadence is assumed to be reasonable for this use case.

The cuzk client is thread-safe. CanAccept() may be called concurrently from multiple scheduler goroutines. The gRPC client must handle concurrent calls without corruption or excessive connection overhead.

The existing "remote proofs" pattern is a foundation to build on, not replace. The original enableRemoteProofs flag and its inverted logic in CanAccept() were designed for a simpler "all-or-nothing" remote proving model. The cuzk integration extends this pattern rather than replacing it, preserving backward compatibility for configurations that don't use the cuzk daemon.

Input Knowledge Required

To understand this edit, one needs knowledge spanning several domains:

Output Knowledge Created

This edit produces several forms of knowledge:

Code change: The CanAccept() method in task_porep.go is modified to query the cuzk daemon's queue depth when the daemon is enabled, returning tasks only when the daemon has capacity.

Architectural precedent: This establishes the pattern for backpressure in remote proving integrations. The same pattern will be applied to SnapDeals and proofshare tasks in subsequent edits.

Integration milestone: This is the third of four edits to the PoRep task file, bringing the integration closer to completion. After this, only TypeDetails() needs to be updated to zero out local GPU/RAM requirements.

Validation of the design: The edit compiles successfully (no LSP errors reported), confirming that the cuzk client type integrates cleanly with the harmony task interface.

The Thinking Process

The assistant's reasoning is visible in the sequence of messages leading to this edit. In msg 3420, the assistant explicitly laid out the four-step plan:

Now wire cuzk into the PoRep task. The approach: 1. Add cuzkClient *cuzk.Client field to PoRepTask 2. In TypeDetails(), when cuzk is enabled, zero out GPU/RAM requirements 3. In CanAccept(), when cuzk is enabled, use backpressure from daemon 4. In Do(), when cuzk is enabled, call PoRepSnarkCuzk instead of PoRepSnark

This plan reflects a clear understanding of the harmony framework's separation of concerns: TypeDetails() advertises resource requirements to the scheduler, CanAccept() gates task acceptance, and Do() executes the work. Each method is modified for its specific role in the cuzk integration.

The assistant also demonstrated careful reading of the existing code. In msg 3379–3381, the assistant investigated the confusingly-named enableRemoteProofs field, tracing it from the config value EnablePoRepProof through to its use in CanAccept(). The assistant noted the inverted logic—enableRemoteProofs == false means "proofs are done remotely"—and confirmed this understanding by reading the constructor call site. This research directly informed the cuzk integration: the assistant understood that the existing "remote proofs" pattern was a simple reject mechanism, and that the cuzk integration needed to extend it with active backpressure rather than passive rejection.

Broader Significance

This edit, while small in isolation, represents a fundamental shift in how Curio manages GPU resources. Previously, GPU proving was tightly coupled to the Curio process—each instance managed its own GPU(s) directly. The cuzk daemon decouples proving from orchestration, allowing GPU resources to be managed by a dedicated process that can be optimized independently. But decoupling introduces coordination challenges: how does the orchestrator know the daemon's capacity? How does it avoid overwhelming the daemon?

The answer, embodied in this edit, is to treat the daemon as a resource with its own scheduling semantics, and to integrate those semantics into Curio's existing scheduler through the CanAccept() backpressure mechanism. This is a design pattern with broad applicability: any external service that Curio delegates work to can be managed the same way, with the scheduler acting as a smart gatekeeper that respects both local and remote capacity constraints.

The edit also demonstrates a key principle of incremental architecture: rather than redesigning the harmony framework to support remote services, the assistant adapted the existing CanAccept() pattern, extending it with backpressure while preserving backward compatibility. The original enableRemoteProofs logic remains for configurations that don't use cuzk, and the new cuzk-specific logic is added alongside it. This minimizes disruption while enabling a fundamentally new capability.

In the broader narrative of this opencode session, msg 3425 is the moment where the cuzk integration transitions from infrastructure (config, client library, helper functions) to core task logic. The config has been added, the gRPC client has been generated, the cuzk_funcs.go helper has been written. Now the actual task implementations are being modified, and CanAccept() is where the scheduler learns to talk to the daemon. It's a small edit with outsized architectural significance.