Phase 7 Lands: The Per-Partition Dispatch Revolution in cuzk's SNARK Proving Engine

Introduction

On the feat/cuzk branch of the Curio Filecoin proving stack, a single commit — f5bfb669 — marks a pivotal architectural transformation. The subject message, a commit summary written by the AI assistant after implementing Phase 7 of the cuzk SNARK proving engine, documents the culmination of a carefully planned optimization cycle. In 578 lines of Rust spread across four files, the proving engine shifts from treating a complete Proof-of-Replication (PoRep) sector as a monolithic work unit to dispatching each of its 10 internal partitions as an independent, pipelineable job. This message is not merely a changelog; it is the formal record of a design decision that rearchitects how GPU compute resources are utilized in one of the most memory-intensive proving pipelines in the Filecoin ecosystem.

Context and Motivation: Why This Message Was Written

To understand why this message exists, one must appreciate the engineering context that produced it. The cuzk project is a multi-month effort to optimize the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) protocol. Earlier phases had already delivered substantial improvements: Phase 6 introduced a slotted partition pipeline that reduced peak memory from ~228 GiB to ~71 GiB, and the team had implemented parallel synthesis via a tokio::sync::Semaphore to better saturate GPU resources. Yet a fundamental structural problem remained.

The core issue was the "thundering herd" pattern in the GPU worker. When a complete sector's worth of 10 partitions arrived at the GPU simultaneously, the CUDA runtime would queue all the work, but the critical b_g2_msm operation — a multi-scalar multiplication on the G2 curve — would take approximately 25 seconds to complete when processing all 10 circuits together. During this time, the GPU was busy, but the CPU-side pipeline was blocked waiting. More critically, between sectors, there were GPU idle gaps where no work was available because the next sector's synthesis had not yet completed. The engine was fundamentally batch-oriented: synthesize everything, then prove everything, then move to the next sector.

Phase 7's design, specified in c2-optimization-proposal-7.md and now implemented in this commit, attacks this problem at its root. Instead of batching all 10 partitions of a sector together, each partition becomes an independent work unit that flows through the synthesis → GPU pipeline on its own. This enables true pipelining: partition 0 can be synthesized and dispatched to the GPU while partitions 1–9 are still being synthesized. The GPU never goes idle waiting for the next partition because there is always one ready.

The message was written to document the completion of this implementation, to provide a structured summary for code review, and to serve as a reference point for the optimization trajectory. It is simultaneously a status update, a design document, and a historical record.

The Message in Full

The subject message, written by the AI assistant after the implementation compiled and was committed, reads in its entirety:

Phase 7 implementation is complete. Committed as f5bfb669 on branch feat/cuzk.

>

## Summary of Changes

>

4 files changed, 578 insertions, 22 deletions:

>

### engine.rs (+534 lines net) - SynthesizedJob: Added partition_index: Option<usize>, total_partitions: Option<usize>, parent_job_id: Option<JobId> fields - PartitionedJobState (new): Tracks ProofAssembler, request metadata, accumulated synth/GPU durations, start_time, and failed flag - PartitionWorkItem (new, #[cfg(feature = "cuda-supraseal")]): Work unit with Arc<ParsedC1Output>, partition index, job ID, SRS params - JobTracker: Added assemblers: HashMap<JobId, PartitionedJobState> - process_batch(): New Phase 7 dispatch path — when partition_workers > 0 and single-sector PoRep C2: parses C1 once, loads SRS, registers assembler, dispatches 10 tokio::spawn tasks each acquiring a partition_semaphore permit then running synthesize_partition() via spawn_blocking, sending results to synth_tx. Returns immediately (non-blocking). Phase 6 fallback retained for partition_workers=0, slot_size>0. - dispatch_batch(): Updated signature to thread partition_workers and partition_semaphore through all 5 call sites - GPU worker: Partition-aware result routing — checks parent_job_id, routes to ProofAssembler, assembles final proof on is_complete(), calls libc::malloc_trim(0) after each partition. Error path marks assembler as failed, notifies callers, subsequent partitions skip work.

>

### pipeline.rs (+20 lines) - ParsedC1Output: Made pub struct with pub(crate) fields (except pub num_partitions) - parse_c1_output(): Made pub - synthesize_partition(): Made pub

>

### config.rs (+27 lines) - SynthesisConfig: Added partition_workers: u32 field (default 20)

>

### cuzk.example.toml (+19 lines) - Documented partition_workers setting with memory guidance per machine size

This is a dense, information-rich summary that serves multiple audiences simultaneously: code reviewers checking the diff, future developers debugging the system, and project managers tracking progress against the optimization roadmap.

The Architectural Vision: Per-Partition Dispatch

The message describes changes across four files, but the heart of the transformation lives in engine.rs, which gained 534 lines net. The architecture can be understood through five key abstractions, each visible in the commit summary.

SynthesizedJob gains partition identity. Three new optional fields — partition_index, total_partitions, and parent_job_id — transform what was previously a sector-level result into a partition-level one. The use of Option<usize> and Option<JobId> is deliberate: the same struct is used for both partitioned and non-partitioned paths, and the presence of these fields signals which path a given job belongs to. This is a pragmatic design choice that avoids duplicating the synthesis result type.

PartitionedJobState is the orchestrator. This new struct tracks the ProofAssembler (the component that collects individual partition proofs and assembles the final 1920-byte Groth16 proof), accumulated synthesis and GPU durations, a start timestamp, and a failure flag. It lives in the JobTracker's assemblers map, keyed by JobId. This is the state machine that knows when a sector's proof is complete: when all 10 partitions have been proved and routed back, is_complete() returns true and the final proof is delivered.

PartitionWorkItem is the work unit. Conditionally compiled with #[cfg(feature = "cuda-supraseal")], it bundles an Arc<ParsedC1Output> (the parsed circuit from Phase 1 of the Groth16 pipeline), the partition index, the job ID, and the SRS (Structured Reference String) parameters. Using Arc for the parsed C1 output is critical: it avoids cloning the potentially large circuit data while allowing multiple partition synthesis tasks to share the same parsed structure.

The dispatch path in process_batch() is the new control flow. The commit summary describes it precisely: when partition_workers > 0 and the request is a single-sector PoRep C2 proof, the engine parses C1 once (not 10 times), loads the SRS once, registers a PartitionedJobState in the tracker, and then dispatches 10 tokio::spawn tasks. Each task acquires a permit from the partition_semaphore (a tokio::sync::Semaphore with a configurable capacity, defaulting to 20), then runs synthesize_partition() via spawn_blocking. The function returns immediately — it is non-blocking, unlike the previous path which waited for all synthesis to complete before proceeding.

The GPU worker becomes partition-aware. On the receiving end, the GPU worker loop now checks for parent_job_id on incoming synthesized jobs. If present, it routes the proof result to the corresponding ProofAssembler rather than treating it as a complete proof. When is_complete() returns true, the assembler delivers the final proof and the worker calls libc::malloc_trim(0) to release any memory freed by the partition's completion back to the operating system.

Decision-Making and Tradeoffs Visible in the Message

The commit summary reveals several deliberate engineering decisions, each with its own rationale.

Why 10 partitions? The number is not arbitrary. Filecoin's Proof-of-Replication protocol divides each sector into 10 "partition proofs" (originally layers in the ZigZag graph), each covering a subset of the sector's data. The Groth16 proving system must produce a separate proof for each partition, and these proofs are aggregated into a single sector-level proof. Phase 7 exploits this natural decomposition.

Why a semaphore with capacity 20? The default partition_workers = 20 allows up to 20 concurrent synthesis tasks. With 10 partitions per sector, this means two sectors' worth of partitions can be in-flight simultaneously, providing headroom for pipelining across sector boundaries. The semaphore prevents unbounded memory growth from too many concurrent synthesis tasks — each of which can consume several GiB of RAM during the constraint-system evaluation phase.

Why retain the Phase 6 fallback? The message explicitly notes: "Phase 6 fallback retained for partition_workers=0, slot_size>0." This is a safety net. If the per-partition path encounters unexpected issues (e.g., a bug in partition routing, or a pathological proof that behaves differently when proved individually), the operator can fall back to the slotted pipeline by setting partition_workers = 0. This dual-path design is characteristic of the project's incremental, measurement-driven approach.

Why malloc_trim(0) after each partition? The synthesis of each partition allocates and frees significant memory. However, the Rust allocator (jemalloc or glibc's malloc) may not release freed pages back to the kernel promptly, leading to RSS (Resident Set Size) growth that mimics a memory leak. Calling malloc_trim(0) after each partition's GPU work completes tells the allocator to return unused pages to the OS, keeping the process's memory footprint stable. This is a practical concern: with ~71 GiB peak memory in Phase 6, any unnecessary retention could push the system toward OOM.

Why Arc<ParsedC1Output> instead of cloning? The parsed C1 output contains the circuit's constraint system, which can be hundreds of megabytes. Cloning it 10 times (once per partition) would multiply memory usage and add allocation overhead. By sharing via Arc, all 10 partition synthesis tasks read from the same parsed structure. The tradeoff is that Arc's atomic reference counting adds a small per-clone cost, but this is negligible compared to the memory savings.

Assumptions Embedded in the Design

The message, read carefully, reveals several assumptions that the implementer made — some explicit, some implicit.

Assumption: Partition synthesis is CPU-bound and independent. The design assumes that synthesizing partition i does not depend on the results of partition j. This is true for PoRep C2: each partition's constraint system is derived from a disjoint subset of the sector's data, so they can be synthesized in any order or in parallel. If this assumption were violated (e.g., if partitions shared mutable state), the entire dispatch architecture would fail.

Assumption: The GPU can handle num_circuits=1 efficiently. In the per-partition path, each GPU call processes exactly one circuit (one partition) rather than 10. The commit summary anticipates that this makes b_g2_msm take ~0.4 seconds instead of ~25 seconds. This assumes that the CUDA kernel launch overhead and the GPU's scheduling do not penalize single-circuit invocations disproportionately. The later benchmarking (visible in the chunk summaries) confirms this assumption holds, though it also reveals that the GPU utilization is "pretty jumpy" — suggesting the assumption is valid but incomplete.

Assumption: 20 concurrent synthesis workers are sufficient. The default semaphore capacity of 20 implies that the system will never need more than 20 in-flight synthesis tasks. With 2 GPUs and 10 partitions per sector, this allows 2 sectors' worth of partitions to be in synthesis simultaneously. If the system were scaled to 4 GPUs or if synthesis became significantly faster than GPU proving, this limit might become a bottleneck. The configurable nature of partition_workers acknowledges this uncertainty.

Assumption: malloc_trim(0) is safe to call from a GPU worker. The GPU worker loop calls malloc_trim(0) after each partition's proof is assembled. This assumes that the Rust allocator's internal locks do not conflict with any concurrent allocation in other threads, and that the performance cost of scanning free lists is acceptable. In practice, malloc_trim can be expensive on large heaps, but the per-partition granularity keeps each invocation's work bounded.

Input Knowledge Required to Understand This Message

A reader needs substantial domain knowledge to fully grasp this message. At minimum:

Output Knowledge Created by This Message

The message produces several forms of knowledge:

  1. A precise specification of the Phase 7 API and data structures. Any developer reading this message can understand the new fields, structs, and control flow without reading the full diff. The bullet-point format serves as reference documentation.
  2. A rationale for the architectural shift. By contrasting the old batch-oriented path with the new per-partition path, the message explains why the change matters: "eliminating the thundering-herd pattern and enabling cross-sector pipelining."
  3. Performance expectations. The message states "Expected steady-state: 42.8s/proof → ~30s/proof (GPU-limited), ~100% GPU utilization, zero cross-sector GPU idle gaps." These are testable hypotheses that later benchmarking will validate or refute.
  4. A historical marker. The commit hash f5bfb669 pins a specific point in the optimization trajectory. Future developers can use git bisect or git log to understand when the per-partition dispatch was introduced and what changed alongside it.
  5. Configuration guidance. The cuzk.example.toml changes document the partition_workers setting with "memory guidance per machine size," helping operators tune the system for their hardware.

The Thinking Process Visible in the Message

Though the message is a summary, it reveals the implementer's thinking through its structure and emphasis. The progression from data structures (SynthesizedJob, PartitionedJobState, PartitionWorkItem) to control flow (process_batch(), GPU worker routing) to configuration (config.rs, cuzk.example.toml) mirrors the implementation order — a bottom-up approach that builds the foundation before the logic.

The attention to error handling is telling. The message notes: "Error path marks assembler as failed, notifies callers, subsequent partitions skip work." This indicates that the implementer thought carefully about partial failures — what happens if partition 3's synthesis fails but partitions 0–2 have already been proved? The failed flag on PartitionedJobState ensures that once any partition fails, the remaining partitions are skipped rather than wasting compute resources on a doomed proof.

The mention of malloc_trim(0) reveals a systems-level awareness. The implementer knows that Rust's default allocator behavior can cause RSS to balloon even when memory is logically freed, and that proactive trimming is necessary in memory-constrained environments. This is not a concern that would occur to a developer focused solely on algorithmic correctness — it comes from experience with production proving workloads.

The retention of the Phase 6 fallback shows intellectual humility. The implementer is confident enough in the new design to commit it, but prudent enough to keep an escape hatch. This dual-path strategy acknowledges that real-world proving workloads can surface unexpected behaviors — perhaps a particular circuit shape causes the per-partition path to deadlock, or a GPU driver bug manifests only with num_circuits=1.

Conclusion

The Phase 7 commit message at message 2088 is far more than a changelog entry. It is the crystallization of weeks of analysis, design, and implementation work — a moment where an architectural insight (treat partitions as independent pipeline units) is translated into concrete Rust code. The message documents 578 lines of changes that fundamentally alter how the cuzk proving engine utilizes GPU resources, shifting from a batch-oriented model that left the GPU idle between sectors to a continuous pipeline that keeps the CUDA hardware saturated.

The assumptions embedded in the design — about partition independence, GPU efficiency with single circuits, and the adequacy of 20 concurrent workers — would be tested in the very next round of benchmarking, leading to the discovery of the static mutex contention problem that would motivate Phase 8. But at this moment, captured in commit f5bfb669, the engine has taken a decisive step toward the project's ultimate goal: a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets. The message stands as both a technical document and a narrative artifact, recording not just what was done, but the engineering judgment behind every decision.