The Blueprint That Drives a Revolution: Deconstructing the cuzk Project Plan Message
Introduction
In the sprawling, intricate world of Filecoin proof generation, where a single Groth16 proof for a 32 GiB sector consumes ~200 GiB of peak memory and takes over ninety seconds of GPU compute time, the difference between a well-architected system and a patchwork of scripts can mean millions of dollars in hardware costs. The message at index 532 of this coding session is, at first glance, deceptively simple: a user typing @cuzk-project.md continue phase 2 and the resulting output of a file read tool that dumps the entire 1213-line project plan document. But this message is far more than a mundane file retrieval. It is a pivotal moment in a months-long engineering effort—a deliberate re-grounding in the architectural vision before embarking on the next critical phase of implementation.
This article examines that single message in microscopic detail, peeling back the layers of context, reasoning, assumption, and consequence that surround it. We will explore why this message was written, what knowledge it presupposes, what decisions it enables, and how it functions as both a status checkpoint and a strategic reset. To understand this message is to understand the entire cuzk project—a pipelined SNARK proving engine designed to transform Filecoin proof generation from a batch-oriented, memory-hungry process into a continuous, efficient, inference-engine-like pipeline.
The Surface: What the Message Actually Contains
The message consists of two parts. The first is the user's instruction: [user] @cuzk-project.md continue phase 2. This is a concise directive, referencing a file (cuzk-project.md) and a phase of work (phase 2). The @ symbol suggests a tool invocation or file reference within the conversation's interface. The second part is the full output of a Read tool call, which retrieved the entire contents of /home/theuser/curio/cuzk-project.md—a 1213-line Markdown document that serves as the comprehensive project plan for the cuzk proving engine.
The document itself is a masterwork of systems architecture writing. It spans seventeen sections covering everything from the high-level motivation ("Why a Daemon") through detailed gRPC protocol definitions, SRS memory manager design, scheduler architecture, GPU worker pipeline designs, a six-phase implementation roadmap, Curio integration paths, key design decisions, open questions, dependency versions, and file references. It is the kind of document that a senior engineer produces after weeks of investigation, analysis, and design—a single source of truth that aligns an entire team around a shared technical vision.
But the message is not about the document. The message is about the act of re-loading that document into the conversation context at a critical juncture. The user is not asking the assistant to read something new; they are asking it to re-read something that already exists, to re-establish the full architectural context before proceeding with Phase 2 implementation. This is a deliberate strategic move, and understanding why requires us to understand the journey that led to this point.
The Context: Six Phases, Six Commits, and a Growing Codebase
The cuzk project, as documented in the plan, is an ambitious effort to build a persistent GPU-resident SNARK proving engine for Filecoin—a "proving server" analogous to how vLLM or TensorRT serve inference models. The project is organized into six phases spanning eighteen weeks:
- Phase 0 (Weeks 1-3): Scaffold the daemon with PoRep C2 proving and SRS residency. Zero upstream modifications.
- Phase 1 (Weeks 3-5): Add all four proof types (PoRep, SnapDeals, WindowPoSt, WinningPoSt), priority scheduling, multi-GPU support.
- Phase 2 (Weeks 5-8): Pipelined synthesis || GPU compute via a bellperson fork that exposes the synthesis/prove split API.
- Phase 3 (Weeks 8-11): Cross-sector batching—multiple sectors proved in one GPU pass.
- Phase 4 (Weeks 11-14): Compute quick wins—targeted optimizations from a catalog of eighteen identified improvements.
- Phase 5 (Weeks 14-18): Pre-Compiled Constraint Evaluator (PCE)—replace circuit synthesis with sparse matrix-vector multiply for a 3-5x synthesis speedup. By the time of this message, the project has completed Phases 0 and 1, and is well into Phase 2. The git history shows six commits on the
feat/cuzkbranch: 1.ae551ee6— Phase 0 scaffold: 24 files, 6859 insertions. The entire gRPC API, engine, scheduler, daemon, and bench tool. 2.f719a710— Phase 0 hardening: tracing, timing breakdown, Prometheus metrics, graceful shutdown. 3.d8aa4f1d— Phase 1 core: all four prover backends, multi-GPU worker pool, enum conversions. 4.9d8453c3— Phase 1 gen-vanilla: vanilla proof generation for PoSt and SnapDeals. 5.f258e8c7— Phase 2 bellperson fork: minimal fork exposingProvingAssignment,synthesize_circuits_batch(), andprove_from_assignments(). 6.beb3ca9c— Phase 2 pipeline implementation:srs_manager.rs(370 lines),pipeline.rs(553 lines), pipeline-aware engine dispatch. The codebase has grown to include six Rust crates (cuzk-proto, cuzk-core, cuzk-server, cuzk-daemon, cuzk-bench, cuzk-ffi), a forked bellperson inextern/bellperson/, and fifteen passing unit tests. The Phase 2 pipeline implementation has been written and compiles cleanly with--no-default-features(non-CUDA mode), but it has not yet been tested end-to-end with a real GPU. That is the immediate next step, and it is the context in which this message arrives.
Why This Message Was Written: The Strategic Re-grounding
The user's decision to re-read the entire project plan at this moment is not accidental. It reflects a specific understanding of how the assistant works and what it needs to proceed effectively. Several factors motivate this re-grounding:
1. Context Window Management
In a long-running coding session that spans dozens of messages, hundreds of tool calls, and thousands of lines of code, the assistant's context window is constantly shifting. Earlier messages—including the original project plan document—may have been pushed out of the active context by newer content. The assistant's "memory" of the project architecture is only as fresh as the most recent messages. By re-reading the full document, the user ensures that the assistant has the complete architectural picture in its immediate context before making decisions about Phase 2 continuation.
This is particularly important because Phase 2 involves complex trade-offs. The pipeline design document (cuzk-phase2-design.md) exists separately, but the master project plan contains the overarching architecture, the scheduler design, the SRS memory manager specifications, the gRPC API, and the phased roadmap that constrains what Phase 2 should and should not do. Without this context, the assistant might make implementation decisions that conflict with later phases—for example, designing a pipeline that doesn't accommodate cross-sector batching (Phase 3) or that requires changes to the gRPC API that would break backward compatibility.
2. The Transition from Design to Testing
Phase 2 has reached a critical inflection point. The code has been written—the bellperson fork exposes the split API, the SRS manager handles parameter residency, the pipeline module orchestrates per-partition synthesis and GPU proving, and the engine dispatches to the pipeline path when enabled. But the code has only been validated in non-CUDA mode (where the pipeline stubs return dummy proofs). The next step is end-to-end GPU testing, which requires:
- Building with
--features cuda-supraseal - Starting the daemon with pipeline mode enabled
- Submitting a real 32 GiB PoRep C2 proof
- Verifying that the proof bytes match the Phase 1 (monolithic) output
- Measuring performance to confirm the pipeline provides a benefit This is a high-risk step. If the pipeline has bugs—incorrect assignment packing, wrong SRS loading, mismatched circuit construction—the GPU test will fail, potentially with hard-to-debug CUDA errors or silent proof invalidity. The user wants the assistant to have the full architectural blueprint fresh in mind before tackling this testing, so that when something goes wrong, the assistant can trace the issue back to the design decisions rather than guessing.
3. The Need to Re-evaluate the Pipeline Design
There is an unspoken tension in the Phase 2 design that the project plan reveals. The plan describes two pipeline models:
Phase 0 (sequential):
GPU Worker loop:
1. Receive job from scheduler
2. Ensure SRS is hot
3. Call filecoin-proofs-api monolithic function
4. Return proof
Phase 2+ (pipelined):
CPU Thread Pool GPU (CUDA)
────────────── ──────────
synth(job N+1) || NTT+MSM(job N)
synth(job N+2) || NTT+MSM(job N+1)
The Phase 2 design assumes that splitting synthesis from GPU proving and overlapping them across different proof jobs will improve throughput. But the current implementation in pipeline.rs applies pipelining within a single proof—synthesizing partition 0, proving partition 0, synthesizing partition 1, proving partition 1, and so on. This is a fundamentally different kind of pipelining, and its performance characteristics are unknown.
The project plan's "Estimated impact" for Phase 2 is "GPU utilization ~40% → ~70%. Throughput ~1.5x over Phase 0." But this estimate assumes the cross-job overlap model, not the per-partition serial model. By re-reading the plan, the user is implicitly asking the assistant to recognize this discrepancy and adjust the implementation strategy accordingly. As we will see in the subsequent messages, this is exactly what happens: the E2E GPU test reveals that per-partition pipelining is 6.6x slower than monolithic, leading to the creation of a batch-all-partitions mode that restores single-proof latency while preserving the architecture for future cross-job overlap.
4. The Importance of the Bellperson Fork Contract
Phase 2's most delicate dependency is the bellperson fork. The fork is minimal—it makes ProvingAssignment public, exposes synthesize_circuits_batch(), and adds a new prove_from_assignments() function. But these changes must be exactly right. If the fork exposes the wrong types, or if the prove_from_assignments() function doesn't correctly handle the assignment data produced by synthesize_circuits_batch(), the entire pipeline breaks.
The project plan contains the exact API contract for the fork:
pub fn synthesize_circuits_batch<E, C>(
circuits: Vec<C>,
) -> Result<(Vec<ProvingAssignment<E::Fr>>, Vec<Arc<Vec<E::Fr>>>, Vec<Arc<Vec<E::Fr>>>)>;
pub fn prove_from_assignments<E, P>(
provers: Vec<ProvingAssignment<E::Fr>>,
input_assignments: Vec<Arc<Vec<E::Fr>>>,
aux_assignments: Vec<Arc<Vec<E::Fr>>>,
params: P,
) -> Result<Vec<Proof<E>>>;
By re-reading this contract alongside the implementation, the assistant can verify that the fork matches the plan. Any divergence—different return types, different parameter ordering, missing generics—would be caught before the GPU test reveals it as a cryptic compilation error or runtime segfault.
The Knowledge Required to Understand This Message
To fully grasp what is happening in message 532, a reader needs a substantial body of background knowledge spanning distributed systems, GPU computing, zero-knowledge proofs, and the Filecoin protocol specifically.
1. Filecoin Proof Generation Architecture
Filecoin uses Groth16 proofs for storage verification. The proving pipeline has three stages:
- C1 (PreCommit1): CPU-only. Constructs the circuit from sector data, generates vanilla proofs (witnesses). Output is a
SealCommitPhase1Outputcontaining per-partition vanilla proofs. - C2 (SealCommitPhase2): GPU-heavy. Takes the C1 output, synthesizes the full R1CS circuit for each partition, then runs NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) on the GPU to produce the Groth16 proof.
- Verify: Checks the proof against the verifying key. The C2 stage is the bottleneck. For a 32 GiB PoRep, it involves ~130 million constraints across 10 partitions, requiring ~200 GiB of peak memory and ~90 seconds on a modern GPU. The current architecture spawns a fresh child process for each proof, which must initialize a CUDA context, load and deserialize the SRS (~47 GiB, taking 30-90 seconds), run one proof, and exit—discarding all state. This means every proof pays the SRS loading cost, even if multiple proofs are run back-to-back.
2. Groth16 and the SRS
Groth16 is a zero-knowledge proving system that requires a Structured Reference String (SRS)—a set of elliptic curve points used in the proof generation. For Filecoin's BLS12-381 curve, the SRS for PoRep is approximately 47 GiB of deserialized points stored in CUDA-pinned host memory. The SRS is loaded from a .params file on disk, deserialized into cudaHostAlloc-pinned memory (which enables high-bandwidth DMA to the GPU), and kept resident for the lifetime of the process.
The SRS size varies dramatically by proof type:
| Proof Type | SRS File Size | |---|---| | PoRep C2 (32 GiB) | ~47 GiB | | WindowPoSt (32 GiB) | ~5 GiB | | WinningPoSt (32 GiB) | ~11 MB | | SnapDeals (32 GiB) | ~626 MB |
This asymmetry drives the entire memory management strategy. PoRep dominates the budget, while PoSt SRS files are small enough to always keep hot alongside PoRep.
3. Bellperson and Supraseal
Bellperson is the Rust library that implements Groth16 proving for Filecoin. It has two backends:
- Native prover: Uses
ec-gpu-genfor GPU acceleration via generated CUDA kernels. Supports all proof types. - Supraseal prover: A specialized CUDA implementation (
supraseal-c2) that is faster but currently only supports PoRep C2. It is activated by a compile-timecuda-suprasealfeature flag. The supraseal prover's internal architecture is critical for Phase 2. Withinbellperson-0.26.0/src/groth16/prover/supraseal.rs, the proof process is already split into two phases: 1. Synthesis:synthesize_circuits_batch()runs the circuit'ssynthesize()method for each circuit in parallel via rayon. This producesProvingAssignmentobjects containing the a/b/c evaluation vectors and input/aux assignment vectors. 2. GPU proving: The assignments are packed into raw scalar arrays and passed tosupraseal_c2::generate_groth16_proof()for NTT + MSM + proof assembly. The split exists but was private. The bellperson fork makes it public, enabling cuzk to call synthesis on one thread while GPU proving runs on another.
4. CUDA Memory Hierarchy and Pinned Memory
CUDA uses a memory hierarchy with different bandwidth characteristics:
- Host RAM: Accessible only by the CPU. Data must be transferred to the GPU before kernels can use it.
- Pinned (page-locked) host memory: Allocated with
cudaHostAlloc. The GPU can DMA directly from pinned memory without going through the CPU. Bandwidth is near the PCIe limit (~12-25 GB/s depending on generation). - Device VRAM: Fastest access for GPU kernels. Limited to 16-80 GB depending on GPU model. The SRS manager in cuzk uses a three-tier system:
- Hot: CUDA-pinned host RAM. Ready for GPU DMA. Immediate access.
- Warm: mmap'd file (OS page cache). Needs re-deserialization into pinned memory (~2-10s).
- Cold: Disk only. Full deserialization from file (~30-90s).
5. The Curio Context
Curio is the Filecoin storage mining software that orchestrates proof generation. It has a Go codebase with tasks for sealing, window PoSt, winning PoSt, and SnapDeals. Currently, each proof type spawns a child process via lib/ffiselect/ to run the Rust FFI prover. The cuzk project aims to replace this with a persistent daemon that communicates over gRPC.
The integration path is carefully designed to be opt-in. Curio gains a feature flag (CuzkEnabled) and a gRPC client (lib/cuzk/client.go). When enabled, Curio sends proof requests to the cuzk daemon instead of spawning child processes. Both paths coexist during the transition period.
The Assumptions Embedded in This Message
Every message in a coding session carries assumptions—about the state of the world, the capabilities of the tools, and the shared understanding between user and assistant. Message 532 is rich with assumptions, some explicit and some implicit.
Assumption 1: The Assistant Has File System Access
The user's instruction @cuzk-project.md triggers a Read tool call that successfully reads the file. This assumes:
- The file exists at the specified path (
/home/theuser/curio/cuzk-project.md) - The assistant has permission to read it
- The file is not locked or corrupted
- The path resolution works correctly (no symlink issues, no mount problems) These are reasonable assumptions in a development environment, but they are not guaranteed. A file could have been moved, renamed, or deleted between sessions. The fact that the read succeeds confirms the assumption was correct.
Assumption 2: The Project Plan Is Still Current
The document is dated by its content—it references Phase 0-1 as completed and Phase 2 as in progress. But the user assumes it has been kept up to date with the actual implementation. If the codebase has diverged from the plan (e.g., if the gRPC API was modified during Phase 1 implementation without updating the document), then re-reading the plan could lead to incorrect decisions.
This assumption is validated by the git history: the six commits are clearly aligned with the phased roadmap. The plan was written before implementation began and has been followed faithfully.
Assumption 3: Phase 2 Continuation Is the Right Next Step
The user says "continue phase 2," implying that Phase 2 is the active work stream and that continuing it is the correct priority. This assumes:
- Phase 0 and Phase 1 are truly complete (not just "good enough for now")
- No blocking issues from earlier phases need resolution first
- The bellperson fork is correct and ready for GPU testing
- The pipeline implementation compiles and is logically sound The assistant's subsequent actions confirm this: it proceeds to plan the E2E GPU test, not to fix Phase 0 or Phase 1 issues.
Assumption 4: The Assistant Can Synthesize the Full Document in One Pass
The document is 1213 lines covering seventeen sections of dense technical content. The user assumes the assistant can read, understand, and retain all of this information in a single context window, and then use it to make sound implementation decisions. This is a test of the assistant's capacity for long-context understanding.
The assistant's response (message 533) shows it has indeed absorbed the document. It immediately begins planning the E2E GPU test, referencing specific sections of the plan (the pipeline architecture, the bellperson fork API, the SRS manager design) without needing to re-read them.
Assumption 5: The gRPC API Is Stable
The project plan defines the gRPC API in detail (Section 4). The user assumes this API has been implemented as specified and that no changes are needed for Phase 2. This is important because Phase 2's pipeline mode is transparent to the API—the same SubmitProof/AwaitProof/Prove RPCs are used regardless of whether the engine runs in monolithic or pipeline mode. The API stability assumption allows the assistant to focus on internal changes without worrying about client compatibility.
Assumption 6: The Test Environment Is Ready
The plan documents the test environment extensively (Section 9): /data/zk/params/ with 29 param files including the 45 GiB PoRep params, /data/32gbench/ with golden test data including c1.json, and generated vanilla proofs for PoSt and SnapDeals. The user assumes this environment is still intact and accessible.
This is a critical assumption for the E2E GPU test. Without the param files, the daemon cannot load the SRS. Without c1.json, there is no valid C1 output to prove. The assistant's subsequent test commands reference these paths directly, confirming the assumption.
The Output Knowledge Created by This Message
While the message itself is a read operation, it creates substantial output knowledge that shapes the subsequent conversation. This knowledge is not written to disk but exists in the assistant's context and influences every decision from this point forward.
1. Re-established Architectural Constraints
The project plan contains specific constraints that the pipeline implementation must respect:
- The gRPC API must not change. Phase 2 is an internal optimization. The
SubmitProofRequestandAwaitProofResponsemessages remain identical. The pipeline mode is configured via the daemon's config file, not via the API. - The SRS manager must handle all four proof types. The
CircuitIdenum insrs_manager.rsmust map to the correct param filenames for PoRep, WindowPoSt, WinningPoSt, and SnapDeals. - The pipeline must support backpressure. The plan specifies: "Pipeline backpressure: if GPU is still busy, synthesis waits before starting next."
- The bellperson fork must remain minimal. The plan says "minimal bellperson fork" — only the changes needed to expose the synthesis/prove split. These constraints are now fresh in the assistant's context, guiding the implementation choices.
2. Confirmed Priority Order
The phased roadmap establishes a clear priority: Phase 2 (pipelining) before Phase 3 (batching) before Phase 4 (quick wins) before Phase 5 (PCE). This means the assistant should not be tempted to implement cross-sector batching now, even if it would improve single-proof latency. The pipeline must work correctly first; batching is a separate concern.
However, the roadmap also shows that Phase 2's estimated impact (1.5x throughput) is modest compared to Phase 3 (4-5x) and Phase 5 (10x+). This suggests that the pipeline implementation should be clean and correct but not over-engineered—it is a stepping stone to more impactful optimizations.
3. Identified Risk Areas
The project plan's "Open Questions" section (Section 14) highlights several risks:
- SnapDeals 16 partitions: SnapDeals has 16 partitions vs PoRep's 10. The supraseal code has
max_num_circuits = 10. This needs to be bumped for Phase 3 batching, but it also affects Phase 2 if the pipeline tries to batch all SnapDeals partitions. - Default Curio build vs cuda-supraseal: The daemon must be built with
--features cuda-suprasealfor GPU proving. The assistant must remember to use this flag. - Multiple sector sizes: Phase 0 only supports 32 GiB. Phase 2 should also focus on 32 GiB. These risks are now top-of-mind for the assistant as it plans the E2E GPU test.
4. The Performance Baseline
The plan includes expected performance numbers:
# Expected output:
# Proof completed in 185.3s (queue: 0.1s, srs: 0.0s, synthesis: 95.2s, gpu: 90.0s)
And the Phase 1 baseline from the "Discoveries" section of the assistant's context:
- Cold SRS (first proof): 116.8s total (includes ~15s SRS load from disk)
- Warm SRS (cached): 92.8s total (SRS in GROTH_PARAM_MEMORY_CACHE)
- Improvement: 20.5% faster with SRS residency These numbers establish the benchmark that the pipeline must beat. If the pipeline is slower than the monolithic Phase 1 approach, it is not providing value and needs to be redesigned.
The Thinking Process: What the User Is Really Doing
Beneath the surface of a simple "continue phase 2" instruction lies a sophisticated reasoning process. The user is performing several cognitive operations simultaneously:
1. Context Restoration
The user knows that the assistant's context is limited and that earlier parts of the conversation may have been evicted. Rather than assuming the assistant remembers the project plan, the user proactively re-supplies it. This is a form of "context priming"—ensuring that the assistant's next response is grounded in the authoritative source document rather than in potentially stale or incomplete memory.
This is particularly important because the project plan is the single source of truth for architectural decisions. If the assistant relies on a summary or a remembered fragment, it might make a decision that contradicts the plan. By re-reading the full document, the assistant gets the exact text, not a paraphrase.
2. Implicit Code Review
By placing the project plan in context alongside the assistant's knowledge of the implemented code (from the previous message, which summarizes the six commits), the user enables an implicit code review. The assistant can mentally compare the plan's specifications against the actual implementation:
- Does
srs_manager.rsimplement the tiered residency model described in Section 5? - Does
pipeline.rsfollow the Phase 2 pipeline design from Section 7? - Does the engine dispatch match the proof type mapping from Section 3?
- Are the gRPC messages implemented exactly as specified in Section 4? Any discrepancies between plan and implementation become visible and can be corrected before the E2E GPU test.
3. Strategic Pause
The message represents a strategic pause before a high-risk activity (GPU testing). Rather than rushing into testing, the user invests time in re-establishing context. This is a classic engineering discipline: "measure twice, cut once." The cost of a failed GPU test is high—it could take hours to debug a CUDA error, rebuild, and retest. The cost of re-reading a document is low (seconds). The user is optimizing for correctness over speed.
4. Delegation of Decision-Making
By providing the full plan, the user is delegating tactical decision-making to the assistant. The user is saying, in effect: "Here is the blueprint. You have all the information you need. Decide what to do next and execute it." This is a high-trust delegation that only works because the assistant has demonstrated competence throughout the earlier phases.
The assistant's response (message 533) validates this trust. It immediately plans the next steps: E2E GPU test, pipelined PoSt/SnapDeals synthesis, and true cross-proof overlap. It creates a todo list with priorities and statuses. It does not ask clarifying questions because the plan provides all the answers.
The Mistakes and Incorrect Assumptions
No engineering effort is free of mistakes, and this message is no exception. Several assumptions embedded in the plan and the user's instruction turn out to be incorrect or incomplete, as revealed by the subsequent conversation.
Mistake 1: Per-Partition Pipelining Is Not a Performance Win
The Phase 2 design assumes that splitting synthesis from GPU proving and running them in parallel will improve throughput. The implementation in pipeline.rs applies this at the partition level: synthesize partition 0, GPU-prove partition 0, synthesize partition 1, GPU-prove partition 1, etc.
The E2E GPU test (which happens in the messages following this one) reveals a devastating result: 611 seconds for the pipelined approach vs 93 seconds for the monolithic approach. The per-partition pipeline is 6.6x slower.
Why? Because the monolithic approach batches all 10 partitions into a single rayon-parallelized synthesis call and a single supraseal GPU call. The per-partition approach serializes work that the monolithic approach parallelizes. Synthesis of partition 1 cannot begin until partition 0's synthesis is complete (because the pipeline code is sequential within a single proof). The GPU cannot start on partition 1 until partition 0's GPU proving is done. There is no overlap benefit within a single proof because there is only one GPU and one synthesis thread working on the same proof.
The mistake is in applying a cross-job optimization (overlap synthesis of job N+1 with GPU of job N) to a within-job context (overlap synthesis of partition K+1 with GPU of partition K). The two scenarios have fundamentally different parallelism profiles.
Mistake 2: Underestimating the Cost of Serialization
The per-partition approach also incurs overhead from repeated function calls, memory allocations, and data movement. Each partition's synthesis creates ProvingAssignment objects that must be passed to the GPU prover. In the monolithic approach, this happens once for all 10 partitions. In the per-partition approach, it happens 10 times, each with its own overhead.
The project plan's memory analysis (Section 7 of the Phase 2 design doc, referenced in the "Discoveries") correctly identifies that per-partition intermediate state is ~13.6 GiB vs ~136 GiB for all 10 partitions. But it misses the performance cost of the serialization overhead. The memory savings are real, but they come at a prohibitive latency cost for single-proof scenarios.
Mistake 3: Assuming the Pipeline Would Be Tested Before Optimization
The project plan's roadmap puts Phase 2 (pipelining) before Phase 3 (batching). The implicit assumption is that pipelining provides a standalone benefit that justifies its implementation effort. But the E2E test shows that per-partition pipelining is worse than monolithic for single proofs. The benefit only appears when there is a stream of proofs to overlap.
This means the "Estimated impact" in the plan (1.5x throughput) is only achievable with multiple concurrent proof jobs. For a single proof, the pipeline is a regression. The plan's throughput estimates assume a workload with multiple proofs in flight, which may not match the actual deployment pattern.
Mistake 4: The Batch-All-Partitions Mode Was Not in the Original Plan
The project plan does not describe a "batch-all-partitions" mode for PoRep C2. It describes per-partition pipelining (Phase 2) and cross-sector batching (Phase 3), but not a mode that synthesizes all partitions at once and proves them in one GPU call. This mode is created in response to the E2E test failure—it is an adaptive fix, not a planned feature.
The fact that the fix works (restoring 91.2s performance, matching the monolithic baseline) shows that the architecture is flexible enough to accommodate new modes. But it also reveals a gap in the original planning: the plan did not consider the single-proof latency case.
Mistake 5: Assuming the Bellperson Fork Is Correct Without Testing
The bellperson fork was created in commit 5 and validated only by compilation (cargo check --workspace --no-default-features). It was not tested with --features cuda-supraseal until the E2E GPU test. This is a risky assumption—the fork could have subtle bugs in the prove_from_assignments() function that only manifest with real GPU data.
Fortunately, the fork turns out to be correct. The E2E test produces a valid 1920-byte proof. But the assumption that "it compiles, therefore it works" is a common engineering pitfall, and the project plan does not explicitly call for pre-GPU-test validation of the fork.
The Deeper Architecture: What the Project Plan Reveals About System Design
Beyond the immediate context of Phase 2, the project plan document is a master class in systems architecture. Let us examine some of its deeper design principles and how they inform the Phase 2 work.
The Inference Engine Analogy
The plan draws a deliberate analogy between cuzk and inference engines like vLLM or TensorRT:
| Inference Concept | cuzk Equivalent | |---|---| | Model weights | SRS / Groth16 parameters | | Model loading/swapping | SRS loading/eviction from pinned memory | | Inference request | Proof request | | KV cache / activations | Witness vectors, a/b/c evaluations | | Continuous batching | Cross-sector proof batching | | Prefill vs decode | Witness synthesis vs GPU compute |
This analogy is not decorative—it drives architectural decisions. The SRS manager's tiered residency (hot/warm/cold) mirrors how inference engines manage model weights across GPU VRAM, host RAM, and disk. The scheduler's batch collector mirrors continuous batching in LLM serving. The GPU worker pipeline mirrors the prefill/decode split in transformer inference.
By framing cuzk as an inference engine for proofs, the plan imports decades of systems research from the ML inference world. This is a powerful design pattern: rather than inventing a new architecture from scratch, cuzk adapts proven patterns from a related domain.
The Memory Hierarchy Design
The SRS memory manager is the heart of the system. Its three-tier design reflects a deep understanding of the cost curves:
- Hot (cudaHostAlloc): Zero latency for GPU access. But expensive in terms of pinned memory budget (~47 GiB for PoRep).
- Warm (mmap): The OS page cache keeps recently accessed pages in RAM. Re-pinning requires deserialization but no disk I/O. Cost: ~2-10s.
- Cold (disk): Full deserialization from the
.paramsfile. Cost: 30-90s. The eviction policy is LRU with a critical constraint: never evict an SRS with active references. This prevents a proof from failing mid-computation because its parameters were swapped out. The reference counting is done withAtomicU32for lock-free reads. The budget management distinguishes between small machines (96-128 GiB RAM) and large machines (256+ GiB RAM). On small machines, only one large SRS can be hot at a time, and switching between PoRep and SnapDeals costs 30-60s. The scheduler mitigates this by grouping same-type proofs. On large machines, all SRS files can be hot simultaneously, eliminating swap overhead entirely. This tiered design is essential for Phase 2 because the pipeline needs to ensure the SRS is hot before synthesis begins. TheSrsManager::ensure_loaded()method is called at the start of each pipeline cycle, and its cost depends on the current tier of the required SRS.
The Scheduler as a Load-Balancing Layer
The scheduler is not just a priority queue—it is a load-balancing layer that considers GPU affinity, SRS residency, and memory budget. Its three dispatch rules are:
- Prefer a GPU that already has the required SRS hot (zero swap cost)
- If no GPU has it, prefer the GPU with the smallest loaded SRS (fastest to evict+reload)
- If all GPUs are busy, queue the job This is critical for Phase 2 because the pipeline adds another dimension to scheduling: the synthesis lookahead. If the scheduler dispatches a proof to a GPU that doesn't have the SRS hot, the pipeline must wait for SRS loading before synthesis can begin. The scheduler's SRS-aware dispatch minimizes this wait. The batch collector (Phase 3) will add yet another dimension: accumulating same-circuit-type proofs until a batch is full or a timeout expires. The pipeline must be able to handle batched proofs—multiple sectors' circuits synthesized together and proved in one GPU call.
The Bellperson Fork Strategy
The plan's approach to the bellperson fork is deliberately minimal. Rather than refactoring bellperson's internals or adding new abstractions, the fork makes three small changes:
- Make
ProvingAssignmentstruct and all its fieldspub - Make the
suprasealmodulepub - Make
synthesize_circuits_batch()puband add a newprove_from_assignments()function This minimalism is a risk-reduction strategy. A larger fork would be harder to maintain, more likely to introduce bugs, and more difficult to upstream. The minimal fork exposes the existing internal split without changing how bellperson works internally. Theprove_from_assignments()function is the key addition. It extracts the GPU-phase code fromcreate_proof_batch_priority_inner(which previously combined synthesis and proving into a single call) into a standalone function that takes pre-synthesized assignments. This function handles: - PackingProvingAssignmentdata into raw scalar arrays - Creatingsupraseal_c2::Assignmentobjects - Callingsupraseal_c2::generate_groth16_proof()- Deserializing the proof bytes By separating this into a public function, cuzk can call it from a different thread than synthesis, enabling the pipeline overlap.
The Phase 2 Implementation: A Deep Dive into pipeline.rs
The pipeline.rs file (553 lines) is the centerpiece of Phase 2. Let us examine its structure and how it relates to the project plan.
The SynthesizedProof Type
pub struct SynthesizedProof {
pub assignments: Vec<ProvingAssignment<Scalar>>,
pub input_assignments: Vec<Arc<Vec<Scalar>>>,
pub aux_assignments: Vec<Arc<Vec<Scalar>>>,
pub timings: PipelinedTimings,
}
This type represents the output of synthesis and the input to GPU proving. It contains the three return values from synthesize_circuits_batch() plus timing information. The type is designed to be passed across threads—it owns its data and can be moved or cloned as needed.
The Per-Partition Synthesis Function
pub fn synthesize_porep_c2_partition(
porep_config: &PoRepConfig,
c1_output: &SealCommitPhase1Output,
sector_id: u64,
partition_index: usize,
groth_params: Arc<SuprasealParameters>,
) -> Result<SynthesizedProof>
This function synthesizes a single partition of a PoRep C2 proof. It:
- Constructs the
PublicInputsfor the partition - Calls
StackedCompound::circuit()to build the circuit - Calls
synthesize_circuits_batch()to run the circuit's synthesis - Returns the
SynthesizedProofwith timing The function is designed to be called in a loop, one partition at a time. This is the per-partition approach that turns out to be 6.6x slower than monolithic.
The GPU Proving Function
pub fn gpu_prove(
synthesized: SynthesizedProof,
groth_params: Arc<SuprasealParameters>,
) -> Result<Vec<u8>>
This function takes a SynthesizedProof and runs the GPU proving phase. It:
- Calls
prove_from_assignments()with the assignments and SRS - Serializes the proof bytes
- Returns the concatenated proof The function is generic—it works for any number of partitions, not just one. This is important because the batch-all-partitions mode (added later) synthesizes all 10 partitions at once and proves them in one call.
The Full Pipeline Orchestrator
pub fn prove_porep_c2_pipelined(
porep_config: &PoRepConfig,
c1_output: &SealCommitPhase1Output,
sector_id: u64,
groth_params: Arc<SuprasealParameters>,
lookahead: usize,
) -> Result<Vec<u8>>
This function orchestrates the full 10-partition pipeline. It:
- Iterates over all 10 partitions
- For each partition, calls
synthesize_porep_c2_partition() - After synthesis, calls
gpu_prove()with the synthesized proof - Concatenates all partition proofs into the final 1920-byte proof The
lookaheadparameter is intended for future use—it controls how many partitions to synthesize ahead of GPU proving. Withlookahead=0, synthesis and proving are sequential (synthesize partition 0, prove partition 0, synthesize partition 1, prove partition 1, ...). Withlookahead>0, multiple partitions could be synthesized before any are proved, enabling overlap. However, the current implementation always useslookahead=0because the GPU worker is single-threaded.
The SRS Manager Integration
The pipeline uses SrsManager::ensure_loaded() to make sure the PoRep SRS is hot before synthesis begins. This is called once at the start of prove_porep_c2_pipelined(), not once per partition. The SRS is kept hot for the duration of the proof via reference counting.
The SrsManager also handles the mapping from CircuitId to param filenames. For PoRep 32 GiB, the filename is:
v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params
This 45 GiB file is loaded via SuprasealParameters::new(path) and kept in pinned memory.
The E2E GPU Test: What It Reveals and What It Changes
The E2E GPU test (which occurs in the messages immediately following this one) is the crucible in which the Phase 2 design is tested. Let us examine what the test involves and what it reveals.
Test Setup
The test requires:
- Building the daemon with
--features cuda-supraseal - Starting the daemon with pipeline mode enabled (
pipeline.enabled=truein config) - Submitting a 32 GiB PoRep C2 proof using
cuzk-bench single --type porep --c1 /data/32gbench/c1.json - Measuring the time and verifying the proof The expected proof size is 1920 bytes (10 partitions × 192 bytes per Groth16 proof). The expected time is comparable to the Phase 1 baseline (~93s with warm SRS).
Test Result: The 6.6x Regression
The per-partition pipeline takes ~611 seconds. This is a catastrophic regression from the ~93 second baseline. The reasons are clear in retrospect:
- Serialized synthesis: The monolithic approach uses rayon to synthesize all 10 partitions in parallel. The per-partition approach synthesizes them one at a time. Synthesis is CPU-bound and highly parallelizable, so serializing it loses a factor of ~10x in wall-clock time.
- Serialized GPU proving: The monolithic approach proves all 10 partitions in a single supraseal GPU call, which internally parallelizes NTT and MSM across the GPU's cores. The per-partition approach makes 10 separate GPU calls, each with overhead for kernel launches, memory transfers, and synchronization.
- No overlap benefit: Within a single proof, there is no opportunity to overlap synthesis of partition N+1 with GPU proving of partition N, because both operations need the same resources (CPU for synthesis, GPU for proving). The overlap only helps across different proofs, where synthesis of proof N+1 can run on the CPU while GPU proving of proof N runs on the GPU.
The Adaptive Fix: Batch-All-Partitions Mode
The response to this regression is the creation of a synthesize_porep_c2_batch() function that synthesizes all 10 partitions in a single rayon-parallelized call and proves them in one GPU call. This restores the monolithic performance while preserving the pipeline architecture for future cross-proof overlap.
The batch mode is implemented by:
- Building all 10 circuits in a
VecusingStackedCompound::circuit()for each partition - Calling
synthesize_circuits_batch()once with all 10 circuits - Calling
prove_from_assignments()once with all 10 assignments - Serializing the combined proof This is essentially the monolithic approach, but using the split API. The result is 91.2 seconds—matching the Phase 1 baseline.
The Revised Architecture
After the fix, the Phase 2 architecture has two modes:
- Batch mode (single proof): Synthesize all partitions at once, prove all partitions at once. Used when there is only one proof to process. Matches monolithic performance.
- Pipeline mode (multiple proofs): Synthesize proof N+1 while GPU proves proof N. Used when there is a stream of proofs. Provides the overlap benefit that Phase 2 was designed for. The engine's dispatch logic chooses between modes based on the queue depth. If there are multiple proofs waiting, it uses pipeline mode. If there is only one, it uses batch mode. This is a pragmatic compromise that preserves the Phase 2 investment while avoiding the single-proof regression. The pipeline architecture (split API, SRS manager,
SynthesizedProoftype) is still needed for the cross-proof overlap case, but the per-partition serialization is abandoned.
The Broader Implications for the cuzk Project
Message 532 and its surrounding context reveal several important lessons about the cuzk project's trajectory.
The Importance of Empirical Validation
The project plan makes performance estimates based on reasoning and analogy (inference engine pipelining). But the actual performance characteristics of per-partition pipelining could only be determined empirically. The E2E GPU test is the first time the pipeline runs on real hardware with real data, and it immediately reveals a flaw in the design.
This validates the project's approach of building and testing early. The alternative—spending weeks optimizing the per-partition pipeline based on theoretical models—would have been wasted effort. Instead, the team discovers the regression quickly and pivots to the batch-all-partitions mode.
The Flexibility of the Split API
The bellperson fork's split API proves its value even though the per-partition pipeline is abandoned. The batch-all-partitions mode uses the same synthesize_circuits_batch() and prove_from_assignments() functions, just with all 10 partitions in one call. The split API is not tied to the per-partition approach—it works for any batch size.
This flexibility means the Phase 2 investment in the bellperson fork is not wasted. The fork enables both the batch mode (for single proofs) and the pipeline mode (for multiple proofs), and it will also enable Phase 3 cross-sector batching (multiple sectors in one GPU call).
The Path to Phase 3
With the batch-all-partitions mode working, the next step is to add cross-sector batching (Phase 3). This involves:
- Bumping
max_num_circuits = 10to30+ingroth16_srs.cuh:62 - Implementing a batch collector in the scheduler that accumulates same-circuit-type proofs
- Calling
synthesize_circuits_batch()with circuits from multiple sectors - Calling
prove_from_assignments()with all assignments combined The Phase 2 split API is the foundation for Phase 3. Without it, cross-sector batching would require modifying the monolithicfilecoin-proofs-apifunctions. With it, batching is just a matter of collecting more circuits before calling the split API.
The Memory Reduction Potential
The project plan's memory analysis shows that per-partition pipelining reduces peak memory from ~200 GiB to ~27 GiB (2 partitions in flight). While the per-partition approach is abandoned for performance reasons, the memory reduction insight is still valuable. The batch-all-partitions mode uses ~200 GiB (same as monolithic), but the pipeline mode (cross-proof overlap) could use less if it processes proofs one partition at a time.
However, the memory reduction benefit is secondary to the performance benefit. In production, the cuzk daemon runs on machines with 256+ GiB RAM, so the ~200 GiB peak is acceptable. The primary goal is throughput, not memory reduction.
Conclusion: The Message as a Microcosm
Message 532 is a microcosm of the entire cuzk project. It captures the tension between planning and execution, between theoretical design and empirical validation, between the ideal architecture and the pragmatic compromise. It shows a user who understands the value of context restoration, who invests in re-grounding before proceeding, and who trusts the assistant to synthesize a complex document into actionable next steps.
The message also reveals the assumptions that underpin the project—some correct (the file exists, the plan is current, the environment is ready) and some incorrect (per-partition pipelining is beneficial, the pipeline provides standalone value). The incorrect assumptions are not failures; they are learning opportunities that drive the project forward. The E2E GPU test that follows this message is the crucible that refines the design.
In the end, the cuzk project is not about building the perfect pipeline on the first try. It is about building a flexible architecture that can adapt to empirical feedback, that can accommodate new modes (batch-all-partitions) without discarding the old ones, and that provides a clear path from Phase 2 to Phase 3 to Phase 4 and beyond. Message 532 is the moment when the blueprint is re-read, the context is refreshed, and the next iteration begins.
The document that the user re-reads is 1213 lines of dense technical content. But the message itself is about something simpler: the discipline of going back to the source, of checking the map before continuing the journey, of ensuring that every step is grounded in the architectural vision. In a world of rapid prototyping and "move fast and break things," this discipline is rare and valuable. It is what makes the cuzk project not just a collection of code, but a coherent system designed to last.## The Knowledge Required: A Deeper Examination
To fully appreciate the significance of message 532, one must understand not just the surface-level content of the project plan, but the deep technical ecosystem in which it operates. This section expands on the knowledge domains that a reader must command to grasp the full implications of this message.
The Filecoin Proof-of-Replication Protocol
Filecoin's proof system is built on a chain of cryptographic constructions. At the foundation is Proof-of-Replication (PoRep), which proves that a storage miner is storing a unique copy of data. PoRep works by having the miner repeatedly encode the data through a depth-robust graph (DRG) and then prove knowledge of the resulting labels. This is the "Stacked DRG" construction referenced throughout the plan.
The proving pipeline has several stages:
- PreCommit1 (PC1): The miner reads the sector data, builds Merkle trees, and generates the initial sealing proof. This is CPU-intensive and can take hours for a 32 GiB sector.
- PreCommit2 (PC2): The miner generates the "vanilla proof" — the witness data that will be used in the SNARK. This involves encoding the sector through multiple layers of the Stacked DRG and producing inclusion proofs for each challenge.
- Commit1 (C1): The miner creates a SNARK circuit that verifies the vanilla proof. This is the first stage that produces a SNARK output — the
SealCommitPhase1Outputthat contains per-partition vanilla proofs. - Commit2 (C2): The miner runs the Groth16 prover on the C1 output to produce the final proof. This is the stage that cuzk targets. The C2 stage is where the ~200 GiB memory footprint comes from. Each of the 10 partitions has ~13 million constraints, and the Groth16 prover must hold the entire constraint system in memory during synthesis. The a/b/c evaluation vectors alone are ~13.6 GiB per partition, and with 10 partitions in flight simultaneously, the peak memory reaches ~200 GiB.
The Role of the SRS in Groth16
The Structured Reference String (SRS) is a set of elliptic curve points used in the Groth16 proving system. For Filecoin's BLS12-381 curve, the SRS consists of:
- Powers of tau: A sequence of curve points
G1 * τ^iandG2 * τ^ifor i = 0 to N, where N is the size of the constraint system. These are used in the MSM (Multi-Scalar Multiplication) operations during proof generation. - Additional parameters: The verifying key, the circuit-specific parameters, and the inner product SRS for SnarkPack aggregation. The SRS is generated through a trusted setup ceremony (or, in Filecoin's case, through a deterministic process based on the sector size and proof type). It is stored in
.paramsfiles that can be tens of GiB in size. Loading the SRS involves: 1. Reading the file from disk (sequential read of 45+ GiB) 2. Deserializing the curve points (parsing the binary format and reconstructing the elliptic curve points) 3. Allocating pinned host memory viacudaHostAlloc4. Optionally transferring the points to GPU VRAM The 30-90 second SRS load time mentioned in the plan is dominated by the deserialization step, which involves thousands of field element inversions and point decompressions.
The CUDA Memory Model for Proof Generation
GPU-accelerated Groth16 proving uses several distinct memory regions:
- SRS in pinned host memory: The ~47 GiB SRS is stored in CUDA-pinned host memory. The GPU can DMA directly from this memory during MSM operations, but the bandwidth is limited by the PCIe bus (typically 12-25 GB/s for PCIe 3.0/4.0).
- Witness data in pinned host memory: The a/b/c evaluation vectors and the input/aux assignments are also stored in pinned memory for fast GPU transfer. These are smaller than the SRS but still significant (~13.6 GiB per partition for PoRep).
- GPU VRAM: The GPU stores intermediate computation results (NTT twiddle factors, MSM buckets, proof points) in its VRAM. For a 16 GB GPU like the RTX 5070 Ti used in testing, this is a tight budget.
- CUDA streams and kernel launches: Each GPU operation (NTT, MSM, point addition) is a separate CUDA kernel launch. The overhead of launching kernels and synchronizing streams adds up, especially when proving many partitions individually vs. in batch. The supraseal CUDA implementation optimizes this memory model by: - Keeping the SRS in pinned host memory and streaming points to the GPU as needed - Using a single CUDA stream for all operations within a proof - Pre-allocating GPU buffers and reusing them across proofs - Using cooperative kernel launches for multi-GPU scenarios
The Bellperson Architecture
Bellperson is a fork of the Bellman library (hence the name "bellperson" — a person who rings bells, playing on "Bellman"). It implements the Groth16 proving system with optimizations for Filecoin's specific circuit shapes.
The key architectural insight is that bellperson's proving pipeline has a natural split point:
Circuit → synthesize() → ProvingAssignment → pack() → RawScalars → generate_groth16_proof() → Proof
The synthesize() method is CPU-only and highly parallelizable via rayon. It walks the circuit's constraint system and evaluates the linear combinations to produce the a/b/c vectors. The generate_groth16_proof() method is GPU-only and runs NTT + MSM on the packed scalars.
In the monolithic API (create_proof_batch_priority_inner), these two phases are combined into a single function call. The Phase 2 bellperson fork separates them, allowing the CPU synthesis of proof N+1 to overlap with the GPU proving of proof N.
The ProvingAssignment type is the bridge between the two phases. It contains:
a,b,c: The evaluation vectors (one scalar per constraint)input_assignments: The public input valuesaux_assignments: The private witness valuesdensity_trackers: Bitmasks indicating which constraints are active (used for optimization) Making this type public (one of the three fork changes) allows cuzk to inspect and manipulate the assignments between synthesis and proving.
The Supraseal CUDA Backend
The supraseal CUDA backend (supraseal-c2) is a specialized implementation of the Groth16 prover that uses handwritten CUDA kernels for the most performance-critical operations:
- NTT (Number Theoretic Transform): Converts polynomials between coefficient and evaluation representations. Used in the MSM precomputation. Supraseal uses a batched NTT implementation that processes multiple polynomials simultaneously.
- MSM (Multi-Scalar Multiplication): Computes
Σ s_i * P_iwheres_iare scalars andP_iare curve points. This is the dominant cost in Groth16 proving. Supraseal uses the "bucket method" with Pippenger's algorithm, parallelized across GPU threads. - Proof assembly: Combines the MSM outputs into the final Groth16 proof (A, B, C points in G1/G2). The supraseal backend is activated by the
cuda-suprasealcompile-time feature flag. When enabled, bellperson'screate_proof_batch_priority_innerdelegates tosupraseal_c2::generate_groth16_proof()instead of the native GPU prover. The Phase 2 fork'sprove_from_assignments()function extracts the GPU-phase code from this delegation path, making it callable independently.
The Thinking Process: A Step-by-Step Reconstruction
To understand the user's thinking in sending message 532, we must reconstruct the cognitive process that led to this action. This is necessarily speculative, but it is grounded in the observable context of the conversation.
Step 1: Assessment of Current State
The user knows that the last substantive work on Phase 2 was commit 6 (beb3ca9c), which implemented the pipeline.rs and srs_manager.rs modules. The code compiles cleanly in non-CUDA mode, but has not been tested with a real GPU. The user also knows that the assistant's context has been accumulating messages since that commit, and the full project plan may no longer be in the active context window.
The user's mental model at this point is something like:
"We have written the Phase 2 pipeline code. It compiles. But we haven't tested it on a GPU yet. Before we do that, I need to make sure the assistant has the complete architectural picture. The project plan is the authoritative source. Let me re-read it into context."
Step 2: Decision to Re-ground
The user decides that the cost of re-reading the project plan (a few seconds of file I/O and context consumption) is far less than the cost of a failed GPU test due to a misunderstanding of the architecture. This is a risk-management decision.
The user also recognizes that the assistant's next actions will be complex and multi-step: building with CUDA, starting the daemon, submitting a proof, measuring performance, debugging any issues. Having the full plan in context reduces the cognitive load on the assistant by providing a single source of truth for all architectural questions.
Step 3: Formulation of the Instruction
The instruction @cuzk-project.md continue phase 2 is concise but information-dense:
@cuzk-project.md— Invokes a tool to read the file. The@syntax suggests this is a recognized command in the conversation interface.continue phase 2— Specifies the work stream. This is not "start phase 2" or "review phase 2" but "continue" — implying that Phase 2 is in progress and the next steps should be taken. The instruction does not specify what "continue" means in detail. It does not say "test the pipeline on GPU" or "fix any issues." It trusts the assistant to determine the appropriate next steps based on the project plan and the current state.
Step 4: Execution and Observation
The user executes the instruction and observes the tool call output. The Read tool successfully retrieves the file and displays all 1213 lines. The user can see that the file is intact and current. They can also see the assistant's next response (message 533), which demonstrates that the assistant has absorbed the document and is planning the E2E GPU test.
If the file had been missing or corrupted, the user would have seen an error instead of the file content. If the assistant had responded with confusion or requests for clarification, the user would have known that the re-grounding was insufficient. But the response is confident and well-structured, confirming that the strategy worked.
Step 5: Evaluation of the Outcome
The user evaluates the success of the re-grounding by observing the assistant's subsequent actions. The assistant:
- Creates a todo list with the E2E GPU test as the top priority
- Plans to build with
--features cuda-supraseal - References specific sections of the project plan (the pipeline architecture, the bellperson fork API)
- Does not ask clarifying questions about the architecture This confirms that the re-grounding was successful. The assistant has the full architectural context and is ready to proceed.
The Mistakes: A Deeper Analysis
The previous section identified several mistakes and incorrect assumptions. Let us examine them in greater depth, including their root causes and their implications for the project.
Root Cause of the Per-Partition Pipeline Regression
The per-partition pipeline regression (6.6x slower than monolithic) has multiple root causes:
1. Misapplication of the inference engine analogy. The inference engine analogy (vLLM/TensorRT) suggests that splitting prefill (synthesis) from decode (GPU proving) and overlapping them improves throughput. But in inference engines, prefill and decode operate on different requests — the prefill of request N+1 overlaps with the decode of request N. In the per-partition pipeline, synthesis and GPU proving operate on the same request (different partitions of the same proof). The overlap is impossible because both phases need the same resources.
2. Underestimation of rayon parallelism. The monolithic approach uses rayon to synthesize all 10 partitions in parallel. The per-partition approach synthesizes them sequentially. The project plan's memory analysis correctly notes that per-partition synthesis reduces peak memory, but it does not account for the loss of parallelism. On a machine with 142 CPU cores (as noted in the "Discoveries" section), the rayon-parallelized synthesis of all 10 partitions is nearly 10x faster than sequential per-partition synthesis.
3. GPU kernel launch overhead. Each GPU proving call involves kernel launches for NTT, MSM, and proof assembly. The monolithic approach launches these kernels once for all 10 partitions. The per-partition approach launches them 10 times. The kernel launch overhead, combined with the overhead of transferring data to/from the GPU for each partition, adds significant latency.
4. Lack of a true pipeline stage. The per-partition pipeline does not actually pipeline anything — it serializes. A true pipeline would have two threads: one synthesizing partition N+1 while another proves partition N. But the implementation uses a single thread that alternates between synthesis and proving. There is no overlap because there is no concurrency.
The Fix: Batch-All-Partitions Mode
The batch-all-partitions mode addresses all four root causes:
- It restores rayon parallelism by synthesizing all 10 partitions in a single
synthesize_circuits_batch()call. - It restores GPU efficiency by proving all 10 partitions in a single
prove_from_assignments()call. - It eliminates the per-partition kernel launch overhead.
- It preserves the split API architecture for future cross-proof overlap. The fix is not a rejection of the pipeline concept — it is a refinement. The pipeline architecture (split API, SRS manager,
SynthesizedProoftype) is still needed for the cross-proof overlap case. The fix simply adds a batch mode for single-proof scenarios where pipelining provides no benefit.
The Missed Opportunity: Pre-Validation of the Pipeline Design
The project plan could have identified the per-partition regression risk earlier through a simple thought experiment:
"If we synthesize partitions one at a time instead of all at once, we lose the rayon parallelism that makes synthesis fast. If we prove partitions one at a time instead of all at once, we pay GPU kernel launch overhead 10 times instead of once. The only benefit is reduced peak memory. Is the memory reduction worth the performance cost?"
This thought experiment would have revealed that the per-partition approach is only beneficial for memory-constrained environments, not for performance. The plan's memory analysis (Section 7 of the Phase 2 design doc) correctly identifies the memory trade-off but does not analyze the performance trade-off.
The lesson is that theoretical analysis should be complemented by quick empirical validation. A micro-benchmark comparing per-partition synthesis time vs. batch synthesis time would have revealed the regression before the full E2E test. The project's testing strategy could be improved by adding such micro-benchmarks to the pipeline.
The Incorrect Throughput Estimate
The project plan estimates Phase 2 throughput improvement at 1.5x over Phase 0. This estimate assumes the cross-proof overlap model, not the per-partition model. But the plan does not explicitly state this assumption, making it easy to misinterpret.
The actual throughput improvement from Phase 2 depends on the workload:
- Single proof: No improvement (batch mode matches monolithic). Actually a regression if per-partition mode is used.
- Multiple proofs with overlap: Potentially 1.5x improvement, but this has not been empirically validated.
- Mixed proof types: The SRS residency benefit (20.5%) applies regardless of pipeline mode. The 1.5x estimate should be treated as a rough upper bound, not a guaranteed improvement. The actual benefit depends on the deployment pattern and the effectiveness of the async overlap implementation.
The Broader Context: Filecoin's Proving Ecosystem
The cuzk project does not exist in isolation. It is part of a larger ecosystem of Filecoin proving software that includes several components:
The Filecoin Proofs Stack
The proving stack has multiple layers:
- storage-proofs-core: The foundational library defining the
CompoundProoftrait, theProvingAssignmenttype, and the parameter cache infrastructure. - storage-proofs-porep: The PoRep-specific circuit definitions, including
StackedCircuitandStackedCompound. - storage-proofs-post: The PoSt circuit definitions, including
FallbackPoStCircuitandFallbackPoStCompound. - storage-proofs-update: The SnapDeals circuit definitions, including
EmptySectorUpdateCircuitandEmptySectorUpdateCompound. - filecoin-proofs: The top-level API that combines the storage-proofs libraries with the parameter management and proof serialization.
- filecoin-proofs-api: The FFI-friendly API that exposes the proving functions to Go via CGO.
- bellperson: The Groth16 proving library that implements the actual proof generation. The cuzk project sits at the same level as
filecoin-proofs-api, providing an alternative interface to the same underlying proving stack. This is why Phase 0 requires zero upstream modifications — it uses the samefilecoin-proofs-apifunctions that Curio already calls.
The Supraseal Project
Supraseal is a separate project that provides a high-performance CUDA implementation of the Groth16 prover. It is located at extern/supra_seal/c2/ in the Curio repository. The supraseal codebase includes:
- C++ CUDA kernels: Handwritten kernels for NTT, MSM, and point arithmetic, optimized for Filecoin's BLS12-381 curve.
- Rust FFI bindings: The
supraseal-c2crate that wraps the CUDA kernels in a Rust API. - SRS management: The
create_SRSfunction with an LRU cache for the deserialized parameters. The supraseal project is a key dependency of cuzk. The Phase 2 bellperson fork'sprove_from_assignments()function callssupraseal_c2::generate_groth16_proof()for the GPU phase. The performance of cuzk's pipeline is directly tied to the performance of supraseal's CUDA kernels.
The Curio Integration Path
The ultimate goal of cuzk is to replace the current ffiselect child-process model in Curio. The integration path is carefully designed to be incremental:
- Phase 0-1: cuzk runs alongside
ffiselect. Curio tasks gain a feature flag to choose between the two. Operators can test cuzk on a subset of proofs while keepingffiselectas the default. - Phase 2-3: cuzk proves its performance and reliability advantages. More operators switch to cuzk. The
ffiselectpath becomes legacy. - Phase 4-5: cuzk becomes the default proving backend.
ffiselectis retired. Curio either embeds the cuzk engine (Mode A) or manages the daemon lifecycle (Mode B). The gRPC API is the key enabler of this integration. By using a standard protocol with strong typing (protobuf), cuzk can communicate with Curio over Unix sockets or TCP, regardless of the deployment mode. The API is designed to be forward-compatible — new fields can be added without breaking existing clients.
The Future: From Phase 2 to Phase 5
Message 532 is a snapshot of the project at a specific point in time. The future trajectory, as outlined in the project plan, involves three more phases of optimization.
Phase 3: Cross-Sector Batching
Cross-sector batching is the next major performance optimization. It involves proving multiple sectors' circuits in a single GPU pass, amortizing the fixed costs (kernel launches, memory transfers) across multiple proofs.
The key technical challenges are:
- Bumping
max_num_circuits: The supraseal code currently limits the batch to 10 circuits. This must be increased to 30+ for batching. - Batch collector in the scheduler: The scheduler must accumulate same-circuit-type proofs until a batch is full or a timeout expires.
- Memory management: Batching increases peak memory because multiple sectors' assignments must be held simultaneously. The scheduler must enforce memory budgets. The estimated impact is 2-3x throughput per GPU with a batch size of 3.
Phase 4: Compute Quick Wins
Phase 4 cherry-picks high-impact, low-effort optimizations from the catalog of 18 identified improvements:
- SmallVec for LC Indexer: Reduces memory allocation overhead during synthesis by using small-vector optimization for the linear combination indexer. Estimated 15-30% synthesis speedup for ~5 lines of code.
- Pre-size vectors: Pre-allocates vectors to their final capacity to avoid repeated reallocations. Estimated 5-10% synthesis speedup.
- Pin a,b,c memory: Uses CUDA-pinned memory for the a/b/c evaluation vectors to improve transfer bandwidth. Estimated +50% transfer bandwidth.
- Parallelize B_G2 MSMs: The B_G2 MSM is currently sequential; parallelizing it reduces its runtime from ~50s to ~5s.
- batch_addition occupancy: Tunes the CUDA kernel launch parameters for the batch addition kernel. Estimated 5-12% MSM speedup.
- Reuse GPU allocs: Reuses GPU memory allocations across proofs to avoid the overhead of repeated allocation and deallocation. The estimated cumulative impact is 30-40% faster per-proof on top of batching.
Phase 5: Pre-Compiled Constraint Evaluator
Phase 5 is the most ambitious optimization. It replaces the entire circuit synthesis phase with a sparse matrix-vector multiply, eliminating the overhead of walking the constraint system.
The key insight is that the R1CS matrices (A, B, C) are fixed for a given circuit topology. They do not change between proofs. The current synthesis process re-derives these matrices from the circuit description every time, which is wasteful. Instead, Phase 5 proposes:
- RecordingCS: A one-time pass that extracts the fixed R1CS matrices into Compressed Sparse Row (CSR) format. This is done once per circuit topology (e.g., once for PoRep 32 GiB).
- WitnessCS: A simplified circuit that only runs the
alloc()closures (to get the witness values) and skips theenforce()calls (since the matrices are already recorded). - Sparse MatVec evaluator: Computes
a = A·w,b = B·w,c = C·wusing sparse matrix-vector multiplication, which is O(constraints) instead of O(constraints × gates). - Coefficient-specialized MatVec: Further optimizes the MatVec by exploiting the fact that many coefficients are ±1 (so multiplication is just addition or subtraction) and many witness values are boolean (so multiplication is just conditional selection). The estimated impact is 3-5x faster synthesis, leading to ~10x total throughput over the baseline.
Conclusion: The Enduring Value of a Good Blueprint
Message 532 is, at its core, an act of discipline. It is the user saying: "Before we take the next step, let us re-read the blueprint. Let us make sure we remember where we are going and why." In a field where the temptation is always to rush forward, to write code and see what happens, this discipline is what separates well-engineered systems from brittle hacks.
The project plan that the user re-reads is a remarkable document. It is comprehensive without being bloated, detailed without being pedantic, aspirational without being unrealistic. It captures the architectural vision, the technical constraints, the phased roadmap, and the key design decisions in a single coherent narrative. It is the kind of document that every complex project should have but few actually do.
The message also reveals something about the relationship between the user and the assistant. The user trusts the assistant to absorb 1213 lines of technical content and synthesize it into actionable next steps. The assistant, in turn, demonstrates that trust is well-placed by immediately planning the E2E GPU test, referencing the plan's specifications, and proceeding without confusion or delay. This is a partnership built on mutual competence and clear communication.
As the cuzk project moves forward — through the E2E GPU test, the batch-all-partitions fix, the PoSt/SnapDeals synthesis expansion, and ultimately the async overlap architecture — message 532 will stand as a reminder of the importance of grounding every step in the architectural vision. The blueprint is not a static artifact; it is a living document that guides the project through uncertainty and change. And the discipline of re-reading it at critical junctures is what ensures that the project stays on course.
In the end, the cuzk project is not just about building a faster proving engine. It is about building a system that is architected for continuous improvement, that can adapt to empirical feedback, and that provides a clear path from today's implementation to tomorrow's optimizations. Message 532 is the moment when that architecture is reaffirmed, the context is refreshed, and the journey continues.