From Investigation to Architecture: Designing cuzk, a Pipelined SNARK Proving Daemon for Filecoin

Introduction

In the span of a single extended coding session, an extraordinary act of systems architecture unfolded. Building on months of prior analysis across seven optimization proposals, the assistant conducted an exhaustive investigation that systematically explored every layer of Filecoin's Groth16 proof generation pipeline—and then synthesized that knowledge into a complete architecture for a new piece of infrastructure: a pipelined, GPU-resident SNARK proving daemon called cuzk. The culmination of this work is the cuzk-project.md document, a 17-section architecture blueprint that lays out a phased implementation plan spanning 18 weeks and promising cumulative throughput improvements from 1.3x to 10x+.

This article synthesizes the entire segment's work, tracing the arc from investigation to design. It examines how the assistant approached the problem, what discoveries shaped the architecture, the key design decisions encoded in cuzk-project.md, and the broader significance of treating SNARK proof generation as a persistent service rather than a stateless function call.

The Starting Point: Seven Proposals and a Pivotal Ask

The segment begins with the assistant already deep in the problem space. Seven prior documents existed: a comprehensive background reference mapping the full SUPRASEAL_C2 call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, five optimization proposals targeting different layers of the pipeline, and a total impact assessment synthesizing them into a unified roadmap. The proposals covered Sequential Partition Synthesis (reducing peak memory from ~200 GiB to ~64 GiB), a Persistent Prover Daemon (eliminating per-proof SRS loading), Cross-Sector Batching (2-3x throughput per GPU), 18 compute-level micro-optimizations, and a Pre-Compiled Constraint Evaluator (PCE) exploiting the deterministic structure of Filecoin circuits.

The user's ask—captured in [msg 67]—was deceptively simple: "look at all proposals, plan a pipelined snark daemon which accepts a pipeline (some rpc) of PoRep/SnapDeals/PoSt snarks (also on various sector sizes), schedules the work, and outputs results." The user also provided specific guidance: "draw inspiration from how inference engines manage models/memory, build as a proving engine," and "Plan a roadmap to first build a basic daemon with minimal changes in upstream libraries, as a scaffold to later iterate on with optimization proposals."

But this simple request landed on a foundation of extraordinary depth. The assistant had already mapped the complete call chain, characterized computational hotpaths at the instruction level, and identified that the SRS (Structured Reference String) parameters consumed ~47 GiB of pinned host memory for PoRep proofs. The challenge was not understanding the system—it was designing a new architecture that could transform it.

The Investigation: Systematic Exploration of Every Layer

What followed was not a rush to write. The assistant spent messages 68-76 conducting an exhaustive investigation that demonstrates a rigorous engineering methodology. Rather than designing in a vacuum, the assistant systematically explored every layer of the existing system to understand the constraints and opportunities.

Exploring the ffiselect Child Process Model

The assistant first examined how Curio currently manages GPU proving. The existing architecture, traced through lib/ffiselect/ffiselect.go, spawns a child process for each GPU operation, communicates via JSON-RPC over stdin/stdout, and tears down the process after each proof. This model means every proof starts from scratch: SRS parameters are loaded from disk (30-90 seconds), GPU context is initialized, and all state is discarded when the process exits. The implications are profound: the GPU sits idle during CPU synthesis, SRS loading is repeated for every proof, and there is no opportunity for batching or pipelining across proofs.

This discovery alone justified the entire cuzk concept. If a persistent daemon could keep SRS resident and pipeline synthesis with GPU compute, the throughput gains would be immediate and substantial.

Mapping SRS/Parameter Loading Paths

The assistant traced how different proof types load their parameters, using subagent tasks to explore the filecoin-proofs crate chain. This revealed a striking asymmetry: PoRep and SnapDeals require enormous ~30-47 GiB SRS files, while PoSt SRS files are tiny (~200 MB - 2 GiB). This asymmetry would become a central driver of the SRS memory management strategy—on a small machine, you can keep all PoSt SRS hot simultaneously while swapping between PoRep and SnapDeals.

The assistant also discovered that the existing filecoin-proofs crate uses a lazy_static HashMap called GROTH_PARAM_MEMORY_CACHE to cache loaded parameters. This was a critical finding: it meant that Phase 0 of cuzk could achieve SRS residency without modifying any upstream code, simply by pre-populating this cache at daemon startup.

Studying the Supraseal C2 API Surface

The assistant examined the five extern "C" functions forming the FFI boundary between Rust and C++/CUDA, reading extern/supra_seal/c2/src/lib.rs and the CUDA source files. Key findings included the max_num_circuits parameter limiting batch size, the SRS cache implementation using cudaHostRegister, and the global mutex serializing generate_groth16_proofs_c() calls. The max_num_circuits discovery was particularly important: it suggested that the existing code could handle multiple circuits in a single GPU invocation, but the limit was conservatively set. Bumping this limit would enable cross-sector batching in Phase 3.

Investigating Bellperson Internals

The assistant traced through bellperson's create_proof_batch_priority_inner function in bellperson-0.26.0/src/groth16/prover/supraseal.rs, which combines synthesis and proving into a single monolithic call. This design prevents pipelining—the GPU cannot start computing until synthesis is complete. The assistant identified that splitting this function into separate synthesize_circuits_batch() and prove_from_assignments() phases would be a critical enabler for pipeline parallelism. This split became the centerpiece of Phase 2.

Studying GPU Inference Engine Architectures

Perhaps the most conceptually important investigation was the study of GPU inference engine architectures. The assistant examined vLLM, Triton Inference Server, and TensorRT-LLM, drawing direct analogies between inference serving and proof generation. The mapping is structurally precise:

• Model weights → SRS parameters: Both are large, immutable data structures loaded into GPU-accessible memory, reused across many computation requests.
• Inference requests → Proof jobs: Both have variable latency, different resource profiles, and arrive asynchronously.
• KV cache → Witness vectors: Both are per-request intermediate representations that dominate working memory.
• Prefill vs. decode → Synthesis vs. GPU compute: Both have a producer-consumer relationship where phases can be overlapped.
• Model serving → Circuit-type serving: Different proof types (PoRep, WindowPoSt, WinningPoSt) have different SRS requirements, analogous to serving different models. This analogy unlocked an entire design vocabulary borrowed from production ML serving systems. The assistant could now think about SRS memory management in terms of model loading/unloading, proof scheduling in terms of request batching, and GPU utilization in terms of continuous batching. The vLLM sleep mode pattern—where models are kept in a warm state and can be quickly re-pinned to GPU memory—directly inspired cuzk's three-tier SRS hierarchy.

Verifying Golden Test Data

The assistant conducted hands-on reconnaissance of the test environment. Listing /data/32gbench/ revealed C1 output files (c1.json, c1-8p.json, c1-single.json), cache directories with 34 GiB data layer files, and metadata files. The assistant decoded pc1out.txt to confirm it contained a StackedDrg32GiBV1_1 proof registration, verifying the data was for 32 GiB sectors. Running --help on every relevant lotus-bench subcommand built a complete map of how vanilla proofs are generated for each proof type.

Discovering Parameter Gaps

A critical discovery came when listing /var/tmp/filecoin-proof-parameters/: only the small-sector (8-0-0) variants were present, not the full 32 GiB (8-8-0) params. The 47 GiB PoRep SRS was not yet on the machine. This finding directly informed the testing strategy—the assistant knew that curio fetch-params 32GiB would need to be run before full-scale testing could begin. The assistant traced the fetch-params command in cmd/curio/main.go to understand how parameters are downloaded.

The Synthesis: Designing the cuzk Architecture

With the investigation complete, the assistant synthesized everything into the cuzk-project.md document ([msg 77]). The architecture that emerged is remarkable for its coherence, its pragmatic incrementalism, and its intellectual honesty.

The Core Insight: SNARK Proving as a Service

The fundamental shift in the cuzk architecture is treating proof generation as a persistent service rather than a stateless function call. The existing architecture calls seal_commit_phase2() as a blocking function—load SRS, synthesize witnesses, compute proof, return result, free everything. Each call is independent, stateless, and starts from scratch. cuzk transforms this into a persistent service with stateful resource management, request scheduling, and pipeline parallelism.

This shift is analogous to the evolution of machine learning inference:

• Phase 1: Load model, run inference, free model (per-request)
• Phase 2: Keep model loaded, run multiple inferences (persistent)
• Phase 3: Schedule requests, batch for efficiency, manage memory (service) Filecoin proof generation was in Phase 1. cuzk proposes jumping to Phase 3.

The Architecture Components

The architecture diagram in cuzk-project.md shows a layered system: a gRPC server (using tonic) receives proof requests from Curio, a scheduler manages priority queues and batch accumulation, GPU workers execute proofs, and a global SRS memory manager tracks pinned memory across all GPUs.

The gRPC API defines seven methods: SubmitProof, AwaitProof, Prove, Cancel, GetStatus, GetMetrics, PreloadSRS, and EvictSRS. The Submit/Await split is the most important design decision—it allows Curio to submit multiple proofs asynchronously, then await them independently, enabling the scheduler to reorder and batch without blocking the submitter. The PreloadSRS and EvictSRS methods give Curio explicit control over SRS lifecycle, allowing it to warm up parameters ahead of known deadlines.

The SRS memory manager introduces a three-tier hierarchy:

• Hot: CUDA pinned host RAM, immediate access for GPU DMA
• Warm: mmap'd file (page cache), ~2-10s to re-pin via cudaHostRegister
• Cold: Disk only, 30-90s for full deserialization The eviction policy is carefully designed: never evict an SRS with in-flight proofs, evict LRU from hot→warm (unpin, keep mmap), and if warm budget is exceeded, evict warm→cold (munmap). The scheduler has three priority levels (CRITICAL for WinningPoSt, HIGH for WindowPoSt, NORMAL for PoRep/SnapDeals), a batch collector that accumulates same-circuit-type jobs, and GPU affinity tracking to route proofs to GPUs that already have the right SRS loaded.

The Phased Roadmap: Pragmatic Incrementalism

The 18-week roadmap across 6 phases is a masterstroke of engineering pragmatism:

Phase 0 (Weeks 1-3): Scaffold with zero upstream modifications. The daemon pre-populates the existing GROTH_PARAM_MEMORY_CACHE at startup, keeping SRS resident across proofs. This alone delivers ~25% throughput improvement by eliminating 30-90s of per-proof SRS loading. No changes to Curio or filecoin-proofs are needed.

Phase 1 (Weeks 3-5): Multi-type scheduling with priority queues, SRS warm tier, GPU affinity tracking, and support for all proof types. WinningPoSt never misses a deadline due to PoRep hogging the GPU.

Phase 2 (Weeks 5-8): Pipelining by splitting bellperson's synthesis and prove functions. GPU utilization increases from ~40% to ~70%, yielding ~1.5x throughput over Phase 0.

Phase 3 (Weeks 8-11): Cross-sector batching with batch sizes up to 3, yielding 2-3x throughput per GPU.

Phase 4 (Weeks 11-14): Compute optimizations—SmallVec for LC Indexer, pre-sized vectors, pinned a/b/c memory, parallelized B_G2 MSMs, improved batch_addition occupancy. Combined: 30-40% faster per-proof.

Phase 5 (Weeks 14-18): Pre-Compiled Constraint Evaluator (PCE)—the most ambitious optimization, eliminating circuit synthesis overhead by pre-compiling R1CS constraint matrices. 3-5x synthesis speedup.

The cumulative throughput table shows the progression: 1.3x after Phase 0, 1.8x after Phase 2, 4-5x after Phase 3, 6-7x after Phase 4, and 10x+ after Phase 5. Each phase has concrete deliverables and stopping points, meaning the project can be paused at any phase and still deliver value.

The Testing Utility: cuzk-bench

The user specifically requested a testing utility for pre-Curio phases ([msg 78]). The assistant designed cuzk-bench with concrete commands for single proof testing, batch testing, stress testing, and vanilla proof generation. The utility uses the existing golden data in /data/32gbench/ and leverages lotus-bench simple commands to generate vanilla proofs for PoSt and SnapDeals. The assistant verified the exact command syntax for each proof type by running --help on the relevant lotus-bench subcommands.

Key Design Decisions and Their Rationale

Several decisions in the architecture deserve close examination.

Why Rust? The assistant chose Rust with tokio/tonic for the core implementation. This is a deliberate departure from Curio's Go codebase. The reasoning: direct access to filecoin-proofs without CGo FFI overhead, better memory control for the SRS manager, and a stronger concurrency model for the scheduler. The trade-off is integration complexity, which the three deployment modes (exec-into, spawn-as-child, external) address.

Why gRPC over unix socket? At ~50 MB per PoRep C1 output, streaming over gRPC on a unix socket takes only ~5ms—well within acceptable latency. This avoids the complexity of shared memory management while enabling the external deployment mode where cuzk runs as a separate process.

Why Phase 0 needs zero upstream modifications? This is strategically critical. By simply pre-populating the existing GROTH_PARAM_MEMORY_CACHE environment variable (which the proof library already checks), Phase 0 keeps SRS parameters resident across proof jobs without changing a single line of Curio or filecoin-proofs code. This means the first deployment can happen immediately, building confidence before more invasive changes.

Why inference engine analogies? The analogy is structurally precise, not superficial. It allows the assistant to import proven solutions from the ML serving world—memory pooling, continuous batching, priority scheduling—into the SNARK proving domain. This cross-domain pattern transfer is a hallmark of good systems design.

Why the Submit/Await split? This decouples submission from result collection, enabling the scheduler to reorder and batch proofs without blocking the submitter. Curio can submit multiple proofs, then await them in any order, or await all at once. This is critical for the batch accumulation strategy in Phase 3.

The Broader Significance

The cuzk architecture represents a new category of infrastructure for the Filecoin ecosystem. It transforms proof generation from a function call into a service, with all the attendant capabilities: persistent resource management, request scheduling, pipeline parallelism, and operational metrics.

The implications extend beyond the immediate project. The architecture demonstrates that SNARK proving systems can benefit from the same patterns that made GPU inference engines successful—patterns that emerged from years of production ML serving experience. This cross-pollination between cryptographic proving and ML inference is itself a significant insight.

The phased approach also models a template for ambitious infrastructure projects: start with immediate value (Phase 0's zero-modification SRS residency), build confidence through incremental deployment, and only then tackle the most aggressive optimizations. This pragmatic incrementalism contrasts with the all-or-nothing approach that often sinks ambitious infrastructure projects.

The investigation methodology itself is worth studying. The assistant did not rush to design. Instead, it systematically explored every layer of the existing system, from the Go task layer through the CUDA kernel internals. It studied the ffiselect child process model, the SRS loading paths, the supraseal C2 API surface, the bellperson prover internals, and the GPU inference engine architectures. It verified test data, explored lotus-bench commands, and discovered parameter gaps. Every finding directly shaped the architecture. There is no substitute for doing the work of understanding the system before designing its replacement.

Conclusion

The work captured in this segment is a masterclass in systems investigation and design. The assistant began with seven prior optimization proposals, conducted an exhaustive investigation spanning GPU kernel characteristics, SRS loading paths, inference engine architectures, and test data layouts, and synthesized everything into a coherent architecture for a pipelined SNARK proving daemon.

The cuzk-project.md document that emerged is not merely a plan—it is a blueprint for transforming how Filecoin proofs are generated. By treating proof generation as a persistent service rather than a stateless function call, by drawing on proven patterns from GPU inference engines, and by designing for incremental deployment with zero-risk initial phases, the cuzk architecture charts a path from the current state of per-proof child processes to a future of continuous, pipelined, GPU-resident proving.

Whether or not the full 18-week roadmap is implemented, the knowledge captured in this segment—the architecture, the design decisions, the open questions, the phased plan—will inform Filecoin proof generation infrastructure for years to come.## Deep Dive: The Investigation Messages

The assistant's investigation unfolded across multiple subagent tasks, each exploring a different layer of the system. Understanding the sequence of these investigations reveals the method behind the architecture.

Message 68: Reading All Seven Proposals

The assistant began by reading all seven existing documents in parallel, using a subagent task to extract architectural decisions, implementation details, and any daemon/service/pipeline concepts already present. This established the baseline: the assistant knew the full call chain, the memory budget breakdown, the circuit characteristics, and the five optimization proposals. The key insight carried forward was that SRS loading was the single largest per-proof overhead (30-90s), and that the existing GROTH_PARAM_MEMORY_CACHE mechanism could be exploited for zero-code-change SRS residency.

Message 69: Exploring the ffiselect Model and Supraseal API

The assistant then explored how Curio currently manages GPU proving, discovering the child process model where each proof spawns a new process. This was the fundamental inefficiency that cuzk would eliminate. Simultaneously, the assistant studied the supraseal C2 C++/CUDA API surface, identifying the five extern "C" functions and the critical max_num_circuits parameter. The discovery that generate_groth16_proofs_c() is globally serialized by a static mutex was particularly important—it meant that even with multiple GPUs, the current code serializes all GPU work at the FFI boundary.

Message 70: Bellperson Internals and Proof Type Differences

The assistant traced through bellperson's supraseal prover, discovering the monolithic create_proof_batch_priority_inner function that combines synthesis and proving. This function would need to be split for pipelining. The assistant also explored the circuit sizes for non-PoRep proof types, discovering the dramatic asymmetry in SRS sizes between PoRep (~47 GiB) and PoSt (~200 MB - 2 GiB).

Message 71: Clarifying Questions

Before proceeding to the architecture, the assistant asked clarifying questions about RPC transport, data transfer, deployment targets, and other design decisions. The user's answers—gRPC, inline C1 data (~50 MB), both small and large machines as targets—directly shaped the architecture. This moment of validation before design is a model of engineering discipline.

Messages 72-76: Ground Truth Verification

The assistant then conducted hands-on verification of critical facts. It checked the actual size of c1.json (50 MB, not "a few MB" as the user suggested, but also not GiB as the proposals assumed). It explored the supraseal C2 directory structure. It traced the SRS parameter construction path in bellperson. Each verification either confirmed an assumption or corrected it, and the corrected facts were incorporated into the architecture.

Messages 79-89: Test Data Reconnaissance

After drafting the initial architecture, the assistant received additional instructions from the user ([msg 78]) about test data locations and the testing utility. The assistant then conducted an extensive reconnaissance of the test environment: listing /data/32gbench/ contents, decoding metadata files, running lotus-bench simple --help for every subcommand, checking the parameter directory, and tracing the curio fetch-params command. This reconnaissance directly informed the cuzk-bench design and the testing strategy in the final document.

The cuzk-bench Testing Utility

One of the most practical outcomes of the investigation is the cuzk-bench testing utility design. The user specifically requested a tool that could be used for pre-Curio testing with easier setup. The assistant designed cuzk-bench with five commands:

• cuzk-bench single: Prove a single proof from a golden C1 file, with timing and memory tracking
• cuzk-bench batch: Submit multiple proofs in sequence or parallel, testing batch accumulation
• cuzk-bench stress: Continuous proof submission with configurable rate and concurrency
• cuzk-bench gen-vanilla: Generate vanilla proofs using lotus-bench simple commands for PoSt and SnapDeals
• cuzk-bench status and cuzk-bench preload: Interact with the daemon for monitoring and SRS management The assistant verified the exact lotus-bench commands needed for each proof type:
• PoRep C2: lotus-bench simple commit2 c1.json
• WindowPoSt: lotus-bench simple window-post sealed cache commR sectorNum
• WinningPoSt: lotus-bench simple winning-post ...
• SnapDeals: lotus-bench simple provereplicaupdate1 and provereplicaupdate2 These verified commands became the foundation of the gen-vanilla subcommand, ensuring that the testing utility can generate fresh vanilla proofs for any proof type without relying solely on golden data.

Deployment Modes and Curio Integration

The architecture defines three deployment modes for integrating cuzk with Curio:

Mode A — exec into daemon: Curio binary includes cuzk-ffi linked in. On startup, if a --prove-daemon flag is set (or detected via os.Args[0] like the current curio ffi pattern), it exec's into the cuzk daemon mode. This is the tightest integration, with zero additional process overhead.

Mode B — spawn as child: Curio runs curio prove-daemon as a subprocess (like the current ffiselect but persistent). The parent connects over gRPC unix socket. This mode requires minimal changes to Curio's process management.

Mode C — external: Operator runs cuzk-daemon independently. Curio is configured with the socket path. This mode enables the proofshare marketplace scenario where proving is a separate service.

All three modes use the same gRPC protocol—the only difference is process lifecycle. This design ensures that the daemon can be deployed incrementally, starting with Mode C for testing and evolving to Mode A for production.

Open Questions and Future Directions

The architecture document honestly acknowledges several open questions that will need to be resolved during implementation:

1. SnapDeals partition count: SnapDeals has 16 partitions vs PoRep's 10. Does the current supraseal code handle num_circuits > 10? The max_num_circuits parameter in groth16_srs.cuh needs verification.
2. Non-supraseal path for PoSt: In default Curio builds, PoSt uses bellperson's native prover (not supraseal). Should cuzk support both prover backends, or require cuda-supraseal for all proof types?
3. SnarkPack aggregation: Should cuzk also handle proof aggregation (CPU-only, no GPU)? The architecture supports it as a separate "proof type" with no GPU requirement, but this is deferred.
4. Remote proving: The gRPC API supports TCP listeners for remote proving (proofshare marketplace). Should this be in scope for Phase 0, or deferred to later phases? These open questions are not weaknesses—they are honest acknowledgments of the unknowns that will be resolved through implementation experience. The architecture provides enough structure to begin building while remaining flexible enough to accommodate the answers.

The Method: A Template for Systems Design

Beyond the specific architecture, the investigation methodology itself is worth studying. The assistant's approach demonstrates several principles of good systems design:

1. Read everything first: Before designing, the assistant read all seven existing documents, ensuring no prior insight was missed.
2. Explore every layer: The assistant traced the call chain from Go through CGO through Rust FFI through C++/CUDA, understanding the full stack before proposing changes.
3. Verify assumptions with data: When the user said C1 output was "a few MB," the assistant checked—and found it was 50 MB. When the proposals mentioned GiB-scale C1 data, the assistant corrected that too. Every assumption was verified.
4. Study analogous systems: The assistant studied GPU inference engines not because they are similar in function, but because they solve the same class of problems: managing large GPU-resident state, scheduling heterogeneous workloads, and maximizing throughput through batching and pipelining.
5. Design for incremental deployment: The phased roadmap ensures that value is delivered at every step, with no single phase requiring a "big bang" deployment.
6. Acknowledge unknowns: The open questions section honestly identifies what is not yet known, providing a roadmap for investigation during implementation. This methodology is applicable far beyond Filecoin proof generation. It is a template for any ambitious infrastructure project.## References [1] "From Investigation to Architecture: The Design of cuzk, a Pipelined SNARK Proving Daemon" — The chunk article for this segment, providing the primary analysis of the cuzk architecture design. [2] Message 66 — The comprehensive summary of the SUPRASEAL_C2 investigation that laid the groundwork for all subsequent optimization proposals. [3] Message 67 — The user's request that triggered the cuzk architecture design: "look at all proposals, plan a pipelined snark daemon." [4] Message 68 — The assistant's initial research phase, reading all seven proposal documents and exploring the ffiselect child process model. [5] Message 69 — Exploration of the supraseal C2 C++/CUDA API surface and the bellperson supraseal prover internals. [6] Message 70 — Investigation of SnapDeals and PoSt circuit sizes and proof parameters. [7] Message 71 — Clarifying questions about RPC transport, data transfer, and deployment targets. [8] Message 72 — Verification of C1 output size (50 MB, not GiB). [9] Message 77 — The core architecture document: cuzk-project.md with full gRPC API, SRS memory manager, scheduler design, and phased roadmap. [10] Message 78 — User instructions for test data locations and the cuzk-bench testing utility. [11] Messages 79-89 — Ground truth reconnaissance of test data in /data/32gbench/, lotus-bench commands, and parameter directory contents. [12] Message 93 — Summary of the completed cuzk-project.md document with its 17 sections. [13] Segment 0 — Deep-dive investigation into SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep. [14] Segment 1 — Compute-level investigation identifying 18 specific micro-optimizations across GPU kernels, CPU synthesis, and memory transfers. [15] Segment 2 — Investigation of constraint-shape exploitation for C2 proof generation, designing Proposal 5 with Pre-Compiled Constraint Evaluator.