From Investigation to Architecture: The Design of cuzk, a Pipelined SNARK Proving Daemon

Introduction

In the span of a single extended coding session, an extraordinary act of systems architecture unfolded. The assistant, building on months of prior analysis across seven optimization proposals, conducted an exhaustive investigation that systematically explored every layer of Filecoin's Groth16 proof generation pipeline—and then synthesized that knowledge into a complete architecture for a new piece of infrastructure: a pipelined, GPU-resident SNARK proving daemon called cuzk. The culmination of this work is the cuzk-project.md document, a 17-section architecture blueprint that lays out a phased implementation plan spanning 18 weeks and promising cumulative throughput improvements from 1.3x to 10x+.

This article synthesizes the entire chunk's work, tracing the arc from investigation to design. It examines how the assistant approached the problem, what discoveries shaped the architecture, the key design decisions encoded in cuzk-project.md, and the broader significance of treating SNARK proof generation as a persistent service rather than a stateless function call.

The Starting Point: Seven Proposals and a Pivotal Ask

The chunk begins with the assistant already deep in the problem space. Seven prior documents existed: a comprehensive background reference mapping the full SUPRASEAL_C2 call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, five optimization proposals targeting different layers of the pipeline, and a total impact assessment synthesizing them into a unified roadmap. The proposals covered Sequential Partition Synthesis (reducing peak memory from ~200 GiB to ~64 GiB), a Persistent Prover Daemon (eliminating per-proof SRS loading), Cross-Sector Batching (2-3x throughput per GPU), 18 compute-level micro-optimizations, and a Pre-Compiled Constraint Evaluator (PCE) exploiting the deterministic structure of Filecoin circuits.

The user's ask—captured in [msg 67] and analyzed in [2]—was deceptively simple: "look at all proposals, plan a pipelined snark daemon which accepts a pipeline (some rpc) of PoRep/SnapDeals/PoSt snarks (also on various sector sizes), schedules the work, and outputs results." But this simple request landed on a foundation of extraordinary depth. The assistant had already mapped the complete call chain, characterized computational hotpaths at the instruction level, and identified that the SRS (Structured Reference String) parameters consumed ~47 GiB of pinned host memory for PoRep proofs.

The Investigation: Systematic Exploration of Every Layer

What followed was not a rush to write. The assistant spent messages 68-76 conducting an exhaustive investigation that demonstrates a rigorous engineering methodology. Rather than designing in a vacuum, the assistant systematically explored every layer of the existing system to understand the constraints and opportunities.

Exploring the ffiselect Child Process Model

The assistant first examined how Curio currently manages GPU proving. The existing architecture spawns a child process for each GPU operation, communicates via JSON-RPC over stdin/stdout, and tears down the process after each proof. This model means every proof starts from scratch: SRS parameters are loaded from disk (30-90 seconds), GPU context is initialized, and all state is discarded when the process exits. The implications are profound: the GPU sits idle during CPU synthesis, SRS loading is repeated for every proof, and there is no opportunity for batching or pipelining across proofs.

Mapping SRS/Parameter Loading Paths

The assistant traced how different proof types load their parameters. This revealed a striking asymmetry: PoRep and SnapDeals require enormous ~30-47 GiB SRS files, while PoSt SRS files are tiny (~200 MB - 2 GiB). This asymmetry would become a central driver of the SRS memory management strategy. The assistant also discovered that the existing filecoin-proofs crate uses a lazy_static HashMap called GROTH_PARAM_MEMORY_CACHE to cache loaded parameters—a mechanism that could be exploited by pre-populating it at daemon startup.

Studying the Supraseal C2 API Surface

The assistant examined the five extern "C" functions forming the FFI boundary between Rust and C++/CUDA. Key findings included the max_num_circuits parameter limiting batch size, the SRS cache implementation using cudaHostRegister, and the global mutex serializing generate_groth16_proofs_c() calls. The max_num_circuits discovery was particularly important: it suggested that the existing code could handle multiple circuits in a single GPU invocation, but the limit was conservatively set.

Investigating Bellperson Internals

The assistant traced through bellperson's create_proof_batch_priority_inner function, which combines synthesis and proving into a single call. This monolithic design prevents pipelining—the GPU cannot start computing until synthesis is complete. The assistant identified that splitting this function into separate synthesis and prove phases would be a critical enabler for pipeline parallelism.

Studying GPU Inference Engine Architectures

Perhaps the most conceptually important investigation was the study of GPU inference engine architectures. The assistant examined vLLM, Triton Inference Server, and TensorRT-LLM, drawing direct analogies between inference serving and proof generation. The mapping is structurally precise:

• Model weights → SRS parameters: Both are large, immutable data structures loaded into GPU-accessible memory, reused across many computation requests.
• Inference requests → Proof jobs: Both have variable latency, different resource profiles, and arrive asynchronously.
• KV cache → Witness vectors: Both are per-request intermediate representations that dominate working memory.
• Prefill vs. decode → Synthesis vs. GPU compute: Both have a producer-consumer relationship where phases can be overlapped.
• Model serving → Circuit-type serving: Different proof types (PoRep, WindowPoSt, WinningPoSt) have different SRS requirements, analogous to serving different models. This analogy unlocked an entire design vocabulary borrowed from production ML serving systems. The assistant could now think about SRS memory management in terms of model loading/unloading, proof scheduling in terms of request batching, and GPU utilization in terms of continuous batching.

Verifying Golden Test Data

The assistant conducted hands-on reconnaissance of the test environment. Listing /data/32gbench/ revealed C1 output files (c1.json, c1-8p.json, c1-single.json), cache directories with 34 GiB data layer files, and metadata files. The assistant decoded pc1out.txt to confirm it contained a StackedDrg32GiBV1_1 proof registration, verifying the data was for 32 GiB sectors. Running --help on every relevant lotus-bench subcommand built a complete map of how vanilla proofs are generated for each proof type.

Discovering Parameter Gaps

A critical discovery came when listing /var/tmp/filecoin-proof-parameters/: only the small-sector (8-0-0) variants were present, not the full 32 GiB (8-8-0) params. The 47 GiB PoRep SRS was not yet on the machine. This finding directly informed the testing strategy—the assistant knew that curio fetch-params 32GiB would need to be run before full-scale testing could begin.

The Synthesis: Designing the cuzk Architecture

With the investigation complete, the assistant synthesized everything into the cuzk-project.md document ([msg 77]), which is analyzed in depth in [12]. The architecture that emerged is remarkable for its coherence, its pragmatic incrementalism, and its intellectual honesty.

The Core Insight: SNARK Proving as a Service

The fundamental shift in the cuzk architecture is treating proof generation as a persistent service rather than a stateless function call. The existing architecture calls seal_commit_phase2() as a blocking function—load SRS, synthesize witnesses, compute proof, return result, free everything. Each call is independent, stateless, and starts from scratch. cuzk transforms this into a persistent service with stateful resource management, request scheduling, and pipeline parallelism.

This shift is analogous to the evolution of machine learning inference:

• Phase 1: Load model, run inference, free model (per-request)
• Phase 2: Keep model loaded, run multiple inferences (persistent)
• Phase 3: Schedule requests, batch for efficiency, manage memory (service) Filecoin proof generation was in Phase 1. cuzk proposes jumping to Phase 3.

The Architecture Components

The architecture diagram shows a layered system: a gRPC server (using tonic) receives proof requests from Curio, a scheduler manages priority queues and batch accumulation, GPU workers execute proofs, and a global SRS memory manager tracks pinned memory across all GPUs.

The gRPC API defines seven methods: SubmitProof, AwaitProof, Prove, Cancel, GetStatus, GetMetrics, PreloadSRS, and EvictSRS. The Submit/Await split is the most important design decision—it allows Curio to submit multiple proofs asynchronously, then await them independently, enabling the scheduler to reorder and batch without blocking the submitter.

The SRS memory manager introduces a three-tier hierarchy:

• Hot: CUDA pinned host RAM, immediate access for GPU DMA
• Warm: mmap'd file (page cache), ~2-10s to re-pin via cudaHostRegister
• Cold: Disk only, 30-90s for full deserialization The eviction policy is carefully designed: never evict an SRS with in-flight proofs, evict LRU from hot→warm (unpin, keep mmap), and if warm budget is exceeded, evict warm→cold (munmap). The scheduler has three priority levels (CRITICAL for WinningPoSt, HIGH for WindowPoSt, NORMAL for PoRep/SnapDeals), a batch collector that accumulates same-circuit-type jobs, and GPU affinity tracking to route proofs to GPUs that already have the right SRS loaded.

The Phased Roadmap: Pragmatic Incrementalism

The 18-week roadmap across 6 phases is a masterstroke of engineering pragmatism:

Phase 0 (Weeks 1-3): Scaffold with zero upstream modifications. The daemon pre-populates the existing GROTH_PARAM_MEMORY_CACHE at startup, keeping SRS resident across proofs. This alone delivers ~25% throughput improvement by eliminating 30-90s of per-proof SRS loading. No changes to Curio or filecoin-proofs are needed.

Phase 1 (Weeks 3-5): Multi-type scheduling with priority queues, SRS warm tier, GPU affinity tracking, and support for all proof types. WinningPoSt never misses a deadline due to PoRep hogging the GPU.

Phase 2 (Weeks 5-8): Pipelining by splitting bellperson's synthesis and prove functions. GPU utilization increases from ~40% to ~70%, yielding ~1.5x throughput over Phase 0.

Phase 3 (Weeks 8-11): Cross-sector batching with batch sizes up to 3, yielding 2-3x throughput per GPU.

Phase 4 (Weeks 11-14): Compute optimizations—SmallVec for LC Indexer, pre-sized vectors, pinned a/b/c memory, parallelized B_G2 MSMs, improved batch_addition occupancy. Combined: 30-40% faster per-proof.

Phase 5 (Weeks 14-18): Pre-Compiled Constraint Evaluator (PCE)—the most ambitious optimization, eliminating circuit synthesis overhead by pre-compiling R1CS constraint matrices. 3-5x synthesis speedup.

The cumulative throughput table shows the progression: 1.3x after Phase 0, 1.8x after Phase 2, 4-5x after Phase 3, 6-7x after Phase 4, and 10x+ after Phase 5. Each phase has concrete deliverables and stopping points, meaning the project can be paused at any phase and still deliver value.

Key Design Decisions and Their Rationale

Several decisions in the architecture deserve close examination.

Why Rust? The assistant chose Rust with tokio/tonic for the core implementation. This is a deliberate departure from Curio's Go codebase. The reasoning: direct access to filecoin-proofs without CGo FFI overhead, better memory control for the SRS manager, and a stronger concurrency model for the scheduler. The trade-off is integration complexity, which the three deployment modes (exec-into, spawn-as-child, external) address.

Why gRPC over unix socket? At ~50 MB per PoRep C1 output, streaming over gRPC on a unix socket takes only ~5ms—well within acceptable latency. This avoids the complexity of shared memory management while enabling the external deployment mode where cuzk runs as a separate process.

Why Phase 0 needs zero upstream modifications? This is strategically critical. By simply pre-populating the existing GROTH_PARAM_MEMORY_CACHE environment variable (which the proof library already checks), Phase 0 keeps SRS parameters resident across proof jobs without changing a single line of Curio or filecoin-proofs code. This means the first deployment can happen immediately, building confidence before more invasive changes.

Why inference engine analogies? The analogy is structurally precise, not superficial. It allows the assistant to import proven solutions from the ML serving world—memory pooling, continuous batching, priority scheduling—into the SNARK proving domain. This cross-domain pattern transfer is a hallmark of good systems design.

The Broader Significance

The cuzk architecture represents a new category of infrastructure for the Filecoin ecosystem. It transforms proof generation from a function call into a service, with all the attendant capabilities: persistent resource management, request scheduling, pipeline parallelism, and operational metrics.

The implications extend beyond the immediate project. The architecture demonstrates that SNARK proving systems can benefit from the same patterns that made GPU inference engines successful—patterns that emerged from years of production ML serving experience. This cross-pollination between cryptographic proving and ML inference is itself a significant insight.

The phased approach also models a template for ambitious infrastructure projects: start with immediate value (Phase 0's zero-modification SRS residency), build confidence through incremental deployment, and only then tackle the most aggressive optimizations. This pragmatic incrementalism contrasts with the all-or-nothing approach that often sinks ambitious infrastructure projects.

Conclusion

The work captured in this chunk is a masterclass in systems investigation and design. The assistant began with seven prior optimization proposals, conducted an exhaustive investigation spanning GPU kernel characteristics, SRS loading paths, inference engine architectures, and test data layouts, and synthesized everything into a coherent architecture for a pipelined SNARK proving daemon.

The cuzk-project.md document that emerged is not merely a plan—it is a blueprint for transforming how Filecoin proofs are generated. By treating proof generation as a persistent service rather than a stateless function call, by drawing on proven patterns from GPU inference engines, and by designing for incremental deployment with zero-risk initial phases, the cuzk architecture charts a path from the current state of per-proof child processes to a future of continuous, pipelined, GPU-resident proving.

The investigation documented in this chunk also demonstrates the value of thoroughness. Every discovery—from the asymmetry of SRS sizes to the existence of GROTH_PARAM_MEMORY_CACHE to the specific contents of /data/32gbench/—directly shaped the architecture. There is no substitute for doing the work of understanding the system before designing its replacement.

Whether or not the full 18-week roadmap is implemented, the knowledge captured in this chunk—the architecture, the design decisions, the open questions, the phased plan—will inform Filecoin proof generation infrastructure for years to come.## References

[1] "The Architecture of a Discovery: Mapping the SUPRASEAL_C2 Groth16 Pipeline for Filecoin Proof Generation" — Analyzes message 66, the comprehensive summary of the SUPRASEAL_C2 investigation that laid the groundwork for all subsequent optimization proposals.

[2] "The Pivotal Ask: From Optimization Proposals to a Unified Proving Engine" — Examines message 67, the user's request that triggered the cuzk architecture design.

[3] "The Research Phase of a Proving Engine: How Systematic Context-Gathering Shaped the cuzk SNARK Daemon" — Covers the research methodology in message 68.

[4] "The Pivot Point: From Research to Architecture in the cuzk Proving Daemon Design" — Analyzes message 69's transition from investigation to design.

[5] "The Gap in the Picture: How a Single Message of Intellectual Honesty Shaped the cuzk Proving Daemon" — Examines message 70's acknowledgment of knowledge gaps.

[6] "The Pivotal Question: How a Single Message Shaped the Architecture of a Pipelined SNARK Proving Daemon" — Analyzes message 71's clarifying questions.

[7] "The 50 MB Truth: How a Single Verification Shaped a Proving Daemon's Architecture" — Covers message 72's verification of C1 output size.

[8] "Ground Truth in the Proving Pipeline: How a 50 MB File Reshaped the cuzk Architecture" — Analyzes message 73's impact on design decisions.

[9] "Ground Truth Verification: Reading the Supraseal-C2 Crate During the cuzk Architecture Design" — Examines message 74's code reading.

[10] "The Final Piece of the Puzzle: Tracing SRS Parameter Construction in the cuzk Proving Daemon Investigation" — Analyzes message 75's SRS tracing.

[11] "The Pivot Point: From Investigation to Synthesis in Designing a Pipelined SNARK Proving Engine" — Covers message 76's transition.

[12] "The Architecture of a SNARK Proving Engine: Designing cuzk — A Pipelined, GPU-Resident Proof Generation Daemon for Filecoin" — The definitive analysis of message 77, the core architecture document.

[13] "The Architect's Blueprint: How a Single User Message Grounded the cuzk Proving Daemon in Reality" — Analyzes message 78's grounding instructions.

[14] "The Reconnaissance That Built a Proving Daemon: How One Message Grounded the cuzk Architecture in Reality" — Covers message 79's reconnaissance.

[15] "The Ground Truth of Test Data: A Methodical Inventory in Service of the cuzk Proving Daemon" — Analyzes message 80's test data inventory.

[16] "Probing the Testing Surface: How Three Bash Commands Shaped the cuzk Proving Daemon" — Covers message 81's lotus-bench exploration.

[17] "Mapping the Proof Type Surface: The Quiet Investigation Behind cuzk-project.md" — Analyzes message 82's proof type mapping.

[18] "The Quiet Reconnaissance: How Two ls Commands Shaped a Proving Daemon's Architecture" — Covers message 83's parameter directory listing.

[19] "The Hidden Infrastructure: How a Single Bash Command Shaped the cuzk Proving Daemon Architecture" — Analyzes message 84's supra_seal.cfg discovery.

[20] "The Moment of Synthesis: Discovering Parameter Gaps in Filecoin's Proof Pipeline" — Covers message 85's parameter gap discovery.

[21] "Verifying Golden Test Data for the cuzk SNARK Proving Daemon" — Analyzes message 86's test data verification.

[22] "The Moment a grep Command Unlocks Architecture: Tracing curio fetch-params for the cuzk Proving Daemon" — Covers message 87's grep exploration.

[23] "The Precision of Verification: A Single Grep in the Cuzk Project" — Analyzes message 88's verification.

[24] "The Moment of Verification: Reading Source Code to Ground Design Documentation" — Covers message 89's source code reading.

[25] "The Pivot Point: From Investigation to Architecture in the cuzk Proving Daemon" — Analyzes message 90's transition.

[26] "The Architecture Document That Took a Thousand Commands to Write" — Covers message 91's document creation.

[27] "The Quiet Signal of Completion: Analyzing a Status Message in the cuzk Proving Daemon Project" — Analyzes message 92's status update.

[28] "The Architecture Emerges: Designing cuzk, a Pipelined SNARK Proving Daemon" — Analyzes message 93's summary of the completed document.

[29] "The Silence That Speaks: Analyzing an Empty Message in a Technical Collaboration" — Examines message 94's empty response.