From Analysis to Architecture: Designing the cuzk SNARK Proving Daemon

Introduction

In the high-stakes world of Filecoin storage proofs, every second of proving time and every gigabyte of memory translates directly into operational cost. The SUPRASEAL_C2 Groth16 proof generation pipeline—responsible for producing the cryptographic proofs that underpin Filecoin's Proof-of-Replication (PoRep) protocol—had been thoroughly analyzed in a preceding investigation that mapped its full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels. That investigation had identified a staggering ~200 GiB peak memory footprint, nine structural bottlenecks, and five optimization proposals promising up to a 96% reduction in per-proof cost [3]. But analysis alone does not change hardware requirements. The next step was to transform these proposals into a concrete, implementable architecture.

This chunk of the coding session documents that transformation. In a dedicated subagent session, the assistant moved from knowledge acquisition to architectural synthesis, ultimately producing the cuzk-project.md document—a comprehensive blueprint for a pipelined SNARK proving daemon that would become the centerpiece of the optimization effort. This article traces that journey: how the assistant internalized seven dense technical documents, explored the existing system's every layer, drew inspiration from GPU inference engine architectures, and wove all these threads into a phased implementation plan spanning 18 weeks.

The Knowledge Foundation: Seven Documents Become One Mental Model

The subagent session began with a precise instruction from the user: read all seven prior documents—the background analysis, five optimization proposals, and total impact assessment—and return comprehensive summaries [1]. This was not a casual request. These documents represented weeks or months of investigation into the SUPRASEAL_C2 pipeline, and the user needed the assistant to internalize every detail before any design work could begin.

The assistant's response was immediate and strategic: "I'll read all 7 files in parallel to gather the complete information" [2]. This seemingly simple decision reveals a sophisticated understanding of the task. By loading all documents simultaneously into working context, the assistant enabled cross-referencing and synthesis that would be impossible with sequential reading. The background document's identification of the ~200 GiB peak memory problem could be immediately connected to Proposal 1's strategy of reducing it to ~52 GiB, Proposal 2's daemon-based SRS residency, and Proposal 3's batching model. The dependency graph—Proposal 1 is independent of Proposal 2, but Proposal 3 requires Proposal 2—emerged naturally from having all documents in context at once.

The comprehensive summary that followed [3] was a masterclass in technical synthesis. The assistant organized each proposal with consistent categories: goal, architecture, memory requirements, performance impact, implementation details, and dependencies. This consistency enabled direct comparison across proposals. The summary also introduced critical cross-document annotations—"Independent of P2 (daemon)" for Proposal 1, "Prerequisites: P2 (persistent daemon) required" for Proposal 3—that transformed seven isolated documents into a coherent dependency graph.

But the summary was only the beginning. The real work—designing a system that could actually implement these proposals—was still ahead.

Exploring the Existing System: Every Layer, Every Path

With the proposals internalized, the assistant turned to empirical investigation of the existing system. This phase was driven by a fundamental question: What exactly would need to change to implement these proposals? The answer required understanding every layer of the current architecture in intimate detail.

The ffiselect child process model was the first target. The assistant examined how Curio's Go task layer spawns a child process for each GPU proof invocation, how the ffiselect mechanism selects between CPU and GPU proving paths, and how the child process loads the ~47 GiB SRS (Structured Reference String) from disk on every invocation. This per-process SRS loading—taking 30-90 seconds and involving deserialization of ~500 million elliptic curve points—was the single most expensive per-proof overhead. The daemon model proposed in Proposal 2 would eliminate it entirely, but only if the assistant understood exactly how the child process model worked and where the hooks for a daemon replacement would need to be inserted.

SRS and parameter loading paths were traced across all four proof types: PoRep (32 GiB and 64 GiB sectors), SnapDeals, WindowPoSt, and WinningPoSt. Each proof type has different circuit sizes and SRS requirements. The assistant mapped the file paths, loading functions, and memory layouts for each. This information was critical for the cuzk architecture's SRS memory manager, which would need to support multiple SRS entries in a hot/warm/cold cache hierarchy.

The supraseal C2 C++/CUDA API surface was examined in detail. The assistant studied the generate_groth16_proofs_c() function signature, the num_circuits parameter that controls how many partition circuits are processed in parallel, and the internal phases: NTT, MSM, and proof assembly. Understanding this API surface was essential for designing the phased API proposed in Proposal 1, which would replace the monolithic function with separate ntt_h_phase() and msm_phase() calls.

Bellperson supraseal prover internals were explored to understand how Rust's bellperson library interfaces with the C++/CUDA code. The assistant traced the FFI boundary, the memory ownership model (who allocates, who frees), and the synchronization points. This was critical for understanding why the current architecture cannot free a/b/c vectors early—Rust holds them until C++ returns from the full proof generation call.

Drawing Inspiration from GPU Inference Engines

Perhaps the most intellectually adventurous part of the investigation was the study of GPU inference engine architectures. The assistant examined three major systems—vLLM, NVIDIA Triton Inference Server, and TensorRT-LLM—drawing direct analogies between LLM serving and SNARK proving.

The parallels are striking. In LLM serving, model weights (hundreds of gigabytes) must be loaded into GPU memory once and kept resident across many inference requests. In SNARK proving, the SRS (~47 GiB) must be loaded into GPU memory once and kept resident across many proof generations. In LLM serving, the KV cache is allocated per request and managed carefully to avoid memory fragmentation. In SNARK proving, witness vectors and auxiliary assignments are allocated per proof and must be managed with similar care. In LLM serving, continuous batching allows multiple requests to be processed in a single GPU invocation. In SNARK proving, cross-sector batching would allow multiple proofs to share a single GPU invocation.

The assistant extracted specific architectural patterns from these systems:

• vLLM's PagedAttention inspired the idea of managing witness vectors in fixed-size blocks, reducing fragmentation and enabling more efficient memory reuse across proofs.
• Triton's model management—with its concept of model versions, concurrent model execution, and request batching—provided a template for the cuzk daemon's SRS memory manager and scheduler.
• TensorRT-LLM's in-flight batching demonstrated how to accumulate requests from multiple clients into a single GPU batch, directly informing the BatchCollector design in Proposal 3. These analogies were not superficial. The assistant mapped specific components: model weights → SRS parameters, inference requests → proof jobs, KV cache → witness vectors, continuous batching → cross-sector batching, and model loading → SRS preloading. This mapping provided both conceptual vocabulary and concrete implementation patterns for the cuzk architecture.

Verifying the Test Infrastructure

Before any architecture could be finalized, the assistant needed to verify that the test infrastructure existed to validate it. The investigation examined the golden test data in /data/32gbench/, confirming that pre-computed C1 outputs and expected C2 proofs were available for testing. The assistant also explored lotus-bench simple commands for generating vanilla proofs for all proof types (PoRep, SnapDeals, WindowPoSt, WinningPoSt), establishing a baseline against which cuzk's performance could be measured.

This verification phase was crucial for the practical design of cuzk-bench, the testing utility that would accompany the daemon. The assistant designed concrete commands for single-proof testing, batch testing, and stress testing, all leveraging the existing golden data. This ensured that the cuzk project would not require new test infrastructure—it could validate against the same data that the current system uses.

The cuzk Architecture: A Complete Blueprint

The culmination of all this investigation was the cuzk-project.md document, written to the repository root. This document is not a summary of proposals—it is a complete architectural blueprint for a new system. Let us examine its key components.

gRPC API Surface

The daemon exposes a clean gRPC API with six methods:

• SubmitProof — Submit a proof request (C1 output, sector type, priority)
• AwaitProof — Block until a submitted proof completes
• Prove — Synchronous RPC combining Submit + Await
• Cancel — Cancel a pending proof
• GetStatus — Query daemon health, queue depth, GPU utilization
• PreloadSRS / EvictSRS — Explicit SRS cache management This API design reflects the lessons learned from GPU inference engines. PreloadSRS and EvictSRS are direct analogs of model loading/unloading in Triton. The separation of SubmitProof and AwaitProof enables asynchronous submission, which is essential for batching—the daemon can accumulate multiple submissions before triggering a GPU invocation.

Three-Tier SRS Memory Manager

The SRS memory manager is one of the most architecturally significant components. It implements a hot/warm/cold cache hierarchy:

• Hot: SRS currently loaded in GPU memory (pinned, ready for proving)
• Warm: SRS loaded in host pinned memory, can be promoted to hot on demand
• Cold: SRS on disk, must be deserialized before use The manager operates under an explicit memory budget, evicting cold entries when the budget is exceeded. This design supports multiple proof types (PoRep 32 GiB, PoRep 64 GiB, SnapDeals, WindowPoSt, WinningPoSt) without requiring all SRS entries to be resident simultaneously. The LRU eviction policy ensures that frequently used proof types stay hot. This three-tier design is directly inspired by vLLM's memory management, which uses a similar hierarchy for KV cache blocks. The assistant recognized that SRS parameters, like model weights, are too large to keep all variants resident, but too expensive to reload on every request. The cache hierarchy provides a middle ground.

Priority-Based Scheduler with Batch Accumulation

The scheduler maintains a priority queue of proof requests, with batch accumulation logic inspired by TensorRT-LLM's in-flight batching:

type BatchCollector struct {
    pending   []ProofRequest
    maxBatch  int
    maxWait   time.Duration
}

The collector accumulates requests until either maxBatch is reached or maxWait expires, then submits the batch to the GPU worker. This design balances latency (individual proofs wait at most maxWait) against throughput (larger batches amortize GPU kernel launch overhead).

The scheduler also tracks GPU affinity—which GPU has which SRS loaded—to minimize SRS transfers. A proof request for PoRep 32 GiB is routed to the GPU that already has the PoRep 32 GiB SRS in its hot cache, avoiding a costly SRS reload.

GPU Worker Pipeline: From Sequential to Pipelined

The GPU worker pipeline is designed to evolve through three phases:

Phase 0 (Sequential, Zero Upstream Changes): The daemon acts as a thin wrapper around the existing supraseal binary. It loads SRS once, then for each proof, invokes the existing proof generation code. This phase delivers immediate value—~25% throughput improvement from eliminating per-proof SRS loading—without any changes to the upstream Curio or bellperson code.

Phase 1 (Pipelined Synthesis): The daemon implements the Sequential Partition Synthesis from Proposal 1. Instead of invoking the monolithic proof generation function, it calls separate NTT and MSM phases, streaming partitions one at a time through the GPU. This reduces peak memory from ~200 GiB to ~64 GiB and enables GPU work to begin ~12s after proof submission instead of ~180s.

Phase 2+ (Fully Pipelined with Batching): The daemon adds cross-sector batching (Proposal 3), compute optimizations (Proposal 4), and the Pre-Compiled Constraint Evaluator (Proposal 5). Multiple proofs are accumulated and processed in a single GPU invocation, with per-proof time dropping to ~35s.

This phased approach is a deliberate architectural decision. Phase 0 delivers immediate ROI with minimal risk. Each subsequent phase builds on the previous one, and the daemon can be deployed at any phase. A storage provider who only deploys Phase 0 still gets a 25% throughput improvement. The full 10.3x improvement requires all phases, but the architecture does not mandate it.

The 18-Week Roadmap

The cuzk-project.md document includes a detailed implementation roadmap organized into six phases:

| Phase | Weeks | Focus | Cumulative Throughput | |-------|-------|-------|---------------------| | 0 | 1-2 | SRS residency, daemon scaffold | 1.3x | | 1 | 3-5 | Sequential partition synthesis | 2.1x | | 2 | 6-8 | Multi-type scheduling, batch accumulation | 3.0x | | 3 | 9-11 | GPU pipelining, NTT/MSM overlap | 5.0x | | 4 | 12-14 | Compute optimizations (Proposal 4) | 6.5x | | 5 | 15-18 | Pre-Compiled Constraint Evaluator (PCE) | 10.0x+ |

Each phase has clear deliverables, dependencies, and success criteria. The roadmap is designed to be interruptible—if resources are diverted after Phase 3, the system still delivers 5.0x improvement over baseline.

The Broader Vision: From Component to Ecosystem

The cuzk architecture is not just a technical design; it is an economic vision. The assistant's investigation of GPU inference engines revealed a deeper pattern: the most successful AI serving systems are not just software components but platforms that manage hardware resources efficiently across multiple tenants. The cuzk daemon applies this same philosophy to SNARK proving.

The two-tier marketplace envisioned in the total impact assessment—GPU machines running cuzk daemons producing proofs at high throughput, feeding CPU-based aggregators that use SnarkPack to compress multiple proofs into a single SNARK—represents a fundamental shift in how Filecoin proving infrastructure is deployed. Instead of each storage provider running their own proof generation on expensive 256 GiB machines, a shared pool of GPU-provisioned daemons can serve multiple providers, achieving economies of scale that drive per-proof cost down by 96%.

This vision is only possible because the cuzk architecture addresses the three fundamental bottlenecks: memory (Sequential Partition Synthesis reduces peak from 200 GiB to 64 GiB), SRS loading (Persistent Prover Daemon eliminates per-proof reloading), and throughput (Cross-Sector Batching multiplies proofs per GPU invocation). Each bottleneck required a different architectural innovation, and each innovation was informed by a different source—the memory analysis from the background document, the daemon pattern from GPU inference engines, the batching model from TensorRT-LLM.

Conclusion

The work in this chunk represents a complete arc from knowledge acquisition to architectural design. The assistant began by reading seven dense technical documents, synthesized them into a unified understanding of the SUPRASEAL_C2 pipeline's bottlenecks and optimization opportunities, explored every layer of the existing system to understand what would need to change, studied GPU inference engine architectures for design patterns, verified the test infrastructure, and finally produced the cuzk-project.md document—a comprehensive blueprint for a pipelined SNARK proving daemon.

What makes this work remarkable is not the individual insights—many of them were present in the source documents—but the synthesis. The assistant connected the memory accounting from the background document to the phased API design in Proposal 1, the daemon model from Proposal 2 to the inference engine patterns from vLLM and Triton, the batching from Proposal 3 to TensorRT-LLM's continuous batching, and all of these to the practical realities of the existing codebase and test infrastructure. The result is an architecture that is both ambitious (10.3x throughput improvement) and pragmatic (Phase 0 delivers value with zero upstream changes).

The cuzk daemon represents a new paradigm for Filecoin proof generation: continuous, memory-efficient, and designed for heterogeneous cloud rental markets where RAM cost dominates. It is the product of an investigation that moved from understanding what is broken to designing what should replace it—and that transition, captured in this single chunk of work, is the essence of architectural engineering.

References

[1] The Architect's Reading List: How a Single Message Launched a SNARK Proving Daemon — Analysis of the user's request to read all seven optimization documents, examining the knowledge transfer process and the assumptions encoded in the instruction.

[2] The Parallel Reading Strategy: A Pivotal Moment in the SUPRASEAL_C2 Optimization Investigation — Examination of the assistant's decision to read all seven files in parallel, revealing strategic thinking about information gathering and synthesis.

[3] The Architect's Blueprint: Deconstructing a 200-GiB Bottleneck in Groth16 Proof Generation — Deep analysis of the assistant's comprehensive summary of all seven documents, covering the background analysis, five optimization proposals, and total impact assessment with memory accounting, performance projections, and economic framework.