The Research Phase of a Proving Engine: How Systematic Context-Gathering Shaped the cuzk SNARK Daemon

In any significant engineering effort, the quality of the output is determined long before the first line of code is written. It is determined during the research phase—the hours spent reading existing code, tracing call chains, and understanding the constraints that define the problem space. Message [msg 68] from a deep-dive investigation into Filecoin's Groth16 proof generation pipeline captures exactly this moment: the transition from having a rich set of optimization proposals to synthesizing them into a coherent architectural plan. This message is not a design document, nor is it a code change. It is the preparatory research that makes the subsequent design possible—four parallel investigations launched simultaneously, each targeting a different dimension of the problem.

The Context: Seven Proposals, One Missing Architecture

To understand why message [msg 68] exists, we must understand what came before it. The preceding session (Segment 0) had produced seven detailed documents: a comprehensive background reference ([msg 66]), five optimization proposals covering everything from sequential partition synthesis to pre-compiled constraint evaluators, and a total impact assessment. These documents represented an enormous amount of analytical work—tracing the full call chain from Go through Rust FFI into CUDA kernels, accounting for ~200 GiB of peak memory usage, identifying 18 micro-optimizations, and proposing a 10x throughput improvement path.

But there was a gap. The proposals were independent—each one described an optimization in isolation, with its own timing estimates and implementation effort. What was missing was the container: the architectural framework that would hold these optimizations together, the deployment model that would make them practical, and the phased roadmap that would turn a collection of good ideas into a shippable product. The user recognized this gap and, in [msg 67], asked for something new: a pipelined SNARK proving daemon—a persistent, GPU-resident service that could accept proof requests over RPC, schedule work across heterogeneous hardware, manage SRS parameter memory like an inference engine manages model weights, and output results. The name "cuzk" was chosen, and the implementation directory was set to extern/cuzk.

Message [msg 68] is the assistant's response to that request. But it is not yet the design. It is the research phase—the systematic gathering of context that must precede any architectural decision.

Four Parallel Investigations

The message opens with a clear statement of intent: "I'll start by re-reading all the proposals and understanding the current architecture, then plan this daemon. Let me gather context in parallel." The assistant then launches four simultaneous research tasks, each with a distinct focus:

Task 1: Reading All Seven Proposal Documents

The first task re-reads every document produced in the prior session. This is not a casual skim—the task description asks for a "comprehensive summary of each, focusing on architectural decisions, implementation details, and any daemon/service/pipeline concepts mentioned." The assistant is specifically looking for prior thinking about the daemon concept, which had been hinted at in Proposal 2 (Persistent Prover Daemon) and Proposal 3 (Cross-Sector Batching) but never fully synthesized.

The result of this task is a structured summary that covers the background analysis (five-layer call chain, memory budget, circuit characteristics), all five optimization proposals, and the total impact assessment. Crucially, it surfaces the key finding that "R1CS matrices are identical for every proof"—a fact that will become central to the daemon's architecture, since it means the same SRS parameters can serve many proofs without reloading.

Task 2: Exploring the ffiselect Child Process Model

The second task dives into the existing deployment architecture. Curio currently uses a pattern where each proof runs in an isolated child process spawned by ffiselect, with GPU device ordinals pinned via CUDA_VISIBLE_DEVICES. This model has profound implications for the daemon design: it means the current system already has a concept of process-per-GPU, but it is ephemeral—each child process loads SRS from scratch, runs one proof, and exits.

The assistant needs to understand this model intimately because the daemon must replace it. The research uncovers the JSON-RPC client-server pattern, the GPU ordinal management, and the resource declaration system in tasks/seal/task_porep.go. This knowledge directly informs the daemon's process model: should it be a long-lived process that replaces the child entirely? Should it support the existing curio ffi exec pattern? The answers to these questions will shape the architecture.

Task 3: Exploring SRS/Params and Proof Types

The third task investigates the parameter landscape. The Groth16 proving system requires Structured Reference Strings (SRS)—large parameter files that encode the circuit structure. Different proof types (PoRep, SnapDeals, WindowPoSt, WinningPoSt) have different circuit sizes and therefore different SRS files. The research maps the SRS loading path from environment variables through async fetching to supraseal's mmap+CUDA pinned memory LRU cache, and documents the circuit sizes per proof type.

This task is essential because the daemon's core value proposition is SRS residency—keeping parameters loaded in GPU-accessible memory across multiple proofs. To design the SRS memory manager, the assistant needs to know exactly how big each SRS file is (PoRep 32GiB: ~47 GiB; WinningPoSt: ~200 MB), how they are loaded, and what the tradeoffs are for keeping multiple SRS files resident simultaneously.

Task 4: Exploring Inference Engine Architectures

The fourth task is the most creative. The user explicitly asked to "draw inspiration from how inference engines manage models/memory, build as a proving engine." The assistant searches the web for patterns from vLLM, TensorRT-LLM, Triton Inference Server, and llama.cpp server mode.

The research produces a structured analysis of inference engine patterns: tiered sleep/hibernate for model management, continuous batching for throughput, prefix caching for KV cache reuse, and explicit model load/unload APIs. The assistant then maps these concepts to the proving domain: model weights become SRS parameters, inference requests become proof requests, KV cache becomes witness vectors, and batch scheduling becomes cross-sector proof batching.

This cross-domain analogy is the intellectual core of the cuzk design. It transforms the problem from "how do we optimize a single proof" to "how do we build a continuous proving pipeline," mirroring the shift that vLLM brought to LLM inference.

The Thinking Process: Systematic and Parallel

What is most striking about message [msg 68] is the parallelism of the research. The assistant does not read the proposals, then explore ffiselect, then study SRS, then research inference engines in sequence. It launches all four tasks simultaneously, recognizing that they are independent and that the synthesis can only happen once all four perspectives are available.

This reveals a sophisticated meta-cognitive strategy. The assistant understands that the daemon design requires four distinct types of knowledge:

  1. What we already know (the proposals) — the optimizations we want to implement
  2. Where we are now (ffiselect) — the deployment model we want to replace
  3. What we need to manage (SRS/proof types) — the resources we must handle
  4. What we can learn from (inference engines) — the architectural patterns we can borrow By gathering all four in parallel, the assistant minimizes the time between receiving the request and producing the design. The subsequent message ([msg 77]) will synthesize these four threads into a comprehensive architecture document, but message [msg 68] is where the raw material is collected.

Assumptions and Their Consequences

The message reveals several assumptions that will shape the design. Some are explicit in the task descriptions; others are implicit in the choice of what to research.

Assumption 1: Inference engines are a valid analogy. The assistant invests significant effort in studying vLLM, Triton, and TensorRT-LLM, implicitly assuming that the patterns used for neural network inference transfer to zero-knowledge proof generation. This assumption turns out to be remarkably productive—the tiered memory management, continuous batching, and scheduling patterns map almost perfectly to the proving domain. But it is an assumption worth examining: inference engines handle models that are orders of magnitude smaller than a 47 GiB SRS file, and they operate on different hardware (tensor cores vs. CUDA cores for MSM/NTT). The analogy is powerful but not exact.

Assumption 2: SRS residency is the primary win. By focusing research on SRS loading paths and the ffiselect child process model, the assistant implicitly prioritizes the "keep SRS loaded" optimization over others. This is a sound judgment—eliminating 30-90 seconds of SRS loading per proof is an immediate 25% throughput gain with no code changes—but it shapes the architecture toward a long-lived daemon model rather than, say, a batch-oriented job system.

Assumption 3: All proof types should be handled by one daemon. The research covers PoRep, SnapDeals, WindowPoSt, and WinningPoSt, assuming a unified daemon that handles all of them. An alternative would be specialized daemons per proof type (a PoRep daemon, a PoSt daemon, etc.), which might simplify SRS management at the cost of operational complexity. The assistant's choice to unify reflects the user's request for "a pipelined snark daemon which accepts a pipeline ... of PoRep/SnapDeals/PoSt snarks."

Assumption 4: The daemon should be written in Rust. The assistant chooses Rust for the daemon implementation, with tokio for async runtime and tonic for gRPC. This is never explicitly debated in the message—it flows naturally from the fact that the existing supraseal C2 code is a Rust crate, and the FFI boundary is already Rust→C++/CUDA. Writing the daemon in Rust avoids adding another language boundary (Go→Rust→C++ becomes Go→Rust→C++, with the daemon in Rust). But it does mean the daemon cannot be directly embedded in Curio's Go process without a CGO bridge, which the assistant addresses by designing a cuzk-ffi crate with extern "C" functions.

Input Knowledge Required

To fully understand message [msg 68], a reader needs substantial context:

Output Knowledge Created

Message [msg 68] produces structured knowledge in four domains:

  1. A comprehensive summary of all prior proposals, organized by architectural relevance. This is not just a re-reading—it is a re-framing of the proposals through the lens of daemon architecture. The assistant is looking for "daemon/service/pipeline concepts" in documents that were written as standalone optimization analyses.
  2. A detailed map of the ffiselect child process model, including the JSON-RPC protocol, GPU ordinal management, and the exec-into-self pattern. This knowledge is essential for designing the daemon's process lifecycle and embedding strategy.
  3. A complete catalog of SRS/parameter files per proof type, including their sizes, loading paths, and cache mechanisms. This becomes the foundation for the SRS memory manager design.
  4. A structured analysis of GPU inference engine patterns, mapped to proving engine equivalents. This cross-domain analogy is the most creative output of the message—it provides the conceptual framework for the entire cuzk architecture.

The Bridge to Design

Message [msg 68] ends with the research complete and the knowledge gathered. The assistant has not yet written the design document—that will come in [msg 77], after additional investigations into the supraseal C2 C++ API, the bellperson supraseal prover internals, and the SnapDeals/PoSt circuit sizes. But the foundation is laid.

What makes this message remarkable is what it reveals about the engineering process. The assistant does not jump to design. It does not start writing code or even architecture diagrams. It spends the time to understand the problem from four different angles, recognizing that the quality of the design depends on the depth of the research. The parallel task execution is not just efficient—it is a deliberate strategy to hold all four perspectives in mind simultaneously, so that the synthesis, when it comes, is informed by the full context.

The cuzk proving daemon that emerges from this research will be a direct product of the work done in message [msg 68]. Its tiered SRS memory manager comes from understanding both the inference engine patterns (vLLM sleep mode) and the SRS loading paths. Its priority scheduler comes from understanding both the proof type profiles (WinningPoSt is critical, PoRep is normal) and the ffiselect GPU management. Its phased roadmap comes from understanding both the optimization proposals (what we want to achieve) and the existing architecture (what we can change without upstream modifications).

In engineering, as in science, the quality of the answer depends on the quality of the question—and the depth of the research. Message [msg 68] is a masterclass in the latter.