The Architecture Emerges: Designing cuzk, a Pipelined SNARK Proving Daemon

The Subject Message

The message under analysis (message 93) was delivered by the assistant after writing the cuzk-project.md file to the repository root. Here is the message in full:

Written cuzk-project.md at the repo root. Here's a summary of what's in it:

>

### Structure (17 sections)

>

1. What Is cuzk — Vision, why a daemon, inference engine analogies 2. Architecture — Full diagram (gRPC server → scheduler → GPU workers → SRS manager), crate layout (cuzk-core, cuzk-proto, cuzk-server, cuzk-daemon, cuzk-bench, cuzk-ffi), three deployment modes (exec-into, spawn-as-child, external) 3. Proof Types & Circuit Profiles — Table of all 4 proof types with constraints, FFT domain, SRS sizes, partition counts, priorities. Key insight: PoRep SRS is 47 GiB, everything else is 1-3 orders smaller 4. gRPC API — Full protobuf schema: SubmitProof/AwaitProof/Prove/Cancel/GetStatus/PreloadSRS/EvictSRS/GetMetrics with priority levels and per-proof timing breakdown in responses 5. SRS Memory Manager — 3-tier (hot/warm/cold), budget management, Phase 0 strategy (pre-populate GROTH_PARAM_MEMORY_CACHE, zero upstream changes), small vs large machine strategies 6. Scheduler — Priority queues (CRITICAL→LOW), batch collector, GPU affinity tracking 7. GPU Worker Pipeline — Phase 0 (sequential, calls existing APIs) → Phase 2+ (pipelined synthesis || GPU) 8. cuzk-bench — Testing utility with single, batch, stress, gen-vanilla, status, preload commands. Concrete usage examples with /data/32gbench/ golden data. Vanilla proof generation via lotus-bench simple commands for PoSt and SnapDeals 9. Environment & Test Data Setup — Params via curio fetch-params 32GiB to /data/zk/params, golden data layout in /data/32gbench/, exact lotus-bench commands for generating vanilla proofs 10. Configuration — Full TOML config with memory budgets, GPU affinity, scheduler tuning, SRS preload 11. Phased Roadmap — 6 phases over 18 weeks with concrete deliverables per phase, stopping points showing 1.3x → 10x+ cumulative improvement 12-17. Curio integration path, design decisions, open questions, dependency versions, file references, related documents

>

### Key design choices - Rust core (tokio/tonic), direct access to filecoin-proofs - gRPC over unix socket, vanilla proofs inline (~50 MB) - Phase 0 needs zero upstream modifications — just pre-populate the existing in-process parameter cache - Library-with-exec mode for flexible embedding in Curio

This compact summary — just a few paragraphs — represents the distillation of hundreds of lines of analysis, dozens of shell commands, and multiple sessions of deep technical investigation. To understand why this message was written and what it means, we must trace the reasoning that produced it.

Introduction

In message 93 of a deep-dive investigation into Filecoin's Groth16 proof generation pipeline, the assistant delivered a milestone artifact: cuzk-project.md, a comprehensive 17-section architecture document for a pipelined SNARK proving daemon called cuzk. This message is the culmination of an exhaustive multi-session investigation spanning GPU kernel micro-optimizations, constraint-shape-aware computation, memory accounting of a ~200 GiB proof generation pipeline, and studies of inference engine architectures. The message itself is a summary — a compact, dense report of what was written, delivered after the actual file was created. But to understand its significance, one must trace the chain of reasoning that led to this moment.

The Context: From Micro-Optimizations to System Architecture

The conversation leading up to message 93 spans three prior segments of work. Segment 0 mapped the entire SUPRASEAL_C2 call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, accounting for the ~200 GiB peak memory footprint and identifying nine structural bottlenecks. Segment 1 drilled into compute-level micro-optimizations, identifying 18 specific opportunities across GPU kernels, CPU synthesis, and memory transfers. Segment 2 investigated exploiting known constraint shapes — SHA-256 dominance and boolean witnesses — to design a Pre-Compiled Constraint Evaluator (PCE) and specialized matrix-vector multiplication.

Then came Segment 3, the immediate precursor to message 93. The user's instruction in message 78 was deceptively simple: "Write cuzk-project.md." But the notes attached revealed the true scope: build a testing utility for pre-Curio phases, use curio fetch-params for parameter fetching, point to /data/zk/params, leverage the golden test data in /data/32gbench, and use or extend lotus-bench simple commands for generating vanilla proofs.

What followed in messages 79–90 was an exhaustive investigation that demonstrates a rigorous engineering methodology. The assistant did not rush to write the document. Instead, it systematically explored every layer of the existing system:

The Architecture Document: What Was Actually Written

Message 93 summarizes a document with 17 sections. Let us examine what each section represents in terms of design thinking.

Section 1: "What Is cuzk" establishes the vision. The name itself — "cuzk" — is a portmanteau of "Curio" and "zk" (zero-knowledge), signaling that this is not a standalone project but an evolution of the existing Curio infrastructure. The section draws direct analogies to GPU inference engines (vLLM, Triton, TensorRT-LLM), which is a powerful conceptual leap: SRS parameters become model weights, proof jobs become inference requests, and witness vectors become KV cache entries. This framing unlocks an entire design vocabulary borrowed from production ML serving systems.

Section 2: Architecture presents the full component diagram: gRPC server → scheduler → GPU workers → SRS manager. The crate layout (cuzk-core, cuzk-proto, cuzk-server, cuzk-daemon, cuzk-bench, cuzk-ffi) reflects a clean separation of concerns. The three deployment modes — exec-into, spawn-as-child, external — show pragmatic thinking about integration: Phase 0 can work with zero upstream Curio modifications by simply pre-populating the existing in-process parameter cache.

Section 3: Proof Types & Circuit Profiles is where the reconnaissance pays off. The assistant tabulates all four proof types (PoRep, SnapDeals, WindowPoSt, WinningPoSt) with their constraints, FFT domains, SRS sizes, partition counts, and priorities. The key insight — PoRep SRS is 47 GiB while everything else is 1–3 orders smaller — directly drives the SRS memory management strategy.

Section 4: gRPC API defines the full protobuf schema with operations like SubmitProof, AwaitProof, Prove, Cancel, GetStatus, PreloadSRS, EvictSRS, and GetMetrics. The inclusion of PreloadSRS and EvictSRS as first-class API operations signals that SRS management is not an internal detail but a core system capability exposed to the orchestrator.

Section 5: SRS Memory Manager introduces a three-tier hierarchy (hot/warm/cold) with explicit budget management. The Phase 0 strategy — pre-populate GROTH_PARAM_MEMORY_CACHE with zero upstream changes — is a masterstroke of incrementalism. It delivers immediate value (SRS stays resident across proofs) without requiring any modifications to the existing proof library. The small vs. large machine strategies acknowledge that the system must work across heterogeneous hardware.

Section 6: Scheduler defines priority queues from CRITICAL to LOW, a batch collector for accumulating proofs that share SRS parameters, and GPU affinity tracking to minimize context-switching overhead.

Section 7: GPU Worker Pipeline shows the evolution from Phase 0 (sequential, calling existing APIs) through Phase 2+ (pipelined synthesis running in parallel with GPU computation). This phased approach is critical: it means the system can ship and deliver value long before the most aggressive optimizations are implemented.

Section 8: cuzk-bench is the testing utility the user explicitly requested. It provides single, batch, stress, gen-vanilla, status, and preload commands with concrete usage examples using the /data/32gbench/ golden data. The gen-vanilla command wraps lotus-bench simple invocations, directly reusing existing infrastructure.

Section 11: Phased Roadmap spans 6 phases over 18 weeks with cumulative throughput improvements from 1.3× to 10×+. Each phase has concrete deliverables and stopping points, meaning the project can be paused at any phase and still deliver value.

The Reasoning Behind Key Design Decisions

Several design decisions in message 93 deserve close examination:

Why Rust? The assistant chose Rust with tokio/tonic for the core implementation. This is a deliberate departure from Curio's Go codebase. The reasoning is that Rust provides direct access to filecoin-proofs (the Rust/C++ proof library) without CGo FFI overhead, better memory control for the SRS manager, and a stronger concurrency model for the scheduler. The trade-off is integration complexity with Curio's Go infrastructure, which the three deployment modes address.

Why gRPC over unix socket? The choice of gRPC rather than a shared library API or HTTP/REST reflects a design for production serving. Unix domain sockets provide low-latency local communication without network overhead. The vanilla proofs are ~50 MB, which is large but manageable for gRPC's streaming capabilities. This also enables the "external" deployment mode where cuzk runs as a separate process.

Why Phase 0 needs zero upstream modifications? This is perhaps the most strategically important decision. By simply pre-populating the existing GROTH_PARAM_MEMORY_CACHE environment variable (which the proof library already checks), Phase 0 keeps SRS parameters resident across proof jobs without changing a single line of Curio or filecoin-proofs code. This means the first deployment can happen immediately, building confidence and demonstrating value before the more invasive pipeline changes are attempted.

Why inference engine analogies? The assistant explicitly studied vLLM, Triton, and TensorRT-LLM architectures. This is not superficial pattern-matching. The analogy is structurally precise: SRS parameters are like model weights (large, slow to load, reusable across many inference/proof jobs), proof jobs are like inference requests (variable latency, different resource profiles), and witness vectors are like KV cache entries (per-request state that can be streamed). This framing allows the assistant to import proven solutions from the ML serving world — memory pooling, continuous batching, priority scheduling — into the SNARK proving domain.

Assumptions and Their Implications

The design in message 93 rests on several assumptions, some explicit and some implicit:

Assumption: SRS loading dominates latency. The entire SRS memory manager architecture assumes that loading parameters from disk is the primary bottleneck. This is validated by the prior analysis showing that PoRep SRS is 47 GiB and takes significant time to load. However, if the bottleneck shifts (e.g., if GPU compute becomes the dominant factor after SRS is always resident), the priority of the SRS manager relative to other optimizations would need reevaluation.

Assumption: The existing proof library API surface is stable. The Phase 0 approach of pre-populating GROTH_PARAM_MEMORY_CACHE assumes that the parameter caching behavior of filecoin-proofs will not change. This is a reasonable assumption for a mature library, but it creates a dependency on internal implementation details.

Assumption: gRPC overhead is acceptable. For proofs that take minutes to generate, the overhead of gRPC serialization/deserialization is negligible. But for the sub-second proof types (WinningPoSt), the gRPC framing could add measurable latency. The design does not explicitly address this concern.

Assumption: The machine has sufficient memory for multiple SRS sets. The three-tier memory manager assumes that at least the "hot" set can be kept resident. On a machine with 256 GiB RAM, keeping the 47 GiB PoRep SRS plus smaller SRS sets for other proof types is feasible. But on memory-constrained machines, the cold tier would trigger frequent disk loads, potentially negating the benefits.

Assumption: The golden test data is representative. The testing strategy relies on /data/32gbench/ data, which represents a specific sector configuration. The assistant verified it's a 32 GiB StackedDrg sector, but real-world deployments may have different characteristics.

Input Knowledge Required

To fully understand message 93, one needs:

  1. Filecoin proof types: Understanding the difference between PoRep (Proof of Replication, used for sealing new sectors), SnapDeals (sector updates), WindowPoSt (ongoing proof of spacetime), and WinningPoSt (election proof) is essential. Each has different circuit sizes, SRS requirements, and latency sensitivity.
  2. Groth16 proof generation: Knowledge of the three-phase C2 pipeline (constraint synthesis → multi-scalar multiplication → Groth16 proving) and the role of SRS (Structured Reference String) parameters as large structured matrices used in the MSM computation.
  3. Curio architecture: Understanding that Curio is a Filecoin storage provider implementation that currently spawns child processes for GPU proving, with each process loading SRS from disk independently.
  4. Inference engine patterns: Familiarity with vLLM's continuous batching, Triton's model management, and TensorRT-LLM's memory pooling helps appreciate the design analogies.
  5. Go/Rust/CUDA ecosystem: The document spans three language ecosystems and their FFI boundaries.

Output Knowledge Created

Message 93 and the associated cuzk-project.md create:

  1. A complete architecture blueprint for a SNARK proving daemon that can be implemented incrementally over 18 weeks.
  2. A phased roadmap with concrete deliverables, stopping points, and cumulative throughput targets at each phase.
  3. A testing infrastructure design (cuzk-bench) that reuses existing golden test data and lotus-bench commands, lowering the barrier to entry for development and validation.
  4. A set of design analogies (inference engines → proving daemons) that provide a shared vocabulary for discussing the system and a source of proven patterns.
  5. A deployment strategy with three modes (exec-into, spawn-as-child, external) that accommodates different integration timelines and risk tolerances.
  6. An explicit SRS memory management strategy with three tiers and budget control, addressing the single largest resource bottleneck identified in prior analysis.

The Thinking Process Visible in the Message

Message 93 is a summary, so the thinking process is compressed. But the structure reveals the assistant's reasoning:

The assistant leads with the document structure (17 sections), then highlights key design choices. This ordering — structure first, then rationale — reflects a "what, then why" communication pattern. The assistant is not just reporting that a document was written; it is teaching the reader how to navigate the document and what matters most.

The choice to enumerate the sections with their content before discussing design decisions shows a prioritization of comprehensiveness over persuasion. The assistant wants the reader to understand the full scope before evaluating individual choices.

The final bullet points on key design choices are notably terse — "Rust core (tokio/tonic), direct access to filecoin-proofs" — suggesting that these decisions were the subject of extensive prior discussion and do not need re-justification. The assistant is speaking to an audience that shares its context.

Potential Mistakes and Open Questions

While the design is thoughtful, several areas warrant scrutiny:

The 18-week roadmap may be optimistic. Six phases with cumulative 10× throughput improvement requires not just building new infrastructure but also implementing all five prior optimization proposals (Sequential Partition Synthesis, Persistent Prover Daemon, Cross-Sector Batching, 18 micro-optimizations, Pre-Compiled Constraint Evaluator). Each of these proposals involves deep changes to the proof generation pipeline, some requiring modifications to CUDA kernels.

The gRPC API for proof submission may need streaming. For large proofs, the ~50 MB payload could cause memory pressure if multiple proofs are in flight. The design mentions gRPC's streaming capabilities but does not detail how streaming would be used for proof input/output.

The SRS memory manager's cold tier strategy is undefined. The document mentions hot/warm/cold tiers but the summary does not detail what happens when memory is exhausted — does it evict the least recently used SRS? Does it fall back to disk loading? The open questions section presumably addresses this.

The interaction between batch accumulation and latency sensitivity is unresolved. WinningPoSt proofs must be generated within tight deadlines (a 30-minute window in Filecoin's consensus protocol). Batching improves throughput but may increase latency for individual proofs. The scheduler design must balance these constraints.

Conclusion

Message 93 represents a pivotal transition in this investigation: from analyzing an existing system to designing a new one. The assistant synthesized months of deep technical analysis — GPU kernel characteristics, memory bottlenecks, constraint shapes, FFI boundaries, test data layouts — into a coherent architecture that addresses the root cause of inefficiency: the lack of a persistent, stateful proving infrastructure.

The decision to model the proving daemon after GPU inference engines is the conceptual breakthrough. It transforms the problem from "how do we make a single proof faster" to "how do we build a continuous proving service that manages resources efficiently across many proof jobs." This reframing unlocks an entirely different set of solutions — memory pooling, continuous batching, priority scheduling — that were invisible when each proof was treated as an isolated computation.

The phased approach, with Phase 0 delivering value with zero upstream changes, demonstrates engineering pragmatism. The assistant recognizes that the most elegant architecture is worthless if it cannot be deployed incrementally. By designing for immediate, low-risk deployment while building toward ambitious throughput gains, cuzk-project.md bridges the gap between analysis and action.

The document is not the end of the investigation. It is the beginning of the implementation phase. But it represents a moment of synthesis where scattered insights — from GPU kernel occupancy to SRS file sizes to lotus-bench command flags — coalesce into a coherent system design. That is the significance of message 93.