The Architecture of a Proving Engine: Synthesizing Knowledge Across 18 Weeks of SNARK Pipeline Design

Introduction

In the world of Filecoin storage proving, few challenges are as computationally formidable as Groth16 proof generation. The C2 phase of a Proof-of-Replication (PoRep) proof consumes approximately 200 GiB of RAM, engages 142 CPU cores during synthesis, and requires the orchestration of multi-gigabyte Structured Reference String (SRS) parameters across GPU memory hierarchies. When the assistant sat down to write message 283 in this coding session, they were not merely composing a status update. They were synthesizing the culmination of a deep investigation spanning optimization proposals, GPU kernel analysis, constraint-structure exploitation, and the full implementation of a working proof generation daemon—all distilled into a single, comprehensive planning document that would serve as the architectural foundation for the next five phases of development.

This article examines message 283 in detail: a message that functions simultaneously as a project charter, a technical reference manual, a performance benchmark report, and a phased roadmap. It is a message that demonstrates how effective technical communication can compress weeks of investigation into actionable knowledge, and how a well-structured planning artifact can align an entire engineering effort around shared understanding.

The Context: From Optimization Analysis to Working System

To understand why message 283 was written, one must first understand the journey that preceded it. The conversation history reveals a multi-segment investigation that began with deep-dive analysis of the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep. The initial work, captured in segment 0, mapped the full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, accounting for the ~200 GiB peak memory footprint and identifying nine structural bottlenecks. This analysis produced five optimization proposals: Sequential Partition Synthesis for memory reduction, a Persistent Prover Daemon to eliminate SRS loading overhead, Cross-Sector Batching for throughput improvement, and two additional proposals focusing on micro-optimizations and constraint-structure exploitation.

The investigation then progressed through increasingly concrete phases. Segment 1 conducted compute-level analysis identifying 18 specific micro-optimizations across GPU kernels, CPU synthesis, and memory transfers. Segment 2 investigated exploitation of known constraint shapes—SHA-256 dominance and boolean witnesses—designing a Pre-Compiled Constraint Evaluator and specialized matrix-vector multiplication. Segment 3 designed the pipelined SNARK proving daemon (cuzk) architecture itself. And segments 4 and 5 implemented Phase 0 of that architecture: a working gRPC-based daemon capable of proving PoRep C2 proofs with SRS residency, validated end-to-end with real GPU proofs achieving a 20.5% speedup.

Message 283 arrives at a pivotal moment. Phase 0 is complete and hardened. The assistant has just committed the second Phase 0 commit (f719a710) containing observability improvements, timing breakdowns, Prometheus metrics, GPU detection, and batch benchmarking. The daemon is proven to work: a 32 GiB PoRep C2 proof completes in approximately 110 seconds with full job-id-correlated logging. Now the assistant must consolidate everything learned into a coherent plan for Phase 1 and beyond.

The Message's Dual Purpose: Handoff and Blueprint

Message 283 serves two distinct but complementary purposes. First, it functions as a knowledge handoff—a comprehensive document that captures every technical discovery made during Phase 0 investigation and implementation. Second, it serves as a blueprint—a structured roadmap that decomposes the remaining work into five additional phases spanning 18 weeks.

The handoff function is evident in the "Discoveries" section, which catalogs findings across five domains: serialization format, SRS/parameter details, build environment, parameter location, and performance baselines. Each discovery represents hours of investigation that would otherwise need to be repeated. The serialization format discovery alone—that the c1.json outer format wraps a base64-encoded JSON of the Rust SealCommitPhase1Output struct—is the kind of detail that can consume days of debugging if undocumented. The assistant explicitly notes: "The base64 decodes to JSON (serde_json) of Rust SealCommitPhase1Output struct" and "Rust C2 FFI expects: Raw JSON bytes via serde_json::from_slice() — NOT bincode." This distinction is critical because it dictates exactly how the daemon must deserialize incoming proof requests.

The blueprint function is equally important. The "What's Next — Phase 1" section enumerates four concrete deliverables: wire up WinningPoSt, WindowPoSt, and SnapDeals provers; build a multi-GPU worker pool; implement priority scheduling with SRS affinity; and add per-GPU SRS tracking. Each of these is a substantial engineering task with its own dependencies and unknowns. The message also references the full project plan in cuzk-project.md, which decomposes the work into Phases 2 through 5: split synthesis/GPU pipeline, Go FFI shim for Curio integration, advanced scheduling and batching, and custom MSM/NTT kernels.

The Discoveries: What Was Learned and Why It Matters

The "Discoveries" section of message 283 is arguably its most valuable component. It represents the distillation of raw investigation into actionable engineering knowledge. Let us examine each discovery category and its significance.

Serialization Format

The discovery about c1.json's serialization format is a textbook example of the kind of knowledge that only emerges from hands-on investigation. The assistant traced the data flow from Go's JSON encoding through the Rust FFI boundary, discovering a two-level JSON wrapping: the outer Go struct with a base64-encoded Phase1Out field, which when decoded reveals a second JSON payload that Rust's serde_json::from_slice() can parse. This is not documented anywhere in the upstream library's public API—it is an emergent property of the Go-Rust FFI boundary in filecoin-ffi.

The practical implication is significant. Any system that wants to submit C2 proof jobs must replicate this two-level deserialization. The assistant's implementation in cuzk-core/src/prover.rs handles this correctly, with a C1OutputWrapper struct that mirrors the Go outer format and a Phase1Out field that contains the inner JSON bytes. The unit test test_parse_c1_wrapper validates this logic.

SRS/Parameter Details

The SRS cache discovery is another critical finding. The GROTH_PARAM_MEMORY_CACHE is a lazy_static Mutex<HashMap<String, Arc<Bls12GrothParams>>> in filecoin-proofs—unbounded, never evicting, populated lazily on first proof call. The cache key for PoRep is "STACKED[{padded_sector_bytes}]". Crucially, there is no explicit preload API; the cache populates only when seal_commit_phase2() is first called. The get_stacked_params() and get_post_params() functions are pub(crate), inaccessible from outside the crate.

This discovery directly shaped the Phase 0 architecture. Since the SRS cannot be preloaded through a public API, the assistant implemented a workaround: the engine pre-populates the cache by calling seal_commit_phase2() with a dummy proof request during startup, which triggers lazy loading. The SRS then remains resident for subsequent real proof requests. This is the mechanism behind the measured 20.5% speedup.

The assistant also discovered the correct parameter filenames. The 32 GiB V1.1 sectors use the 8-8-0 variant files, not 8-8-2 which correspond to V1.2. This distinction matters because using the wrong parameter file would produce incorrect proofs. The specific filename is documented: v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params at 45 GiB.

Build Environment

The build environment discoveries reflect the practical challenges of integrating with a complex dependency tree. Rust 1.86.0 is required because filecoin-ffi uses this version and blake2b_simd v1.0.4 needs edition 2024. The home crate must be pinned to 0.5.11 because 0.5.12 requires rustc 1.88. The supraseal-c2 crate comes from crates.io (v0.1.2), not from the local extern/supra_seal/c2/ directory which contains v0.1.0. These are the kind of versioning constraints that can derail a build for days if not documented.

The assistant also discovered that no workspace Cargo.toml exists at the repo root—extern/cuzk/ is a standalone workspace. This means the cuzk project is self-contained within its directory, with its own dependency resolution independent of the parent Curio project. This isolation is both a blessing (no dependency conflicts with Curio) and a curse (shared dependencies must be duplicated).

Performance Baselines

The performance baselines are perhaps the most actionable knowledge in the entire message. The assistant measured three key metrics on the RTX 5070 Ti GPU:

Architectural Decisions Embedded in the Message

While message 283 is primarily a summary and planning artifact, it contains several implicit architectural decisions that deserve examination.

The Deployment Model Decision

The message states: "Library with exec mode — can be embedded in Curio OR run as standalone daemon." This dual-mode design is a deliberate architectural choice that maximizes flexibility. By building the proving logic as a library (cuzk-core) with a separate daemon executable (cuzk-daemon), the assistant ensures that the same code can be deployed in two fundamentally different ways: as an embedded component within Curio's Go process (via a future FFI shim in Phase 3) or as a standalone network service that Curio communicates with over gRPC.

This decision has implications for every subsequent phase. The gRPC API must be designed to support both local (UDS) and remote (TCP) communication. The worker pool must be configurable to run in-process or as separate processes. The SRS cache must be shareable when embedded but isolated when standalone.

The gRPC Message Size Decision

The message notes: "Vanilla proofs (~50 MB for PoRep) sent inline over gRPC (128 MiB message limit set)." This is a pragmatic decision driven by the constraints of gRPC's default message size limits. By setting the limit to 128 MiB, the assistant ensures that even the largest vanilla proofs (which include Merkle tree paths and other data beyond the C1 output) can be transmitted without resorting to streaming or chunking. This simplifies the API at the cost of increased memory pressure on both client and server during submission.

The No-Upstream-Modifications Constraint

Perhaps the most important architectural constraint is stated explicitly: "Phase 0 requires ZERO upstream library modifications." This constraint shaped every Phase 0 decision. The SRS pre-population workaround exists because of this constraint. The lack of fine-grained timing breakdown (the message notes "Phase 0 can't split SRS/synthesis/GPU inside seal_commit_phase2") is a direct consequence. The assistant is deliberately building a layer on top of existing libraries rather than modifying them, which preserves compatibility and simplifies deployment but limits optimization opportunities.

This constraint is relaxed in later phases. Phase 2's "split synthesis/GPU pipeline" and Phase 5's "custom MSM/NTT kernels" would require upstream modifications or parallel implementations. The phased approach allows the project to deliver value early while deferring invasive changes.

Assumptions and Potential Mistakes

Every engineering artifact contains assumptions, and message 283 is no exception. Identifying these assumptions is crucial for understanding the message's limitations.

Assumption: The SRS Cache is Sufficient

The entire SRS residency strategy depends on the GROTH_PARAM_MEMORY_CACHE being a global, process-wide cache. This is confirmed by the source code analysis, but the assistant assumes that this cache will remain stable across proof types. When Phase 1 adds WinningPoSt, WindowPoSt, and SnapDeals, each proof type may use different SRS parameters with different cache keys. The assistant has not yet verified that the cache handles multiple key types correctly, or that switching between proof types doesn't cause cache thrashing.

Assumption: GPU Isolation via CUDA_VISIBLE_DEVICES is Sufficient

The multi-GPU worker pool design uses CUDA_VISIBLE_DEVICES to isolate workers to specific GPUs. This is a common pattern, but it assumes that the CUDA runtime respects this environment variable correctly when set after process start. Some CUDA operations cache device properties at initialization time, and setting CUDA_VISIBLE_DEVICES after CUDA driver initialization may not produce the expected isolation. The assistant may need to verify this with multi-GPU testing.

Assumption: The 128 MiB gRPC Limit is Sufficient

The message states that vanilla proofs are approximately 50 MB for PoRep, fitting within the 128 MiB limit. However, WinningPoSt and WindowPoSt proofs may require multiple vanilla proofs per request (the message notes "one per sector" in the chunk summary). If a single request includes vanilla proofs for dozens of sectors, the total payload could exceed 128 MiB. The assistant acknowledges this in the protobuf definition with repeated bytes vanilla_proofs, but the size implications are not fully addressed.

Potential Mistake: Parameter File Path

The assistant documents the parameter cache path as /data/zk/params and notes that 29 files including the 45 GiB PoRep params are stored there. However, the FIL_PROOFS_PARAMETER_CACHE environment variable must be set before the SETTINGS lazy_static is first accessed. The engine sets this via std::env::set_var() in the GPU worker, but if any code path accesses SETTINGS before this point, the parameter cache path may resolve to the default instead. The assistant's testing suggests this works correctly, but it is a fragile initialization dependency.

Input Knowledge Required

To fully understand message 283, a reader needs knowledge spanning multiple domains:

  1. Filecoin Proof Architecture: Understanding of PoRep, WinningPoSt, WindowPoSt, and SnapDeals proof types; the distinction between vanilla proofs and SNARK proofs; the C1/C2 phase split in PoRep.
  2. Groth16 and SNARKs: Familiarity with Groth16 proving systems, SRS parameters, the role of MSM and NTT operations in GPU-accelerated proving, and the concept of constraint synthesis.
  3. Rust and gRPC: Understanding of Rust's async runtime (tokio), gRPC protocol design with tonic/prost, protobuf message definitions, and Cargo workspace management.
  4. CUDA and GPU Computing: Knowledge of GPU memory hierarchies, CUDA device management, the role of CUDA_VISIBLE_DEVICES for device isolation, and GPU kernel execution models.
  5. Go-Rust FFI: Understanding of how Go and Rust interact through C FFI boundaries, including serialization format mismatches and memory management.
  6. The Curio Project: Context about Curio as a Filecoin storage provider implementation, its task orchestration model, and how proof generation fits into the sealing pipeline.

Output Knowledge Created

Message 283 creates several forms of output knowledge:

  1. A Verified Performance Baseline: The 116.8s cold / 92.8s warm measurements on RTX 5070 Ti establish a reference point for all future optimization work. Anyone implementing an optimization can compare against these numbers.
  2. A Complete File Inventory: The message enumerates every file in the cuzk workspace across five crates, with descriptions of each file's purpose. This serves as a living documentation of the codebase structure.
  3. A Serialization Specification: The two-level JSON wrapping format for C1 outputs is documented with enough precision that anyone implementing a client can correctly serialize proof requests.
  4. A Dependency Map: The version pinning requirements (Rust 1.86.0, home 0.5.11, supraseal-c2 0.1.2) are documented, preventing future build failures from dependency drift.
  5. A Phased Roadmap: The decomposition of remaining work into five phases with explicit deliverables provides a shared understanding of the project trajectory.
  6. A Set of Test Vectors: The golden test data at /data/32gbench/ is documented with file sizes and contents, enabling reproducible testing.

The Thinking Process: From Investigation to Synthesis

The thinking process visible in message 283 reveals a mind that has moved through multiple stages of understanding. The earliest messages in the conversation show exploratory investigation—reading source code, running experiments, discovering serialization formats. By message 283, the assistant has synthesized all of this into a coherent mental model.

The structure of the message itself reflects this synthesis. The "Discoveries" section is organized by domain, not by chronology. Each discovery is presented with its practical implication, not just its factual content. The serialization format discovery includes the note about what it means for the daemon's deserialization logic. The SRS cache discovery includes the workaround strategy. This is the hallmark of mature technical understanding: the ability to connect facts to their consequences.

The "Accomplished" section demonstrates a different kind of thinking: retrospective organization. The assistant has grouped 24 files and 6859 insertions into two coherent commits with clear themes. The first commit is "scaffold"—the initial structure. The second is "hardening"—observability, correctness, and tooling. This categorization is not merely descriptive; it is prescriptive about what Phase 0 means as a milestone. It establishes a pattern that later phases can follow.

The performance baseline section reveals analytical thinking. The assistant didn't just run one proof and record the time. They ran cold and warm measurements, calculated the improvement percentage, and decomposed the total time into components (SRS loading, deserialization, proving). The 20.5% figure is not just a number—it is a validation of the SRS residency strategy and a benchmark for future optimization.

The Broader Significance: What This Message Represents

Message 283 is more than a project update. It is an artifact of a particular moment in the engineering process: the transition from exploration to construction. The assistant has completed the investigative phase—reading source code, running experiments, discovering constraints—and is now ready to build systematically. The message captures everything learned during exploration so that the construction phase can proceed without repeating investigations.

This transition is one of the most critical moments in any engineering project. It is the point at which uncertainty is converted into knowledge, and knowledge is converted into plan. A poorly executed transition leaves the team repeating investigations or building on incorrect assumptions. A well-executed transition, like the one documented in message 283, provides a foundation that supports months of subsequent work.

The message also demonstrates a particular philosophy of engineering documentation: that the most valuable documentation is not abstract architecture diagrams but concrete, verified knowledge about how the system actually behaves. The serialization format discovery, the SRS cache key format, the parameter filenames, the build environment constraints—these are the details that determine whether a project succeeds or fails. Abstract architecture is important, but it is useless without the concrete knowledge that makes it implementable.

Conclusion

Message 283 represents the culmination of a deep investigation into Filecoin proof generation, compressed into a single planning artifact that serves as both knowledge handoff and construction blueprint. It captures discoveries across serialization formats, SRS caching behavior, build environment constraints, and performance baselines—each discovery the product of hours of investigation that would otherwise need to be repeated. It decomposes the remaining work into five phases with explicit deliverables, providing a shared understanding of the project trajectory.

The message's greatest strength is its synthesis of concrete, verified knowledge. It does not speculate about how the system might work; it reports how the system actually works, based on source code analysis and empirical measurement. This grounding in reality is what makes the message valuable not just as a status update but as a permanent reference document for the entire cuzk project.

For anyone joining the cuzk project—whether as a contributor, reviewer, or user—message 283 is the essential starting point. It contains everything needed to understand what has been built, what remains to be built, and the technical context in which the building occurs. It is, in short, the kind of message that every complex engineering project needs and too few produce.