The Architecture of a Memory Crisis: Deconstructing Filecoin's ~200 GiB Groth16 Proving Pipeline

Introduction

In the world of zero-knowledge proofs, few systems push the boundaries of computational scale quite like Filecoin's Proof-of-Replication (PoRep) pipeline. When a storage miner seals a 32 GiB sector, the Groth16 SNARK proof generation consumes approximately 200 GiB of peak memory — a staggering figure that dwarfs most cryptographic workloads. This article chronicles a deep-dive investigation into the SUPRASEAL_C2 proof generation pipeline that sought to understand every byte of that memory footprint, trace the full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, and ultimately design a new architecture for continuous, memory-efficient proving.

The investigation, spanning a subagent session within a larger analysis pipeline, produced four comprehensive documents: a background reference capturing the full pipeline architecture with nine identified structural bottlenecks, plus three composable optimization proposals — Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching. This article synthesizes that work, examining how the team moved from open-ended exploration to targeted architectural design.

The Exploration Mandate: Five Questions That Defined an Investigation

The investigation began with a precisely scoped exploration mandate [3]. The user posed five questions that would define the entire inquiry: (1) how Groth16 proving parameters (SRS) are loaded across different proof types, (2) the circuit sizes and constraint counts for each proof type, (3) the SnapDeals proving path and how it differs from PoRep C2, (4) whether PoSt code paths also use supraseal CUDA, and (5) the parameter/SRS memory footprint for each proof type.

These five questions were not arbitrary. They were designed to fill specific knowledge gaps that stood between the team's high-level understanding of the proof pipeline and the concrete, line-number-level facts needed to architect a replacement system — a pipelined SNARK proving daemon codenamed cuzk. As the exploration mandate document notes, "The overarching goal was to shift from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets" [3].

The questions formed a logical progression from mechanism (how are parameters loaded?) to scale (how large are the circuits?) to variants (how do different proof types map onto the infrastructure?) to resource cost (what is the memory footprint?). Each question fed into the next, creating a complete investigative framework.

Systematic File Discovery: Building the Map

The first step was methodical file discovery. The assistant began by confirming the existence of key files through glob searches [2], locating the Rust-side FFI API (api.rs), the Go-side FFI bindings (proofs.go), and the CUDA-side SRS caching header (groth16_srs.cuh). These three files formed the pillars of the proof system — the Go orchestration layer, the Rust FFI bridge, and the GPU kernel implementation.

The assistant then read these three foundational files [18], establishing the three-layer architecture of the proof system. The Rust api.rs file revealed imports for filecoin_proofs_api::seal and filecoin_proofs_api::update, hinting at the distinction between PoRep and SnapDeals proof paths. The Go proofs.go file showed the CGo linker directives linking against libfilcrypto.a. And the CUDA groth16_srs.cuh header revealed the verifying_key struct with its six elliptic curve points — the data structure that defines the Groth16 verification key.

From there, the investigation expanded outward. The assistant read the parameters.json manifest [21], which maps parameter file names to sector sizes and content identifiers (CIDs). This file was the ground truth for understanding which parameter sets exist for which proof types. The glob results revealed a complete inventory of task files under tasks/seal/, tasks/snap/, tasks/winning/, and tasks/window/ — the directories corresponding to each proof type.

A pivotal moment came when the assistant searched for the "SealCalls layer" [10] — the bridging code between Curio's task orchestrators and the low-level FFI implementations. The glob for lib/ffi/*.go and lib/ffiselect/*.go revealed the FFI wrapper layer and the GPU selection/dispatch mechanism. This was the moment the investigation shifted from examining external dependencies to understanding Curio's own internal orchestration layer.

Tracing the Call Chain: From Go Tasks to CUDA Kernels

With the file map established, the assistant traced the call chain for each proof type. For PoRep C2, the path went from tasks/seal/task_porep.go through lib/ffi/sdr_funcs.go to the supraseal CUDA prover via the ffiselect GPU subprocess mechanism [5]. For SnapDeals, the path diverged: the SNARK proof went through standard filecoin-ffi (bellperson), not supraseal, while only the TreeR generation used supraseal's CUDA kernels [14]. For WindowPoSt and WinningPoSt, the assistant confirmed that neither used the supraseal C2 prover — they went through the standard filecoin-ffi path with much smaller circuits [9].

The GPU subprocess bridge, discovered in lib/ffiselect/ffidirect/ffi-direct.go [7], was a critical architectural finding. All GPU-bound proof computations are dispatched to a separate subprocess (curio ffi) with explicit CUDA_VISIBLE_DEVICES or GPU_DEVICE_ORDINAL settings. This subprocess mechanism provides resource isolation but also introduces complexity: each subprocess potentially loads its own SRS cache, multiplying the memory footprint.

The assistant also examined the lib/supraffi package [12], which provides the Go-side interface to the Supranational GPU-accelerated proving library. The build constraints (//go:build linux && !nosupraseal) revealed that supraseal is a Linux-only feature that can be disabled at build time. The linker dependencies — libsupraseal, libcudart_static, libblst, libconfig++, libgmp — painted a picture of a complex native library stack.

The Memory Accounting: Where 200 GiB Goes

The investigation's central achievement was a detailed memory accounting of the ~200 GiB peak footprint [20]. The analysis broke down the memory consumption by component:

SRS Memory in Supraseal (PoRep C2): From groth16_srs.cuh lines 257-264, the memory calculation formula was extracted:

total = round_up(L * sizeof(affine_t)) + round_up(A * sizeof(affine_t)) 
      + round_up(B_G1 * sizeof(affine_t)) + round_up(B_G2 * sizeof(affine_fp2_t))
      + round_up(H * sizeof(affine_t))

For a 32 GiB PoRep circuit with ~130M constraints, this translates to:

The Null Result That Redirected the Investigation

A critical moment came when the assistant searched for explicit circuit size constants [6]. The grep for constraint|num_inputs|num_aux|fft_domain|circuit_size returned "No files found". This null result was itself a discovery: the Supraseal codebase encodes circuit size information in the SRS binary format rather than in source code constants. The number of H points, L points, A points, B_G1 points, and B_G2 points in the SRS parameter files is the circuit size — there is no separate metadata file or constant declaration.

This forced the investigation to derive circuit sizes through indirect means. The assistant turned to the CUDA kernel code, reading groth16_ntt_h.cu [17] to understand the relationship between FFT domain size and constraint count. The kernel takes lg_domain_size as a runtime parameter, computing limit = (size_t)1 << lg_domain_size. For a 32 GiB PoRep circuit with ~130M constraints, the FFT domain must be at least 2^28 = 268,435,456, and likely 2^28 or 2^29.

The assistant also discovered the aux_assignment_size field [19] in both the Rust SRS struct (lib.rs line 181) and the CUDA kernel code (groth16_cuda.cu line 58). The assertion assert(points_l.size() == p.aux_assignment_size) confirmed that the L point count equals the number of auxiliary variables — the primary dimension determining circuit size.

The Nine Bottlenecks: Structural Inefficiencies in the Pipeline

The background reference document identified nine structural bottlenecks in the existing pipeline. While the full list is detailed elsewhere, several key bottlenecks emerged from the investigation:

  1. All-10-Partitions-in-Parallel Model: The current architecture processes all 10 partitions of a PoRep proof simultaneously, each requiring its own circuit synthesis and SRS allocation. This is the primary driver of the ~200 GiB peak memory.
  2. SRS Loading Overhead: Each proof generation loads the SRS from disk, involving mmap, deserialization, and CUDA pinned memory allocation. This takes approximately 60 seconds per proof and is repeated for every proof, even when the same SRS could be reused.
  3. LRU Cache Inefficiency: The SRS cache holds up to 3 entries, but the eviction policy is naive (pop back when size > 3). There is no coordination between the cache and the proof scheduler, leading to unnecessary reloads.
  4. GPU Subprocess Isolation: Each proof type runs in a separate GPU subprocess, preventing SRS sharing across proof types and multiplying the memory footprint.
  5. No Cross-Sector Batching: Each sector's proof is generated independently, even when multiple sectors could share circuit synthesis or GPU kernel launches.
  6. CPU Synthesis Hotpaths: The enforce() loop during circuit synthesis involves SHA-256 bit manipulation and Fr field arithmetic that are not optimized for the specific constraint structure.
  7. H-to-D Transfer Patterns: Data transfer between host and device follows a pattern that could be optimized with asynchronous transfers and better overlap.
  8. Materialization of a/b/c Vectors: The witness vectors for the a, b, and c polynomials are fully materialized in memory, consuming significant space that could be saved by recomputing on-the-fly.
  9. Lack of Persistent Proving Infrastructure: The entire proof pipeline is ephemeral — processes are spawned, SRS is loaded, proof is generated, and everything is torn down. There is no persistent daemon that could maintain loaded SRS and GPU state across proofs.

The Three Optimization Proposals

Building on the bottleneck analysis, the team developed three composable optimization proposals, each targeting a different dimension of the problem.

Proposal 1: Sequential Partition Synthesis

The most impactful proposal breaks the all-10-partitions-in-parallel model. Instead of processing all partitions simultaneously — each with its own circuit synthesis and SRS allocation — Sequential Partition Synthesis streams partitions one-at-a-time through the GPU. This reduces peak memory from ~200 GiB to approximately 64-103 GiB, depending on the specific implementation.

The key insight is that the 10 partitions are independent: they can be processed sequentially, with only one partition's circuit and SRS data in memory at a time. The trade-off is increased latency per proof (since partitions are serialized), but the dramatic memory reduction enables other optimizations like Cross-Sector Batching.

Proposal 2: Persistent Prover Daemon

This proposal eliminates the ~60-second per-proof SRS loading overhead by keeping the proving process alive across proofs. Instead of spawning a new GPU subprocess for each proof, a persistent daemon maintains the SRS in CUDA pinned memory and accepts proof jobs via a gRPC API.

The daemon architecture draws direct inspiration from GPU inference engines like vLLM, Triton, and TensorRT-LLM [1], drawing analogies between model weights and SRS parameters, inference requests and proof jobs, and KV cache and witness vectors. The daemon manages a three-tier SRS memory hierarchy (hot/warm/cold) with explicit budget control, allowing it to serve multiple proof types without reloading.

Proposal 3: Cross-Sector Batching

This proposal exploits the memory headroom freed by Sequential Partition Synthesis to batch multiple sectors' circuits into single GPU invocations. By processing multiple sectors' proofs in a single GPU kernel launch, the proposal projects 2-3× throughput per GPU and 5-6× reduction in $/proof when combined with the other proposals.

The batching is enabled by the observation that the GPU's compute capacity (NTT, MSM) is underutilized in the single-sector case. By packing multiple sectors' data into larger kernel launches, the GPU's parallelism is more fully exploited.

The Micro-Optimization Analysis

A final round of micro-optimization analysis examined the computational hotpaths at the instruction level [17]. The CPU synthesis hotpaths — the enforce() loop, SHA-256 bit manipulation, and Fr field arithmetic — were characterized to identify optimization opportunities. The GPU NTT and MSM compute characteristics were analyzed to understand kernel occupancy and memory bandwidth utilization.

The analysis also evaluated the feasibility of recomputing a/b/c vectors on-the-fly rather than materializing them in memory. This would trade compute for memory, potentially reducing the peak footprint further at the cost of increased computation time. The analysis concluded that for the PoRep circuit's specific structure, selective recomputation could save significant memory without unacceptable latency impact.

From Analysis to Architecture: The cuzk Design

The culmination of this investigation was the design of the cuzk pipelined SNARK proving daemon [1]. The architecture document, written to the repo root as cuzk-project.md, lays out a complete architecture and phased implementation plan covering:

Conclusion

The investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline represents a comprehensive deconstruction of one of the most memory-intensive cryptographic workloads in production. By tracing the full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, the team produced a detailed memory accounting that explains exactly where the ~200 GiB peak goes: ~75 GiB per SRS in pinned CUDA memory, multiplied by a 3-entry LRU cache, plus working memory for circuit synthesis and proof computation.

The three optimization proposals — Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching — offer a composable path from the current memory-hungry architecture to a continuous, memory-efficient proving pipeline. The cuzk design document translates these proposals into a concrete implementation plan with an 18-week roadmap.

The overarching achievement is a shift in mindset: from optimizing individual proof generation toward architecting a continuous proving infrastructure optimized for heterogeneous cloud rental markets where RAM cost dominates. In a world where GPU instances charge a premium for memory, reducing peak footprint from 200 GiB to 64 GiB is not just an engineering optimization — it's an economic necessity.## References

[1] "Bridging the Gap: How the Curio FFI Wrapper Layer Orchestrates Filecoin Proof Generation" — Article detailing the FFI wrapper layer connecting Curio's task layer to the underlying proof implementations.

[2] "The Opening Move: Systematic File Discovery in a Codebase Investigation" — Article covering the initial file discovery phase of the investigation.

[3] "The Exploration Mandate: Deconstructing the Curio Codebase for SNARK Proving Architecture Design" — Article analyzing the five-question exploration mandate that defined the investigation.

[5] "The Pivot Point: How an AI Investigator Navigates a Multi-Language Codebase" — Article examining the strategic decision to read the four key task files.

[6] "The Null Result That Shaped an Investigation: Tracing Circuit Sizes in the Supraseal Groth16 Pipeline" — Article analyzing the null grep result that redirected the investigation.

[7] "Tracing the GPU Subprocess Bridge: A Pivotal Discovery in the Curio Proof Pipeline" — Article covering the discovery of the GPU subprocess handler in ffi-direct.go.

[9] "The Pivot Point: Reading the Parameter Fetcher and CUDA Kernel in a Filecoin Proof System Investigation" — Article examining the paramfetch and CUDA kernel reads.

[10] "The Pivot Point: How a Single Glob Command Uncovered the SealCalls Layer in Curio's Proof Pipeline" — Article analyzing the discovery of the lib/ffi and lib/ffiselect layers.

[12] "The Pivot Point: Reading the Supraseal Go Wrapper in a Filecoin Proof Pipeline Investigation" — Article covering the supraffi package analysis.

[14] "Tracing the SnapDeals Proof Path: A Methodical Grep Through Filecoin's FFI Layer" — Article examining the SnapDeals proof path discovery.

[17] "The Microscope Turned on the NTT Kernel: Decoding Circuit Sizes from a CUDA Source File" — Article analyzing the NTT kernel read and FFT domain size derivation.

[18] "Opening the Investigation: Reading the Three Pillars of Groth16 Proving in Curio" — Article covering the foundational file reads.

[19] "The Pivot Point: How a Single Exploration Message Unlocked the Memory Architecture of Filecoin's Groth16 Prover" — Article analyzing the aux_assignment_size discovery.

[20] "Anatomy of a Knowledge Synthesis: Mapping Filecoin's Groth16 Proving Pipeline" — Article covering the comprehensive findings document.

[21] "Navigating the Filecoin Proving Pipeline: A Deep Dive into Parameter Discovery and Task Code Paths" — Article examining the parameters.json read and task file globs.