The Architecture of a Discovery: Mapping the SUPRASEAL_C2 Groth16 Pipeline for Filecoin Proof Generation

Introduction

In the landscape of modern cryptographic proving systems, few challenges rival the sheer scale of Filecoin's Proof-of-Replication (PoRep) protocol. Generating a single Groth16 proof for a 32 GiB sector requires orchestrating roughly 130 million constraints across 10 partitions, consuming nearly 200 GiB of system memory, and coordinating CPU synthesis with GPU-accelerated NTT and MSM operations. The message at index 66 in this conversation represents a remarkable artifact: a comprehensive summary of an exhaustive investigation into the SUPRASEAL_C2 code path, produced after the assistant had already written seven detailed optimization proposals and a total impact assessment. This message is not merely a status report—it is a synthesis of deep systems knowledge, a catalog of architectural discoveries, and a blueprint for transforming how Filecoin proofs are generated.

This article examines message 66 in detail: why it was written, what decisions it encodes, the assumptions it makes, the knowledge it both consumes and produces, and the thinking process that shaped it. For readers unfamiliar with the conversation, this message is the culmination of a multi-session investigation into the Curio Filecoin mining stack's proof generation pipeline, specifically the SUPRASEAL_C2 implementation of Groth16 proving accelerated by CUDA. The message captures the assistant's complete understanding of a system spanning Go orchestration, Rust FFI boundaries, C++ CUDA kernels, and the mathematical structure of Rank-1 Constraint Systems (R1CS).

The Context and Motivation: Why This Message Was Written

To understand why message 66 exists, one must understand the broader arc of the conversation. The user—a developer working on the Curio Filecoin mining software—had been engaged in a deep-dive investigation spanning multiple sessions. The work had already produced a background reference document and five optimization proposals, each targeting a different layer of the proving pipeline. Message 66 appears after the assistant had written both c2-optimization-proposal-5.md (constraint-shape-aware optimizations) and c2-total-impact-assessment.md (combined impact across all proposals). The preceding messages show the assistant extracting timing estimates from prior proposals and composing the total impact assessment.

Message 66 serves multiple purposes simultaneously. First, it is a knowledge consolidation artifact—a single document capturing everything learned across the entire investigation. Second, it is a handoff document that could serve as an executive summary for someone new to the project. Third, it is a decision record documenting which approaches were investigated, which were ruled out, and why. Fourth, it is a task completion summary demonstrating that the assigned work has been comprehensively addressed. Fifth, and perhaps most importantly, it is a foundation for future work—the "What Could Be Done Next" section explicitly identifies the next engineering priorities.

The motivation for writing this message stems from the user's ambitious goal: to maximize proof throughput per dollar of system cost for a "proofshare marketplace" deployment with a heterogeneous GPU fleet. This is not a typical optimization exercise. The user is not asking to make a single proof faster; they are asking to architect an entire proving infrastructure optimized for cloud rental markets where GPU instances cost different amounts, memory is a first-class cost driver, and the system must handle diverse proof types (PoRep, SnapDeals, WindowPoSt, WinningPoSt) across different sector sizes. The economic framing—"proofs per dollar"—permeates every decision documented in this message.

The Instructions: What the Assistant Was Asked to Investigate

The message reproduces the original instructions, which are worth examining for what they reveal about the investigation's scope and depth:

"Start from extern/filecoin-ffi Commit2 entry point and trace the full code path deeply" "Understand proof synthesis step (CPU before GPU), memory usage, and look for opportunities to reduce data / pipeline harder / merge/parallelize work" "Think big about optimization — not just individual proofs but constant proving pipeline throughput" "Optimize for maximum proof throughput per $ of system cost" "Go deep on PCE (Pre-Compiled Constraint Evaluator) implementation design, constraint shape exploitation, and boolean witness MSM optimization" "Keep SnarkPack transpositions at architectural level within the same document" "Investigate whether constraint-shape-aware optimizations... can yield significant gains" "Investigate mathematical transpositions when output is SnarkPack-ed"

These instructions reveal a sophisticated understanding of the problem space. The user already knew that the GPU pipeline existed (SUPRASEAL_C2), that SnarkPack aggregation was relevant, and that constraint shape might be exploitable. The instructions are not generic—they are targeted at specific known unknowns. The user wanted the assistant to go deep on the Pre-Compiled Constraint Evaluator (PCE) concept, to investigate whether the deterministic structure of Filecoin circuits (identical R1CS matrices across proofs) could be exploited, and to understand whether SnarkPack's mathematical structure could reduce C2 computation.

The instruction to "think big about optimization—not just individual proofs but constant proving pipeline throughput" is particularly revealing. It signals that the user was already dissatisfied with the existing architecture where each proof runs in an isolated child process, SRS is loaded from scratch every time, and the GPU sits idle during CPU synthesis. The user wanted architectural transformation, not micro-optimization.

The Discoveries: What the Investigation Uncovered

The discoveries section of message 66 is the heart of the document, and it contains findings that range from the architectural to the microscopic. Let me examine each major discovery in detail.

The Full Call Chain

The assistant traced the complete path from Go to CUDA kernels:

"Go → ffiselect spawns child process per proof with pinned GPU → CGO → Rust FFI (api.rs:284) → filecoin-proofs-api → filecoin-proofs → storage-proofs-core compound_proof → bellperson 0.26.0 → supraseal-c2 crate → C++ CUDA (groth16_cuda.cu:104)"

This call chain is significant because it reveals the complexity budget of the system. Every layer transition—Go to CGO, CGO to Rust FFI, Rust to bellperson, bellperson to supraseal-c2, supraseal-c2 to CUDA—represents a potential optimization boundary. The fact that each proof runs in an isolated child process (curio ffi) with SRS loaded from scratch every time (30-90 seconds of deserialization) is identified as a major structural bottleneck. This discovery directly motivates Proposal 2 (Persistent Prover Daemon) and Proposal 1 (Sequential Partition Synthesis).

The Memory Budget

The memory accounting is precise and damning:

"10 × ProvingAssignment (a,b,c vectors): ~120 GiB" "10 × aux_assignment: ~40 GiB" "SRS in CUDA pinned memory: ~47 GiB" "C++ split vectors/temp: ~5-15 GiB" "Total peak: ~195-220 GiB host; GPU peak: ~12-16 GiB"

This breakdown reveals that the single largest memory consumer is the ProvingAssignment vectors (a, b, c) for all 10 partitions simultaneously—120 GiB of the ~200 GiB peak. The insight that drives Proposal 1 is that these vectors are only needed one partition at a time: synthesize partition 0, transfer to GPU, NTT+H, free vectors, repeat. This single insight reduces peak memory from 200 GiB to 64 GiB, which has enormous cost implications for cloud deployments where memory is priced per GiB.

Circuit Characteristics and the Key Finding

The circuit analysis is where the investigation makes its most impactful discovery:

"~99% of aux_assignment values are boolean (0 or 1) — dominated by SHA-256 internal bits" "~88% of constraints are SHA-256, ~10-11% Poseidon, <1% other" "The circuit structure is fully deterministic — the constraint graph is identical for every 32 GiB sector PoRep proof. Only the witness values change."

The key finding is stated with emphasis:

"KEY FINDING: R1CS Matrices Are Fixed" "The R1CS constraint matrices A, B, C are identical for every proof. Only the witness vector changes." "Currently, every proof re-runs circuit.synthesize(), rebuilding ~130M LinearCombination objects from scratch (780M heap allocations per partition) just to evaluate them against a new witness — pure waste."

This is the foundational insight that motivates the entire Pre-Compiled Constraint Evaluator (PCE) proposal. The assistant recognized that the current approach rebuilds the entire constraint graph for every proof, even though the constraint graph never changes. Only the witness values differ between proofs. This is analogous to recompiling a program every time you want to run it with different input data, rather than compiling once and executing repeatedly.

The 780M heap allocations per partition figure is staggering. With 10 partitions, that's 7.8 billion heap allocations per proof, all of which are immediately discarded after evaluation. The PCE eliminates this entirely by extracting the fixed R1CS matrices into CSR (Compressed Sparse Row) format once and then evaluating them against each new witness using a sparse matrix-vector multiply.

GPU Pipeline Details

The assistant's analysis of the GPU pipeline reveals several bottlenecks:

"generate_groth16_proofs_c() is globally serialized by static mutex" "NTT+H uses 2-3 CUDA streams with partial overlap; a,b,c transferred from pageable (not pinned) memory" "GPU sits idle during entire synthesis phase (1-3 minutes)"

The pageable memory transfer is a particularly subtle but important finding. Pageable memory achieves ~10-15 GiB/s transfer speeds versus ~25 GiB/s for pinned memory. Since the a,b,c vectors total ~12 GiB per partition, this difference adds roughly half a second to transfer time—not enormous, but every millisecond counts when optimizing for throughput.

The finding that the GPU sits idle for 1-3 minutes during synthesis is the primary motivation for the pipeline architecture proposed in cuzk. If the GPU could be kept busy while the CPU synthesizes the next partition, overall throughput would nearly double.

SnarkPack Analysis

The SnarkPack analysis is notable for what it rules out:

"No mathematical transposition from SnarkPack can reduce C2 computation" "The win is purely architectural: GPU does C2 while CPU does aggregation (CPU-dominated: pairings + MSMs)"

This is an important negative result. The user had asked the assistant to "investigate mathematical transpositions when output is SnarkPack-ed," and the assistant traced the aggregation code thoroughly to determine that SnarkPack operates exclusively on output proof triplets. There is no way to make C2 faster by exploiting SnarkPack's structure. However, the architectural win—running C2 on GPU while aggregation runs on CPU—is still valuable and enables the two-tier marketplace concept where GPU machines handle proving and cheap CPU machines handle aggregation.

NTT/MSM Kernel Details

The kernel-level analysis reveals specific GPU micro-bottlenecks:

"8-way shared memory bank conflicts for 32-byte Fr elements" "GS Step 1 butterflies have stride 2^26 — global data dependency, cannot stream" "batch_addition uses cooperative kernel (grid().sync()) — prevents GPU concurrency" "batch_addition occupancy: ~12.5% (196 registers/thread)" "B_G2 MSM runs on CPU sequentially across circuits — major bottleneck (50s)"

These findings directly feed into Proposal 4's 18 compute-level optimizations. The bank conflicts in shared memory, the global data dependency preventing streaming NTT, the low occupancy of batch_addition, and the sequential CPU B_G2 MSM are all concrete optimization targets.

The Seven Documents: What Was Produced

Message 66 catalogs seven completed documents, each targeting a different layer of the optimization stack:

c2-improvement-background.md — The comprehensive reference document. This is the foundation that all other proposals build upon. It covers the full call chain, feature activation, synthesis internals, GPU pipeline, SRS loading, memory budget breakdown, circuit value distribution, Curio orchestration model, nine structural constraints/bottlenecks, and a complete file reference table.
c2-optimization-proposal-1.md — Sequential Partition Synthesis with GPU Pipelining. The key insight: synthesize one partition at a time, immediately transfer to GPU for NTT+H, free the a/b/c vectors, repeat. Reduces peak memory from 200 GiB to 64 GiB and increases GPU utilization from ~30% to ~70%.
c2-optimization-proposal-2.md — Persistent Prover Daemon with SRS Residency. Replace the per-proof child process with a long-lived daemon that loads SRS once and keeps it in pinned memory. Eliminates 30-90 seconds of SRS loading per proof.
c2-optimization-proposal-3.md — Cross-Sector Proof Batching. Batch multiple sectors' partition circuits into a single GPU invocation to amortize fixed costs. Achieves 2-3x throughput per GPU.
c2-optimization-proposal-4.md — Compute-Level Optimizations. 18 specific optimizations across 5 categories: SmallVec for Indexer, pre-size vectors, pin a/b/c memory, parallelize B_G2 MSMs, NTT kernel fusions, MSM cooperative kernel elimination, and more. Combined: 30-43% faster per-proof.
c2-optimization-proposal-5.md — Constraint-Shape-Aware Optimizations & SnarkPack Transpositions. Five parts: (A) Pre-Compiled Constraint Evaluator (3-5x faster synthesis), (B) Specialized MatVec (~16x on inner loop), (C) Pre-Sorted Split MSM (15-25% faster GPU MSM), (D) SnarkPack Co-Design (two-tier marketplace), (E) Approaches Ruled Out.
c2-total-impact-assessment.md — Combined impact across all proposals: 10x throughput improvement, ~20x cost reduction, memory reduction from 256 GiB to 96 GiB, 13-week phased implementation plan. The documents form a coherent optimization narrative. Each proposal builds on the discoveries of the background document, and the total impact assessment synthesizes them into a unified implementation roadmap. The assistant explicitly identifies the 8-week mark (through P5A, the Pre-Compiled Constraint Evaluator) as the natural stopping point, capturing 72% of the total improvement with 62% of the effort.

The Thinking Process: How the Assistant Approached the Problem

The thinking process visible in message 66 reveals a systematic, multi-layered approach to understanding and optimizing a complex system.

Layer 1: Architectural Tracing. The assistant started by tracing the full call chain from Go to CUDA kernels. This is the foundational step—without understanding how the pieces fit together, any optimization would be guesswork. The tracing revealed the child process model, the SRS loading overhead, and the feature activation chain.

Layer 2: Quantitative Accounting. Every claim in the message is backed by specific numbers: 130M constraints per partition, 780M heap allocations, 47 GiB SRS, 200 GiB peak memory, 30-90s SRS load time, 1-3 minutes GPU idle during synthesis. This quantitative rigor is essential for prioritizing optimization efforts. Without these numbers, one might optimize the wrong bottleneck.

Layer 3: Structural Pattern Recognition. The key finding—that R1CS matrices are fixed across proofs—came from recognizing a structural pattern in the system. The assistant noticed that the circuit is deterministic (same constraint graph for every 32 GiB sector proof) and that the current implementation rebuilds it every time. This pattern recognition is the most valuable cognitive skill demonstrated in the message.

Layer 4: Mathematical Boundary Analysis. The assistant carefully analyzed what SnarkPack does and does not affect, concluding that "no mathematical transposition from SnarkPack can reduce C2 computation." This is an important negative result that prevents wasted engineering effort. The assistant also ruled out several other approaches (pre-computing INTT of matrix columns at 540 PiB, streaming NTT due to global data dependency, tensor cores due to no modular arithmetic mapping, proof recycling due to completely different witnesses).

Layer 5: Economic Framing. Throughout the message, the assistant frames findings in terms of cost: "maximize proof throughput per $ of system cost," "$0.083 → $0.004 per proof," "256 GiB → 96 GiB minimum machine." This economic lens is what transforms a technical investigation into a business decision document.

Layer 6: Implementation Roadmapping. The "What Could Be Done Next" section and the 13-week phased plan in the total impact assessment show that the assistant was thinking about execution, not just analysis. The identification of the PCE as the "highest single-week-of-effort impact at 1.0x throughput multiplier per week" is a classic prioritization framework.

Assumptions and Potential Mistakes

While message 66 is remarkably thorough, it rests on several assumptions that deserve examination.

Assumption 1: The circuit structure is truly identical across all proofs. The assistant states that "the circuit structure is fully deterministic—the constraint graph is identical for every 32 GiB sector PoRep proof." This is likely true for PoRep, but the message also mentions other proof types (SnapDeals, WindowPoSt, WinningPoSt). Are those circuits also deterministic? The assistant's analysis focused on PoRep, and the generalization to other proof types may not hold perfectly. Different sector sizes (32 GiB vs 64 GiB) might produce different circuit structures.

Assumption 2: The timing estimates are accurate. The message reports specific numbers like "3-5x faster synthesis" and "~16x on MatVec inner loop" and "15-25% faster GPU MSM." These are estimates based on analysis, not measurements. The assistant acknowledges this in "What Could Be Done Next": "Benchmarking/profiling to validate the timing estimates with actual measurements." Until validated, these numbers should be treated as educated guesses.

Assumption 3: The PCE implementation is feasible within bellperson's architecture. The assistant identifies that WitnessCS exists in bellperson and "completely no-ops enforce() while still running alloc() closures." However, implementing the PCE requires extracting the R1CS matrices from KeypairAssembly (which captures them in CSC format) and converting them to CSR for efficient MatVec. The assistant also notes that SHA-256 labeling has "NO is_witness_generator() fast path." Creating this fast path is non-trivial engineering that may require changes to upstream bellperson.

Assumption 4: The proofshare marketplace model is viable. The entire optimization framework is built around the assumption that a two-tier marketplace (GPU machines for C2, CPU machines for aggregation) is the right deployment model. This is an architectural assumption that may not hold for all operators. Some may prefer single-machine deployments where the memory reduction (256 GiB → 96 GiB) is the primary benefit.

Potential Mistake: Underestimating integration complexity. The message lists 7 documents and 13 weeks of implementation, but the actual integration effort may be higher. The system spans Go, CGO, Rust, C++, and CUDA—each with its own build system, dependency management, and testing infrastructure. Changes to bellperson require coordination with upstream maintainers. The child process model in ffiselect would need significant rework for the Persistent Prover Daemon.

Potential Mistake: Overlooking memory bandwidth constraints. The message identifies that a,b,c vectors are transferred from pageable memory at 10-15 GiB/s, but doesn't fully account for PCIe bandwidth contention when multiple GPU operations are happening concurrently. In a fully pipelined system with overlapping transfers and computations, PCIe bandwidth could become a bottleneck that the current estimates don't capture.

Input Knowledge Required

To fully understand message 66, a reader needs knowledge spanning several domains:

Cryptographic Proof Systems: Understanding of Groth16, R1CS, NTT (Number Theoretic Transform), MSM (Multi-Scalar Multiplication), and SnarkPack aggregation. The message assumes familiarity with these concepts and uses them without explanation.

GPU Architecture: Understanding of CUDA kernels, shared memory bank conflicts, occupancy, cooperative kernels, pinned vs pageable memory, PCIe transfer characteristics, and GPU stream concurrency.

Filecoin Protocol: Understanding of PoRep (Proof-of-Replication), sector sizes, partition structure, and the role of C1/C2 proof phases. The message references "32 GiB PoRep 10-partition SNARK" without explaining what partitions are or why there are 10 of them.

Software Architecture: Understanding of FFI boundaries, child process models, CGO bridges, and Rust crate dependency chains. The message traces a path through Go → CGO → Rust FFI → bellperson → supraseal-c2 → CUDA, which requires familiarity with multi-language systems.

Systems Optimization: Understanding of memory budgets, heap allocation patterns, throughput vs latency optimization, and the economics of cloud computing. The message's framing in terms of "proofs per dollar" requires economic as well as technical literacy.

Output Knowledge Created

Message 66 creates substantial new knowledge that did not exist before the investigation:

Architectural Knowledge: The complete call chain from Curio's Go layer through to CUDA kernels, documented with specific file paths and line numbers. This is knowledge that was distributed across multiple codebases and required significant effort to assemble.

Quantitative Knowledge: Precise memory budgets, timing estimates, and circuit characteristics. Before this investigation, the 200 GiB peak memory was likely known only as "a lot" rather than as the specific sum of 120 GiB (a/b/c vectors) + 40 GiB (aux) + 47 GiB (SRS) + 15 GiB (temp).

Structural Knowledge: The insight that R1CS matrices are fixed across proofs, and that 99% of witness values are boolean. This knowledge enables entirely new optimization approaches that were not previously considered.

Negative Knowledge: The message explicitly documents what does NOT work: SnarkPack cannot reduce C2 computation, INTT of matrix columns is infeasible at 540 PiB, streaming NTT is impossible due to global data dependency, tensor cores cannot do modular arithmetic, proof recycling doesn't work with different witnesses. This negative knowledge is valuable because it prevents wasted engineering effort.

Decision Knowledge: The message provides a clear implementation roadmap with prioritized phases, diminishing returns analysis, and recommended stopping points. This transforms raw technical findings into actionable engineering decisions.

Economic Knowledge: The cost analysis ($0.083 → $0.004 per proof, 256 GiB → 96 GiB memory) provides a clear business case for the optimization work. This is knowledge that can justify engineering investment to stakeholders who care about economics rather than technical elegance.

Conclusion

Message 66 is a remarkable artifact of systems investigation. It demonstrates what thorough, systematic analysis of a complex software system can produce: not just a list of bugs or performance issues, but a complete understanding of how the system works, where its resources go, what structural patterns constrain its performance, and how to transform it into something fundamentally better.

The message's greatest strength is its integration of multiple perspectives. It is simultaneously a technical deep-dive (tracing individual CUDA kernel characteristics), an architectural analysis (mapping the full call chain), an economic framework (proofs per dollar), and an engineering roadmap (13-week phased implementation). This multi-perspective approach is what distinguishes a truly useful investigation from a merely thorough one.

The investigation documented in message 66 also demonstrates the value of negative results. The assistant explicitly ruled out several promising-sounding approaches (SnarkPack transpositions, streaming NTT, tensor cores, proof recycling) with clear reasoning. In optimization work, knowing what not to do is often as valuable as knowing what to do.

Finally, message 66 exemplifies a particular kind of technical communication: the comprehensive summary that serves as both documentation of past work and foundation for future work. It is written with the assumption that the reader may not have been present for the investigation, may not be familiar with all the codebases involved, and needs to understand not just what was found but why it matters. This is a difficult balance to strike, and the message achieves it through precise language, quantitative rigor, and a clear narrative arc from discovery to decision.

The seven documents summarized in this message represent one of the most thorough optimizations of a cryptographic proving pipeline that this author has encountered. Whether or not the implementation proceeds through all 13 weeks of the roadmap, the knowledge captured in message 66 and its companion documents will inform Filecoin proof generation architecture for years to come.