Orchestrating Parallel Discovery: How an AI Agent Mapped the SUPRASEAL_C2 Groth16 Pipeline

Introduction

In the opening move of a deep-dive investigation into Filecoin's proof-of-replication (PoRep) system, an AI assistant received a single, dense directive from a user: "Dive into rust fil proofs / supraseal code — Investigate SUPRASEAL_C2 code path, especially understand proof synthesis step (cpu before GPU used), the memory usage (nearly 200GB peak for PoRep 10 partition Snark) and look for opportunities to reduce data / pipeline harder / merge/parallelise work. Mostly understand the current code deeply. Entry is extern/filecoin-ffi Commit2."

What followed was not a linear, step-by-step code reading exercise but a carefully orchestrated burst of parallel exploration. The assistant's response — message index 1 in the conversation — represents a critical methodological moment: the decision to decompose a complex, multi-layered codebase into four concurrent investigative threads, each targeting a distinct layer of the system. This article examines that single message in depth, exploring the reasoning behind its structure, the assumptions embedded in its design, the knowledge it consumed and produced, and the strategic thinking that shaped it.

The Message in Full

The assistant's response began with a clear statement of intent:

I'll dive deep into the SUPRASEAL_C2 code path, starting from the FFI entry point and tracing through to understand proof synthesis, memory usage, and the CPU/GPU pipeline. Let me launch several exploration agents in parallel to map out the key areas.

It then launched four parallel exploration tasks, each with a specific focus:

Explore FFI Commit2 entry point: Trace the Go-side entry for SealCommitPhase2 through CGO into Rust FFI, mapping the call chain from Curio's task layer down to the Rust boundary.
Explore supraseal C2 Rust code: Analyze the Rust/CUDA implementation layers — Go, Rust FFI (bellperson), and CUDA/C++ — focusing on architecture, memory management, and the CPU-to-GPU handoff.
Explore proof synthesis details: Investigate the CPU-intensive circuit synthesis step that runs before GPU multiexp, looking at synthesize(), ProvingAssignment, constraint generation, and the enforce() loop.
Explore supraseal build system: Map how supraseal is built and integrated into the Curio/filecoin-ffi build system, including build tags, CGO bridges, and feature flag propagation. Each task returned comprehensive results that collectively painted a complete picture of the pipeline from Go orchestration down to CUDA kernel launches. The message thus served as the foundational data-gathering phase for the entire investigation.

Why This Message Was Written: The Strategic Rationale

The message was written to satisfy a specific, high-stakes engineering need. The user had identified a critical problem: the SUPRASEAL_C2 proof generation pipeline for Filecoin PoRep was consuming approximately 200 GiB of peak memory for a 32 GiB sector with 10 partitions. This is an extraordinary memory footprint — far beyond what typical cloud instances offer, and a significant barrier to cost-effective operation. Understanding why the memory was so high required tracing through a deeply layered system spanning Go, Rust, C++, and CUDA, connected by FFI boundaries, conditional compilation flags, and complex build system integration.

The assistant's strategic insight was that this system could not be understood by reading code linearly. The pipeline's complexity arose from the interaction of multiple independently complex layers: Curio's Go task orchestration, filecoin-ffi's CGO bridge, bellperson's Rust proof system, supraseal-c2's CUDA kernels, and the build system that conditionally compiles the whole thing. Each layer had its own memory model, allocation patterns, and concurrency semantics. To understand the 200 GiB peak, one needed to account for memory simultaneously across all layers — the Go heap, the Rust heap, CUDA pinned host memory, GPU device memory, and mmap'd SRS files.

The assistant's decision to launch four parallel agents was therefore not just a convenience optimization but a methodological necessity. The system was too large and too interconnected for any single sequential read to capture. By decomposing the investigation into four orthogonal axes — entry point, Rust implementation, synthesis internals, and build system — the assistant ensured that each agent could focus deeply on its domain while the results could be synthesized into a unified understanding.

How Decisions Were Made: The Architecture of Parallel Exploration

The most visible decision in this message is the choice of four task scopes. Each task was carefully bounded to avoid overlap while ensuring complete coverage:

Task 1 (FFI Entry Point) was scoped to the Go-to-Rust boundary. This was the natural starting point — the user had specified "Entry is extern/filecoin-ffi Commit2." By tracing from SealCommitPhase2 in proofs.go through CGO into the Rust seal::seal_commit_phase2, this task would establish the complete call chain that any optimization would need to respect. It would also identify the feature flag chain (FFI_USE_CUDA_SUPRASEAL=1 → Cargo feature cuda-supraseal) that controls which code path is taken.

Task 2 (Supraseal C2 Rust Code) was scoped to the Rust/CUDA implementation layers. This task would cover the bellperson prover (supraseal.rs), the supraseal-c2 crate's Rust FFI bindings, and the CUDA C++ entry points. It would map the three-layer architecture (Go → Rust FFI → CUDA/C++) and identify the key data structures and memory allocations.

Task 3 (Proof Synthesis Details) was scoped to the CPU-intensive circuit synthesis step. This was the most memory-critical phase — the user had specifically called out "proof synthesis step (cpu before GPU used)." The task would investigate ProvingAssignment structure, the synthesize() method, constraint generation via enforce(), and the density tracking that enables the split-MSM optimization.

Task 4 (Build System) was scoped to the integration mechanics. Understanding how supraseal is built — the build tags, build.rs scripts, CGO bridges, and feature flag propagation — was essential because the conditional compilation means that code paths can differ dramatically between builds. Memory accounting requires knowing exactly which code is compiled in.

The decision to run all four tasks in parallel rather than sequentially reflects an understanding that these investigations are independent in terms of data sources (different files, different directories) but interdependent in terms of interpretation. The assistant could not wait for Task 1 to finish before starting Task 2 because the files were physically separate and the agents would not conflict. This parallelization reduced the time-to-understanding from what would have been four sequential read-and-report cycles to a single parallel burst.

Assumptions Embedded in the Approach

Several assumptions underpin this message's design, and examining them reveals both the strengths and potential blind spots of the approach.

Assumption 1: The four axes are orthogonal and jointly sufficient. The assistant assumed that understanding the entry point, the Rust implementation, the synthesis details, and the build system would together provide complete coverage of the memory problem. This is a reasonable assumption — these four areas map to the four major layers of the system. However, it implicitly excludes other potentially relevant areas: the Go-side task scheduling and subprocess management (which could affect memory isolation), the CUDA kernel implementation details (which affect GPU memory but not host memory), and the protocol-level parameters (which determine circuit size). Later messages in the conversation would indeed dive into the CUDA kernel code and the memory accounting of individual data structures, suggesting that the initial four-task decomposition was a starting point rather than a complete map.

Assumption 2: The agents can operate independently without shared context. Each task prompt was written as a standalone exploration request, without references to findings from other tasks. This means each agent had to rediscover certain facts — for example, both Task 1 and Task 2 independently identified the feature flag chain and the three-layer architecture. This duplication is acceptable for a first pass but creates the risk of inconsistent findings if the agents draw different conclusions from the same evidence.

Assumption 3: The build system is a separate concern from the runtime code. By isolating build system exploration into its own task, the assistant assumed that build configuration is orthogonal to runtime behavior. In practice, the build system directly controls which code paths are active (e.g., whether the CUDA supraseal path or the native Rust prover is used), and memory behavior can differ dramatically between configurations. The build system task did discover this — it identified the FFI_USE_CUDA_SUPRASEAL flag and its propagation — but the separation means that the runtime analysis tasks may have missed build-dependent code paths.

Assumption 4: The "200 GB peak" claim is accurate and refers to host memory. The user stated "nearly 200GB peak for PoRep 10 partition Snark" as a given fact. The assistant accepted this without verification and structured the investigation around explaining it. This is pragmatically correct — the user is likely reporting observed behavior — but it means the investigation did not independently validate the memory measurement methodology. In later messages, the assistant would produce a detailed memory budget table showing ~200-230 GiB estimated peak, confirming the user's claim while adding precision.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs familiarity with several domains:

Filecoin PoRep Protocol Knowledge: Understanding that PoRep (Proof-of-Replication) proves that a storage provider is storing a unique copy of data, and that it uses a "vanilla" (non-zero-knowledge) Merkle proof in phase 1 (C1) followed by a Groth16 zk-SNARK in phase 2 (C2) to compress the proof. The "10 partition" configuration means the sector is divided into 10 independent circuits, each proving a subset of challenges.

Groth16 zk-SNARK Architecture: Understanding that Groth16 proof generation involves circuit synthesis (translating the computation into constraint equations), followed by a multi-exponentiation (MSM) to produce the proof. The "split-MSM" optimization mentioned in the task results exploits the fact that many witness values are 0 or 1, allowing cheaper point additions instead of full scalar multiplications.

FFI and Cross-Language Memory Models: Understanding how Go, Rust, and C++ interact through CGO and C FFI boundaries, and how memory allocated in one language must be pinned or copied for access by another. The CUDA pinned host memory (cudaHostAlloc) is particularly important for understanding the SRS memory footprint.

Rayon Parallelism and Rust Ownership: Understanding how rayon::par_iter creates parallel work, how .collect() creates a synchronization barrier, and how Arc (atomic reference counting) enables shared ownership of large data structures across threads.

CUDA GPU Programming Model: Understanding device memory vs. host memory, pinned memory for DMA transfers, kernel launches, and stream synchronization. The NTT (Number Theoretic Transform) and MSM operations are GPU-accelerated in this pipeline.

A reader without this background would see the task descriptions and results but miss the deeper significance of the findings — for example, why the .collect() barrier on 10 parallel ProvingAssignment allocations is the root cause of the 160 GiB memory spike, or why the SRS being in pinned host memory matters for system RAM availability.

Output Knowledge Created

This message produced a comprehensive, multi-layered map of the SUPRASEAL_C2 pipeline that did not exist in any single document before. The key outputs include:

Complete Call Chain: From Curio task → ffiselect.FFISelect.SealCommitPhase2() → ffi.SealCommitPhase2() → C.seal_commit_phase2() → seal::seal_commit_phase2() → filecoin-proofs::seal_commit_phase2_circuit_proofs() → bellperson::create_random_proof_batch_in_priority() → supraseal_c2::generate_groth16_proof() → generate_groth16_proofs_c() (C++ CUDA entry). This chain spans five programming languages and three FFI boundaries.

Memory Accounting Framework: The message established that the 200 GiB peak breaks down into ~160 GiB for 10 parallel ProvingAssignment structures (each ~16 GiB containing a, b, c, and aux_assignment vectors of ~130M Fr elements), plus ~48 GiB for the SRS in pinned host memory, plus overhead for density trackers, split vectors, and tail MSM bases. This accounting would be refined in later messages but the framework was established here.

Identification of the Synthesis Barrier: The critical finding that all 10 circuits must complete synthesis before any GPU work begins, due to the .collect() barrier in synthesize_circuits_batch(). This is the root cause of the peak memory — if circuits could be pipelined to the GPU one-by-one, peak memory would drop dramatically.

Feature Flag Architecture: The complete chain of conditional compilation from FFI_USE_CUDA_SUPRASEAL=1 environment variable through Cargo features to Rust cfg gates, controlling whether the supraseal CUDA path or the native Rust prover is used.

SRS Loading and Caching Model: The SRS file (~72 GiB on disk) is loaded into a single contiguous CUDA pinned host allocation (~48 GiB in internal representation) with an LRU cache of up to 4 entries. Deserialization happens on a background thread using a dedicated thread pool.

Initial Optimization Landscape: The message identified several optimization opportunities, ranked by impact: pipelining synthesis with GPU proving (highest impact, highest difficulty), reducing batch size (medium impact, medium difficulty), dropping a/b/c vectors after NTT per-circuit (medium impact, medium difficulty), and moving B_G2 MSM to GPU (lower impact, lower difficulty).

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning is visible in the structure and content of the four task prompts. Each prompt is carefully scoped to answer specific questions while avoiding overlap:

The FFI entry point task prompt asks for "the Go-side entry point for Commit2," "how it calls into C/Rust code," "the feature flags that control which path is taken," and "the CGO bridge setup." This reveals a mental model of the system as a series of layers connected by well-defined interfaces, where understanding the entry point means tracing the control flow through each interface.

The proof synthesis task prompt asks for "the synthesize() function," "how ProvingAssignment is built up," "the enforce() method," and "the constraint generation for the StackedCircuit." This reveals an understanding that the memory problem is fundamentally about data structure sizes — each ProvingAssignment holds vectors of field elements whose size is determined by the circuit's constraint count.

The build system task prompt asks for "build tags, Makefiles, or build.rs files," "the CGO bridge between Go and Rust," and "how the CUDA supraseal feature is toggled." This reveals an understanding that the system's behavior is configuration-dependent, and that memory accounting must account for which code paths are actually compiled in.

The most sophisticated reasoning is in the absence of certain questions. The assistant did not ask about the protocol-level parameters that determine circuit size (e.g., why 10 partitions? why ~130M constraints per partition?). It did not ask about the Curio task scheduling model that isolates GPU work in subprocesses. It did not ask about alternative proving systems or cryptographic primitives. These omissions reflect a pragmatic scoping decision: the user asked for deep understanding of the current code, not a redesign of the protocol. The assistant correctly inferred that the investigation should focus on the implementation as it exists, within the constraints set by the Filecoin protocol.

Mistakes and Incorrect Assumptions

While the message is remarkably thorough, several potential issues deserve examination:

The SRS memory estimate is imprecise. The task results report "~30-70 GiB" for the SRS pinned memory, which is a wide range. Later messages would refine this to ~48 GiB. The initial uncertainty reflects the difficulty of estimating internal representation sizes from on-disk sizes — G1 affine points are 96 bytes on disk but may be stored in Montgomery form (64 bytes) or Jacobian form (96 bytes) in memory, and G2 points similarly vary.

The density tracker sharing assumption is stated but not verified. The task results note that "the C++ code already exploits this: all Assignment structs share the same density pointers" but this is inferred from code structure rather than confirmed by tracing the actual pointer assignments. In later messages, the assistant would verify this by reading the specific code paths.

The "sequential synthesis barrier" is identified as the primary bottleneck, but the analysis does not fully account for the split-MSM dependency. The split-MSM optimization requires OR-ing bit vectors across all circuits to determine the "significant" scalar set. If circuits were pipelined one-by-one, this OR operation would need to be redesigned — either by using a conservative estimate from the first circuit or by doing a two-pass approach. The initial analysis acknowledges this difficulty but does not quantify the performance impact of losing the split-MSM optimization.

The build system analysis does not fully connect to memory behavior. The build system task discovered the feature flag chain and the conditional compilation, but the memory accounting in the other tasks assumes a specific build configuration (CUDA supraseal enabled). If the system were built without CUDA support, the memory profile would be completely different (using the native Rust prover instead). This configuration dependence is noted but not explored.

Conclusion

Message index 1 is a masterclass in investigative decomposition. Faced with a sprawling, multi-language codebase exhibiting a critical memory problem, the assistant did not attempt to read code linearly. Instead, it decomposed the investigation into four parallel, orthogonal axes — entry point, Rust implementation, synthesis internals, and build system — each targeted at a specific layer of the system. The results of these parallel investigations would form the foundation for all subsequent analysis: the detailed memory budget, the nine identified bottlenecks, the three optimization proposals (Sequential Partition Synthesis, Persistent Prover Daemon, Cross-Sector Batching), and the micro-optimization analysis of CPU and GPU hotpaths.

The message's lasting contribution is not just the information it gathered but the framework it established for understanding the pipeline. The call chain from Curio to CUDA, the memory accounting categories (ProvingAssignment, SRS, density trackers, split vectors, GPU workspace), the identification of the synthesis barrier as the root cause of peak memory — these conceptual structures would guide the entire investigation. Later messages would refine the numbers, add detail to the bottlenecks, and propose specific optimizations, but the foundational map was drawn in this first response.

For anyone seeking to understand how to approach a complex, multi-layer system with a critical performance problem, this message offers a template: decompose the system into its architectural layers, launch parallel investigations of each layer, and synthesize the results into a unified model. The parallel exploration strategy reduced time-to-understanding from hours to minutes, and the four-axis decomposition ensured that no major layer was overlooked. It is a demonstration of how systematic, structured investigation can tame complexity — and a reminder that the first step to solving a hard problem is often not to solve it, but to map it.