Tracing the FFI Boundary: A Research Probe into Rust–C++ Coordination for Groth16 Optimization

In the middle of a deep optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), the assistant issues a brief but consequential research message. The message at index 2720 consists of a single sentence of intent followed by a file read tool call:

Now let me also check how the Rust-side b_g2_msm/prep_msm timing looks to understand the FFI call structure: [read] /home/theuser/curio/extern/bellperson/src/groth16/prover/supraseal.rs

The tool returns lines 340–347 of the file, which show Rust code assembling vectors of pointers (input_assignments_ref, aux_assignments_ref, a_ref) to pass across the foreign function interface (FFI) into C++/CUDA. On its surface, this is a trivial operation—a developer reading a source file. But in the context of the broader optimization campaign, this message represents a critical moment of verification: the assistant is confirming its mental model of the Rust–C++ boundary before committing to a set of three optimization interventions that must operate across that seam.

The Strategic Context: Phase 11 Design

To understand why this message matters, one must understand the optimization trajectory that led to it. The assistant and user had been systematically working through phases of optimization on the cuzk SNARK proving engine. Phase 9 had achieved a 14.2% throughput improvement through PCIe transfer optimization but revealed DDR5 memory bandwidth contention as the next bottleneck. Phase 10 attempted a two-lock GPU interlock architecture but was abandoned after discovering fundamental CUDA device-global synchronization conflicts—cudaDeviceSynchronize and cudaMemPoolTrimTo operate at the device level, defeating any attempt to split locks per-worker.

After reverting to Phase 9's proven single-lock approach, the assistant ran comprehensive benchmarks across concurrency levels (c=5 through c=20) and performed a waterfall timing analysis from daemon logs. This revealed that at high concurrency, GPU utilization reached 90.8% but throughput plateaued at ~38 seconds per proof. The bottleneck was DDR5 memory bandwidth contention: the synthesis phase (SpMV evaluation) and the prep_msm/b_g2_msm phases were competing for the same memory channels, causing both to inflate under load.

The assistant then researched the memory-bandwidth-heavy code paths and discovered a critical insight: prep_msm with num_circuits=1 (the per-partition pipeline mode) is actually single-threaded. The par_map(1, ...) call with a single item runs on exactly one pool thread regardless of pool size. This meant that the original plan for a full semaphore interlock between prep_msm and synthesis was overkill—one thread competing with 192 synthesis threads has negligible bandwidth impact. The real contention points were:

1. TLB shootdown storms from concurrent munmap() calls in async deallocation threads
2. b_g2_msm using all 192 groth16_pool threads for Pippenger MSM, directly competing with 192 rayon synthesis threads
3. L3 cache pressure from the enormous working sets (~49 GiB CSR for synthesis, ~4.2 GiB G2 bases for b_g2_msm) This led to a revised three-intervention plan: bound async_deallocation to a single thread, reduce the groth16_pool thread count, and add a lightweight atomic throttle to briefly pause synthesis workers during the b_g2_msm window. The user approved this plan.

The Message's Purpose: Verifying the FFI Seam

The subject message sits at the transition between analysis and implementation. The assistant has the conceptual plan approved. Now it needs to write the detailed design specification (c2-optimization-proposal-11.md) and eventually implement the changes. But before writing the spec, the assistant is systematically verifying its understanding of every code path that the interventions will touch.

The message targets the FFI call structure for b_g2_msm and prep_msm—the two phases that run inside the C++ GPU lock and that will be affected by Interventions 2 and 3 (reducing groth16_pool size and adding the synthesis throttle). The assistant needs to understand:

• How Rust prepares and passes data to the C++ side
• Where the timing instrumentation lives (Rust side vs. C++ side)
• How the pointer assembly works for the multi-circuit case
• Whether the FFI boundary introduces any constraints on the proposed changes The file read returns lines 340–347, which show the Rust code building Vecs of raw pointers (as_ptr()) for input_assignments, aux_assignments, and the a vectors. This is the standard pattern for passing Rust-owned data across FFI: collect pointers into a contiguous array, pass the array and its length to C++. The code is truncated (ending with let mut a_ref = Vec::with_capacity(num_c...)), but the pattern is clear.

Input Knowledge Required

To understand this message, one needs substantial context about the SUPRASEAL_C2 pipeline:

• The overall architecture: Curio (Go) dispatches proof tasks to a Rust FFI layer in bellperson, which calls into C++/CUDA code in supraseal-c2. The cuzk engine orchestrates partition-level parallelism.
• The per-partition pipeline: Each partition goes through synthesis (CPU, SpMV evaluation using rayon parallelism), then prep_msm (CPU, bitmap classification and base/scalar population), then GPU work (batch addition, tail MSM), then b_g2_msm (CPU, Pippenger MSM on G2 curve using the groth16_pool thread pool).
• The FFI boundary: Rust code in bellperson/src/groth16/prover/supraseal.rs prepares data and calls into C++ functions via extern "C" declarations. The C++ code in groth16_cuda.cu runs the GPU lock, dispatches CUDA kernels, and manages thread pools.
• The thread pool architecture: There are two main thread pools—rayon (used by synthesis, default 192 threads) and groth16_pool (used by prep_msm and b_g2_msm, also default 192 threads). The groth16_pool is a custom thread pool from the sppark library.
• The config system: The cuzk-core crate has a GpuConfig struct with fields like gpu_threads that control the groth16_pool size.
• The Phase 10 post-mortem: The two-lock design failed because CUDA device-global synchronization APIs cannot be partitioned per-worker, and 16 GB VRAM cannot hold pre-staged buffers from multiple workers simultaneously. The assistant also needs to know the specific file paths and code structure—that supraseal.rs in bellperson is the Rust-side FFI layer, that lines 340+ contain the pointer assembly for passing assignments to C++, and that the timing instrumentation for b_g2_msm and prep_msm lives in the C++ side (in groth16_cuda.cu).

Output Knowledge Created

This message produces a narrow but essential piece of knowledge: confirmation of the FFI call structure at the specific lines where Rust passes assignment data to C++. The assistant now knows:

• The Rust side builds arrays of pointers using as_ptr() on each assignment vector
• The arrays are pre-allocated with Vec::with_capacity(num_circuits) and populated in a loop
• The pattern is consistent across input_assignments, aux_assignments, and the a vectors
• The FFI boundary is clean—Rust passes pointers, C++ receives them—with no complex ownership transfer This knowledge feeds directly into the Phase 11 design document. For Intervention 2 (reducing groth16_pool), the assistant needs to know that the pool size is controlled through the config system, not hardcoded in the FFI layer. For Intervention 3 (synthesis throttle), the assistant needs to know that the synthesis runs in rayon threads on the Rust side, while b_g2_msm runs in groth16_pool threads on the C++ side—meaning the throttle mechanism must cross the FFI boundary (e.g., a Rust-side atomic flag that synthesis workers check, set by the C++ side before b_g2_msm starts).

The Thinking Process

The message reveals a disciplined, systematic approach to optimization. The assistant is not jumping into implementation. It is methodically verifying each piece of the puzzle before committing to a design. The sequence of reads tells the story:

1. Message 2717: Analyze parallelism structure, discover prep_msm is single-threaded, revise the plan
2. Message 2718: Read C++ async deallocation code (groth16_cuda.cu lines 1037+)
3. Message 2719: Read Rust async deallocation code (supraseal.rs lines 388+)
4. Message 2720 (subject): Read FFI call structure (supraseal.rs lines 340+)
5. Message 2721: Check config system for gpu_threads Each read targets a specific concern: async deallocation (Intervention 1), FFI structure (Interventions 2 and 3), config system (Intervention 2). The assistant is building a complete mental model of the codebase before writing the design document. The choice of what to read next shows prioritization: the assistant starts with the most critical uncertainty (the FFI structure for b_g2_msm/prep_msm, which determines whether the interventions are even feasible), then moves to the config system (which determines how easy Intervention 2 is to implement).

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message:

1. That the FFI call structure is the same for both b_g2_msm and prep_msm: The message groups them together ("b_g2_msm/prep_msm timing"), assuming they share the same call pattern. This is reasonable—both are called from the same C++ function (generate_groth16_proofs_c) and receive the same assignment data.
2. That the timing instrumentation is on the Rust side: The message says "Rust-side b_g2_msm/prep_msm timing," implying the assistant expects to find timing code in supraseal.rs. In fact, the detailed timing (TIMELINE events) is on the C++ side in groth16_cuda.cu. The Rust side only has a coarse proof_time measurement. This is a minor assumption that doesn't affect the intervention design.
3. That the pointer assembly pattern is representative: The assistant reads only lines 340–347, which show the beginning of pointer assembly. The pattern is consistent enough that the truncated view is sufficient.
4. That the FFI boundary doesn't introduce synchronization constraints: The assistant is implicitly assuming that adding a throttle mechanism across the FFI boundary is feasible (e.g., a Rust atomic flag checked by synthesis workers, set by C++ code). This is a reasonable assumption—Rust atomics are accessible from C++ via FFI—but it's not verified in this message.

Significance in the Larger Arc

This message, while brief, represents a crucial phase in the optimization workflow: the transition from "what to do" to "how to do it." The assistant has the conceptual plan (three interventions) and the user's approval. Now it is gathering the implementation-level details needed to write a precise, actionable design specification.

The message also illustrates a key principle of systems optimization: understanding the boundaries. The FFI seam between Rust and C++ is where many optimization interventions must be implemented, and getting the details wrong here could lead to subtle bugs (dangling pointers, memory corruption, undefined behavior). By reading the actual pointer assembly code, the assistant grounds its design in the concrete reality of the codebase rather than working from abstract assumptions.

The next messages in the conversation show the fruit of this research: the assistant writes c2-optimization-proposal-11.md with detailed implementation steps for each intervention, updates cuzk-project.md with the Phase 10 post-mortem and Phase 11 roadmap, and commits the documentation. The research probe at message 2720 was a necessary precondition for that work—without understanding the FFI structure, the design document would have been speculative rather than grounded.

In the broader narrative of the optimization campaign, this message is a quiet but essential moment: the point where analysis becomes design, where understanding becomes action, and where the abstract three-intervention plan begins to take concrete shape in the actual codebase.