The Final Rust Plumb: Exporting the Split API from Bellperson's Groth16 Module

Subject Message: [edit] /home/theuser/curio/extern/bellperson/src/groth16/mod.rs Edit applied successfully.

Introduction

At first glance, message [msg 2894] appears to be the most mundane of tool outputs: a single line confirming that an edit was applied to a Rust module file. But in the context of the Phase 12 split API implementation — a multi-hour, multi-file refactoring spanning C++ CUDA kernels, Rust FFI bindings, and application-level orchestration — this brief confirmation represents the completion of a critical milestone. It is the moment when the Rust-side plumbing for the split proving API was finalized, making the new PendingProof, prove_start, and prove_finish symbols publicly accessible from the bellperson::groth16 module. Without this export, the higher-level pipeline and engine code in cuzk-daemon would have no way to call the split API, and the entire refactoring effort would remain stranded at the FFI boundary.

The Broader Context: Phase 12 Split API

To understand why this message matters, one must trace the chain of reasoning that led to it. The Phase 12 split API was born from a detailed memory-bandwidth analysis conducted during Phase 11 ([chunk 29.0]). The assistant and user had identified that the b_g2_msm operation — a multi-scalar multiplication on the G2 curve — ran for approximately 1.7 seconds after the GPU lock was released but still blocked the GPU worker from picking up the next synthesis job. The insight was that this CPU-bound post-processing step could be deferred: instead of having the GPU worker thread wait for b_g2_msm to complete before returning to the job queue, the worker could package the intermediate state into an opaque handle, spawn a separate thread to finalize the proof, and immediately loop back to grab the next circuit synthesis task.

This architectural change required coordinated modifications across three layers:

  1. C++ CUDA layer (groth16_cuda.cu): Refactor generate_groth16_proofs into a split start/finish pair, with a heap-allocated groth16_pending_proof struct that holds all intermediate state (MSM results, split vectors, tail bases, the running prep_msm_thread, etc.).
  2. Rust FFI layer (supraseal-c2/src/lib.rs): Declare the new C++ functions as extern "C" symbols, adding generate_groth16_proofs_start_c and finalize_groth16_proof with appropriate Rust signatures.
  3. Bellperson wrapper layer (bellperson/src/groth16/prover/supraseal.rs): Create Rust-safe wrappers (prove_start, prove_finish, PendingProof) that manage the opaque handle, handle errors, and present a clean interface to the rest of the Rust codebase. Message [msg 2894] sits at the very end of this chain. It is the step that takes the new types from the wrapper module and makes them visible to the outside world through the bellperson::groth16 module's public API.## What the Message Actually Does The edit targets the file bellperson/src/groth16/mod.rs, which is the module-level re-export file for the Groth16 implementation. Prior to this edit, the pub use block at lines 33–36 exported only the original (non-split) API symbols:
#[cfg(feature = "cuda-supraseal")]
pub use self::prover::supraseal::{
    alloc_gpu_mutex, free_gpu_mutex, prove_from_assignments, synthesize_circuits_batch,
    synthesize_circuits_batch_with_hint, GpuMutexPtr, SendableGpuMutex, SynthesisCapacityHint,
};

The edit added PendingProof, prove_start, and prove_finish to this export list. These three symbols were defined in the preceding message ([msg 2892]) where the assistant added them to supraseal.rs:

The Reasoning: Why This Export Matters

The assistant's decision to add these exports was driven by a clear architectural understanding of Rust's module system. The bellperson::groth16 module is the public face of the Groth16 implementation. Downstream consumers — primarily the cuzk-daemon crate's pipeline.rs and engine.rs — import from bellperson::groth16, not from the deeply nested bellperson::groth16::prover::supraseal submodule. If the new split API symbols were not re-exported at the groth16 level, the engine code would either need to add a deeper import path or, worse, the assistant would need to restructure the engine to use the old monolithic API for some paths and the new split API for others, creating an inconsistent and confusing codebase.

The export also reflects an implicit design decision: the split API is meant to replace the monolithic prove_from_assignments for the GPU worker's critical path, not coexist alongside it as a separate code path. By exporting prove_start and prove_finish at the same module level as prove_from_assignments, the assistant signals that these are first-class citizens of the proving API. The engine will be restructured to use prove_start/prove_finish exclusively in the worker loop, while prove_from_assignments may remain for non-split use cases (e.g., single-shot proving outside the worker loop).

Assumptions and Input Knowledge

To understand this message, the reader must be familiar with several pieces of context:

  1. Rust module system and re-exports: The pub use statement in mod.rs is the standard Rust idiom for making symbols from submodules available at a higher module level. Without it, the symbols are technically public but require a longer import path.
  2. The FFI boundary between Rust and C++: The prove_start and prove_finish functions in supraseal.rs call extern "C" functions declared in supraseal-c2/src/lib.rs, which in turn link to the compiled C++ CUDA code. The PendingProof struct wraps a raw pointer (*mut c_void) that represents the C++ heap-allocated groth16_pending_proof.
  3. The conditional compilation flag: All of this code is gated behind #[cfg(feature = "cuda-supraseal")], meaning it only compiles when the CUDA supraseal feature is enabled. This is a build-system concern that ensures the code doesn't break on non-CUDA platforms.
  4. The preceding refactoring work: The C++ side had already been restructured across multiple edit rounds ([msg 2863] through [msg 2886]), with careful attention to memory stability — ensuring that results, batch_add_res, and the split vectors lived in the heap-allocated handle so that references captured by threads remained valid. The assistant also made an implicit assumption: that the three new symbols are sufficient for the engine's needs. This assumption is reasonable given the design discussions in the preceding chunk, but it carries risk. If the engine later needs to inspect the pending proof's state mid-flight (e.g., to check progress or handle cancellation), the current opaque-handle design would need extension.## Mistakes, Pitfalls, and Correctness Considerations While the edit itself is straightforward, the path to this message was paved with several hard-won lessons that deserve examination. The most significant challenge was the memory-stability problem that the assistant wrestled with in messages [msg 2863] through [msg 2866]. The original monolithic generate_groth16_proofs function declared results, batch_add_res, and the split vectors as local variables on the stack. These were captured by reference in both the GPU worker threads (per_gpu) and the prep_msm_thread. When the assistant attempted to split the function, it initially tried to move these locals into the heap-allocated groth16_pending_proof handle after the GPU threads had joined but while prep_msm_thread was still running. This would have created dangling references — a classic C++ undefined-behavior trap. The solution, arrived at through iterative reasoning across several edit rounds, was to allocate the groth16_pending_proof on the heap first, before any threads were spawned, and to have all threads reference the handle's fields directly. This required restructuring the variable declarations so that pp (the handle pointer) was allocated at line 378, before the split-msm flags at lines 364–372 that reference pp->l_split_msm etc. The assistant initially got the ordering wrong ([msg 2882]), triggering a compilation error: identifier "pp" is undefined. This was a straightforward but instructive mistake — it arose because the assistant was editing in a nonlinear fashion, adding the flag aliases before ensuring the handle allocation preceded them. The mult_pippenger type mismatch ([msg 2880]) was another subtle issue. When the assistant changed tail_msm_b_g2_bases from a local std::vector<affine_fp2_t> to an alias referencing pp->tail_b_g2, the .data() method's return type changed from a non-const pointer to a const pointer (because the alias went through a const reference in some paths). The ternary expression that chose between tail_msm_b_g2_bases.data() and points_b_g2.data() then had mismatched types, confusing the template deduction. The fix — casting to const affine_fp2_t* — was simple but highlights the fragility of template-heavy C++ code when reference semantics change. These mistakes are not failures; they are the natural byproduct of a complex cross-language refactoring. The assistant's debugging process — read the error, identify the root cause, apply a targeted fix, rebuild — is a model of disciplined iterative development.

Output Knowledge Created by This Message

The immediate output of message [msg 2894] is a modified mod.rs file that re-exports three new symbols. But the knowledge created extends beyond this single file:

  1. A public API contract: The bellperson::groth16 module now publicly advertises that it supports split proving. Any code that imports from this module can now use groth16::prove_start and groth16::prove_finish. This establishes a new pattern for how GPU proving is done in the cuzk-daemon.
  2. A boundary for the engine refactoring: With the split API exported, the assistant can now proceed to restructure the GPU worker loop in engine.rs. The engine code will call prove_start at the top of the loop, spawn a tokio task for prove_finish, and immediately loop back to grab the next synthesis job. The export in mod.rs is the prerequisite for this work.
  3. A precedent for future API splits: If other GPU operations (e.g., the b_g1_msm or the L/MSM tail computations) are later identified as deferrable, the same pattern — opaque handle, start/finish pair, module-level export — can be replicated. The split API design is now a documented pattern in the codebase.
  4. A testable compilation unit: With the C++ compiling ([msg 2886]) and the Rust FFI declarations in place ([msg 2888]), the next build of cuzk-daemon will attempt to link the entire split API. Any linker errors (e.g., missing symbol definitions, ABI mismatches) will surface immediately, allowing the assistant to fix them before the engine refactoring proceeds.

The Thinking Process: From Problem to Plumb

The reasoning visible across the message sequence leading to [msg 2894] reveals a systematic, layered approach to problem-solving. The assistant begins with a clear goal: "hide b_g2_msm latency by decoupling GPU worker critical path from CPU post-processing" ([chunk 29.1]). It then works backward from this goal:

  1. Identify the bottleneck: b_g2_msm runs for ~1.7s after GPU lock release, blocking the worker.
  2. Design the solution: Split the proving function into start/finish, with a heap-allocated handle for intermediate state.
  3. Implement the C++ side: Refactor groth16_cuda.cu, dealing with memory-stability issues and type errors.
  4. Plumb the Rust FFI: Declare extern "C" functions in lib.rs.
  5. Wrap in bellperson: Create prove_start, prove_finish, PendingProof in supraseal.rs.
  6. Export from module: Add the new symbols to mod.rs — this is message [msg 2894]. Each step depends on the previous one. The assistant never jumps ahead; it builds the chain link by link. This is particularly visible in the ordering of the Rust edits: first lib.rs (the raw FFI declarations), then supraseal.rs (the safe wrappers), then mod.rs (the public export). A less disciplined approach might have added all three in a single massive edit, but the assistant's incremental method allows each step to be verified independently. The assistant also demonstrates a strong awareness of Rust's module visibility rules. The fact that it added the exports to mod.rs rather than expecting downstream code to use the full path bellperson::groth16::prover::supraseal::prove_start shows an understanding of how Rust API design works in practice: the public face of a module should be convenient and minimal.

Conclusion

Message [msg 2894] is a small edit with outsized significance. It is the capstone of the Phase 12 split API implementation, the moment when a complex, multi-layered refactoring becomes usable by the rest of the codebase. The edit itself is trivial — adding three identifiers to a pub use block — but the reasoning that led to it spans dozens of earlier messages, encompassing memory-stability analysis, C++ template debugging, FFI design, and Rust module architecture.

In the broader narrative of the cuzk-daemon optimization project, this message marks the transition from "designing and implementing the split API" to "integrating it into the engine worker loop." The next steps — restructuring engine.rs to use prove_start/prove_finish, benchmarking the throughput improvement, and iterating on any performance gaps — will build directly on the foundation laid here. The split API is not the end of the optimization journey, but it is a crucial enabler for the next phase of latency hiding and throughput improvement.

For the reader studying this session, message [msg 2894] serves as a reminder that in complex software engineering, the most important edits are often the smallest ones. A single line added to a module file can unlock an entire architectural transformation, provided the groundwork has been laid with care and precision.