The Split API Decision: Decoupling GPU Workers from CPU Post-Processing in Phase 12

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) generation, every millisecond counts. The SUPRASEAL_C2 pipeline, a CUDA-accelerated Groth16 proving engine, had been through eleven phases of optimization — from memory-bandwidth interventions to GPU interlock refinements — each shaving seconds off the proof generation time. By Phase 11, the pipeline was producing a proof every ~36.7 seconds, a 3.4% improvement over baseline. But the optimization team had identified a persistent structural inefficiency: the GPU worker thread, after releasing its lock on the GPU, was spending ~1.7 seconds running a CPU-bound operation called b_g2_msm before it could loop back and pick up the next synthesis job. Message 2854 captures the exact moment when the assistant, having thoroughly analyzed the dependency chain, begins implementing the Phase 12 split API — a design that would decouple the GPU worker's critical path from CPU post-processing by returning an opaque "pending proof" handle and deferring finalization to a separate thread.

The Optimization Context: Eleven Phases of Iterative Refinement

To understand why message 2854 matters, one must appreciate the journey that led to it. The SUPRASEAL_C2 pipeline is a cross-language beast: Go-based Curio orchestration calls into Rust FFI, which in turn invokes C++ CUDA kernels for the heavy lifting of Groth16 proof generation. Each proof requires synthesizing circuit assignments, running multi-scalar multiplications (MSMs) and number-theoretic transforms (NTTs) on the GPU, then performing CPU-side post-processing to compute the final proof elements.

The optimization effort had been methodical. Phase 8 introduced a dual-worker GPU interlock that improved throughput by 13-17%. Phase 9 optimized PCIe transfers, yielding 14.2% improvement in single-worker mode. Phase 10 attempted a two-lock architecture but was abandoned after discovering fundamental CUDA device-global synchronization conflicts. Phase 11 implemented three memory-bandwidth interventions — serializing async deallocations, reducing thread pool sizes, and adding a global atomic throttle — achieving a 3.4% improvement to 36.7 seconds per proof.

But throughout these phases, a nagging question persisted: could the GPU worker be unblocked faster? The worker's cycle was: acquire GPU lock → run GPU kernels (~1.8s for one partition) → release lock → run b_g2_msm (~1.7s) → run epilogue → loop back for next job. The 1.7 seconds of b_g2_msm after GPU unlock meant the worker couldn't pick up the next synthesis job until it finished. With per-partition proving producing partitions at 3-5 second intervals, every fraction of a second saved in the worker's cycle time reduced the risk of GPU idle gaps.

The Critical Realization: Per-Partition Proving Changes the Calculus

The breakthrough came in message 2839, where the assistant walked through the timing analysis. With gw=2 (two GPU workers) and num_circuits=1 (one partition per call), the GPU kernel time was ~1.8 seconds per call, not the ~18 seconds for all ten partitions. This meant the worker cycle was:

Worker A: [lock][GPU 1.8s][unlock][b_g2 1.7s][epilogue][loop]
Worker B:                 [lock][GPU 1.8s][unlock][b_g2 1.7s][epilogue][loop]

Worker A unlocks at t=1.8s, Worker B immediately acquires the lock. Worker A finishes b_g2_msm at t=3.5s, while Worker B unlocks at t=3.6s. The 1.7s of b_g2_msm is partially hidden behind Worker B's GPU time, but it still adds 1.7s to Worker A's cycle before it can pick up the next job. If synthesis hasn't produced a new partition by the time the worker loops back, the GPU sits idle. Cutting b_g2_msm from the critical path would let the worker loop ~1.7s faster, reducing GPU idle gaps and improving overall throughput.

The user confirmed this analysis and chose the "fire-and-forget in C++" approach: spawn a detached thread for b_g2_msm + epilogue, return an opaque handle, and let a finalizer thread complete the proof while the GPU worker immediately picks up the next job.

Message 2854: The Implementation Begins

Message 2854 is the turning point where analysis becomes implementation. The assistant writes:

Now I have the complete picture. Let me implement the split API. The key insight is that the pending proof handle needs to own:

>

1. The prep_msm_thread (still running b_g2_msm) 2. results (GPU writes h/l/a/b_g1, b_g2_msm writes b_g2) 3. batch_add_res (from GPU) 4. vk pointer (from SRS — lives forever) 5. l_split_msm, a_split_msm, b_split_msm flags 6. num_circuits 7. Copy of r_s, s_s (32 bytes per circuit) 8. The dealloc data (split_vectors + tail_msm_bases) — moved here 9. caught_exception flag 10. tv_func_entry for timing

Then it reads the C++ source file from line 1 to begin the actual code changes.

This message is deceptively short — just a list of ten items and a file read — but it represents the culmination of extensive analysis across multiple prior messages. The assistant had read the epilogue code (msg 2842), identified the msm_results and batch_add_results types (msg 2843-2847), examined the verifying_key structure (msg 2850-2851), and traced the full dependency chain of what the epilogue needs. The ten-item list is a precise inventory of every piece of state that must survive beyond the GPU worker's critical path.

The Thinking Process: What the List Reveals

The list in message 2854 is not arbitrary — it's the result of careful reasoning about ownership and lifetime. Each item represents a design decision:

Item 1: The prep_msm_thread. The assistant recognizes that b_g2_msm runs in a separate thread spawned during prep_msm. This thread is still running when the GPU worker returns from the GPU phase. The handle must own the thread object so the finalizer can join it.

Items 2-3: results and batch_add_res. These are the outputs from GPU computation (MSM results and batch addition results). They're written by GPU kernels and read by the epilogue. Moving them into the handle ensures they survive until finalization.

Item 4: The vk pointer. The verifying key lives in the SRS (Structured Reference String), which is loaded once and persists for the lifetime of the process. The assistant notes "lives forever" — this is an assumption that the SRS is not unloaded while pending proofs exist. It's a safe assumption given the daemon architecture.

Items 5-6: Flags and circuit count. These are small scalar values computed during prep_msm that control the epilogue's behavior. They must be captured because they determine which split-MSM paths to execute.

Item 7: Copy of r_s, s_s. This is a critical ownership decision. The randomness values r_s and s_s are owned by Rust (allocated as Vec<Fr>). If the GPU worker returns the handle to a finalizer thread and then drops its own references, those Rust buffers would be freed. The assistant explicitly notes "32 bytes per circuit" — with num_circuits=1, this is a trivial copy that eliminates the lifetime problem entirely. This is a pragmatic choice: rather than engineering a complex ownership transfer across the FFI boundary, just copy the two 32-byte scalars into the C++ handle.

Item 8: Dealloc data. The split vectors and tail MSM bases are large allocations that must be freed after the proof is complete. Moving them into the handle ensures they're cleaned up by the finalizer, not leaked when the GPU worker returns early.

Items 9-10: Exception flag and timing. These are diagnostic state that must be preserved for error reporting and performance measurement.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this design:

  1. The vk pointer remains valid indefinitely. This is reasonable for a daemon process that loads SRS once at startup, but it's worth noting — if the SRS were ever reloaded while a pending proof existed, the pointer would dangle.
  2. Copying r_s and s_s is sufficient. The epilogue reads these values from the handle's copy rather than from the original Rust buffers. This is safe because the epilogue only reads them — it doesn't modify them or pass them back to Rust.
  3. The prep_msm_thread can be safely moved into the handle. In C++, std::thread is movable but not copyable. Moving it into the handle transfers ownership, and the finalizer joins it. This is correct as long as the thread is joinable (not already joined or detached).
  4. The GPU worker can return before b_g2_msm completes. This is the entire point of the split — the worker doesn't wait for the thread to finish. But it means the handle must be valid for the duration of the thread's execution. The assistant ensures this by moving all necessary state into the handle. One potential mistake is not explicitly accounting for the provers (the circuit assignment vectors a, b, c). These are uploaded to the GPU during kernel execution and are not needed after the GPU kernels complete. The existing dealloc mechanism handles them separately, so they don't need to be in the pending proof handle. The assistant correctly excludes them from the list.

Input Knowledge Required

To fully understand message 2854, one needs:

  1. The Groth16 proving protocol. Knowledge of how MSM results (h, l, a, b_g1, b_g2) combine with randomness (r, s) and verifying key elements (delta_g1, alpha_g1, etc.) to produce the final proof.
  2. The SUPRASEAL_C2 architecture. Understanding that the pipeline is split across Go/Rust/C++/CUDA, with FFI boundaries at each layer. The groth16_cuda.cu file is the C++ entry point that orchestrates GPU kernels and CPU post-processing.
  3. CUDA GPU programming concepts. Understanding that GPU kernels are asynchronous, that MSM results must be copied back to host memory, and that b_g2_msm is a CPU-side MSM (not a GPU kernel).
  4. The optimization history. Knowing that Phase 11 achieved 36.7s/proof, that the GPU worker cycle is ~1.8s GPU + ~1.7s b_g2_msm, and that the goal is to hide the 1.7s behind other work.
  5. C++ threading and ownership semantics. Understanding std::thread ownership, move semantics, and the importance of joining threads before destruction.
  6. Rust FFI patterns. Knowing how opaque pointers (the handle) are passed across the C/Rust boundary, and how Rust manages ownership of resources that live beyond a single function call.

Output Knowledge Created

Message 2854 creates the blueprint for the Phase 12 implementation. The ten-item list is the specification for the groth16_pending_proof struct that will be allocated in C++, populated during the GPU phase, and consumed by the finalizer. The subsequent file read ([read] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu) begins the actual code modification — the assistant is loading the file to insert the new struct definition and the two new FFI functions (generate_groth16_proofs_start_c and finalize_groth16_proof).

The knowledge created by this message flows downstream into:

The Broader Significance

Message 2854 exemplifies a pattern that recurs throughout high-performance systems optimization: the moment when you realize that the bottleneck isn't raw compute speed but structural coupling between pipeline stages. The GPU worker was fast at its primary job (running GPU kernels), but it was structurally coupled to a CPU post-processing step that delayed its return to the dispatch loop. The split API is a classic latency-hiding technique — instead of making b_g2_msm faster (which would be difficult — it's already a highly optimized MSM), make it invisible to the critical path.

The assistant's approach is methodical: first understand the full dependency chain (messages 2842-2853), then enumerate every piece of state that must survive the split (message 2854), then implement the struct and functions (subsequent messages). The ten-item list is a precise, minimal inventory — nothing extraneous, nothing missing. Each item was verified against the actual source code, not assumed from high-level knowledge.

This message also highlights the challenges of cross-language optimization. The split API must work across C++ (where the thread and GPU state live), Rust (where the FFI boundary and worker orchestration live), and the application layer (where the engine loop lives). The handle is an opaque pointer from C++'s perspective, but Rust must manage its lifetime carefully — ensuring it's not dropped before the finalizer completes, and that the underlying C++ memory is properly freed.

Conclusion

Message 2854 is a pivotal moment in the SUPRASEAL_C2 optimization journey. It transforms the abstract insight "we should hide b_g2_msm latency" into a concrete, implementable design. The ten-item list is the bridge between analysis and code — a precise specification of what the pending proof handle must own, derived from careful reading of the epilogue code and understanding of ownership semantics across the C++/Rust boundary. The subsequent implementation would restructure the GPU worker loop, add new FFI functions, and ultimately achieve the goal of decoupling GPU execution from CPU post-processing. In the broader narrative of systems optimization, this message captures the critical transition from "what should we do?" to "here's exactly how we'll do it."