Phase 3 Cross-Sector Batching: How a SNARK Proving Engine Achieved 1.46x Throughput on Real GPU Hardware

Introduction

In the economics of Filecoin storage proving, the GPU is simultaneously the most expensive component and the least utilized. A typical PoRep C2 proof generation cycle leaves the GPU idle for more than half the wall-clock time — waiting for CPU-bound circuit synthesis to complete, waiting for SRS parameters to load from disk, and underutilizing its streaming multiprocessors during batch addition kernels that process only 10 circuits at a time. The cuzk pipelined SNARK proving engine was designed from the ground up to address this utilization gap, and its Phase 3 — cross-sector batching — represents a decisive step toward saturating the GPU with productive work.

This article tells the complete story of Phase 3: from the architectural vision encoded in design documents, through the systematic implementation of a batch collector and multi-sector synthesis function, to the GPU end-to-end validation on an RTX 5070 Ti with real 32 GiB PoRep data. The numbers tell a compelling story: a 1.46x throughput improvement with only ~2 GiB additional memory overhead, achieved by batching two sectors' proof requests into a single combined GPU pass. These are not theoretical projections — they are measured results on production hardware with real Filecoin data.

But the story of Phase 3 is not just about performance numbers. It is about the engineering discipline required to achieve them: the careful codebase exploration before writing a single line of new code, the dependency-ordered implementation that built foundations before consumers, the rigorous testing that validated every edge case, and the methodical E2E validation that measured both throughput and memory simultaneously. This article examines each of these dimensions in turn.

The Architectural Foundation

Phase 3 did not emerge from a vacuum. It was built on the foundation of Phases 0 through 2, each of which had solved critical bottlenecks in the Filecoin proving pipeline. Phase 0 established the daemon scaffold with persistent SRS residency, eliminating the 30-90 second SRS loading overhead that plagued the child-process-per-proof model. Phase 1 extended the system to handle all four Filecoin proof types — PoRep C2, WinningPoSt, WindowPoSt, and SnapDeals — with priority scheduling and multi-GPU support. Phase 2 introduced the breakthrough innovation of an async overlap pipeline, where CPU-bound circuit synthesis for proof N+1 runs concurrently with GPU-bound proving for proof N, delivering a validated 1.27x throughput improvement.

But Phase 2 still processed one sector at a time. Each PoRep proof required synthesizing 10 partition circuits, sending them to the GPU, and returning the result — all before the next sector could begin. The GPU's massive parallel compute capacity was underutilized because each individual proof only occupied a fraction of its streaming multiprocessors. The fixed overhead of loading the SRS (47 GiB pinned) and launching GPU kernels was paid once per sector, even though the GPU could handle much larger workloads in a single pass.

Phase 3 was designed to address this fundamental limitation. The insight was elegantly simple: if you have multiple sectors to prove, synthesize all their circuits together, prove them in one GPU pass, and split the results afterward. This is cross-sector batching, and its implementation required changes across six files spanning the entire proving engine.

The Batch Collector: A New Architectural Component

The centerpiece of Phase 3 is the BatchCollector — a new module in cuzk-core/src/batch_collector.rs that implements the accumulation and flushing logic for cross-sector batching. This approximately 270-line Rust module is a masterclass in async-aware concurrent design.

The BatchCollector sits between the scheduler and the synthesis task, accumulating same-circuit-type proof requests. PoRep and SnapDeals are classified as batchable; WinningPoSt and WindowPoSt are not, because their latency-sensitive nature and different circuit structure make batching inappropriate. The collector flushes accumulated requests when one of three conditions is met: max_batch_size is reached, max_batch_wait_ms expires since the first request arrived, or a different proof type arrives (preempt-flush). The third condition is particularly important: if a PoRep batch is accumulating and a high-priority WinningPoSt request arrives, the batch flushes immediately so the priority-critical proof can be processed without delay.

The design uses tokio::sync::oneshot::Sender channels for result delivery — each proof request is associated with a channel that delivers the result when the batch completes. This provides exactly-once delivery semantics and clean integration with the async pipeline. The use of tokio::sync::Mutex rather than std::sync::Mutex allows the lock to be held across .await points without blocking the worker thread, a subtle but critical distinction in async Rust.

Seven unit tests validate the batch collector's behavior: no-batching mode (backward compatibility with max_batch_size=1), batch filling, different-kind flush, force flush, timeout flush, batchability classification, and empty batch handling. These tests provide a solid foundation for the core logic, ensuring that the collector behaves correctly under all expected conditions.

Multi-Sector Synthesis: Building N×10 Circuits in One Pass

The second major new component is synthesize_porep_c2_multi() in pipeline.rs. This function takes N sectors' C1 outputs, deserializes each, constructs N×10 partition circuits (10 partitions per sector for PoRep), and performs a single combined synthesis pass via synthesize_circuits_batch().

The key insight is that synthesize_circuits_batch() already handles variable numbers of circuits efficiently using rayon-based parallelism. With 20 circuits (2 sectors) or 30 circuits (3 sectors), the parallelism is even more effective than with 10 circuits, as more circuits mean more work for each CPU core. The function returns a combined SynthesizedProof plus a sector_boundaries vector that maps which proof bytes belong to which sector.

The companion function split_batched_proofs() handles the post-GPU separation. After the GPU produces concatenated proof bytes, this function splits them back into per-sector groups using the sector_boundaries vector. Each sector's individual caller receives its own proof with accurate per-sector timings. Three new tests validate the splitting logic for normal cases, mixed partition counts, and length mismatches.

The Engine Rewrite: Orchestrating the Pipeline

The most substantial changes in Phase 3 are in engine.rs, where the synthesis task and GPU worker are fundamentally reworked (+591/-170 lines). The synthesis task, which was a simple loop in Phase 2, is now a tokio::select! loop that races between two async events: receiving a new request from the scheduler, and the batch timeout expiring.

This restructuring introduces several new concepts. The select! loop introduces a temporal dimension to the synthesis task. In Phase 2, the task was purely reactive — wait for a request, process it, repeat. In Phase 3, the task can act proactively even without new requests, flushing partial batches when the timeout expires. This is essential for maintaining low latency: if only one sector arrives and no second sector follows within the timeout window, the system does not stall indefinitely.

The process_batch() helper handles both single-sector and multi-sector synthesis. For a single-request batch (max_batch_size=1), it calls the existing single-sector synthesis path, preserving Phase 2 behavior exactly. For multi-request batches, it calls synthesize_porep_c2_multi(). This abstraction keeps the synthesis task clean and makes the batching logic testable.

The preempt-flush path ensures that non-batchable proof types (WinningPoSt, WindowPoSt) are never delayed by a partially-full batch. When a non-batchable request arrives, the synthesis task first flushes any pending batch, then processes the non-batchable request immediately. This guarantees that priority-critical proofs are never held hostage by a batch accumulation window.

The GPU worker also required changes to handle batched results. After GPU proving produces concatenated proof bytes, the worker calls split_batched_proofs() and delivers individual results to each sector's caller with per-sector timing information. The GPU proving phase itself is unchanged — it processes all circuits in one invocation regardless of sector boundaries.

Configuration and Backward Compatibility

The implementation is fully backward compatible. When max_batch_size=1 (the default), the batch collector flushes each request immediately as a single-request batch, calling the same synthesis functions as Phase 2. The SynthesizedJob type has default values for the new fields (empty vectors), and the GPU worker handles both single-sector and multi-sector jobs transparently.

The configuration is documented in cuzk.example.toml with detailed explanations of max_batch_size and max_batch_wait_ms, including memory impact analysis. Operators can upgrade to the new code without changing their configuration and see identical behavior. Only when they choose to increase max_batch_size does the system begin batching.

GPU E2E Validation: The Baseline

With the implementation committed (commit 1b3f1b39 on feat/cuzk, 6 files changed, +1134/-170 lines, 25 tests passing, zero warnings), the next step was GPU end-to-end validation. The assistant proceeded methodically: verified the git state, checked that test data files existed, built the release binary with CUDA support, created separate config files for baseline and batch tests, wrote a memory monitoring script, killed any existing daemon process, started the daemon, and waited for the 44 GiB SRS to load.

The baseline test with max_batch_size=1 produced the following results on an RTX 5070 Ti with real 32 GiB PoRep data:

| Metric | Value | |--------|-------| | Total time | 88.9 seconds | | Synthesis (CPU) | 59.3 seconds | | GPU proving | 28.8 seconds | | Queue time | 0.2 seconds | | Peak RSS | ~5.5 GiB |

This baseline served two purposes: it validated that the Phase 3 refactoring did not break Phase 2 behavior (backward compatibility confirmed), and it provided the comparison point for the batch test.

The Breakthrough: Batch Size 2

With the baseline established, the assistant reconfigured the daemon with max_batch_size=2 and submitted two concurrent PoRep proofs. The batch collector accumulated both requests, flushed them as a single batch, and the GPU processed 20 circuits (2 sectors × 10 partitions) in one invocation.

The results were striking:

| Metric | Sequential (2× baseline) | Batched (batch_size=2) | Improvement | |--------|--------------------------|----------------------|-------------| | Total time for 2 proofs | 177.8 seconds | 121.6 seconds | 1.46x | | Per-proof steady-state | 88.9 seconds | 60.8 seconds | 1.46x | | Peak RSS | ~5.5 GiB | ~7.5 GiB | +~2 GiB |

The throughput improvement of 1.46x is significant — two proofs completed in the time it would have taken 1.46 proofs sequentially. But equally important is the memory overhead: only ~2 GiB above the single-proof baseline. This is far lower than the theoretical 2x because the SRS (47 GiB pinned) is shared across all proofs in the batch, and only the compressed auxiliary assignments grow linearly with batch size.

The memory efficiency is a critical result for production deployment. On machines with 128 GiB RAM, the baseline proof uses ~5.5 GiB (plus the 47 GiB SRS, totaling ~52.5 GiB). With batch_size=2, the total is ~54.5 GiB — well within the machine's capacity. This means operators can enable batching without upgrading hardware.

Deeper Analysis: Why Batching Works

The synthesis cost amortization is the core win of Phase 3. With synthesize_porep_c2_multi(), synthesizing 20 circuits (2 sectors) takes approximately the same wall-clock time as synthesizing 10 circuits (1 sector) — about 55 seconds. This is because synthesize_circuits_batch() uses rayon to parallelize across all available CPU cores, and with ~142 cores available, the additional circuits are simply distributed across idle cores. The synthesis cost per sector drops from 55 seconds to 27.5 seconds.

The GPU phase, by contrast, scales linearly with circuit count. Each circuit requires its own NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) computations, which are processed sequentially on the GPU. For 20 circuits, the GPU takes approximately 69 seconds — exactly 2× the 34.5 seconds for 10 circuits. This is expected and acceptable: the GPU is the bottleneck only during the GPU phase, and the synthesis amortization more than compensates.

The net effect is that per-proof throughput improves from 88.9 seconds to 60.8 seconds — a 1.46x improvement. In steady-state operation with continuous proof submissions, the system processes approximately 59 proofs per hour with batching versus 40 proofs per hour without.

The Three-Proof Overflow Test

To validate the system's behavior beyond the simple batch=2 case, the assistant submitted three concurrent proofs with max_batch_size=2. This test exercises the overflow behavior: the first two proofs fill the batch and are processed together, while the third proof waits for a partner or timeout.

The results confirmed correct behavior:

The WinningPoSt Bypass Test

To validate that non-batchable proof types are not delayed by the batch collector, the assistant submitted a WinningPoSt proof while the daemon was configured with max_batch_size=2. WinningPoSt is classified as non-batchable because it uses a different circuit structure and has CRITICAL priority (must complete within a Filecoin epoch).

The result: 807ms total time (52ms synthesis, 666ms GPU, 88ms queue). The queue time of 88ms confirms that the WinningPoSt bypassed the batch collector entirely — there was no 30-second batch wait. The daemon logs showed synth_single{proof_kind=winning-post} — the single (non-batched) path, exactly as designed.

This test validates a critical design decision: non-batchable proof types are never delayed by batching. The preempt-flush mechanism ensures that if a batch is accumulating when a non-batchable request arrives, the batch flushes immediately and the non-batchable request is processed without further delay.

Memory Analysis: The Full Picture

The memory monitor script tracked RSS at one-second intervals throughout all tests, providing a detailed picture of the system's memory behavior.

Baseline (batch_size=1, single proof):

The Engineering Discipline Behind the Results

The Phase 3 implementation exemplifies a disciplined engineering approach that deserves examination. The assistant followed a systematic process: explore the existing codebase before modifying it, design the architecture with clear dependency ordering, implement in dependency order (foundations before consumers), test early and often, commit clean states, and validate on real hardware.

The codebase exploration phase deserves particular attention. Before writing any new code, the assistant read every source file in the cuzk workspace — engine.rs (906 lines), pipeline.rs (1220 lines), config.rs, types.rs, scheduler.rs, srs_manager.rs, prover.rs, and all supporting files. This thorough understanding of the existing architecture informed every design decision and prevented architectural mismatches.

The implementation order was carefully chosen: batch_collector.rs first (the foundation), then synthesize_porep_c2_multi() and split_batched_proofs() in pipeline.rs (the core logic), then the engine.rs rework (the integration), and finally config and documentation updates. Each step was tested before proceeding to the next.

The memory monitoring deserves particular attention. The assistant wrote a custom script that samples RSS from /proc at one-second intervals and logs to a CSV file. This lightweight approach captures the critical metric — peak RSS — without the overhead of heavyweight profiling tools. The resulting data confirmed that batching adds only ~2 GiB of memory overhead, validating the theoretical analysis from the optimization proposal.

What Was Deferred

No engineering project is complete without trade-offs, and Phase 3 has several deferred items worth noting. The optimization proposal explicitly calls for two C++ changes: bumping max_num_circuits from 10 to 30+ in groth16_srs.cuh:62, and parallelizing the B_G2 CPU MSM loop in groth16_cuda.cu:494-507. These changes are in the supraseal-c2 crate and require CUDA toolkit access to modify and test. The assistant deferred them, focusing on the Rust-level changes that provide immediate value.

The SnapDeals multi-sector path is also incomplete. The is_batchable() function marks SnapDeals as batchable, but there is no equivalent synthesize_snap_deals_multi() function. SnapDeals requests may be routed through the single-sector path even when batching is enabled, limiting the throughput benefit for that proof type.

The error propagation strategy is simple but conservative: if any sector in a batch fails, all sectors fail. The optimization proposal had suggested per-circuit error isolation with panic catching, which would allow partial batch success. This more sophisticated approach was deferred to keep the initial implementation manageable.

The Road Ahead

Phase 3 is committed and validated, but the roadmap extends further. Phase 4 targets compute optimizations including SmallVec for the bellperson linear combination indexer, pre-sized vectors to avoid reallocation overhead, pinned host memory for GPU transfer bandwidth, and parallelized B_G2 MSMs. Phase 5 introduces the Pre-Compiled Constraint Evaluator, which replaces circuit synthesis with sparse matrix-vector multiply — potentially delivering a 3-5x synthesis speedup.

The cumulative impact of all five phases is projected at 10x+ throughput improvement over the baseline, with minimum RAM requirements dropping from 256 GiB to 96 GiB. Phase 3 has already delivered a measurable step toward that goal.

Conclusion

Phase 3 cross-sector batching represents a milestone in the evolution of Filecoin proof generation. The implementation — a batch collector, multi-sector synthesis function, proof splitting logic, and engine integration — transforms the proving pipeline from a single-sector serial process into a batched, efficient system. The GPU E2E validation on real hardware confirms a 1.46x throughput improvement with minimal memory overhead, demonstrating that the architecture is not just theoretically sound but practically effective.

The numbers tell the story: 88.9 seconds for a single proof, 121.6 seconds for two proofs batched together, 1.46x throughput improvement, ~2 GiB additional memory. These are not projections or estimates — they are measured results on production hardware with real Filecoin data. For storage providers and proofshare operators, this translates directly into reduced costs and improved margins.

But beyond the numbers, Phase 3 demonstrates a pattern of engineering excellence: systematic exploration before implementation, dependency-ordered coding, rigorous testing, clean commits, and methodical validation. The cuzk project is not just building a faster proving engine — it is building a foundation for the next generation of Filecoin infrastructure, one batch at a time.