The Commit That Sealed Phase 3: Documenting 1.42× Throughput in a SNARK Proving Pipeline

Introduction

In the course of a multi-month engineering effort to build cuzk — a persistent, GPU-resident SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) circuits — a single git commit marked the official close of Phase 3. Message 767 in the conversation is that commit: a one-line bash command that stages and commits updated documentation of end-to-end test results for cross-sector batching. On its surface, it is a mundane act of housekeeping. But beneath that surface lies the culmination of a systematic validation campaign spanning four distinct test scenarios, memory profiling, throughput measurement, and a careful trade-off analysis between memory consumption and proof generation throughput. This article unpacks what that commit represents, the reasoning that led to it, the assumptions it enshrines, and the knowledge it produces for the next phase of work.

The Message Itself

The subject message is a single bash invocation:

git -C /home/theuser/curio add cuzk-project.md && git -C /home/theuser/curio commit -m "docs(cuzk): Phase 3 E2E test results — batch=2 validated at 1.42x throughput

All Phase 3 E2E tests pass on RTX 5070 Ti:
- Timeout flush: BatchCollector correctly flushes after 30s wait
- Batch=2: 2 sectors synthesized as 20 circuits in 55s (same as 10),
  GPU 69s, yielding 62.7s/proof (1.42x vs baseline 89s)
- Overflow: 3 proofs with batch=2 shows correct batch+overflow+pipeline
- Non-batchable: WinningPoSt bypasses BatchCollector (0.8s total)

Memory: batch=2 peaks at 360 GiB (vs 203 GiB for single proof).
Updated roadmap table with measured numbers."

The output confirms success: [feat/cuzk 353e4c2a] docs(cuzk): Phase 3 E2E test results — batch=2 validated at 1.42x throughput. The commit hash 353e4c2a now permanently anchors this milestone in the project's history.

Context: What Phase 3 Was

To understand why this commit matters, one must understand what Phase 3 of the cuzk project set out to accomplish. The cuzk engine is designed to replace the monolithic, one-shot proof generation in Filecoin's supraseal-c2 with a persistent daemon that keeps GPU state (SRS parameters, CUDA contexts) resident across proof requests. Phase 1 established the basic daemon infrastructure and the gen-vanilla command for test data. Phase 2 introduced pipelining — splitting the proof process into a CPU-bound synthesis phase and a GPU-bound proving phase that could overlap asynchronously — and validated that the pipeline achieved a 1.27× throughput improvement over the monolithic baseline.

Phase 3 went further by implementing cross-sector batching. The key insight is that PoRep C2 proofs for multiple sectors share the same circuit structure. If two sectors' proofs are submitted concurrently, their synthesis phases can be merged: instead of synthesizing 10 constraint circuits for sector A and then 10 for sector B sequentially, the BatchCollector waits for a second request to arrive and then synthesizes all 20 circuits together in a single rayon-parallelized pass. Because CPU synthesis is already saturating all available cores at 10 circuits, adding 10 more circuits costs negligible additional wall-clock time — the synthesis completes in the same ~55 seconds whether it is 10 circuits or 20. The GPU phase, which processes circuits sequentially, does double: ~69 seconds for 20 circuits versus ~34 seconds for 10. The net effect is that two proofs complete in ~125 seconds total, yielding an amortized 62.7 seconds per proof — a 1.42× throughput improvement over the 89-second single-proof baseline.

Why This Message Was Written

The commit serves several distinct purposes, each reflecting a deliberate engineering decision.

First, it marks a clean stopping point. The conversation leading up to this message shows the assistant methodically working through a todo list: stop the baseline daemon, analyze baseline memory data, start a new daemon with max_batch_size=2, test timeout flush, test batched proofs, test overflow with 3 proofs, test non-batchable WinningPoSt, analyze memory data, compile results, and finally update the project documentation. Each test was designed to validate a specific behavioral requirement of the BatchCollector architecture. The commit crystallizes that all four requirements are met. It draws a line under Phase 3 before the assistant pivots to Phase 4 (compute-level optimizations), which begins in the very next chunk of the conversation.

Second, it creates a permanent record of empirical validation. The commit message contains specific, measurable claims: 55-second synthesis for 20 circuits, 69-second GPU time, 62.7-second amortized proof time, 360 GiB peak memory for batch=2. These numbers are not speculative — they were measured on real hardware (an RTX 5070 Ti) with real 32 GiB PoRep test data. By committing them to the project's git history alongside the documentation, the assistant ensures that future developers (including the assistant itself in later phases) can refer back to these baselines. If Phase 4 optimizations change performance characteristics, the Phase 3 numbers provide a reference point for regression detection.

Third, it communicates the trade-off explicitly. The commit message juxtaposes throughput gain (1.42×) against memory cost (360 GiB vs 203 GiB). This is a candid acknowledgment that batching is not a free lunch — it trades memory for throughput. The ~360 GiB peak for batch=2 closely matches the ~408 GiB prediction from earlier analysis (the difference being attributable to SRS overhead and measurement granularity). By documenting this trade-off in the commit itself, the assistant ensures that anyone reading the git log understands the cost of the optimization.

The Testing Campaign That Produced These Numbers

The commit message summarizes four tests, but each one reveals something about the assistant's reasoning process.

Timeout flush: The BatchCollector is designed to hold proofs until either the batch is full (2 proofs for max_batch_size=2) or a timeout expires (max_batch_wait_ms=30000). The timeout flush test submitted a single proof and measured 30,258ms before the collector flushed it. This confirms that the timeout mechanism works correctly and that proofs are not stuck indefinitely when batching partners are slow to arrive. The 258ms overshoot is attributable to scheduling granularity and is well within acceptable bounds.

Batch=2: This was the core validation. Two proofs submitted concurrently should trigger an immediate synth_batch_full flush (no timeout wait). The logs confirmed synth_batch_full{batch_size=2 proof_kind=porep-c2} and synthesize_porep_c2_multi{num_sectors=2} with total_circuits=20. The synthesis time of 55.3 seconds for 20 circuits was virtually identical to the 55.6 seconds for 10 circuits in the single-proof case — proving that synthesis cost is fully amortized. The GPU time of 69.4 seconds was approximately 2× the single-proof GPU time of 34.4 seconds, confirming that the GPU processes circuits sequentially and does not benefit from batching (nor does it regress). The output proofs were 3840 bytes total, correctly split into two 1920-byte Groth16 proofs.

Overflow (3 proofs): This test validated the interaction between batching and the async pipeline. When three proofs arrive simultaneously, the first two form a batch and begin synthesis+GPU immediately. The third proof waits in the collector until the timeout fires (~30 seconds later), at which point it is flushed as a singleton. Critically, the third proof's synthesis phase began while the batch-of-2 was still in its GPU phase, demonstrating that the async overlap pipeline (synthesis ∥ GPU, implemented in Phase 2) continues to function correctly alongside batching. The 3-proof throughput was 0.964 proofs/min, consistent with the batch=2 steady-state rate.

Non-batchable bypass (WinningPoSt): Some proof types, like WinningPoSt, have fundamentally different circuit structures and cannot be batched with PoRep C2 proofs. The BatchCollector must recognize these types and bypass the collector entirely, sending them directly to the single-proof synthesis path. The WinningPoSt test completed in 807ms total (52ms synthesis, 666ms GPU, 88ms queue), confirming that non-batchable types skip the collector with negligible overhead.

Assumptions and Trade-Offs Embedded in This Commit

The commit message, and the Phase 3 work it documents, rests on several assumptions that deserve scrutiny.

Assumption 1: The 1.42× throughput improvement is worth the memory cost. The memory jumps from 203 GiB to 360 GiB for batch=2 — a 77% increase for a 42% throughput gain. Whether this trade-off is acceptable depends on the deployment environment. On a machine with 512 GiB or 1 TiB of RAM, the trade-off is clearly worthwhile. On a machine with 256 GiB, batch=2 may not fit. The assistant implicitly assumes that the target deployment has sufficient memory, or that the user is willing to provision accordingly. The commit does not discuss memory-constrained fallbacks, though the architecture supports configuring max_batch_size=1 to disable batching entirely.

Assumption 2: The RTX 5070 Ti results generalize. All measurements were taken on a single GPU model with 32 GiB PoRep data. The assistant assumes that the relative performance characteristics — synthesis cost being fully amortized, GPU time scaling linearly with circuit count — hold across other GPU models and sector sizes. This is a reasonable assumption given that synthesis is CPU-bound (rayon thread pool) and GPU time is dominated by NTT/MSM operations that scale linearly with circuit count, but it remains untested.

Assumption 3: The BatchCollector's timeout-based flush is sufficient. The 30-second timeout was chosen arbitrarily (it was the default in the configuration). The assistant does not analyze whether 30 seconds is optimal for different workloads. A shorter timeout would reduce batching opportunities but improve latency for singleton proofs; a longer timeout would increase batching probability but delay proofs that arrive alone. The commit does not explore this parameter space.

Assumption 4: The 360 GiB peak is the correct number to report. The memory analysis revealed that the 420 GiB peak occurred during the 3-proof overflow test, where the third proof's synthesis overlapped with the batch's GPU phase. The assistant chose to report 360 GiB as the "batch=2" peak, implicitly attributing the extra 60 GiB to the overlap artifact rather than to batching itself. This is a defensible choice — steady-state batch=2 without overlap does peak at ~360 GiB — but it is an interpretation, not a raw measurement.

Input Knowledge Required to Understand This Message

A reader who encounters this commit in the git log needs substantial domain knowledge to interpret it:

Output Knowledge Created by This Message

The commit produces several forms of knowledge that persist beyond the conversation:

  1. Empirical validation of the batching hypothesis: The core claim of Phase 3 — that synthesis cost can be amortized across sectors — is now backed by measured data. This transforms the idea from a design speculation into a proven result.
  2. Baseline numbers for Phase 4: The 89-second single-proof baseline and 62.7-second batch=2 amortized time provide reference points against which future optimizations can be measured. If Phase 4's compute optimizations (SmallVec, pre-sizing, parallel B_G2, pinned memory, per-MSM window tuning) improve or regress performance, the Phase 3 numbers define the starting point.
  3. Memory accounting validation: The earlier prediction of ~408 GiB peak for batch=2 was very close to the measured 360-420 GiB range. This validates the memory model used throughout the project and increases confidence in predictions for larger batch sizes.
  4. A documented stopping criterion: The commit defines what "Phase 3 done" means: all four test scenarios passing, with documented numbers. This prevents scope creep and provides a clear handoff to Phase 4.
  5. A reusable test methodology: The four-test pattern (timeout flush, batch fill, overflow, non-batchable bypass) can be applied to test future batching changes or different batch sizes without redesigning the validation approach.

The Thinking Process Visible in the Reasoning

The assistant's thinking is visible in the structure of the commit message itself. The four test scenarios are listed in a specific order that reflects a logical progression: first verify the basic timeout mechanism works (safety), then verify the core batching path works (functionality), then verify overflow behavior with the pipeline (integration), then verify non-batchable types are unaffected (isolation). This is classic incremental validation — each test builds on the confidence established by the previous one.

The decision to include both the throughput number (1.42×) and the memory number (360 GiB vs 203 GiB) in the commit summary shows a balanced awareness that optimization is about trade-offs, not just speedups. The assistant does not present the result as an unqualified win; it presents the cost alongside the benefit.

The commit message's reference to "Updated roadmap table with measured numbers" reveals another layer of thinking: the assistant is not just dumping raw data but integrating it into the project's planning artifacts. The roadmap table (section 11 of cuzk-project.md) was updated to replace speculative estimates with actual measurements, tightening the feedback loop between design and validation.

Conclusion

Message 767 is a commit that closes a chapter. It documents the successful validation of cross-sector batching in the cuzk SNARK proving engine, with concrete numbers that confirm a 1.42× throughput improvement at the cost of increased memory consumption. But it is more than a record — it is a reasoning artifact that encodes the assistant's testing strategy, trade-off awareness, and preparation for the next phase of work. For anyone studying this conversation, the commit serves as a boundary object between Phase 3's empirical validation and Phase 4's compute-level optimization, preserving the knowledge gained from one phase to inform the next. In a project where memory consumption reaches hundreds of gigabytes and proofs must be correct down to the byte, such documentation is not a luxury — it is the difference between a prototype and a production system.