Benchmarking the Partitioned Pipeline: Memory–Throughput Trade-offs in SNARK Proving

Introduction

In the course of developing a high-performance SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol, the engineering team had just completed a significant architectural redesign: replacing the "slotted pipeline" with a new "partitioned pipeline" for the cuzk proving engine. Message [msg 1770] captures the moment when that redesign meets reality—the assistant checks the benchmark results of the newly implemented pipeline by grepping the summary table from a previous tool output. The message itself is deceptively simple: a single bash command that extracts a table of results. But the table it reveals tells a nuanced story about parallelism, memory efficiency, and the often-surprising behavior of complex GPU-accelerated pipelines.

This article examines that message in depth: why it was written, what decisions it reflects, the assumptions embedded in the benchmark design, and the insights that the results provide about the partitioned pipeline's performance characteristics.

Context: The Pipeline Redesign

To understand the significance of message [msg 1770], we need to understand what came before it. The cuzk proving engine processes Filecoin PoRep proofs by splitting the work into two phases: synthesis (CPU-bound circuit construction) and GPU proving (the heavy cryptographic computation). A single proof involves 10 "partitions," each requiring substantial computation and memory.

The earlier "slotted pipeline" (Phase 6) had grouped partitions into slots, where each slot contained multiple partitions synthesized together and then proved on the GPU. But this design had limitations: the slot_size parameter controlled how many partitions were grouped, and the interaction between slot grouping and GPU call characteristics was suboptimal. Specifically, when multiple circuits were passed to a single GPU call (num_circuits >= 2), the b_g2_msm operation—normally fast in multi-threaded CPU mode (~0.4s)—became slow in single-threaded GPU dispatch mode (~23s).

The redesign, implemented across messages [msg 1741] through [msg 1768], replaced the slotted pipeline with a partitioned pipeline that always processes one partition per GPU call. The key parameters shifted from slot_size to max_concurrent_slots, which controls how many synthesized partitions can be in-flight simultaneously (bounded by channel capacity). The design uses a producer-consumer architecture: multiple synthesis workers produce partitions in parallel via rayon, sending results through a bounded channel to a single GPU consumer thread. The channel backpressure naturally limits memory usage.

After implementing the new pipeline and verifying that both cuzk-bench and cuzk-daemon built successfully (messages [msg 1766] and [msg 1767]), the assistant ran the benchmark in message [msg 1769]:

RUST_LOG=info FIL_PROOFS_PARAMETER_CACHE=/data/zk/params \
  /home/theuser/curio/extern/cuzk/target/release/cuzk-bench \
  slotted-bench --c1 /data/32gbench/c1.json --slot-sizes "1,2,3,10"

The output was truncated, so in message [msg 1770], the assistant retrieves the summary table by grepping the tool output file.

The Benchmark Results

The summary table extracted in message [msg 1770] is the centerpiece of the analysis:

========== Summary ==========
max_concur    total_s  synth_sum  gpu_sum    gpu_%  overlap peak_GiB  proof_B
------------------------------------------------------------------------------------
1                72.0      351.4     38.6      54%    5.42x     71.3     1920
2                72.7      353.1     38.0      52%    5.38x     86.8     1920
3                71.9      347.8     38.2      53%    5.37x     86.8     1920
10(batch)        62.3       36.2     25.5      41%    0.99x    228.5     ...

Each row represents a different max_concurrent configuration:

Surprising Findings and Their Implications

The most striking result is that max_concurrent=1 performs identically to max_concurrent=3 (~72s total time), while using 18% less memory (71.3 GiB vs 86.8 GiB). This contradicts the intuition that allowing more in-flight partitions would improve throughput by keeping the GPU better fed. The data suggests that with max_concurrent=1, the pipeline already achieves sufficient parallelism to keep the GPU busy most of the time, and adding more concurrent slots doesn't help because the bottleneck lies elsewhere.

The overlap metric of 5.4x for the partitioned pipeline versus 0.99x for batch mode is revealing. It confirms that the partitioned pipeline achieves substantial parallelism between synthesis and GPU proving—synthesis of one partition overlaps with GPU proving of another. In batch mode, there is essentially no overlap: all synthesis completes before any GPU work begins, resulting in a sequential execution pattern.

However, the partitioned pipeline is ~15% slower than batch mode (72s vs 62.3s). This is the cost of finer-grained pipelining: individual partition synthesis is less efficient than batched synthesis (35s per partition vs 3.6s for the batch), and the overhead of per-partition GPU calls adds up (38.6s total GPU time vs 25.5s for the batch). The batch mode benefits from GPU-side optimizations when proving multiple circuits simultaneously.

The memory trade-off is dramatic: the partitioned pipeline at max_concurrent=1 uses 71.3 GiB, compared to 228.5 GiB for batch mode—a 3.2× reduction. This is the primary value proposition of the partitioned design. For deployments with memory constraints (e.g., cloud instances with 128 GiB RAM), the partitioned pipeline makes it feasible to run the proving engine where batch mode would require prohibitively expensive hardware.

Assumptions and Potential Misconceptions

The benchmark design reveals several assumptions:

  1. That varying max_concurrent would show a throughput sweet spot. The assumption was that some intermediate value of max_concurrent would balance memory pressure against GPU starvation. The results show this assumption was incorrect: throughput is flat across max_concurrent=1,2,3, suggesting the bottleneck is not GPU starvation but rather the serial portions of the pipeline or synthesis efficiency.
  2. That the partitioned pipeline could match batch throughput. The 15% gap between partitioned and batch modes suggests that the per-partition overhead is significant. The synthesis of individual partitions is less efficient than batched synthesis, and per-partition GPU calls have overhead that batched calls avoid.
  3. That GPU utilization would increase with more concurrent slots. The GPU utilization hovers around 52-54% regardless of max_concurrent, suggesting that the GPU is not the primary bottleneck. Instead, synthesis throughput appears to be the limiting factor.
  4. That the slot-sizes parameter name could be reused. The benchmark command still uses --slot-sizes even though the parameter has been renamed to max_concurrent_slots in the configuration. This is a minor naming inconsistency that could cause confusion.

The Thinking Process Visible in the Message

Message [msg 1770] shows the assistant in a diagnostic mode. The previous command's output was truncated, so the assistant retrieves the full results by grepping the tool output file. This is a practical debugging technique: when terminal output is too long, the assistant knows that opencode saves tool output to a file and can be accessed later.

The assistant's choice of grep -A 30 (showing 30 lines after the match) indicates an expectation that the summary table is approximately 30 lines long, including header and separator. The path /home/theuser/.local/share/opencode/tool-output/tool_c6f486cce001YSEQCglbWd16GT reveals the internal file naming convention of the opencode environment.

The message does not include any analysis or commentary—it simply presents the raw data. This suggests that the assistant is in a data-gathering phase, collecting results before proceeding to interpretation. The subsequent messages in the conversation would likely analyze these results and draw conclusions about the pipeline's performance.

Conclusion

Message [msg 1770] captures a pivotal moment in the optimization of a complex GPU-accelerated proving pipeline. The benchmark results reveal that the partitioned pipeline achieves its primary design goal—significant memory reduction (3.2×) compared to batch mode—but at the cost of 15% lower throughput. The surprising finding that max_concurrent=1 performs identically to higher values while using less memory simplifies the configuration: for memory-constrained deployments, the optimal setting is simply max_concurrent=1.

The results also highlight a fundamental tension in the pipeline design: the partitioned approach trades synthesis efficiency (which is optimal when all partitions are processed together) for memory efficiency (which benefits from streaming partitions one at a time). The 5.4× overlap metric confirms that the pipeline successfully parallelizes synthesis and GPU work, but the overall throughput is still limited by the serial portions of the pipeline and the overhead of per-partition processing.

These findings would inform the next phase of optimization: either reducing synthesis time further (already near-minimum with the Pre-Compiled Constraint Evaluator), or exploring more radical architectural changes such as running multiple synthesis tasks in parallel across separate GPU streams.