The Benchmark That Defined a Memory Scaling Law

A Single Bash Command That Mapped the Frontier of Practical Groth16 Proving

In the sprawling investigation of the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, there comes a moment where intense engineering gives way to systematic characterization. Message <msg id=3300> is that moment — a single bash command launching a benchmark, yet one that sits at the inflection point between months of optimization work and the production deployment guidance that would follow. The message itself is deceptively simple:

/tmp/cuzk-lowmem-bench.sh /tmp/cuzk-lowmem-pw5-gw2.toml lowmem-pw5-gw2 5 5

This command starts a benchmark of the cuzk proving engine configured with partition_workers=5 and gpu_workers_per_device=2 (pw=5 gw=2), running five proofs at concurrency five. To understand why this single invocation matters, we must trace the reasoning that led to it, the assumptions it encodes, and the knowledge it would produce.

The Context: From Optimization to Characterization

The message belongs to Segment 32 of the opencode session, a phase the analyzer summary describes as a shift "from raw optimization to production readiness and configuration characterization." The preceding segments — 27 through 31 — were a whirlwind of architectural engineering. Phase 9 had identified PCIe optimization bottlenecks. Phase 10 attempted a two-lock GPU interlock design but was abandoned after discovering fundamental CUDA device-global synchronization conflicts. Phase 11 introduced three targeted interventions to reduce DDR5 memory bandwidth contention. Phase 12 implemented a split GPU proving API that decoupled b_g2_msm latency from the critical path, along with a memory backpressure system involving early deallocation of evaluation vectors, channel capacity auto-scaling, and a partition semaphore permit-through-send fix.

By the time we reach <msg id=3300>, the assistant has just completed a major documentation consolidation. The complex Phase 12 architecture was formally recorded in cuzk-project.md, the example configuration file cuzk.example.toml was updated with optimal defaults (gpu_threads=32, partition_workers=12), and all changes were committed as 9bb657e5 on the feat/cuzk branch. The engine worked, and it worked well — at pw=12 gw=2, it could produce a proof in 37.7 seconds with a peak memory footprint of approximately 400 GiB.

But 400 GiB is not a practical number for many deployment scenarios. The user's request in <msg id=3272> was explicit: "Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory." This was a practical question from someone who needed to deploy this proving engine on hardware with varying memory capacities — perhaps heterogeneous cloud rental markets, as the broader session context suggests.

The Reasoning Behind the Sweep

The assistant's response to the user's request reveals a methodical approach. Before launching any benchmarks, the assistant first explored the configuration structure in <msg id=3273>, using a subagent task to read cuzk-core/src/config.rs and understand all memory-affecting knobs. This was essential input knowledge: the assistant needed to know which configuration parameters controlled memory usage before designing the experiment.

The key parameters identified were partition_workers (the number of CPU threads performing parallel synthesis of circuit partitions) and gpu_workers_per_device (the number of GPU worker threads that hold in-flight GPU memory). The assistant correctly reasoned that reducing partition_workers would directly reduce peak memory because each partition worker holds a working set of evaluation vectors during synthesis. Reducing gpu_workers_per_device would reduce the GPU-side memory buffers that remain allocated while waiting for GPU operations to complete.

The assistant then created config files for pw=1/2/5/7 with gw=1 in <msg id=3276>, noting: "For low-memory testing, I'll use gw=1 to minimize GPU-side in-flight memory, along with each pw value. I'll also test gw=2 for comparison at the lower pw values since that's the production default." This decision to test both gw=1 and gw=2 at each pw level was critical — it would reveal whether the second GPU worker provided any throughput benefit when synthesis was the bottleneck.

The benchmark script itself, written in <msg id=3292>, automated the entire workflow: start the daemon, wait for SRS loading, record baseline RSS, run the bench tool, monitor peak RSS via /proc/PID/status, and report results. This automation was necessary because the manual approach attempted in <msg id=3282> had failed — the background RSS monitor process was killed when the shell timed out, leaving empty log files.

What This Message Actually Does

The subject message <msg id=3300> is the launch of the pw=5 gw=2 configuration. It follows a specific sequence of prior runs:

  1. pw=1 gw=1 (<msg id=3282>): 104 GiB peak, ~290s/proof wall time, 33s prove time. The extreme low end — almost entirely serialized synthesis, but minimal memory.
  2. pw=2 gw=1 (<msg id=3294>): 110 GiB peak, 152s/proof throughput, 33.7s prove time. A 2× improvement in throughput for only 6 GiB more memory.
  3. pw=5 gw=1 (<msg id=3296>): 170 GiB peak, 68.4s/proof, 34.0s prove time. A dramatic throughput improvement — synthesis parallelism was clearly effective.
  4. pw=7 gw=1 (<msg id=3298>): 208 GiB peak, 53.3s/proof, 34.0s prove time. Continuing to scale. Each of these runs used gw=1 — a single GPU worker. The assistant's stated rationale for now running pw=5 gw=2 was: "Now let me run gw=2 comparisons for pw=5 and pw=7 to see the memory impact of the second GPU worker." This reveals an important assumption: that the second GPU worker might increase memory without providing commensurate throughput benefit at low partition_worker counts. The assistant suspected that synthesis starvation would make the second GPU worker idle — if synthesis can't produce work fast enough, having two GPU workers just means they both sit waiting.

The Thinking Process Visible in the Message

The message itself is a tool call — a bash invocation — but the reasoning behind it is visible in the surrounding messages. The assistant's thinking follows a clear pattern:

  1. Hypothesis formation: The assistant believes that gw=2 will show no benefit at low pw because synthesis is the bottleneck, not GPU throughput. This is stated explicitly in <msg id=3299>: "to see the memory impact of the second GPU worker."
  2. Experimental design: The assistant systematically varies one parameter at a time. First, sweep pw at gw=1 to establish the memory-throughput curve. Then, test gw=2 at selected pw values to isolate the GPU worker effect.
  3. Anticipation of results: The assistant expects that gw=2 will either increase memory without improving throughput (if synthesis-starved) or provide marginal improvement (if the GPU can overlap operations). The subsequent message <msg id=3301> confirms the hypothesis: "Interesting! pw=5 gw=2 vs gw=1: same peak RSS (~170 GiB), nearly identical throughput (68.6 vs 68.4 s/proof), but gw=2 has higher prove time (58.2 vs 34.0s). The dual worker adds no benefit at low pw because synthesis can't keep the GPU fed fast enough."
  4. Iterative refinement: After completing the planned sweep, the assistant extends it in <msg id=3303> by running pw=10 gw=1 and pw=10 gw=2 to fill in the upper end of the curve, recognizing that the pattern might change once synthesis is fast enough to feed two GPU workers.

Assumptions and Potential Mistakes

Several assumptions underpin this message:

Assumption 1: Peak RSS is the right memory metric. The assistant uses /proc/PID/status VmHWM (peak resident set size) as the memory measurement. This captures the maximum physical memory used by the daemon process. However, it doesn't capture GPU memory (VRAM), which could be a separate constraint. The assistant's focus on RSS is reasonable for system RAM sizing but incomplete for GPU memory budgeting.

Assumption 2: The benchmark script's RSS monitoring is accurate. The script reads /proc/PID/status every two seconds. This sampling rate could miss short-lived memory spikes between samples. The assistant acknowledges this limitation implicitly by running multiple proofs and looking for consistent patterns.

Assumption 3: Five proofs at concurrency five is sufficient for steady-state measurement. This assumes that the engine reaches a stable memory plateau within a few proofs. For pw=1, where each proof takes ~290 seconds, only four proofs completed before timeout. The assistant correctly notes the pattern is "clear" even with partial data.

Assumption 4: The baseline RSS of 69 GiB (after SRS loading but before proving) is constant across configurations. This is reasonable — the SRS (Structured Reference String) is a 44 GiB file loaded into memory regardless of configuration. The assistant subtracts this baseline to compute the per-partition-worker memory cost.

Potential mistake: Not measuring GPU memory. The assistant never queries GPU memory usage via nvidia-smi or CUDA APIs. The peak RSS numbers capture host memory but not the ~20+ GiB of GPU VRAM used for NTT/MSM operations. For a complete deployment picture, GPU memory constraints are equally important.

Potential mistake: Confounding concurrency effects. Running five proofs at concurrency five means five proofs are in-flight simultaneously. This could increase memory beyond what a single proof would require, especially if the engine doesn't fully serialize per-proof resources. The assistant's measurements reflect peak multi-proof memory, not single-proof memory.

Input Knowledge Required

To understand this message, one needs:

  1. The cuzk engine architecture: Knowledge that partition_workers controls CPU-side synthesis parallelism, that each worker holds evaluation vectors in memory, and that gpu_workers_per_device controls GPU-side worker threads.
  2. The Phase 12 split API: Understanding that the GPU proving pipeline was restructured to decouple b_g2_msm from the critical path, and that memory backpressure mechanisms (early a/b/c free, channel capacity auto-scaling) were implemented to control peak memory.
  3. The benchmark infrastructure: Knowledge of the cuzk-bench tool, its batch command, the --count and --concurrency flags, and the C1 input format.
  4. Linux memory monitoring: Understanding that /proc/PID/status VmHWM reports peak resident set size, and that RSS is measured in kB.
  5. The Filecoin PoRep context: Knowledge that each proof involves 10 partitions of a 32 GiB sector circuit, and that synthesis of each partition takes approximately 29 seconds.

Output Knowledge Created

This message, as part of the broader sweep, produced several critical pieces of knowledge:

The memory scaling formula: The sweep revealed a clean linear relationship: peak RSS ≈ 69 GiB (baseline) + pw × ~20 GiB. This formula allows operators to predict memory requirements for any pw value.

The gw=2 inflection point: The comparison showed that gw=2 provides no throughput benefit below pw=10 because synthesis cannot keep two GPU workers busy. At pw=10, gw=2 begins to show marginal improvement (45.4s vs 42.5s). At pw=12, gw=2 is optimal (37.7s).

Deployment guidance: The sweep produced concrete recommendations:

The Deeper Significance

This message represents a transition from engineering to science. The preceding phases were about making the engine work — fixing bugs, reducing memory, improving throughput. This benchmark sweep is about understanding the engine's behavior systematically, characterizing its performance envelope, and producing actionable guidance for deployers.

The assistant's approach is a model of experimental methodology: form hypotheses, design experiments that isolate variables, automate measurement, validate results across multiple runs, and derive quantitative models from the data. The single bash command in <msg id=3300> is not just running a benchmark — it's testing a hypothesis about the relationship between synthesis parallelism, GPU worker count, memory footprint, and throughput.

The result — a clean linear memory scaling law and a clear inflection point for GPU worker utilization — is the kind of knowledge that transforms a complex system from an opaque black box into a predictable, tunable machine. For anyone deploying Groth16 proving infrastructure on real hardware with finite memory, this knowledge is invaluable.

In the broader narrative of the opencode session, this message is the pivot point where the months of optimization work crystallize into deployable knowledge. The engine is no longer just fast — it is understood.