The Art of the Benchmark Sweep: Characterizing Memory Scaling in the SUPRASEAL_C2 Proving Engine

Introduction

In the course of a deep engineering session on the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, there comes a moment when the intense cycle of optimization, debugging, and refactoring gives way to something quieter but equally important: systematic characterization. Message <msg id=3304> captures this transition perfectly. It is a single bash command — the execution of a benchmark script for a specific configuration (pw=10 gw=1) — but it sits at the confluence of weeks of engineering effort, a carefully designed experimental methodology, and a strategic shift from raw optimization toward production readiness.

The Message

The message is deceptively simple:

[assistant] [bash] /tmp/cuzk-lowmem-bench.sh /tmp/cuzk-lowmem-pw10-gw1.toml lowmem-pw10-gw1 5 5
=== Starting benchmark: lowmem-pw10-gw1 ===
  Config: /tmp/cuzk-lowmem-pw10-gw1.toml
  Count: 5, Concurrency: 5
  Daemon log: /home/theuser/cuzk-lowmem-pw10-gw1.log
  Daemon PID: 1714908
  Daemon ready after 28s
  Baseline RSS: 69 GiB
  Running benchmark...
[2m2026-02-20T16:22:42.042697Z[0m [32m INFO[0m [2mcuzk_bench[0m[2m:[0m loading C1 output [3mpath[0m[3m=[0m/data/32gbench/c1.json
=== Batch Benchmark ===
proof type:  porep
count:       5
concurrency: 5

  [1/5] COMPLETED — 75.6s...

At first glance, this is just another benchmark run in a series. But to understand why this particular message matters, we must understand the journey that led here and the questions this specific data point was meant to answer.

Context and Motivation: Why This Message Was Written

The message is part of a systematic low-memory benchmark sweep ordered by the user in <msg id=3272>, who asked: "Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory."

This request came immediately after the completion of Phase 12 — a major engineering effort that introduced a split GPU proving API, memory backpressure mechanisms (early a/b/c vector deallocation, channel capacity auto-scaling, partition semaphore permit-through-send), and various bug fixes (including a use-after-free in the C++ prep_msm_thread). Phase 12 had achieved a stable configuration at pw=12 gw=2 that produced proofs in ~37.7s with a peak RSS around 400 GiB. But 400 GiB is a lot of memory — it restricts deployment to large, expensive machines.

The user's question was pragmatic: could the engine run on smaller, cheaper systems? What would the tradeoffs be? The assistant had just spent weeks optimizing for throughput at high memory budgets. Now the task was to map the other end of the curve: how does memory scale down, and what performance does one get at each point?

The assistant's response was to design and execute a systematic sweep across partition_workers values of 1, 2, 5, 7, 10, and 12, each tested with both gpu_workers_per_device=1 (gw=1) and gw=2 where relevant. The subject message is the run for pw=10 gw=1 — a configuration that sits in the middle-to-upper range of the sweep, bridging the low-memory configs already tested (pw=1 at 104 GiB, pw=2 at 110 GiB, pw=5 at 170 GiB, pw=7 at 208 GiB) toward the high end (pw=12 gw=2 at ~400 GiB).

The Methodology: A Purpose-Built Benchmark Script

A critical detail visible in this message is the use of a custom benchmark script: /tmp/cuzk-lowmem-bench.sh. This script was written in <msg id=3292> after the assistant discovered that the initial ad-hoc approach to RSS monitoring was unreliable — a background subshell was killed when the benchmark timeout hit, leaving an empty RSS log. The assistant recognized the need for a robust, repeatable methodology and wrote a script that:

1. Kills any existing daemon to ensure a clean state
2. Starts the daemon with the specified config file
3. Waits for the daemon to be ready (SRS loaded, baseline RSS established)
4. Records baseline RSS (the "69 GiB" seen in the message — this is the SRS and daemon overhead before any proving work)
5. Runs the benchmark with specified proof count and concurrency
6. Monitors RSS in the background during the run
7. Captures peak RSS from /proc filesystem
8. Kills the daemon after completion This methodological rigor is important. Without it, the RSS measurements would be unreliable — a daemon left running from a previous test would contaminate results, and peak RSS captured after the fact (as happened in the pw=1 run) would miss the actual maximum. The assistant learned from early mistakes and built a proper harness.

The Configuration: Why pw=10 gw=1?

The choice of pw=10 gw=1 is not arbitrary. It reflects a deliberate experimental design. The sweep had already covered pw=1, 2, 5, and 7 with gw=1. The assistant had also tested pw=5 gw=2 and pw=7 gw=2 and discovered an important result: at low partition_worker counts, adding a second GPU worker (gw=2) provided no throughput benefit because synthesis could not keep the GPU fed fast enough — the second worker starved.

The pw=10 configuration was the next logical data point. It would show whether the linear memory scaling trend continued (baseline ~69 GiB + pw × ~20 GiB per worker), and whether the throughput curve was still climbing steeply or beginning to plateau. The gw=1 variant was chosen first to isolate the effect of partition_workers alone, without the confounding factor of GPU worker count. Only after establishing the gw=1 baseline would the assistant test pw=10 gw=2 (in <msg id=3306>) to see whether at higher pw values the second GPU worker finally became useful.

Assumptions Made

Several assumptions underpin this message and the broader sweep:

1. Linear memory scaling. The assistant implicitly assumes that peak RSS scales roughly linearly with partition_workers. The baseline RSS of ~69 GiB (SRS + daemon overhead) plus ~20 GiB per partition worker had been observed in earlier tests, and the sweep was designed to validate this formula across a wider range.

2. Concurrency of 5 is sufficient for steady-state measurement. The assistant uses count=5, concurrency=5 for most runs. This assumes that 5 concurrent proofs will saturate the engine and produce representative peak RSS and throughput numbers. For the low-pw configs this was fine; for pw=12 gw=2, the assistant later noted that j=5 produced lower RSS than j=20 (271 GiB vs ~400 GiB), revealing that concurrency itself is a memory-affecting parameter.

3. The benchmark script captures true peak RSS. The script monitors RSS every 2 seconds via /proc/PID/status. This assumes that the memory allocation pattern is slow enough that 2-second sampling captures the peak. For a pipeline that allocates and deallocates memory rapidly, there is a risk of missing transient spikes.

4. gpu_threads=32 is optimal across all configurations. The assistant carried forward the Phase 12 default of gpu_threads=32 without re-tuning it for lower-memory configs. This assumption is reasonable — the GPU thread count primarily affects L3 cache partitioning and was optimized for the split API — but it might not be optimal at very low pw values where GPU utilization is already low.

Input Knowledge Required

To understand this message, one needs considerable context:

• The SUPRASEAL_C2 architecture: Knowledge that proof generation involves 10 partitions, each requiring CPU synthesis (~29s) followed by GPU proving (NTT/MSM operations). The partition_workers parameter controls how many partitions can be synthesized concurrently.
• The Phase 12 split API: Understanding that the proving pipeline was restructured so that b_g2_msm runs outside the GPU worker lock, and that memory backpressure mechanisms (early a/b/c free, channel capacity auto-scaling, permit-through-send) were implemented to control RSS.
• The config file format: Knowing that pw = partition_workers, gw = gpu_workers_per_device, gt = gpu_threads, and how these interact.
• The benchmark infrastructure: Understanding that cuzk-bench batch runs proofs against a running daemon, that C1 outputs are pre-computed circuit values, and that the RSS baseline of 69 GiB represents the SRS loaded into memory.
• The prior results: Knowing that pw=1 achieved 104 GiB at ~290s/proof, pw=2 achieved 110 GiB at 152s/proof, pw=5 achieved 170 GiB at 68.4s/proof, and pw=7 achieved 208 GiB at 53.3s/proof. This context makes the pw=10 result meaningful as the next point on the curve.

Output Knowledge Created

This message produced a critical data point in the memory/throughput characterization:

pw=10 gw=1: 310 GiB peak RSS, 45.4s/proof throughput, 39.5s average prove time.

This result, combined with the other data points, enabled several key insights:

1. The memory scaling formula holds: 69 GiB baseline + 10 × ~20 GiB = ~269 GiB predicted, and actual was 310 GiB. The additional ~41 GiB comes from in-flight proof data at concurrency 5, confirming that memory scales as baseline + pw × per_partition_memory + concurrency × per_proof_memory.
2. The throughput curve is flattening: From pw=7 (53.3s) to pw=10 (45.4s), the improvement is ~7.9s for 3 additional workers. From pw=5 (68.4s) to pw=7 (53.3s), the improvement was ~15.1s for 2 additional workers. The marginal return per worker is diminishing, suggesting the system is approaching a throughput ceiling determined by GPU compute capacity rather than synthesis parallelism.
3. gw=2 becomes viable above pw=10: The subsequent test of pw=10 gw=2 (in <msg id=3306>) showed 271 GiB peak and 42.5s/proof — slightly lower RSS (because concurrency effects dominate) and slightly better throughput. This established the threshold where a second GPU worker stops being wasted.

The Thinking Process

The reasoning visible in the surrounding messages reveals a methodical, data-driven approach. The assistant does not jump to conclusions or guess at optimal configurations. Instead, it:

1. Recognizes the need for a systematic sweep after the user's request
2. Designs the sweep with specific pw/gw combinations
3. Builds proper tooling (the benchmark script) when ad-hoc methods fail
4. Executes methodically, running one configuration at a time, collecting results, and using those results to inform the next choice
5. Adapts the plan based on intermediate findings — e.g., adding gw=2 comparisons after seeing the gw=1 results, and extending the sweep to pw=10 when the lower values showed clear trends
6. Validates assumptions by running reference configurations (pw=12 gw=2) with the same methodology to ensure comparability The assistant also demonstrates a healthy skepticism about its own measurements. When the pw=1 RSS monitor produced an empty log, it diagnosed the root cause (background subshell killed by timeout) and built a more robust solution. When comparing pw=10 gw=2 results at j=5 vs prior j=20 results, it noted the concurrency effect explicitly rather than treating all measurements as directly comparable.

Conclusion

Message <msg id=3304> is, on its surface, a single benchmark execution. But it represents the culmination of a systematic, well-designed experimental campaign to characterize the memory/throughput tradeoff space of a complex GPU proving engine. The assistant moved from ad-hoc testing to a structured sweep with a purpose-built harness, generating data that would directly inform deployment decisions: what hardware to buy, how to configure it, and what performance to expect.

The result — 310 GiB peak RSS at 45.4s/proof for pw=10 gw=1 — became one of nine data points that together told a coherent story about linear memory scaling, diminishing returns on partition workers, and the threshold where dual GPU workers become useful. This is the kind of knowledge that transforms a clever optimization into a production-ready system that operators can configure with confidence.