Systematic Memory Characterization: The Low-Memory Benchmark Sweep of Phase 12

In the course of a months-long engineering campaign to optimize the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, a critical transition occurred at message [msg 3296]. After twelve phases of increasingly sophisticated optimization—spanning PCIe bandwidth analysis, two-lock GPU interlock architectures, DDR5 memory bandwidth contention diagnosis, split GPU proving APIs, and intricate memory backpressure mechanisms—the assistant turned from building to measuring. The message at [msg 3296] is the execution of a pw=5 gw=1 benchmark, the third data point in a systematic low-memory sweep designed to answer a fundamentally practical question: How small a system can run this pipeline, and how fast will it be?

Context: From Optimization to Characterization

The conversation leading to this message reflects a mature engineering project that had just completed its most complex phase. Phase 12 had introduced a split GPU proving API that decoupled the b_g2_msm computation from the GPU worker loop, along with memory backpressure mechanisms including early deallocation of NTT evaluation vectors, channel capacity auto-scaling, and a partition semaphore permit-through-send fix. These changes had been implemented, debugged, benchmarked at pw=12 gw=2 (achieving ~37.7s per proof at ~400 GiB peak memory), and documented in the project's master document cuzk-project.md.

But one question remained unanswered. The production configuration targeted a 768 GiB system, but what about smaller deployments? The user's request at [msg 3272] was direct: "Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory." This was not a request for a single data point—it was a request for a characterization sweep across the configuration space, producing a memory-versus-throughput curve that would allow system integrators to choose the right hardware for their budget.

The Message Itself: Executing the pw=5 gw=1 Benchmark

The subject message is a single bash command invocation:

/tmp/cuzk-lowmem-bench.sh /tmp/cuzk-lowmem-pw5-gw1.toml lowmem-pw5-gw1 5 5

This launches the benchmark script with the configuration file for partition_workers=5, gpu_workers_per_device=1, and gpu_threads=32. The script starts the daemon, waits for it to be ready (28 seconds, as shown by "Daemon ready after 28s"), records a baseline RSS of 69 GiB (the memory consumed by the SRS and daemon infrastructure before any proof work begins), then runs 5 proofs at concurrency 5.

The output shows the benchmark beginning: the C1 output is loaded from /data/32gbench/c1.json, and the first proof completes in 81.5 seconds. The message is truncated in the conversation view (ending with "81.5s (p..."), but the full results—visible in the subsequent message at [msg 3297]—show that this configuration achieved 170 GiB peak RSS, 68.4 seconds per proof throughput, and 34.0 seconds prove time.

Reasoning and Decision-Making

This message represents a deliberate methodological choice. The assistant had earlier attempted a more ad-hoc approach: starting the pw=1 benchmark manually with a background RSS monitor script, only to discover that the monitor log was empty because the background subshell was killed when the benchmark timeout hit ([msg 3286]). The failure of that approach—"The RSS monitor log is empty"—prompted a redesign.

The assistant's response was to write a proper benchmark script (/tmp/cuzk-lowmem-bench.sh at [msg 3292]) that handles the entire lifecycle: start daemon, monitor RSS via /proc/$PID/status, run the bench, collect results, then kill the daemon. This script became the standardized tool for the entire sweep, ensuring consistent methodology across all nine configurations tested.

The choice of pw=5 gw=1 as the third configuration (after pw=1 and pw=2) reflects an understanding of the pipeline's scaling behavior. With 10 partitions per proof and 5 workers, each proof requires approximately 2 rounds of synthesis (10/5=2), so the expected wall time is roughly 2 × ~29s per partition = ~58s synthesis, plus ~34s GPU work = ~92s total. The actual result of 68.4s/proof throughput (with 5 concurrent proofs) is better than this rough estimate because concurrency hides some of the serialization.

Assumptions and Implicit Knowledge

The assistant operates under several assumptions in this message. First, that gpu_workers_per_device=1 (gw=1) is the right choice for low-memory characterization. The reasoning, articulated in [msg 3276], is that gw=1 "minimizes GPU-side in-flight memory." A second GPU worker would double the GPU-side buffers (GPU proof data, MSM scratch, NTT scratch) without improving throughput when synthesis is the bottleneck—an assumption later validated when pw=5 gw=2 showed identical throughput at the same RSS ([msg 3301]).

Second, the assistant assumes that 5 proofs at concurrency 5 is sufficient for steady-state measurement. This is a reasonable assumption given that each proof takes ~68 seconds and the daemon's memory should stabilize within the first few proofs. The baseline RSS of 69 GiB is measured before any proofs run, and the peak RSS is captured by polling /proc/$PID/status every 2 seconds during the benchmark.

Third, the assistant assumes that the benchmark script's RSS monitoring is reliable. The script reads VmRSS from /proc/$PID/status—this gives the current RSS, not the peak. To capture the peak, the script polls every 2 seconds and the maximum observed value is reported. This is an approximation: the true peak could occur between polls. However, for a multi-minute benchmark with relatively stable memory usage, 2-second polling provides adequate accuracy.

Input Knowledge Required

To fully understand this message, one must know several things about the cuzk engine architecture:

• Partition workers (pw): The number of parallel CPU threads performing Groth16 synthesis. Each partition is an independent computation that produces the a/b/c vectors for one segment of the circuit. With 10 partitions per proof, pw=5 means 5 partitions are synthesized concurrently, and the remaining 5 wait for worker threads to become available.
• GPU workers (gw): The number of parallel GPU command queues. Each worker can hold one proof's GPU data in flight. With gw=1, only one proof's GPU buffers are allocated at a time; with gw=2, two proofs' buffers coexist, doubling GPU-side memory.
• Baseline RSS: The 69 GiB baseline represents the SRS (Structured Reference String) loaded into memory (~44 GiB for the porep-32g circuit), plus the daemon's runtime overhead, CUDA driver allocations, and pinned memory pools. This is the floor below which no configuration can go.
• The prove time vs. throughput distinction: "Prove time" (~34s) is the GPU computation time for a single proof. "Throughput" (68.4s/proof) is the wall-clock time per proof under concurrency, which includes queueing and synthesis delays. The gap between these numbers reveals that synthesis is the bottleneck at pw=5.

Output Knowledge Created

This message produced a critical data point in the memory scaling curve. Combined with the pw=1 (104 GiB, 290s/proof), pw=2 (110 GiB, 152s/proof), and subsequent pw=7 (208 GiB, 53.3s/proof), pw=10 (310 GiB, 45.4s/proof), and pw=12 (~400 GiB, 37.7s/proof) results, the sweep revealed a clean linear relationship: ~69 GiB baseline + pw × ~20 GiB per partition worker. This formula became the basis for deployment guidance in the updated cuzk.example.toml and cuzk-project.md documentation.

The sweep also demonstrated that gw=2 provides no benefit below pw=10—a finding with direct practical implications. A system with 128 GiB of RAM can run at pw=2 gw=1 achieving 152s/proof, while a 256 GiB system can run at pw=7 gw=1 achieving 53.3s/proof. This data-driven guidance replaced the previous rule-of-thumb recommendations.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the benchmark sweep itself. The progression from pw=1 through pw=12 is not random—it follows a deliberate pattern designed to map the memory/throughput Pareto frontier. The pw=1 and pw=2 runs establish the low end (minimal memory, maximal time). The pw=5 and pw=7 runs fill in the mid-range. The pw=10 and pw=12 runs (with both gw=1 and gw=2) establish the high end. The gw=2 comparisons at pw=5 and pw=7 answer the specific question: "Does a second GPU worker help when synthesis is slow?" The answer—no—is a valuable insight for system configuration.

The assistant also shows awareness of measurement methodology. The failure of the initial ad-hoc RSS monitor led to the creation of a robust benchmark script. The choice of 5 proofs at concurrency 5 balances statistical significance against time cost. The reporting of both peak RSS and baseline RSS allows readers to separate fixed overhead from per-worker memory.

Conclusion

Message [msg 3296] is, on its surface, a simple bash command running a benchmark. But in the context of the broader engineering effort, it represents a critical shift from optimization to characterization—from "how do we make this faster?" to "how do we help others deploy this?" The low-memory sweep produced actionable deployment guidance that would be essential for anyone trying to run the SUPRASEAL_C2 pipeline on systems smaller than the 768 GiB reference configuration. It transformed the Phase 12 engineering成果 into a practical tool for system integrators, completing the arc from research prototype to production-ready software.