The Fourth Data Point: Systematic Characterization in the cuzk Low-Memory Benchmark Sweep

Introduction

In the course of a months-long engineering effort to optimize the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), a single bash command at message index 3298 represents a pivotal moment of transition. The assistant executes:

/tmp/cuzk-lowmem-bench.sh /tmp/cuzk-lowmem-pw7-gw1.toml lowmem-pw7-gw1 5 5

This is the fourth benchmark in a systematic low-memory sweep, testing the configuration of partition_workers=7 with gpu_workers_per_device=1. On the surface, it is a routine invocation of a shell script. But beneath this mundane command lies a carefully orchestrated investigation that would ultimately produce a clean linear memory scaling formula, concrete deployment guidance for systems ranging from 128 GiB to 768 GiB, and a fundamental understanding of how the engine's memory footprint behaves under varying workloads. This message is the pivot point where raw optimization yields to production readiness.

Why This Message Was Written: Reasoning, Motivation, and Context

The immediate trigger for this message was the user's request at <msg id=3272>: "Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory." This request came immediately after the assistant had completed Phase 12 — a complex set of memory backpressure fixes including a split GPU proving API, early deallocation of NTT evaluation vectors, channel capacity auto-scaling, and a partition semaphore permit-through-send fix. The Phase 12 work had been benchmarked at pw=12 gw=2 achieving 37.7s/proof with ~400 GiB peak memory, but the question of how the engine behaved at lower memory configurations remained entirely unexplored.

The deeper motivation was production readiness. The cuzk engine was being developed for deployment on heterogeneous hardware — not everyone has 768 GiB of RAM available. To provide meaningful deployment guidance, the assistant needed to characterize the full performance/memory tradeoff curve. The user's request for pw=1/2/5/7 was a deliberate sampling strategy: these values span the range from fully serialized synthesis (pw=1, where 10 partitions are processed one at a time) through moderate parallelism (pw=7, where most partitions can be processed concurrently with some contention).

The assistant's response at <msg id=3273> shows immediate understanding of the task's scope. Rather than blindly running benchmarks, it first spawned a subagent task to explore the cuzk config structure and identify all memory-affecting knobs. This exploration revealed that partition_workers (pw) controls how many CPU threads concurrently synthesize partition proofs, while gpu_workers_per_device (gw) controls how many GPU-side work items can be in-flight simultaneously. Both directly impact peak memory: each partition worker holds a complete synthesis state (~20 GiB), and each GPU worker holds GPU buffers and pinned memory for its in-flight proof.

The assistant then created config files for pw=1/2/5/7 with gw=1 and gt=32 (<msg id=3276>), wrote a proper benchmark script (<msg id=3292>), and systematically executed the sweep. By the time it reached message 3298, it had already completed pw=1 (104 GiB, ~290s/proof), pw=2 (110 GiB, 152s/proof), and pw=5 (170 GiB, 68.4s/proof). Each result was immediately analyzed and the pattern was becoming clear: prove time was remarkably stable at ~33-34s regardless of pw, while wall-clock throughput scaled inversely with pw and memory grew linearly.

How Decisions Were Made

Several design decisions are visible in this message and its surrounding context. The first is the choice of benchmark methodology. The assistant wrote a dedicated script (/tmp/cuzk-lowmem-bench.sh) that handled the entire lifecycle: kill any existing daemon, start a fresh daemon with the target config, wait for SRS loading (28s baseline), record baseline RSS, run the benchmark with controlled concurrency, capture peak RSS from /proc, and clean up. This automation was essential for consistency across nine configurations.

The second decision was the choice of benchmark parameters. For pw=7, the assistant used count=5, concurrency=5 — five proofs with five concurrent workers. This was a deliberate choice to measure steady-state throughput under load, not just single-proof latency. The concurrency level matching the proof count ensured the system was fully utilized.

The third decision was the choice of gw=1 for the initial sweep. The user had suggested "lower queues/gpu workers that would also hold a lot of memory," and the assistant correctly inferred that gw=1 would minimize GPU-side memory pressure, giving the cleanest picture of how partition_workers alone affects memory. The gw=2 comparison runs would come later, after the pw sweep established the baseline.

The fourth decision, visible in the progression from <msg id=3294> through <msg id=3298>, was the order of execution. The assistant started with pw=1 (slowest, lowest memory), then pw=2, pw=5, and pw=7. This ascending order was pragmatic: the slower configurations took longer to run (pw=1 took ~5 minutes per proof), so running them first allowed the faster configurations to complete more quickly as the session progressed.

Assumptions Made by the User and Agent

The user's request at <msg id=3272> contained an implicit assumption that lower partition_workers would proportionally reduce memory. This turned out to be correct, but the exact scaling was unknown — it could have been sub-linear (if there were fixed memory costs independent of pw) or super-linear (if there were overheads that grew faster than linearly). The assistant's benchmark sweep was designed to measure this empirically.

The assistant made several assumptions in its methodology. It assumed that RSS (Resident Set Size) from /proc/PID/status was an accurate measure of the engine's memory footprint. RSS includes shared libraries and file-backed mappings, so the "baseline RSS" of ~69 GiB after SRS loading but before any proofs was partly the SRS parameter file (44 GiB on disk) mapped into memory. The assistant treated this as a fixed baseline and subtracted it to compute the per-proof memory overhead, which was a reasonable engineering approximation.

The assistant also assumed that 5 proofs with concurrency 5 was sufficient to reach steady-state memory usage. For pw=7, with 10 partitions per proof and 7 workers, each proof takes about 2 rounds of partition synthesis (10/7 ≈ 1.43, rounded up to 2). With 5 concurrent proofs, the system would have up to 7×5 = 35 partition synthesis tasks in flight, which should be enough to saturate memory. This assumption proved valid — the results showed clean, repeatable patterns.

Mistakes or Incorrect Assumptions

One notable methodological issue was the RSS monitoring approach. In the first benchmark (pw=1, <msg id=3282>), the assistant attempted to run a background RSS monitor using a subshell that polled /proc/PID/status every 2 seconds. This monitor failed silently — the log file was empty (<msg id=3285>). The assistant had to fall back to reading VmHWM (Peak Resident Set Size) from /proc after the benchmark completed, which only provides the single peak value rather than a time series. For the subsequent benchmarks, the assistant wrote a proper benchmark script that handled RSS capture more robustly.

Another issue was the initial attempt to run 5 proofs for pw=1. The assistant quickly realized this was wasteful — each proof took ~290 seconds wall time, and the pattern was clear after just 2-3 proofs. It killed the benchmark early and moved on, learning to use fewer proofs for the slowest configurations.

The assistant also initially struggled with PID detection when multiple processes matched the grep pattern (<msg id=3286-3287>). The daemon had spawned a child process, and pgrep -f "cuzk-daemon --config" returned two PIDs. This was quickly fixed by piping through head -1.

None of these were fundamental errors — they were the normal friction of automated benchmarking in a live environment. Each was diagnosed and corrected within one or two messages, demonstrating the assistant's ability to learn from operational feedback.

Input Knowledge Required

To understand this message, one needs knowledge of several domains. First, the Groth16 proving pipeline for Filecoin PoRep: a sector proof requires processing 10 "partitions" of circuit constraints, each requiring CPU synthesis (generating the circuit's evaluation vectors a, b, c) followed by GPU proving (multi-scalar multiplication and number-theoretic transform). The partition_workers parameter controls how many partitions can be synthesized concurrently on CPU cores.

Second, the cuzk engine architecture: the split GPU proving API (Phase 12) decouples GPU work from CPU post-processing, allowing b_g2_msm to overlap with the next proof's synthesis. The memory backpressure system uses channel capacity and semaphore permits to bound in-flight work. The gpu_workers_per_device parameter controls how many GPU work items can be queued simultaneously.

Third, Linux memory accounting: RSS from /proc/PID/status includes both private anonymous memory and shared file-backed mappings. The SRS parameter file is 44 GiB and is memory-mapped, contributing to RSS even when not actively used. The assistant's "baseline RSS" of ~69 GiB includes this SRS mapping plus the daemon's own heap.

Fourth, benchmarking methodology: the distinction between "prove time" (GPU compute time for a single proof) and "wall-clock throughput" (total time including queueing and contention). The assistant consistently reports both, and the stable prove time (~33-34s) across all pw values is a key finding — it means GPU compute is not the bottleneck at lower pw values.

Output Knowledge Created

This message, as part of the sweep, contributed to a body of knowledge that would be synthesized in the following messages. The immediate output was the pw=7 gw=1 data point: 208 GiB peak RSS, 53.3s/proof throughput, 34.0s prove time (revealed in <msg id=3299>). Combined with the previous data points, this enabled the assistant to derive the linear memory scaling formula: ~69 GiB baseline + pw × ~20 GiB.

The full sweep (including the gw=2 comparisons that followed) produced a comprehensive deployment guide:

| System RAM | Configuration | Throughput | |------------|--------------|------------| | 128 GiB | pw=2 gw=1 | 152s/proof | | 192 GiB | pw=5 gw=1 | 68.4s/proof | | 256 GiB | pw=7 gw=1 | 53.3s/proof | | 400 GiB | pw=12 gw=1 | ~45s/proof | | 768 GiB | pw=12 gw=2 | 42.5s/proof |

The critical insight was that gw=2 provides no throughput benefit below pw=10 because synthesis starvation (not enough partition workers to keep the GPU fed) becomes the bottleneck. This finding directly informs deployment decisions: a system with 256 GiB RAM should use pw=7 gw=1, not pw=5 gw=2, because the second GPU worker would waste memory without improving throughput.

The broader output knowledge was a validated understanding of the engine's memory architecture. The ~20 GiB per partition worker matches the expected size of a single partition's synthesis state (circuit evaluation vectors, intermediate buffers). The ~69 GiB baseline includes the SRS mapping (~44 GiB), the daemon's working set, and GPU pinned memory buffers. This decomposition allows system integrators to predict memory usage for any configuration.

The Thinking Process Visible in Reasoning

The assistant's reasoning is visible in the progression of messages leading to and following this one. At <msg id=3273>, the assistant immediately recognized that it needed to understand the config structure before designing the benchmark sweep — it spawned a subagent to explore config files rather than guessing at parameters. This is a hallmark of systematic thinking: understand the knobs before turning them.

At <msg id=3275>, the assistant read an existing config file to use as a template, showing attention to detail and reuse of proven configurations. At <msg id=3276>, it created config files for all four pw values before running any benchmarks — a batch-processing approach that minimized context switching.

The most revealing thinking is visible in the assistant's response to the pw=1 results (<msg id=3283>): "The pw=1 run is very slow — each proof takes ~292s wall time because only 1 partition can be synthesized at a time (10 partitions × ~29s each = ~290s sequential). But the prove time is excellent (~33s). Let me kill this, get the RSS data, and move on to faster configs." This shows real-time analysis: the assistant immediately understood why pw=1 was slow, recognized that the prove time being stable at ~33s was the important signal, and made a pragmatic decision to abort the remaining proofs rather than waste time confirming an already-obvious pattern.

At <msg id=3294>, after pw=2 completed, the assistant noted: "pw=2 gw=1: 110 GiB peak, 152s/proof throughput, 33.7s prove time." The pattern was emerging: memory grew by ~6 GiB from pw=1 to pw=2 (104→110), and throughput improved by ~2x (290→152s). The assistant was building the mental model that would become the linear scaling formula.

At <msg id=3297>, after pw=5: "Excellent! pw=5 gw=1: 170 GiB peak, 68.4s/proof, 34.0s prove time." The memory jump from pw=2 (110 GiB) to pw=5 (170 GiB) is 60 GiB for 3 additional workers = ~20 GiB/worker. This is the moment the linear scaling became apparent. The assistant's "Excellent!" reflects the satisfaction of seeing a clean, predictable pattern emerge from noisy real-world measurements.

Conclusion

Message 3298 is, on its face, a single bash command — the fourth invocation of a benchmark script in a systematic sweep. But it represents something much larger: the transition from engineering to science. The Phase 12 memory backpressure fixes were the engineering achievement; the low-memory benchmark sweep was the scientific characterization that made those fixes useful in practice. Without this sweep, system integrators would have no way to predict whether their hardware could run the engine, or how to configure it for optimal throughput within their memory budget.

The assistant's methodical approach — understand the knobs, automate the measurements, run in ascending order of expected duration, analyze each result immediately, and synthesize the findings into a predictive model — is a textbook example of performance characterization. The resulting deployment guidance, committed as 9bb657e5, ensures that the months of optimization work can be effectively deployed on real hardware in production.