The Low-Memory Benchmark: Characterizing Production Boundaries After Phase 12

Message: [user] Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory.

This short user message, appearing at message index 3272 in the conversation, marks a critical inflection point in the cuzk proving engine development. It arrives immediately after an intensive documentation consolidation effort (messages 3246–3271) where the Phase 12 split GPU proving API and its memory backpressure fixes were formally recorded in cuzk-project.md and cuzk.example.toml. The Phase 12 work had achieved a breakthrough: 37.7 seconds per proof at 400 GiB peak memory with partition_workers=12 and gpu_threads=32. But the user, thinking like a production engineer rather than a researcher, asks a question that the optimization work had not yet answered: What about smaller systems?

The Motivation: From Optimization to Deployment

The Phase 12 engineering sprint had been laser-focused on maximizing throughput. The split API decoupled b_g2_msm from the GPU critical path, the use-after-free fix in prep_msm_thread eliminated crashes, early deallocation of a/b/c evaluation vectors reduced peak memory, channel capacity auto-scaling prevented GPU starvation, and the partition semaphore permit-through-send fix resolved a deadlock that had been masking memory pressure. The result was a finely tuned engine optimized for a 400+ GiB, multi-GPU server — exactly the kind of machine a Filecoin storage provider might deploy in a data center.

But the user recognizes a gap in this characterization. The cuzk engine is designed for heterogeneous cloud rental markets, as noted in the segment 0 analyzer summary. Not every deployment target has 768 GiB of RAM. Some operators might run on 128 GiB or 256 GiB machines. Some might want to run multiple proving instances on a single large machine. The question "how low can we get RSS requirements on smaller systems" is fundamentally about understanding the engine's memory scalability — its ability to gracefully degrade to smaller footprints while maintaining acceptable throughput.

The choice of parallelism values — 1, 2, 5, 7 — is deliberate and systematic. These are partition_workers values, controlling how many of the 10 Groth16 partitions are synthesized and proved concurrently. At pw=12 (the Phase 12 optimum), the engine keeps 12 partitions in flight, using a pool of 12 synthesis threads feeding a GPU that processes partitions sequentially. The user wants to explore the lower end of this spectrum: pw=1 (fully serial, one partition at a time), pw=2 (minimal parallelism), pw=5 (moderate, half the partitions), and pw=7 (approaching the Phase 11/12 territory). The request to "also lower queues/gpu workers that would also hold a lot of memory" shows an understanding that memory consumption isn't driven solely by partition workers — the GPU worker channels, staging buffers, and result queues all contribute to the peak RSS.

Assumptions Embedded in the Request

The message makes several implicit assumptions worth examining. First, it assumes a roughly linear relationship between partition_workers and peak memory — that cutting pw from 12 to 6 should roughly halve the per-partition memory overhead. This assumption would prove largely correct (the subsequent benchmarks revealed a clean formula: ~69 GiB baseline + pw × ~20 GiB), but it glosses over the baseline memory that doesn't scale. The SRS data, circuit parameters, and GPU allocations total about 69 GiB regardless of parallelism, meaning the lowest achievable memory is bounded by this baseline.

Second, the user assumes the engine can operate stably at these low parallelism settings. Phase 12's memory backpressure fixes were designed for high-throughput operation; whether the channel capacity auto-scaling and permit system work correctly at pw=1 or pw=2 is an open question. The user implicitly trusts that the architecture is robust across the full range.

Third, there is an assumption that lower parallelism will produce proportionally lower throughput — that a pw=2 configuration will take roughly 6x longer than pw=12. This assumption would be challenged by the results, which showed that at very low pw values, the GPU becomes severely underutilized, and the throughput degradation is worse than linear due to fixed overheads like SRS loading and kernel launch latency.

Input Knowledge Required to Understand This Message

To fully grasp what the user is asking, one needs substantial context from the preceding engineering work. The concept of "parallelsms" (a typo for "parallelism" or "parallel SMs") refers to partition_workers, the number of concurrent partition synthesis threads. The "queues/gpu workers" refer to the Tokio channel capacities and GPU worker pool sizes that were tuned during Phase 12's memory backpressure work. The "RSS" (Resident Set Size) metric is the key memory measurement, distinct from the GPU VRAM usage that had been the focus of earlier phases.

One must also understand the Groth16 proof structure: each Filecoin PoRep proof requires 10 partitions, each undergoing circuit synthesis (CPU), NTT (GPU), and MSM (GPU) phases. The partition_workers setting controls how many of these 10 partitions are processed concurrently, directly multiplying the memory needed for synthesis intermediates, evaluation vectors, and GPU staging buffers.

The Output Knowledge Created

The benchmark sweep that followed this message produced one of the most practically valuable outputs of the entire cuzk development effort: a clean, data-driven deployment guide for systems of varying memory capacities. The results revealed a linear memory scaling formula — peak_RSS ≈ 69 GiB + pw × 20 GiB — that allowed operators to predict memory requirements precisely. The throughput results showed that gw=2 (two GPU workers per device) provides no benefit below pw=10 because synthesis becomes the bottleneck, not GPU compute.

Concrete deployment guidance emerged: a 128 GiB system can run at pw=2 gw=1 achieving ~152 seconds per proof; a 256 GiB system at pw=7 gw=1 achieves ~77 seconds; a 512 GiB system at pw=12 gw=1 achieves ~49 seconds; and a 768 GiB system at pw=12 gw=2 achieves ~42.5 seconds. These numbers transformed the cuzk engine from a single-point optimization into a configurable platform adaptable to diverse hardware budgets.

The Thinking Process Visible in the Message

The user's thinking reveals a production-oriented mindset. Rather than asking "how fast can we go?" — the question that drove Phases 1 through 12 — they ask "how low can we go?" This is the question of a system architect preparing for real-world deployment, where hardware costs matter and not every customer has a 768 GiB server. The specific values (1, 2, 5, 7) show a systematic approach: start at the floor (pw=1), double it (pw=2), then take intermediate steps toward the known optimum. The inclusion of "also lower queues/gpu workers" shows holistic thinking about all memory contributors, not just the obvious one.

This message, though brief, represents the shift from engineering to productionization — from proving a design works at its peak to understanding how it behaves across the full operating envelope. It is the kind of question that separates a research prototype from a deployable system.