The First Benchmark in the Sweep: Starting the Low-Memory Characterization of a Groth16 Proving Engine

Introduction

In the lifecycle of any high-performance computing project, there comes a pivotal moment when the intense, focused work of optimization gives way to the quieter but equally critical task of characterization. The engineer must step back from the heat of bug-fixing and feature development to ask: what have we actually built, and how does it behave across the full spectrum of deployment scenarios? Message [msg 3280] captures exactly this transition. After weeks of engineering the Phase 12 split GPU proving API and its memory backpressure system—fixing use-after-free bugs, implementing channel capacity auto-scaling, and wrestling with CUDA device-global synchronization conflicts—the assistant executes a single, seemingly mundane command: it starts the cuzk daemon with partition_workers=1 and gpu_workers_per_device=1, launching the first data point in a systematic low-memory benchmark sweep.

The Message

The message reads:

That's just the shell process. Good. Now start the daemon with pw=1 gw=1:

>

``bash FIL_PROOFS_PARAMETER_CACHE=/data/zk/params nohup /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon --config /tmp/cuzk-lowmem-pw1-gw1.toml > /home/theuser/cuzk-lowmem-pw1-gw1.log 2>&1 & echo "Daemon PID: $!" Daemon PID: 1449837 ``

On its surface, this is a routine command: start a daemon in the background, redirect output to a log file, record the PID. But in the context of the broader project, this command represents the opening move of a carefully planned experimental campaign that would ultimately produce a clean linear memory scaling formula and concrete deployment guidance for systems ranging from 128 GiB to 768 GiB of RAM.

Context and Motivation

To understand why this message was written, we must trace back through the preceding conversation. The project is the cuzk engine—a CUDA-accelerated Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). The engine had just completed Phase 12, a major architectural revision that introduced a split GPU proving API to decouple the critical GPU worker loop from CPU post-processing, along with a suite of memory backpressure mechanisms. The assistant had spent the previous messages ([msg 3249] through [msg 3271]) consolidating this work into documentation: updating cuzk-project.md with the Phase 12 architecture, revising cuzk.example.toml with optimal defaults (gpu_threads=32, partition_workers=12), and committing everything as 9bb657e5.

Then came the user's request at [msg 3272]: "Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory."

This request reveals a crucial shift in perspective. The Phase 12 optimization had been benchmarked at pw=12 (12 partition workers), achieving 37.7 seconds per proof with a peak memory footprint around 400 GiB. That's excellent throughput, but it targets large, expensive systems. The user is asking: what about smaller deployments? What if someone only has 128 GiB or 256 GiB of RAM? Can the engine still be useful? The request is fundamentally about production readiness—not just raw speed, but configurability across hardware tiers.

The assistant's response at [msg 3273] shows the systematic engineering approach: first, explore the config structure to understand which knobs affect memory. A subagent task reads config.rs and returns the full configuration surface. The assistant then creates a todo list ([msg 3274]) planning config files for pw=1/2/5/7 with gw=1 and gt=32, followed by individual benchmark runs. At [msg 3275], it reads the existing production config to use as a template. At [msg 3276], it writes four config files. At [msg 3277], it updates the todo list with the first run marked "in_progress." At [msg 3278], it kills any existing daemon and verifies the system is clean.

Then at [msg 3279], the assistant runs pgrep and confirms that only the shell process itself is running—no stale daemon. This is the critical precondition check. The stage is set.

The Decision to Start with pw=1 gw=1

The choice of pw=1 gw=1 as the first configuration is deliberate. partition_workers=1 is the absolute minimum—only one partition is synthesized at a time, which should produce the lowest possible memory footprint. gpu_workers_per_device=1 means only one GPU worker is active, minimizing GPU-side in-flight memory. This combination represents the "anchor point" of the sweep: the configuration that will tell us the irreducible baseline memory cost of the engine. Every subsequent run with higher pw or gw values will add memory on top of this baseline.

The assistant's comment—"That's just the shell process. Good."—confirms that the precondition (no existing daemon) is satisfied. The pkill from the previous message may or may not have killed anything, but the pgrep confirms the system is clean. This attention to experimental hygiene is important: a stale daemon from a previous run could hold GPU resources, occupy pinned memory, or interfere with the new daemon's port binding, corrupting the benchmark results.

Input Knowledge Required

To fully understand this message, the reader needs several layers of context:

  1. The cuzk project architecture: The engine is a CUDA-accelerated Groth16 prover for Filecoin PoRep. It uses a partition-based synthesis approach where the circuit is divided into partitions, each synthesized independently and then aggregated. The partition_workers config controls how many partitions are synthesized concurrently, which directly drives memory usage.
  2. The Phase 12 split API: The recently completed Phase 12 introduced a split GPU proving API that decouples the GPU worker loop from CPU post-processing. This changed the memory profile by allowing earlier deallocation of evaluation vectors and introducing channel capacity limits.
  3. The memory backpressure system: Phase 12 also implemented early a/b/c vector freeing, channel capacity auto-scaling, and a partition semaphore permit-through-send mechanism. These changes made the engine stable at higher partition worker counts but their effect at low counts was untested.
  4. The configuration surface: The partition_workers (pw), gpu_workers_per_device (gw), and gpu_threads (gt) parameters are the primary knobs affecting memory and throughput. The assistant had just explored config.rs via a subagent task to confirm this.
  5. The benchmark infrastructure: The daemon is started with nohup and redirected to a log file, implying that benchmarks are driven by a separate client (likely cuzk-bench) that connects to the daemon's listen address. The log file captures daemon-side diagnostics including RSS measurements.

Assumptions Embedded in the Message

Several assumptions underpin this seemingly simple command:

Assumption 1: The daemon will start successfully with pw=1 gw=1. This is not guaranteed. The daemon had been tested extensively at pw=12, but lower partition counts exercise different code paths. For instance, with only one partition worker, the partition semaphore logic might behave differently, and the GPU pipeline might stall waiting for work. The assistant is implicitly assuming that the code is robust across the full range of partition counts.

Assumption 2: The benchmark will produce meaningful data. Starting the daemon is only the first step. The assistant assumes that a benchmark client will connect, submit work, and that the resulting log will contain both timing and RSS information. If the benchmark client isn't compatible with the low-memory config, or if the daemon's memory tracking isn't accurate at low allocations, the data could be misleading.

Assumption 3: The system has sufficient resources for even this minimal config. The daemon requires GPU access, CUDA pinned memory, and SRS parameter files. The FIL_PROOFS_PARAMETER_CACHE environment variable points to /data/zk/params, which must contain the porep-32g parameters. The assistant assumes these are present and accessible.

Assumption 4: The log file path is valid. The output is redirected to /home/theuser/cuzk-lowmem-pw1-gw1.log. The assistant assumes this path is writable and that the filesystem has sufficient space for the daemon's logging.

Mistakes and Incorrect Assumptions

At this point in the conversation, no mistakes are visible in this specific message—it's a straightforward daemon launch. However, the broader sweep would later reveal nuances. The chunk summary tells us that the sweep ultimately tested nine configurations (pw=1/2/5/7/10/12 × gw=1/2), and the results showed that gw=2 provides no throughput benefit below pw=10 due to synthesis starvation. This finding implicitly corrects an assumption that might have been carried from the Phase 12 benchmarks: that gw=2 (the production default) is always beneficial. The low-memory sweep revealed that for smaller systems, the extra GPU worker just sits idle because the CPU-side synthesis can't feed it fast enough.

The summary also reveals the clean linear memory scaling formula: ~69 GiB baseline + pw × ~20 GiB. This formula is the key output of the sweep, and it validates the assistant's experimental design—the data was clean enough to extract a simple predictive model. The 69 GiB baseline represents the irreducible cost of loading SRS parameters, circuit structures, and GPU state. Each additional partition worker adds approximately 20 GiB for its in-flight synthesis data.

Output Knowledge Created

This message, as the first benchmark run, creates the foundation for all subsequent analysis. The daemon process with PID 1449837 will:

  1. Load the SRS parameters into pinned memory
  2. Initialize the GPU context
  3. Listen for benchmark connections
  4. Process proof requests using exactly one partition worker and one GPU worker
  5. Log timing and memory statistics to the designated file The resulting log file (cuzk-lowmem-pw1-gw1.log) becomes the first data point in a characterization that will eventually span nine configurations. When combined with the other runs, it enables the derivation of the memory scaling formula and the deployment guidance that is the ultimate deliverable of this chunk.

The Thinking Process

The assistant's reasoning is visible in the sequence of actions leading up to this message. The todo list at [msg 3274] shows a methodical plan: create configs, then run benchmarks one by one. The config creation at [msg 3276] shows the assistant reasoning about which parameters to vary: "I'll use gw=1 to minimize GPU-side in-flight memory, along with each pw value. I'll also test gw=2 for comparison at the lower pw values since that's the production default." This reveals an awareness that the sweep needs to isolate the effect of each variable.

The precondition check at [msg 3278]-[msg 3279] shows the assistant thinking about experimental validity: a stale daemon would contaminate results. The comment "That's just the shell process. Good." confirms the assistant correctly interpreted the pgrep output—the only process matching "cuzk-daemon" was the pgrep command itself, not an actual daemon.

The choice of nohup and backgrounding with & indicates the assistant is thinking about the asynchronous nature of the benchmark: the daemon must run in the background while the assistant proceeds to launch the benchmark client in a subsequent message. The log file redirection ensures that daemon output is preserved even if the terminal session ends.

Conclusion

Message [msg 3280] is a deceptively simple command that marks the beginning of a systematic characterization effort. After weeks of intense optimization and bug-fixing, the assistant transitions to the equally important work of understanding the engine's behavior across its configuration space. The low-memory benchmark sweep that starts here would ultimately produce a clean linear memory model and concrete deployment guidance, transforming the cuzk engine from a single-point optimization into a configurable tool adaptable to hardware ranging from 128 GiB to 768 GiB systems. The message embodies the engineering discipline of experimental design: verify preconditions, start at the baseline, and build understanding incrementally.