Mapping the Memory Landscape: Inside cuzk's Configuration Knobs for Groth16 Proving

Introduction

In the high-stakes world of Filecoin storage proving, memory is not just a resource—it is the dominant constraint. The SUPRASEAL_C2 Groth16 proof generation pipeline, responsible for producing the cryptographic proofs that underpin Filecoin's Proof-of-Replication (PoRep), can consume upwards of 200 GiB of peak memory for a single proving operation. Understanding how to control this memory footprint is essential for operators running proving infrastructure on heterogeneous hardware, from memory-rich dedicated servers to memory-constrained cloud instances.

This article examines a single message from an opencode coding session—message index 8—in which an AI assistant delivers a comprehensive summary of memory-affecting configuration knobs in the cuzk proving engine. The message is the culmination of a targeted exploration task, synthesizing findings from across multiple source files into a coherent map of how each configuration parameter influences memory behavior. It is a document of discovery, revealing not only what the knobs do, but the engineering reasoning behind their design, the historical bugs they fixed, and the gaps where intended features remain unimplemented.

The message in question is an assistant response that collates findings from reading config.rs, engine.rs, pipeline.rs, and srs_manager.rs in the cuzk-core crate. It was produced in response to a user request (message 0) asking for a deep investigation into "memory-affecting configuration knobs." The result is a dense, information-rich report that serves as both a reference document and a diagnostic tool for anyone seeking to tune the cuzk engine's memory behavior.

The Context: Why This Message Was Written

To understand why this message exists, we must understand the broader investigation it belongs to. The root session was a "deep-dive investigation into SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, mapping the full call chain from Curio to CUDA kernels, accounting for ~200 GiB peak memory." The goal was to produce four documents including a background reference with nine bottlenecks and three optimization proposals. The session had already completed substantial work—documenting the Phase 12 architecture, running low-memory benchmark sweeps, and committing changes to git.

The subagent task that produced this message was specifically: "Explore cuzk config and engine (agent: explore)." The user's request was precise and practical: they needed to understand which configuration knobs affect memory, how they interact, and what their default values mean. This is not academic curiosity—it is the groundwork for making informed deployment decisions. When you are operating a proving cluster where each machine might have 256 GiB, 512 GiB, or 754 GiB of RAM, knowing that partition_workers should be scaled down from 20 to 5 for smaller systems is not a nice-to-have; it is essential to avoid OOM kills.

The user's request specifically asked for:

  1. The full config.rs file contents
  2. Channel capacity calculation code from engine.rs
  3. How gpu_workers_per_device is used
  4. The synthesis_lookahead usage
  5. Any other memory-related config knobs The assistant's response (message 8) delivers exactly this, but goes further—it contextualizes each finding with the engineering rationale, the memory formulas, and the historical lessons learned from OOM incidents.

The Message Itself: A Structural Analysis

The message is organized into six major sections plus a "Other Memory-Related Details" catch-all. Let us walk through each section and unpack what it reveals.

Section 1: Full config.rs Contents

The message begins by laying out the top-level configuration structure. The Config struct has eight sub-sections, but the assistant focuses on the four that are memory-affecting: [memory], [gpus], [synthesis], and [pipeline]. Each is presented in a table format with field name, type, default value, and purpose.

The [memory] section is particularly revealing. It contains two fields:

Section 2: Channel Capacity Calculation

This section is a masterclass in why configuration defaults matter and how they interact. The assistant reproduces the channel capacity calculation from engine.rs lines 729-750, including the extensive comment that explains the reasoning.

The channel is a tokio::sync::mpsc::channel<SynthesizedJob> that connects the synthesis pipeline to the GPU workers. Its capacity determines how many completed synthesis jobs can be queued waiting for GPU processing. The key insight is the auto-scaling rule:

let lookahead = if pw > 0 {
    configured_lookahead.max(pw)   // channel capacity = max(lookahead, partition_workers)
} else {
    configured_lookahead           // channel capacity = synthesis_lookahead
};

This means that when partition mode is active (which it is by default with partition_workers=20), the effective channel capacity is max(synthesis_lookahead, partition_workers). Since synthesis_lookahead defaults to 1 and partition_workers defaults to 20, the channel is sized to 20 slots.

The comment in the code explains the history behind this decision: "If the channel is too small, completed syntheses block on send() while still holding their full memory allocation (~16 GiB each pre-a/b/c-free, ~4 GiB after). This caused OOM at pw=12 with channel capacity=1 (28 provers piled up)."

This is a fascinating piece of engineering archaeology. The original design had a channel capacity of 1, assuming that the GPU would consume jobs as fast as they were produced. But in practice, when synthesis was faster than GPU processing (which happens when you have many CPU cores working on partition synthesis), completed jobs would pile up, blocked on the full channel, while still holding their large memory allocations. The fix was to auto-scale the channel to max(lookahead, partition_workers), allowing completed partitions to drain into the channel buffer without blocking the synthesis tasks. The comment notes that "once the channel is full, backpressure kicks in naturally."

This is a textbook example of the producer-consumer backpressure problem in concurrent systems. The channel acts as a buffer that decouples the producer (synthesis) from the consumer (GPU proving). If the buffer is too small, the producer blocks while holding memory. If the buffer is too large, memory usage grows unbounded. The solution here is elegant: size the buffer to the number of producers, ensuring that each producer can always enqueue its output without blocking, while the total number of in-flight jobs is bounded by the number of producers.

Section 3: How gpu_workers_per_device Is Used

The message explains that gpu_workers_per_device controls how many tokio tasks are spawned per physical GPU. In pipeline mode, the default is 2; in monolithic mode, it is forced to 1.

The dual-worker interlock is a clever design: each GPU gets a C++ std::mutex shared by all workers on that GPU. Workers acquire the mutex only during CUDA kernel execution, allowing CPU preprocessing (pointer setup, bitmap population, b_g2_msm) to run concurrently. This means that while one worker is holding the CUDA mutex and executing GPU kernels, another worker can be preparing data in CPU memory. The result is better GPU utilization without requiring additional GPU hardware.

The memory impact is direct: with 2 workers per GPU, up to 2 SynthesizedJobs can be in the GPU proving stage simultaneously per GPU. Each SynthesizedJob holds synthesized circuit data (~4 GiB after a/b/c vectors are freed). This doubles the GPU-side memory compared to a single worker.

The message also notes that gpu_threads (default 0, meaning auto = all CPUs) controls the CPU thread pool for GPU-side preprocessing. Setting this lower reserves CPU cores for synthesis, reducing contention. This is a subtle but important knob: CPU cores are a shared resource between synthesis (which is CPU-bound) and GPU preprocessing (which is also CPU-bound for certain operations like b_g2_msm).

Section 4: synthesis_lookahead Usage

The message describes two uses of synthesis_lookahead:

  1. Channel capacity: As described above, it participates in the auto-scaling formula.
  2. Synthesis concurrency semaphore: When synthesis_concurrency > 1, a Semaphore with synthesis_concurrency permits limits how many proof-level syntheses run in parallel. The message notes that even with N concurrent syntheses, only lookahead completed proofs can be queued—the channel provides downstream backpressure. The partition semaphore is particularly important: "The partition permit is held across both synthesis AND the channel send. This bounds total in-flight synthesis outputs to partition_workers—preventing unbounded memory growth when synthesis is faster than GPU consumption." This is a two-level backpressure system. At the top level, the synthesis_concurrency semaphore limits how many proof-level syntheses run simultaneously. At the bottom level, the partition_workers semaphore limits how many partition-level syntheses run simultaneously within each proof. The channel sits between them, providing a buffer that decouples the two levels.

Section 5: Buffer Flight Counters

This section describes the diagnostic system built into the pipeline. Five global AtomicUsize counters track in-flight large buffers at each pipeline stage. The counters are:

| Counter | Size Estimate | Meaning | |---|---|---| | SYNTH_IN_FLIGHT | -- | Partitions currently being synthesized | | PROVERS_IN_FLIGHT | ~12 GiB each | ProvingAssignment sets (a/b/c vectors + density bitvecs) | | AUX_IN_FLIGHT | ~4 GiB each | aux_assignment buffers | | PROVERS_SHELL_IN_FLIGHT | ~0.05 GiB each | Post-prove provers (a/b/c freed) | | PENDING_HANDLES | ~2 GiB each | C++ side pending proof handles |

The log_buffers(event) function computes a rough RSS estimate using the formula:

est_gib = (provers * 12) + (aux * 4) + (shells * 0.05) + (pending * 2)

The lifecycle transitions are documented with function calls:

Section 6: PendingGpuProof Type

The message briefly notes the PendingGpuProof type alias:

pub type PendingGpuProof = (PendingProofHandle<Bls12>, Option<usize>, Instant);

This is a tuple of: the C++-side pending proof handle, an optional partition index, and the GPU start timestamp. Each handle holds C++ split_vectors and tails (~2 GiB estimated). This type is the bridge between the Rust pipeline and the C++/CUDA proving backend. The PendingProofHandle&lt;Bls12&gt; is an opaque handle that references GPU-side memory allocations, which are freed when the handle is consumed.

Section 7: Other Memory-Related Details

The message concludes with three additional findings:

  1. malloc_trim(0) is called after each GPU partition prove (pipeline.rs:2025) and after dealloc in engine.rs (lines 150, 167). This forces glibc to return freed memory to the OS. Without this call, glibc's allocator may hold onto freed memory in per-thread arenas, causing RSS to remain high even after logical deallocation. This is a critical detail for memory-constrained environments.
  2. Capacity Hint Cache (pipeline.rs, lines 168-238): After the first synthesis, Vec capacity hints are cached in OnceLock statics (POREP_32G_HINT, etc.). This pre-allocates vectors to full size on subsequent runs, eliminating ~27 reallocation cycles and ~250 GB of redundant memcpy for 10-circuit PoRep batches. This is a performance optimization that also reduces memory fragmentation by ensuring vectors are allocated to their final size from the start.
  3. working_memory_budget is confirmed as unused—a placeholder for a future memory admission controller.
  4. pinned_budget (default &#34;50GiB&#34;) is advisory only—SRS loads that would exceed the budget produce a warning but are not blocked.

The Unused Knob: A Design Gap

One of the most interesting findings in this message is the working_memory_budget field. It exists in the config struct with a default of "80GiB" and a parser (working_memory_budget_bytes() presumably), but no code reads it at runtime. This is a classic example of design drift—a feature that was planned, maybe even partially implemented, but never completed.

Why would this knob be useful? A memory admission controller would allow the engine to dynamically throttle synthesis or GPU work when the working memory budget is approached. Instead of relying on static configuration (like partition_workers and slot_size), the engine could monitor actual memory usage and adapt. This would be particularly valuable in shared environments where memory pressure varies.

The fact that it remains unimplemented suggests that either:

Assumptions and Their Implications

The message reveals several assumptions embedded in the code:

  1. Synthesis is faster than GPU consumption: The channel capacity auto-scaling assumes that synthesis can produce partitions faster than the GPU can prove them. This is why the channel is sized to max(lookahead, partition_workers)—to prevent synthesis from blocking. But this assumption may not hold in all configurations. If GPU proving is faster (e.g., with very fast GPUs or very slow CPUs), the channel may be oversized, leading to unnecessary memory usage.
  2. Memory scales linearly with partition workers: The message states that each in-flight partition holds ~13.6 GiB and that partition_workers directly controls how many are in flight. This assumes that all partition workers are actively synthesizing simultaneously. In practice, some workers may be idle due to work-stealing or load imbalance, but the worst-case memory is bounded by partition_workers.
  3. GPU workers serialize on CUDA execution: The dual-worker interlock assumes that the CUDA mutex is the primary bottleneck and that CPU preprocessing can overlap with GPU execution. This is a reasonable assumption for modern GPUs where kernel launch is asynchronous and CPU work can proceed while the GPU computes.
  4. Buffer flight counters provide useful diagnostics: The counters assume that the size estimates (12 GiB per prover, 4 GiB per aux, etc.) are accurate and stable across different proving configurations. If these estimates drift (e.g., due to changes in circuit size or proof parameters), the diagnostic value degrades.
  5. malloc_trim(0) is safe and effective: The code assumes that calling malloc_trim(0) after deallocation is both safe (no correctness impact) and effective (actually returns memory to the OS). This is generally true for glibc, but the call has performance cost and may not work as expected with alternative allocators (e.g., jemalloc, tcmalloc).

The Thinking Process Revealed

The message itself is a summary, but it reveals the thinking process of the code's authors through the comments it quotes. The channel capacity comment is particularly rich:

"In partition mode (partition_workers > 0), up to pw partitions complete synthesis concurrently. If the channel is too small, completed syntheses block on send() while still holding their full memory allocation (~16 GiB each pre-a/b/c-free, ~4 GiB after). This caused OOM at pw=12 with channel capacity=1 (28 provers piled up)."

This comment tells a story: someone ran the engine with partition_workers=12 and a channel capacity of 1, and observed an OOM. Investigation revealed that 28 provers had piled up—the synthesis workers had completed their partitions but were blocked on the full channel, still holding their memory allocations. The fix was to auto-scale the channel to max(lookahead, partition_workers).

The comment also reveals the memory lifecycle: before a/b/c vectors are freed, each prover holds ~16 GiB. After freeing, it drops to ~4 GiB. This is why blocking on send() before the free is so dangerous—the memory footprint is much larger.

Another thinking artifact is the buffer flight counter lifecycle. The transitions are carefully designed to track memory at each stage:

The Broader Significance

This message is more than a reference document—it is a case study in memory-aware systems engineering. The cuzk engine deals with allocations measured in gigabytes, where a single misconfiguration can cause an OOM kill that takes down the entire proving pipeline. Every knob, every channel capacity, every semaphore permit is a tool for controlling memory pressure.

The message reveals several engineering principles in action:

  1. Explicit backpressure: Rather than relying on implicit synchronization (e.g., blocking I/O), the engine uses explicit semaphores and channel capacities to bound memory usage. Every in-flight job is accounted for.
  2. Diagnostic instrumentation: The buffer flight counters provide real-time visibility into memory allocation. This is essential for debugging memory issues in production.
  3. Graceful degradation: The advisory budget for pinned memory (warn but proceed) is a deliberate choice to prioritize availability over strict enforcement. The engine will attempt to work even under memory pressure, logging warnings for operators to act on.
  4. Historical learning: The channel capacity fix is a direct response to an OOM incident. The code comments preserve this history, ensuring that future developers understand why the design is the way it is.
  5. Configurability with defaults: Every knob has a sensible default, but the defaults are documented with their memory impact. Operators can make informed trade-offs between throughput and memory.

Conclusion

Message 8 of this opencode session is a remarkable piece of technical communication. It takes the raw findings from reading multiple source files and synthesizes them into a coherent, actionable reference. It answers the user's specific questions while also providing the context needed to understand why the knobs exist and how they interact.

For someone deploying the cuzk engine, this message is a roadmap to memory tuning. It tells you which knob to turn (lower partition_workers for less memory), which knob to ignore (working_memory_budget is a ghost), and which diagnostic tools to use (the buffer flight counters). It also tells you the memory formulas: (slot_size + 1) × 13.6 GiB for partitions, partition_workers × 13.6 GiB for synthesis, gpu_workers_per_device × 4 GiB for GPU-side memory.

But beyond its practical utility, the message is a window into the engineering process. It shows how a complex system evolves through real-world experience—the OOM incident that led to the channel capacity fix, the diagnostic counters added to track memory, the unused knob that represents a deferred feature. This is the kind of knowledge that is rarely documented in official specifications but is essential for understanding how a system actually behaves.

In the broader context of the root session—a deep-dive investigation into the SUPRASEAL_C2 pipeline—this message serves as the foundation for the optimization proposals that follow. You cannot optimize what you do not understand, and this message provides the understanding. It maps the memory landscape, identifies the pressure points, and equips the reader with the knowledge to make informed trade-offs. It is, in short, exactly what a good technical investigation should produce: clarity.