Taming Memory in the GPU Proving Pipeline: The Phase 12 Memory Backpressure Commit

The Message

The subject message (global index 3240) is a single git commit command issued by the AI assistant during an opencode coding session. In its entirety:

git commit -m "feat(cuzk): Phase 12 memory backpressure — channel capacity + semaphore fix

Three improvements on top of the Phase 12 split API (99c31c2c):

1. Early a/b/c free: After prove_start returns (GPU done with NTT+MSM),
   clear prover.a/b/c evaluation vectors (~12 GiB per partition). Only
   density bitvecs + assignment data are kept for background b_g2_msm.

2. Channel capacity auto-scaling: Size the synthesis→GPU channel to
   max(synthesis_lookahead, partition_workers) instead of hardcoded 1.
   Completed syntheses drain into the channel buffer without blocking,
   preventing synthesis output pile-up on send().

3. Partition permit held through send: The partition semaphore permit
   is now held until AFTER the channel send succeeds (not just through
   synthesis). With channel capacity = pw, sends are non-blocking so
   this adds zero latency. Bounds total in-flight synthesis outputs
   to partition_workers, preventing unbounded memory growth.

Also adds buffer flight counters (atomic, tracing::debug) for memory
diagnostics and converts eprintln to log::debug in bellperson dealloc.

Benchmark results (20 proofs, j=20, gw=2, gt=32):
  pw=10: 38.5s/proof, 321 GiB peak RSS (was 367 GiB)
  pw=12: 37.7s/proof, 400 GiB peak RSS (was OOM at 668 GiB!)
  pw=14: 37.8s/proof, 457 GiB peak RSS
  pw=16: 38.4s/proof, 510 GiB peak RSS

Optimal config: pw=12 — best throughput with bounded memory."

The commit succeeded, producing hash 98a52b33 on branch feat/cuzk, modifying 3 files with 142 insertions and 6 deletions.

Context: The Proving Pipeline and Its Memory Problem

To understand why this message was written, one must grasp the architecture of the system being optimized. The codebase implements Groth16 proof generation for Filecoin's Proof of Replication (PoRep) — a cryptographic proof that a storage provider is genuinely storing a client's data. This is computationally intensive work, and the system under development (cuzk) is a high-performance proving daemon that orchestrates GPU-accelerated proof generation across multiple workers.

The pipeline works in stages. First, a "synthesis" phase runs on CPU, producing "partitions" — chunks of circuit evaluation data. Each partition is approximately 12 GiB in size. These partitions are then sent to GPU workers for the heavy number-theoretic transform (NTT) and multi-scalar multiplication (MSM) operations. The Phase 12 split API (commit 99c31c2c, the immediate parent) had decoupled the GPU proving into an asynchronous interface, allowing synthesis to run ahead of GPU consumption. This pipelining improved throughput but created a dangerous dynamic: when synthesis outpaced GPU processing, completed partitions piled up in memory, waiting to be sent. The system had no mechanism to bound this accumulation, leading to out-of-memory (OOM) conditions.

The previous message in the conversation ([msg 3239]) shows the assistant staging the files for commit. The message before that ([msg 3238]) shows the assistant reviewing the full diff. And the messages leading up to this point (segments 29–31) document the entire Phase 12 journey: designing the split API, implementing it, fixing bugs, diagnosing memory pressure, and finally arriving at the backpressure solution that this commit crystallizes.

Why This Message Was Written: The Reasoning and Motivation

The commit message exists because the assistant had just completed a multi-hour investigation and implementation cycle to solve a critical memory management problem. The motivation is stated implicitly in the benchmark results: "pw=12: 37.7s/proof, 400 GiB peak RSS (was OOM at 668 GiB!)." The exclamation mark is not mere punctuation — it signals a breakthrough. The system was crashing at 668 GiB, and the fix brought it down to 400 GiB while maintaining (and even slightly improving) throughput.

But the deeper motivation is architectural. The assistant recognized that the Phase 12 split API had introduced a fundamental resource management problem. By decoupling synthesis from GPU consumption, the system gained throughput but lost the natural memory bounding that came from synchronous operation. The synthesis phase could run ahead arbitrarily, filling memory with partitions waiting for GPU slots. The assistant needed to re-establish a memory bound without sacrificing the throughput gains — a classic producer-consumer backpressure problem.

The reasoning visible in the commit message shows three distinct interventions, each targeting a different mechanism by which memory could grow unbounded:

  1. Early a/b/c free targets the lifetime of evaluation vectors within each partition. Once the GPU kernels complete their NTT and MSM work, the a/b/c vectors (~12 GiB per partition) are no longer needed. But the code was holding them until the entire proof finalized. Freeing them immediately after prove_start returns reclaims this memory earlier.
  2. Channel capacity auto-scaling targets the communication channel between synthesis (producer) and GPU workers (consumer). The channel had a hardcoded capacity of 1, meaning only one completed partition could be buffered. When a synthesis finished and the channel was full, the send() call would block, holding the entire 12 GiB partition in the sender's stack frame. By sizing the channel to max(synthesis_lookahead, partition_workers), the assistant ensured the channel could absorb all in-flight partitions without blocking.
  3. Partition permit held through send targets the semaphore mechanism that limits how many synthesis tasks can run concurrently. Previously, the semaphore permit was released as soon as synthesis completed, before the partition was sent to the GPU. This meant the number of in-flight partitions could exceed the semaphore count — the semaphore only bounded active synthesis, not buffered outputs. By holding the permit until the channel send succeeds, the assistant ensured that total in-flight partitions (synthesizing + buffered) never exceeds partition_workers. These three interventions work together as a coherent system: early free reduces per-partition memory lifetime, channel capacity ensures sends don't block, and the permit-through-send bounds the total number of partitions in flight.

How Decisions Were Made

The decisions in this commit were not made in a vacuum — they emerged from an iterative process of measurement, hypothesis, and validation visible in the preceding messages ([msg 3195] through [msg 3239]). The assistant systematically benchmarked different configurations (pw=10, pw=12, pw=14, pw=16), measuring both throughput and peak RSS for each. This empirical approach reveals a decision-making pattern:

First, the assistant established a baseline. The Phase 12 split API without backpressure achieved ~37.1s/proof but OOM'd at high partition counts. The assistant then implemented the three fixes and tested pw=10, finding 38.5-38.9s/proof at 317 GiB — a regression from the baseline but dramatically better memory. Rather than accepting this, the assistant dug deeper, comparing GPU times between runs ([msg 3201]) and hypothesizing that memory fragmentation or deallocation overhead was responsible.

The decision to try pw=12 was driven by reasoning about synthesis parallelism: if the GPU is the bottleneck, more synthesis workers won't help; but if synthesis is the bottleneck, more workers could improve throughput. The assistant's comment "pw=12 has more synthesis parallelism" ([msg 3204]) shows this reasoning. The result — 37.7s/proof at 400 GiB — validated the hypothesis and became the optimal configuration.

The assistant then pushed further to pw=14 and pw=16, not because they expected improvement, but to characterize the trade-off curve. The results (pw=14: 37.8s at 457 GiB; pw=16: 38.4s at 510 GiB) confirmed that pw=12 was indeed the sweet spot, with diminishing returns beyond it. The assistant correctly identified the limiting factor: "Higher pw means more memory pressure competing for the same DDR5 bandwidth" ([msg 3234]).

Assumptions Made

Several assumptions underpin this commit. The most critical is that the GPU kernels (NTT + MSM) are truly complete when prove_start returns. The commit message states this as fact: "The GPU kernels (NTT + MSM) are done and cudaHostUnregister has run." If this assumption were wrong — if the GPU were still using the a/b/c vectors after prove_start returns — then freeing them early would cause use-after-free bugs. The assistant had previously fixed such a bug ([msg 3210] of segment 30 mentions a "use-after-free bug in groth16_cuda.cu"), suggesting this assumption was validated through careful code review and testing.

Another assumption is that the channel capacity auto-scaling formula — max(synthesis_lookahead, partition_workers) — is sufficient to prevent blocking on send(). This assumes that the GPU workers consume partitions at least as fast as synthesis produces them, or that the channel buffer can absorb any surplus. If GPU consumption were much slower than synthesis, even a large channel would eventually fill, and the permit-hold mechanism would cause synthesis to block at the semaphore level anyway. The benchmark results validate this assumption for the tested configurations.

The assistant also assumes that the tracing::debug instrumentation for buffer flight counters adds negligible overhead. Converting eprintln to log::debug was a deliberate choice to keep production logs clean while retaining diagnostic capability.

Mistakes and Incorrect Assumptions

The most notable mistake visible in the surrounding conversation is the initial belief that the throughput regression from 37.1s to 38.5s was caused by the early a/b/c free deallocation overhead. The assistant hypothesized that freeing 120 GiB (10 partitions × 12 GiB) on spawn_blocking threads might add latency ([msg 3200]). However, further investigation revealed the real cause: the Phase 12 baseline benchmark had run 20 proofs (200 partitions) while the comparison run had only 15 proofs (150 partitions), and the GPU times were ~0.5s slower per partition in the smaller sample. The assistant eventually realized this was a statistical artifact, not a regression from the code change.

Another subtle error was the initial configuration of the daemon for pw=12 testing. The assistant created a config file but the daemon failed to start because the previous daemon still held the port (<msg id=3205-3207>). This required retrying with a sleep delay, a minor operational hiccup that didn't affect the final results.

The assistant also initially assumed that pw=14 or pw=16 might improve throughput further, but the benchmarks showed they did not. This wasn't a mistake — it was a hypothesis tested and disproven through measurement.

Input Knowledge Required

To understand this commit message, a reader needs knowledge of several domains:

  1. Groth16 proofs: The cryptographic proof system used by Filecoin. Understanding that proof generation involves circuit synthesis (CPU work) followed by NTT and MSM (GPU work) is essential.
  2. Producer-consumer backpressure: The classic concurrency pattern where a bounded buffer prevents a fast producer from overwhelming a slow consumer. The semaphore + channel design is a specific implementation of this pattern.
  3. CUDA memory management: The concept of cudaHostRegister/cudaHostUnregister for pinned memory, and the fact that GPU kernels access host memory asynchronously. The early free optimization depends on knowing when GPU access is truly complete.
  4. Filecoin PoRep: The specific proof type being generated (32 GiB sector proofs). The "porep-32g" SRS parameter preload and the C1 input format are domain-specific.
  5. The cuzk architecture: The daemon-worker model, the split API, the partition-based pipeline. The commit message references these without explanation, assuming familiarity.
  6. Rust async patterns: The channel-based communication, spawn_blocking tasks, semaphore permits, and the tracing logging framework.

Output Knowledge Created

This commit creates several forms of knowledge:

  1. A working memory-bounded pipeline: The primary output is a system that can run pw=12 without OOM, achieving 37.7s/proof at 400 GiB. This is immediately usable knowledge for anyone running the cuzk daemon.
  2. Benchmark characterization: The pw=10/12/14/16 sweep provides empirical data on the throughput-memory trade-off. The finding that pw=12 is optimal, with diminishing returns beyond, is valuable operational knowledge.
  3. A reusable backpressure pattern: The three-intervention design (early free + channel capacity + permit-through-send) is a general solution to producer-consumer memory problems in GPU pipelines. It could be applied to other systems facing similar issues.
  4. Diagnostic instrumentation: The buffer flight counters provide ongoing visibility into pipeline behavior, enabling future optimization work.
  5. A documented commit: The detailed commit message itself is knowledge — it explains not just what was changed but why, preserving the reasoning for future developers.

The Thinking Process

The commit message reveals the assistant's thinking process through its structure. It begins with a clear title identifying the change ("Phase 12 memory backpressure"), then enumerates three specific interventions, each with a rationale. The benchmark results are presented as evidence, and the optimal configuration is identified explicitly.

But the deeper thinking is visible in what the commit message doesn't say — the experimental process that led to these conclusions. The assistant didn't implement all three interventions at once. The preceding messages show an iterative approach: first implementing the early a/b/c free and semaphore fix, then benchmarking, then discovering the channel capacity issue, then fixing that, then re-benchmarking across multiple pw values. The commit message presents the final integrated solution, but the thinking behind it was shaped by hours of measurement and debugging.

The assistant's decision to include buffer flight counters as tracing::debug (rather than removing them entirely) shows a forward-thinking mindset: the counters may be needed for future diagnostics, but they shouldn't clutter production logs. This is a mature engineering judgment.

Conclusion

The Phase 12 memory backpressure commit (98a52b33) represents the culmination of a deep optimization cycle in the cuzk proving pipeline. What appears as a single git commit is actually the synthesis of multiple debugging sessions, benchmark iterations, and design decisions. The assistant transformed an OOM-prone system into a stable, memory-bounded pipeline, reducing peak memory by 40% (from 668 GiB to 400 GiB at pw=12) while maintaining throughput. The three interventions — early a/b/c free, channel capacity auto-scaling, and partition permit held through send — form a coherent backpressure system that could serve as a template for similar GPU pipeline optimizations. The commit message itself, with its detailed rationale and benchmark evidence, exemplifies the kind of documentation that makes open-source development sustainable.