Taming the Memory Beast: How Three Targeted Interventions Solved Phase 12's OOM Crisis

Introduction

In the high-stakes world of Filecoin proof generation, where a single Groth16 proof can consume hundreds of gigabytes of memory and take over a minute to compute, the line between a working system and a crashing one is razor-thin. The Phase 12 split GPU proving API [1] had unlocked meaningful throughput gains by decoupling CPU synthesis from GPU proving — but it introduced a silent killer: when CPU synthesis outran GPU consumption, completed partitions piled up in memory until the process exhausted its 755 GiB RAM budget and the OOM killer stepped in.

This article synthesizes the work of an intensive optimization session that solved this memory crisis through three carefully coordinated interventions: early deallocation of evaluation vectors, auto-scaling of channel capacity, and restructuring of semaphore permit ownership [33] [34] [44]. The result transformed a configuration that previously crashed at 668 GiB into a stable pipeline running at 37.7 seconds per proof with 400 GiB peak RSS — a 40% memory reduction with no throughput loss [57] [110].

The Architecture: Phase 12's Split GPU Proving API

To understand the memory problem, one must first understand the pipeline it plagued. The cuzk SNARK proving engine generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol [1]. Each proof requires synthesizing circuit evaluations across multiple partitions — chunks of computation that can be parallelized across CPU cores. These partitions, each holding approximately 12 GiB of evaluation vectors (the a, b, and c polynomials), are then consumed by GPU workers that perform the heavy number-theoretic transforms (NTT) and multi-scalar multiplications (MSM) required for the proof.

Phase 12 introduced a "split" API that decoupled the GPU worker's critical path from CPU post-processing. Previously, the GPU worker would hold the GPU lock for the entire duration of proving a partition, including a 1.7-second CPU-bound computation called b_g2_msm. During those 1.7 seconds, the GPU sat idle. The split API allowed prove_start to return immediately after GPU work completed, releasing the GPU lock so the worker could pick up the next partition, while b_g2_msm continued in a background thread. This overlapping of CPU and GPU work improved throughput from ~40 seconds per proof to ~37.1 seconds.

But the split architecture created a new vulnerability. The pipeline now had two stages — CPU synthesis (producer) and GPU proving (consumer) — connected by a bounded channel. With 10 or 12 partition workers running concurrently, synthesis could complete partitions at a rate far exceeding GPU consumption. The channel, originally hardcoded to capacity 1, was meant to provide backpressure: when full, it would block send() calls, preventing further accumulation. But the backpressure mechanism had a critical flaw.

The Root Cause: A Permit Released Too Early [33] [34]

The partition semaphore — a tokio::sync::Semaphore with partition_workers permits — was designed to limit how many CPU synthesis tasks could run concurrently. Each task acquired a permit before starting synthesis and released it when done. The channel's send() call happened after the permit was released, inside the async task that awaited the spawn_blocking result.

This ordering had a devastating consequence. When all 12 partition workers completed synthesis faster than the GPU could consume them, the semaphore permits were immediately returned to the pool, allowing 12 new synthesis tasks from the next proof to start. The completed outputs, still holding their ~12 GiB evaluation vectors, would either block on send() (if the channel was full) or pile up in the channel buffer (if the channel had room). Either way, the system had no mechanism to bound the total number of in-flight synthesis outputs. The semaphore limited active synthesis but not completed-but-unconsumed outputs.

The assistant's buffer flight counters revealed the shocking truth: at peak, there were 28 synthesized ProvingAssignment sets alive simultaneously, each holding ~12 GiB of evaluation vectors. That's 336 GiB just for the prover data, plus auxiliary buffers pushing the total toward the 755 GiB ceiling.

Three Interventions, One Coherent Design [107] [110]

The solution emerged not as a single fix but as three coordinated changes, each targeting a different mechanism by which memory could grow unbounded. The assistant implemented them iteratively, testing each hypothesis against benchmark data before proceeding to the next.

1. Early a/b/c Free: Reclaiming ~12 GiB Per Partition [107]

The first intervention was the simplest and most immediately effective. After prove_start returns from the GPU, the evaluation vectors (a, b, and c polynomials) are no longer needed on the host side — the GPU has already copied them to device memory for the NTT and MSM operations. Yet the code was holding them until the entire proof finalized, keeping ~12 GiB per partition alive for no reason.

The fix was a surgical edit: immediately after prove_start returns, the vectors are cleared with Vec::new(). This freed ~12 GiB per partition from the peak working set, providing immediate memory relief. For a configuration with 12 partitions in flight, this alone saved up to 144 GiB.

2. Channel Capacity Auto-Scaling: Preventing Blocked Sends [44] [107]

The second intervention addressed the channel bottleneck. The synthesis→GPU channel was hardcoded to capacity 1, meaning only one completed partition could be buffered. When a synthesis finished and the channel was full, the send() call would block, holding the entire 12 GiB partition in the sender's stack frame. With multiple syntheses completing simultaneously, this created a pile-up of blocked senders, each holding large allocations.

The fix auto-scaled the channel capacity to max(synthesis_lookahead, partition_workers). With capacity equal to the number of partition workers, all completed syntheses could drain into the channel without blocking. This eliminated the bottleneck where synthesis tasks blocked holding large allocations.

However, the channel capacity increase alone was not enough — and in some ways made things worse. With room for 12 outputs in the channel, up to 12 completed partitions could accumulate in the buffer, pushing RSS even higher. The assistant's benchmarks showed that the channel-only fix actually increased peak memory from 367 GiB to 390 GiB at pw=10, with a slight throughput regression from 37.1s to 38.8s per proof.

3. Partition Permit Held Through Send: The Critical Insight [33] [34] [44]

The third intervention was the most subtle and the most impactful. The assistant realized that the channel capacity increase had addressed the symptom (blocked sends) but not the root cause (early permit release). The semaphore permit was released inside spawn_blocking — as soon as synthesis completed — before the channel send(). This meant a new partition could start synthesizing even though the previous partition's output was still waiting to be consumed.

The fix restructured the permit ownership: instead of moving the permit into the spawn_blocking closure (where it would be dropped when synthesis finished), the assistant held the permit in the outer async task until after the channel send() succeeded. The key insight was that with channel capacity equal to partition_workers, the send() call would never block — there was always room in the channel. So holding the permit through send() added zero latency while guaranteeing that at most partition_workers completed outputs could exist at any time.

This was the moment of synthesis. The assistant had previously tried holding the permit through send() with channel capacity 1, and it had killed throughput (40.5s per proof) because the single channel slot caused send() to block, which held the permit, which prevented new syntheses from starting — effectively serializing the pipeline. The combination of larger channel and permit held through send was the magic formula that neither change alone could achieve.

The Benchmarking Campaign: Finding the Sweet Spot

With the three interventions implemented, the assistant embarked on a systematic benchmarking campaign across four configurations of the partition_workers parameter:

| pw | Throughput | Peak RSS | Notes | |----|-----------|----------|-------| | 10 | 38.5 s/proof | 321 GiB | Stable, baseline comparison | | 12 | 37.7 s/proof | 400 GiB | Previously OOM at 668 GiB | | 14 | 37.8 s/proof | 457 GiB | No throughput gain, more memory | | 16 | 38.4 s/proof | 510 GiB | Throughput regression, massive memory |

The results told a clear story. pw=12 emerged as the optimal configuration, delivering the best throughput (37.7s) with bounded memory (400 GiB). The higher values — pw=14 and pw=16 — consumed significantly more memory without improving throughput, hitting what the assistant identified as the DDR5 bandwidth wall: more CPU cores competing for the same memory bus, generating contention that offset any parallelism gains.

The assistant's benchmarking methodology was meticulous. Each configuration was tested with 20 proofs at concurrency 20, with GPU workers=2 and GPU threads=32. RSS was monitored continuously via a background logging loop, and buffer flight counters provided real-time visibility into the pipeline's internal state. The assistant even investigated a 1.8-second throughput regression between runs, comparing GPU timing distributions and testing hypotheses about eprintln! overhead before concluding it was within normal variance.

From Debug Instrumentation to Production Readiness

Throughout the optimization work, the assistant had added extensive debug instrumentation — atomic buffer flight counters (PROVERS_IN_FLIGHT, AUX_IN_FLIGHT, SYNTH_IN_FLIGHT) that logged the state of every buffer allocation at key events. These counters were invaluable for diagnosing the memory pressure problem, revealing the peak of 28 in-flight provers that confirmed the unbounded accumulation.

But eprintln! calls in production code are unacceptable — they add synchronous I/O overhead and clutter logs. In the final phase of the work, the assistant converted all buffer counter logging from eprintln! to tracing::debug!, preserving the diagnostic capability while ensuring production cleanliness. This change was verified by benchmarking: the conversion had no measurable impact on throughput (38.8s per proof before and after), disproving the hypothesis that eprintln! overhead was causing the regression.

The Commit: 98a52b33

The culmination of this work was commit 98a52b33 on the feat/cuzk branch, with the message:

feat(cuzk): Phase 12 memory backpressure — channel capacity + semaphore fix

Three improvements on top of the Phase 12 split API (99c31c2c):

1. Early a/b/c free: After prove_start returns (GPU done with NTT+MSM),
   clear prover.a/b/c evaluation vectors (~12 GiB per partition).

2. Channel capacity auto-scaling: Size the synthesis→GPU channel to
   max(synthesis_lookahead, partition_workers) instead of hardcoded 1.

3. Partition permit held through send: The partition semaphore permit
   is now held until AFTER the channel send succeeds.

The diff spanned 142 insertions across three files — supraseal.rs (the Rust FFI layer), engine.rs (the engine orchestration), and pipeline.rs (the pipeline implementation). The distribution of changes reflected the layered nature of the fix: the early free touched the FFI boundary, the channel capacity auto-scaling touched the engine coordinator, and the permit restructuring touched the core pipeline logic.

Lessons for Systems Engineering

The Phase 12 memory backpressure saga offers several enduring lessons for engineers building high-performance concurrent systems:

1. Semaphore permits must cover the entire resource lifetime. A permit that gates only the computation phase, not the output's lifetime, is no permit at all. The assistant's key insight was that the partition semaphore was bounding synthesis but not in-flight outputs — and the latter was what mattered for memory.

2. Channel capacity and permit discipline interact non-trivially. The assistant's first attempt at holding the permit through send failed because the channel was too small. The channel capacity increase alone failed because the permit was released too early. Only the combination worked. This is a classic example of how two seemingly independent mechanisms can interact in ways that neither alone predicts.

3. Benchmarking is not optional. Without the benchmark data showing a 5% regression from the channel-only fix, the assistant might have declared victory and moved on. The regression forced a re-examination that led to the combined fix — and ultimately to a better solution.

4. Memory backpressure is a design problem, not a tuning problem. The OOM at pw=12 was not a configuration issue that could be solved by reducing worker counts. It was a fundamental flaw in the pipeline's backpressure architecture. The fix required understanding the lifecycle of every allocation and ensuring that each was bounded by a mechanism that couldn't be circumvented.

Conclusion

The Phase 12 memory backpressure work transformed a failing, OOM-prone configuration into a stable, high-throughput pipeline. Three interventions — early a/b/c free, channel capacity auto-scaling, and partition permit held through send — worked together to bound memory without sacrificing the throughput gains of the split API. The result, pw=12 at 37.7s/proof with 400 GiB peak RSS, became the new production baseline [57] [110] [112].

But the deeper achievement was the design itself. By understanding that the semaphore permit's lifetime determined the effective pipeline depth, and that channel capacity determined whether the permit added latency, the assistant created a backpressure system that was both efficient and robust. The fix was not a hack — it was a principled solution grounded in a clear understanding of the system's dynamics. And that, ultimately, is what separates a temporary workaround from a lasting architectural improvement.

References

[1] The Checkpoint: How a Memory Crisis Reshaped a SNARK Proving Engine's Architecture [33] The Permit That Wasn't Held: A Case Study in Memory Backpressure Design [34] Holding the Line: How a Single Permit Restructuring Fixed Memory Backpressure in a GPU Proving Pipeline [44] Validating the Semaphore+Channel Fix: How a Combined Approach Tamed Memory Pressure Without Sacrificing Throughput [57] The Moment Memory Backpressure Proved Itself: pw=12 at 38.4s Without OOM [107] The Moment Before Commit: Reviewing a Memory Backpressure Breakthrough [110] Taming Memory in the GPU Proving Pipeline: The Phase 12 Memory Backpressure Commit [112] The Status Checkpoint: How a Structured Todo List Captured the Culmination of Phase 12's Memory Backpressure Optimization