Taming the Memory Beast: How a Semaphore+Channel Fix Unlocked pw=12 Without OOM

Introduction

In the relentless pursuit of optimizing Filecoin's Groth16 proof generation pipeline, the cuzk team had been iterating through a series of increasingly sophisticated phases. Phase 12 introduced a "split API" that decoupled GPU proving from CPU post-processing, hiding the latency of the b_g2_msm computation. But with every optimization came new challenges. The split API, while promising throughput gains, created a dangerous memory accumulation problem: synthesized partitions could pile up in the channel faster than the GPU could consume them, leading to catastrophic Out of Memory (OOM) failures. Message 3188 in this conversation marks the triumphant moment when that problem was finally solved — and the beginning of the next investigation.

The Message in Full

The subject message, delivered by the AI assistant at index 3188, reads:

pw=12: 383.8 GiB peak RSS (vs 668 GiB before → OOM). The semaphore+channel fix works perfectly: - pw=12 now runs successfully with peak RSS 383.8 GiB (well within 755 GiB budget) - 38.4s/proof — slightly better than pw=10's 38.9s - Peak provers = 18 (bounded)

>

Now let me investigate the ~1.8s throughput regression from the Phase 12 baseline (37.1s). The buffer counters add eprintln! calls at every synthesis start/done and prove start/finalize. That's ~600+ synchronous stderr writes per run. Let me disable them and re-test: [todowrite] {"todos":[{"content":"Increase synthesis_lookahead channel capacity to match partition_workers (pw)","priority":"high","status":"completed"},{"content":"Hold partition permit through channel send (semaphore fix)","priority":"high","status":"completed"},{"content":"Build and verify compilation","priority":"high","status":"completed"},{"content":"Benchmark pw=10 with semaphore+channel fix — 38.9s/proof, 314.7 GiB peak RSS","priority":"high","status":"completed"},{"content":"Benchmark pw=12 with semaphore+channel fix — 38.4s/proof, 383.8 GiB peak RSS","priority":"high","status":"completed"},{"content":"Disable buffer counters (eprintln!) and re-test for baseline throughput","priority":"high","status":"pending"},{"content":"Benchmark pw=14, pw=16 to find optimal partition_workers","priority":"medium","status":"pending"}]}

This message is a milestone marker. It reports success, quantifies the improvement, and immediately pivots to the next bottleneck. It is simultaneously a victory lap and a launch pad.

The Memory Backpressure Problem

To understand why this message matters, one must grasp the memory dynamics of the Phase 12 split API. The Groth16 proof pipeline for Filecoin's Proof-of-Replication (PoRep) operates on partitions — chunks of the circuit that can be synthesized independently and then aggregated. Each partition synthesis produces evaluation vectors (a, b, c) that occupy approximately 12 GiB of memory. The GPU then consumes these vectors to generate the proof, but GPU processing is fast (~3 seconds per partition) relative to synthesis (~29 seconds per partition).

The split API introduced a channel-based producer-consumer architecture: synthesis workers produce completed partition data and send it through a channel to GPU workers. The original design used a channel capacity of 1, which meant that if synthesis completed faster than GPU consumption (which it did, by roughly 5×), the extra synthesis outputs would block on send(). But crucially, the partition semaphore — which controlled how many synthesis tasks could run concurrently — was released before the send() call. This meant a new partition could start synthesizing even while the previous one's output was still waiting to be sent. The result was unbounded accumulation: completed partitions piled up in memory, each holding 12 GiB of evaluation vectors, until the system ran out of memory.

The previous attempt to fix this (Phase 10's two-lock architecture) had failed due to fundamental CUDA device-global synchronization conflicts, forcing a reversion to Phase 9's single-lock design. Phase 11 addressed memory bandwidth contention with three targeted interventions. Now Phase 12's split API had reintroduced the memory accumulation problem in a new form.

The Semaphore+Channel Fix

The fix implemented in this round was elegant in its simplicity. Instead of the original design where the partition permit was released inside spawn_blocking (before the channel send), the permit was moved to be held until after the channel send succeeded. This single change meant that the total number of in-flight partition outputs was bounded by the number of semaphore permits (partition_workers).

But holding the permit through the send would have created a throughput problem if the channel were full — the synthesis task would block on send while holding the permit, preventing a new synthesis from starting. The companion fix was to auto-scale the channel capacity to match partition_workers. With channel capacity equal to pw, the send operation never blocks (there's always room in the channel), so holding the permit through the send adds zero latency. The permit gates only the synthesis itself, not the handoff.

This two-part fix — "partition permit held through send" plus "channel capacity auto-scaling" — created a natural backpressure system. The semaphore bounds the total number of in-flight outputs, while the channel provides the buffering needed to keep the GPU fed without blocking.

Results: From OOM to Production-Ready

The numbers tell the story. Before the fix, pw=12 (12 concurrent partition workers) would OOM at 668 GiB peak RSS — far exceeding the 755 GiB system budget and causing the process to be killed. After the fix, pw=12 completed successfully with a peak RSS of 383.8 GiB, well within the memory budget. The peak number of in-flight "provers" (completed partitions waiting for or being processed by the GPU) dropped from 19 to 18 — tightly bounded.

The throughput was 38.4 seconds per proof, slightly better than pw=10's 38.9 seconds. This is a critical data point: it shows that pw=12 provides a marginal throughput improvement over pw=10, but at a significantly higher memory cost (383.8 GiB vs 314.7 GiB). The optimal configuration for memory efficiency might be pw=10, while pw=12 is the choice when maximum throughput is desired and memory is available.

The Throughput Regression Investigation

No victory is ever complete in optimization work. The message immediately pivots to the next question: why is the throughput 38.4s/proof when the Phase 12 baseline was 37.1s/proof? That ~1.8 second regression (about 4.8%) demands investigation.

The assistant's hypothesis is that the buffer counters — instrumentation added to track memory pressure — are themselves causing the slowdown. These counters call eprintln! at every synthesis start, synthesis done, prove start, and prove finalize event. With 10 partitions per proof and 15 proofs in the benchmark, that's approximately 600+ synchronous stderr writes. Each write involves a system call, kernel transition, and potential lock contention on stderr. In a highly concurrent Rust async runtime, these synchronous I/O operations can create unexpected bottlenecks.

This hypothesis is testable: disable the buffer counters and re-run the benchmark. If throughput returns to ~37.1s, the counters are the culprit. If not, the regression lies elsewhere — perhaps in the channel capacity increase itself, or in the permit-holding restructuring.

Assumptions and Decisions

Several assumptions underpin this message. First, the assistant assumes that the 37.1s Phase 12 baseline is a reliable reference point. But earlier messages in the conversation show variance: the first run gave 37.1s, subsequent runs gave 38.8-39.3s. The 37.1s result might have been a statistical outlier — a "lucky run" where memory pressure was minimal and the glibc allocator happened to perform well. The assistant acknowledges this implicitly by planning to "disable buffer counters and re-test for baseline throughput," which will establish whether the regression is real or illusory.

Second, the assistant assumes that the buffer counters are the primary cause of the regression. This is a reasonable engineering hypothesis — synchronous stderr writes in a concurrent system are a known performance antipattern. But it's also possible that the channel capacity increase itself introduces overhead (more memory pressure from buffered outputs, more cache pressure, more TLB misses). The experiment will distinguish these possibilities.

Third, the assistant assumes that pw=12 is worth pursuing. The data shows only a marginal improvement over pw=10 (38.4s vs 38.9s, about 1.3% faster) at a 22% higher memory cost (383.8 GiB vs 314.7 GiB). The decision to benchmark pw=14 and pw=16 suggests the assistant is still searching for the optimal throughput point, even though the DDR5 memory bandwidth wall identified in Phase 11 may ultimately cap performance regardless of partition count.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The Groth16 proving system and its role in Filecoin's Proof-of-Replication is the foundation. The concept of partition synthesis — splitting a large circuit into independently provable chunks — is essential. The Rust async programming model with tokio, channels, and semaphores provides the implementation framework. CUDA GPU programming concepts (kernel launches, memory transfers, device synchronization) explain the GPU side. System-level memory management (RSS tracking, OOM behavior, glibc allocator fragmentation) explains the constraints. And the specific history of this optimization project — Phases 9 through 12, the failed two-lock architecture, the memory bandwidth interventions — provides the context for why this particular fix was pursued.

Output Knowledge Created

This message creates several pieces of knowledge. It establishes that the semaphore+channel fix is effective: pw=12 is now viable where it previously OOM'd. It provides quantitative benchmarks (38.4s/proof, 383.8 GiB RSS, 18 peak provers) that can be compared against future optimizations. It identifies the buffer counters as a suspected performance regression source, setting up the next investigation. And it updates the project roadmap, marking five tasks as completed and two as pending.

The Thinking Process

The message reveals a disciplined optimization methodology. The assistant does not simply celebrate the success — it immediately quantifies the gap to the theoretical best (37.1s baseline) and formulates a testable hypothesis for the discrepancy. This is the hallmark of systematic performance engineering: every result is interrogated, every regression is chased, every assumption is tested.

The todo list structure is revealing. The completed items trace the exact sequence of the fix: increase channel capacity, hold permit through send, build, benchmark pw=10, benchmark pw=12. The pending items show the next steps: disable counters, test higher pw values. This is not random exploration but a directed search along the dimensions of memory pressure (counters) and parallelism (pw).

The mention of "755 GiB budget" is also significant — it reveals the system constraint that defines success. The fix doesn't just reduce memory; it brings memory within a specific budget, making the configuration production-deployable. This is engineering under real-world constraints, not theoretical optimization.

Conclusion

Message 3188 captures a pivotal moment in the cuzk optimization journey. The semaphore+channel fix solved the Phase 12 memory accumulation problem, transforming pw=12 from an OOM death sentence into a viable configuration. But the message's true value lies not in the victory itself but in what follows: the disciplined investigation of the remaining throughput gap, the formulation of testable hypotheses, and the relentless pursuit of the next bottleneck. In performance engineering, every answer spawns a new question — and this message embodies that cycle perfectly.