The Channel Capacity Fix: How a One-Line Change Unlocked Memory-Efficient GPU Proving at Scale

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, memory is the unforgiving constraint. The SUPRASEAL_C2 pipeline, responsible for generating Groth16 zero-knowledge proofs for Filecoin sectors, routinely consumes upwards of 200 GiB of memory during a single proof cycle. When the Curio team introduced Phase 12's split GPU proving API — which decoupled GPU lock acquisition from proof finalization to hide latency — they unlocked throughput gains but also triggered a subtle memory pressure crisis. At partition_workers=12, the daemon would exhaust system memory and crash with an out-of-memory (OOM) error, peaking at 668 GiB RSS. The fix, delivered in a single message (msg 3144), was deceptively simple: auto-scale the channel capacity to match the number of partition workers. But behind that one-line change lies a deep story about backpressure design, concurrency models, and the hidden costs of bounded channels.

The Message

[assistant] Now I'll make the channel capacity auto-scale when partition workers are configured. The key insight: when partition_workers > 0, up to partition_workers partitions can complete synthesis concurrently and then compete to send on the channel. We should size the channel to partition_workers so they can all enqueue without blocking: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

The Architecture That Created the Problem

To understand why this change was necessary, we must first understand the pipeline architecture that preceded it. The SUPRASEAL_C2 proving engine operates as a two-stage pipeline. Stage 1 is CPU-bound synthesis: the system takes vanilla Filecoin proofs, builds circuits, and runs synthesize_circuits_batch() to produce intermediate state — a/b/c evaluation vectors, density trackers, and witness assignments. Stage 2 is GPU-bound proving: the GPU takes these synthesized partitions and runs the heavy multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations required for Groth16 proof generation.

Between these two stages sits a bounded channel. The channel's purpose is to provide backpressure: if the GPU workers are busy and the channel is full, the synthesis task blocks, preventing unbounded memory growth from pre-synthesized proofs piling up. This is a classic producer-consumer pattern, and it worked well for the original design where synthesis_lookahead (the channel capacity) was set to 1, meaning at most one completed proof could be waiting for the GPU.

The complication came from the introduction of partition workers. Filecoin PoRep proofs are large — a single 32 GiB sector requires processing that can be split across multiple partitions. The partition_workers parameter controls how many partitions can be synthesized concurrently, exploiting CPU parallelism to reduce wall-clock time. With partition_workers=10, up to 10 synthesis tasks run simultaneously, each producing a partition's worth of intermediate state.

The critical insight that the assistant identified is the mismatch between these two concurrency parameters. The channel capacity remained at 1 (the default synthesis_lookahead), but up to partition_workers syntheses could complete at roughly the same time. When 10 partitions finished synthesis nearly simultaneously, 9 of them would block on synth_tx.send(), each holding roughly 16 GiB of allocated memory (or about 4 GiB after an earlier "early a/b/c free" optimization). This pile-up was the root cause of the OOM: at partition_workers=12, the memory spike reached 668 GiB before the kernel's OOM killer stepped in.

The Reasoning Behind the Fix

The assistant's thinking process, visible in the messages leading up to the fix (msg 3140–3142), reveals a careful diagnostic procedure. First, it confirmed the current configuration by reading the TOML files used for benchmarking, finding that none of them set synthesis_lookahead explicitly, meaning it defaulted to 1. Next, it traced the interaction between the partition semaphore and the channel by reading the engine source code. It then articulated the core problem in a structured analysis:

pw=10: 10 partitions synthesize concurrently, but channel capacity is 1. Completed syntheses pile up blocked on synth_tx.send(), each holding ~16 GiB (or ~4 GiB with early a/b/c free). This works at pw=10 (~367 GiB peak) but OOMs at pw=12 (668 GiB). Semaphore fix (holding permit until after send) capped memory but killed throughput (40.5s → regression). The right fix: Increase the channel capacity so completed syntheses flow into the channel buffer instead of blocking while holding their full memory allocations.

This analysis demonstrates a sophisticated understanding of the system's dynamics. The assistant considered and rejected an earlier approach (the semaphore fix) that had already been tried and found to cause a throughput regression. It recognized that the semaphore fix — holding the partition permit until after the channel send succeeded — effectively serialized the pipeline, negating the benefits of parallel synthesis. The correct approach was to give the channel enough capacity to absorb the burst of completed syntheses, allowing them to enqueue without blocking while still maintaining backpressure: when the channel is full, the (pw+1)th send blocks, bounding memory at roughly 2 × pw × per_partition_size.

The Assumptions Embedded in the Fix

The fix rests on several assumptions that are worth examining. First, it assumes that the GPU workers can consume partitions at roughly the rate that synthesis produces them. If GPU consumption is slower, the channel will fill up and synthesis will block — which is the desired backpressure behavior. But if GPU consumption is much slower, the channel capacity of partition_workers could still allow significant memory accumulation. The assistant's analysis of earlier benchmarks (documented in Phase 9 and Phase 11) had established that the GPU was not the bottleneck; rather, DDR5 memory bandwidth was the limiting factor.

Second, the fix assumes that the memory bound of ~2 × pw × per_partition_size is acceptable for the target hardware. With partition_workers=12 and roughly 4 GiB per partition (after early a/b/c free), this gives ~96 GiB of in-flight partition data, plus the GPU's own allocations (~12 GiB for NTT evaluation vectors), plus the SRS parameters (~40 GiB for porep-32g). The total of ~150 GiB fits comfortably in a 256 GiB machine, and indeed the benchmark results showed 400 GiB peak RSS at pw=12 — higher than the estimate due to other allocations, but well within the system's limits.

Third, the fix assumes that the synthesis_lookahead configuration parameter should be overridden when partition mode is active. This is a design choice: rather than requiring users to manually set synthesis_lookahead to match partition_workers, the system auto-scales. This is sensible because the two parameters are coupled — there is no scenario where you want synthesis_lookahead < partition_workers in partition mode, as it would recreate the original OOM problem.

What Knowledge Was Required

Understanding this message requires knowledge at multiple levels. At the system level, one must understand the producer-consumer pattern with bounded channels, and how backpressure prevents unbounded memory growth. At the application level, one must understand the SUPRASEAL_C2 pipeline architecture: the split between CPU synthesis and GPU proving, the role of partitions, and the memory footprint of each stage. At the code level, one must be familiar with Rust's tokio::sync::broadcast and bounded channel APIs, the configuration system in config.rs, and the engine initialization code in engine.rs.

The assistant had built this knowledge incrementally over the course of the session. Earlier phases (9, 10, 11) had established the memory accounting framework, the PCIe transfer optimizations, the DDR5 bandwidth wall analysis, and the split API design. The "early a/b/c free" optimization in Phase 12 had already reduced per-partition memory from ~16 GiB to ~4 GiB by deallocating evaluation vectors immediately after prove_start returned. The channel capacity fix was the final piece that made the memory profile manageable.

The Outcome

The fix was committed as part of 98a52b33 and benchmarked at partition_workers=12, achieving 37.7 seconds per proof with 400 GiB peak RSS — down from 668 GiB at OOM. The optimal configuration was pw=12 with gpu_workers_per_device=2 and gpu_threads=32. Higher values (pw=14, pw=16) consumed more memory without improving throughput, confirming that the DDR5 bandwidth wall, not the channel capacity, was the ultimate bottleneck.

What makes this message remarkable is its economy. The assistant identified a fundamental design flaw — a mismatch between two concurrency parameters that caused a catastrophic memory explosion — and fixed it with a single edit. The edit itself was likely a one-line change to the channel creation code, replacing a hardcoded or config-driven capacity with max(synthesis_lookahead, partition_workers). But the reasoning behind that line required tracing through the entire pipeline, understanding the memory lifecycle of synthesized partitions, and rejecting a previous incorrect fix. It is a textbook example of how deep system understanding enables minimal, precise interventions.