Segment 31

This sub-session addressed the critical memory backpressure problem in Phase 12's split GPU proving API, where synthesized partitions piled up in memory when synthesis outpaces GPU consumption. Three key changes were implemented: early freeing of ~12 GiB/partition evaluation vectors after prove_start, auto-scaling the synthesis→GPU channel capacity to match partition_workers, and holding the partition semaphore permit until after channel send succeeds. These changes eliminated OOM conditions at pw=12, achieving 37.7s/proof with 400 GiB peak RSS. The optimal configuration was determined to be pw=12 with gw=2, gt=32. Buffer counters were converted to tracing::debug for production readiness, and the fix was committed as 98a52b33.

Implement early a/b/c freeAuto-scale channel capacityHold permit through channel sendTune partition_workers parameterConvert buffer counters to tracing::debug

Taming the Memory Beast: How Three Targeted Interventions Solved Phase 12's OOM Crisis 2463 words

Chunks