Chunk 31.0

## Summary This chunk implemented the memory backpressure mechanism for Phase 12's split GPU proving API, addressing the critical problem of synthesized partitions piling up in memory when synthesis outpaces GPU consumption. Three key changes were made: (1) **Early a/b/c free** — clearing ~12 GiB/partition of evaluation vectors immediately after `prove_start` returns, since the GPU no longer needs them; (2) **Channel capacity auto-scaling** — sizing the synthesis→GPU channel to `max(synthesis_lookahead, partition_workers)` instead of the hardcoded 1, preventing completed syntheses from blocking on `send()` while holding large allocations; (3) **Partition permit held through send** — the semaphore permit is now released only after the channel send succeeds (not just after synthesis), bounding total in-flight outputs to `partition_workers` without adding latency since the channel has room for all of them. The results are dramatic: pw=12 now completes successfully at 37.7s/proof with 400 GiB peak RSS, whereas previously it OOM'd at 668 GiB. The optimal configuration is pw=12 (partition_workers=12) with gw=2, gt=32, delivering the best throughput-to-memory ratio. Higher pw values (14, 16) consumed more memory without improving throughput, hitting the DDR5 bandwidth wall. The fix was committed as `98a52b33` with all buffer counters converted to `tracing::debug` for production cleanliness. The overarching theme is that careful memory backpressure design — using channel capacity as the natural throttle rather than coarse semaphore gating — can eliminate OOM conditions while preserving the throughput gains from the split API.

Taming the Memory Beast: How Three Targeted Interventions Solved Phase 12's OOM Crisis 2265 words

Message Articles