Chunk 24.0

## Summary This chunk focused on deploying and debugging the pinned memory pool fix for GPU underutilization, uncovering and addressing two critical issues that prevented it from working effectively. First, the pinned pool's budget integration caused silent fallback to heap allocations: `PinnedAbcBuffers::checkout()` called `budget.try_acquire()` for memory already accounted for in per-partition budget reservations, so with 5 jobs × 16 partitions consuming ~362 GiB, the pinned allocations were denied and every synthesis completed with `is_pinned=false`. Removing budget from the pool entirely (pinned2) fixed this, and subsequent logs confirmed `pinned prover created` and `is_pinned=true` completions. Second, the same budget exhaustion prevented PCE caching — `insert_blocking` looped forever trying to acquire 15.8 GiB against the 5 GiB remaining, forcing all synthesis through the slow `enforce()` path instead of the fast PCE path. To address the root cause of budget pressure, the team implemented a GPU queue depth throttle (`max_gpu_queue_depth = 8`) that pauses synthesis dispatch when too many partitions are queued waiting for GPU, freeing budget for PCE caching and reducing memory bandwidth contention. This was deployed as pinned3 alongside a config update. The throttle successfully allowed PCE to be cached (385 GiB budget used, 15 GiB for PCE), but the initial batch of 80 partitions was already dispatched before PCE was available, so the pipeline is still draining those slow unpinned partitions. The user observed that when 20+ syntheses run simultaneously, GPU activity drops to near zero (likely blocked on pinned pool allocation contention), and suggested the dispatch should modulate more smoothly — starting synthesis incrementally as GPU consumes slots rather than dumping everything at once. The PCE disk save race (rename failures on overlay FS) is benign since the in-memory cache works, and the pinned pool is now allocating successfully, but the PCE fast path still doesn't use pinned backing for a/b/c vectors, leaving H2D transfers as the remaining bottleneck once PCE synthesis becomes active.

From Silent Fallback to Near-Zero Transfers: The Full Arc of GPU Pinned Memory Pool Optimization 2508 words

Message Articles