The Semaphore That Didn't: Diagnosing a Memory Explosion in GPU Proof Generation

In a single, tightly-focused message within a months-long optimization campaign for Filecoin's Groth16 proof generation pipeline, an AI assistant identified a subtle concurrency bug that was silently consuming hundreds of gigabytes of memory. The message, part of the Phase 12 split GPU proving API implementation, reads in its entirety:

Line 1163: let _permit = permit; // held until synth complete — the permit is dropped at the end of the spawn_blocking closure (line 1181), i.e., when synthesis completes. Then the synth_tx.send(job).await (line 1199) can block while the permit is already released. So another synthesis task can grab the semaphore while this one is sitting on its ~16 GiB SynthesizedJob waiting for the channel.

The fix: hold the permit until after synth_tx.send() completes. Move the permit out of the spawn_blocking:

This is a masterclass in diagnostic reasoning. The message is deceptively short — a single paragraph of analysis followed by an edit command — but it represents the culmination of an intricate debugging journey through a distributed GPU proving system, and it reveals a fundamental misunderstanding about how a concurrency control mechanism was actually behaving.

The Context: A Pipeline Under Memory Siege

To understand why this message was written, one must understand the system it operates within. The codebase implements SUPRASEAL_C2, a high-performance Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) consensus mechanism. The pipeline is structured as a multi-stage assembly line: CPU-bound synthesis tasks produce large constraint evaluation vectors (the "a", "b", "c" NTT polynomials), which are then shipped to GPU kernels for multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations, and finally post-processed into compact Groth16 proofs.

The Phase 12 optimization had introduced a "split API" that decoupled the GPU worker's critical path from CPU post-processing by offloading the b_g2_msm computation to a background thread. This improved throughput modestly (from 38.0s to 37.1s per proof), but it exposed a severe memory pressure problem. When the assistant attempted to increase synthesis parallelism by raising partition_workers (pw) from 10 to 12, the process would exhaust the 755 GiB of available system memory and crash with out-of-memory (OOM) errors. The RSS peaked at 668 GiB — dangerously close to the system's physical limit.

The assistant had just spent several rounds building a global buffer tracker with atomic counters to gain visibility into every large buffer class in flight. This instrumentation was the key that unlocked the diagnosis.

The Diagnostic Trail: What the Buffer Counters Revealed

The buffer tracker logged events at every stage of the pipeline: when synthesis started (buf_synth_start), when the a/b/c vectors were freed after GPU submission (buf_abc_freed), and when the Rust dealloc thread completed (buf_dealloc_done). The peak readings were devastating:

provers=28: 28 ProvingAssignment sets alive, each holding ~12 GiB of a/b/c evaluation vectors = 336 GiB
aux=97-99: 97-99 aux_assignment buffers alive, each ~4 GiB = ~392 GiB
Total estimate: 727-732 GiB The provers counter measured how many synthesized partitions had completed CPU synthesis but had not yet been processed through prove_start (the GPU submission phase). With only 12 partition workers allowed to synthesize concurrently (controlled by a semaphore), the fact that 28 provers were alive meant that 16 partitions had finished synthesis and were sitting in memory, waiting to be consumed by the GPU. The channel between synthesis and GPU had a capacity of only 1 (synthesis_lookahead=1), so the bottleneck was obvious: partitions were completing synthesis faster than the GPU could consume them, and the backlog was accumulating in memory. But why was the backlog growing so large? With pw=12, at most 12 partitions should be in-flight at any time. If the GPU takes ~3.5 seconds per partition and synthesis takes ~5 seconds per partition, the system should roughly balance. The observed provers=28 indicated something was fundamentally wrong with the concurrency control.

The Core Insight: The Permit That Released Too Early

The assistant traced the problem to a single line of code:

let _permit = permit; // held until synth complete

This line was inside a tokio::task::spawn_blocking closure. The permit was acquired from a semaphore that was supposed to limit concurrent partition synthesis to pw (12). The comment said "held until synth complete" — but that was the bug. The permit was indeed held until synthesis completed, but it was dropped inside the spawn_blocking closure. After the closure returned, the code continued with an asynchronous synth_tx.send(job).await — but the permit was already released.

The consequence was devastating: as soon as a partition finished synthesis, the semaphore permit was returned to the pool, allowing another synthesis task to start. But the just-completed partition's SynthesizedJob — holding ~16 GiB of data — was still sitting in memory, blocked on the synth_tx.send() call because the GPU channel had capacity for only one item. The semaphore was supposed to limit the number of in-flight partitions, but because it released before the data was delivered to the next pipeline stage, it effectively only limited the number of actively synthesizing partitions — not the number of synthesized-but-not-yet-GPU-processed partitions.

This is a classic pitfall in pipeline design: the semaphore's scope was too narrow. It controlled the entry to the synthesis stage but not the exit from it. The pipeline could have up to pw partitions actively synthesizing, plus an unbounded number of completed partitions waiting to be sent through the channel. With pw=12 and a GPU that processes ~3 partitions in the time it takes 12 to synthesize, the backlog grew at roughly 9 partitions per cycle, quickly reaching 28 and consuming 336 GiB of memory just for the a/b/c vectors.

The Fix: Extending the Semaphore's Lifetime

The fix was conceptually simple but structurally significant: move the permit out of the spawn_blocking closure so that it remains alive until after the synth_tx.send().await completes. This changes the semaphore from limiting "number of partitions being synthesized" to limiting "number of partitions between start-of-synthesis and acceptance-into-GPU-channel." The permit now covers the entire critical section where memory is held.

The assistant applied the edit directly, restructuring the code to hold the permit across both the synthesis and the channel send. This single change would dramatically reduce peak memory because the semaphore now provides genuine backpressure: when the GPU channel is full, synthesis tasks cannot even start, preventing the backlog from accumulating.

Deeper Implications: What This Reveals About the System

This bug is revealing on multiple levels. First, it shows how easy it is for a concurrency primitive to silently fail at its intended purpose. The semaphore was working — it limited concurrent synthesis to 12 — but the actual resource that needed limiting was not "concurrent synthesis" but "in-flight synthesized data awaiting GPU consumption." The programmer who wrote let _permit = permit; // held until synth complete was correct about where the permit was held but wrong about what resource needed protection.

Second, it demonstrates the critical importance of instrumentation. Without the buffer tracker, the assistant would have seen only the OOM symptom and the high RSS. The buffer counters pinpointed exactly which buffers were accumulating (provers=28) and revealed the disconnect between the semaphore's intended behavior and its actual effect. The assistant built the instrumentation specifically to diagnose this problem, and it paid off immediately.

Third, the bug highlights a fundamental tension in pipelined systems: the desire to overlap computation (synthesis and GPU work) conflicts with the need to bound memory usage. The original design allowed synthesis to run ahead of the GPU, assuming the channel capacity would provide backpressure. But because the semaphore released before the channel send, the backpressure never materialized. The fix restores the intended behavior, but it also serializes synthesis and channel delivery to some degree — a trade-off the assistant would later need to revisit by increasing the channel capacity instead.

The Thinking Process: A Model of Diagnostic Reasoning

What makes this message remarkable is the clarity of the reasoning. The assistant didn't just notice that memory was high and try random fixes. It:

Instrumented the system with buffer counters to gain visibility
Collected data from a live run and identified the peak values
Formulated a hypothesis about why provers=28 exceeded pw=12
Traced the code path to find the exact line where the permit was dropped
Validated the hypothesis by reasoning about the timing: "the permit is dropped at the end of the spawn_blocking closure... Then the synth_tx.send(job).await can block while the permit is already released"
Designed the fix with surgical precision: "hold the permit until after synth_tx.send() completes" Each step builds on the previous one, and the reasoning is presented with economy and precision. The assistant doesn't explain what a semaphore is or why holding memory matters — it assumes the reader (or the user) has the necessary background. Instead, it focuses on the specific mechanism by which the bug manifests and the specific change needed to fix it.

Conclusion

This message captures a moment of diagnostic clarity in a complex optimization effort. A single misplaced semaphore permit was causing hundreds of gigabytes of unnecessary memory pressure, and the assistant identified it through careful instrumentation, data analysis, and code tracing. The fix — moving the permit out of the spawn_blocking closure — is a one-line structural change with profound consequences for memory usage. It's a reminder that in high-performance systems, the difference between a correctly functioning concurrency primitive and a subtly broken one can be measured in hundreds of gigabytes and the difference between a working system and an OOM crash.