The Moment of Suspicion: Tracing a Memory Leak Through Rust's Async Dealloc

A Single Message in the Phase 12 Optimization Campaign

Subject Message (msg id=3001): `` [assistant] OK, 200 C++ deallocs for 200 partitions — that's fine. The C++ side is cleaning up. Let me now think about the Rust side. In finish_pending_proof`, the provers/assignments are moved into a dealloc thread. But does that thread actually get to run timely?

>

Actually, let me re-read the finish_pending_proof carefully: [read] /home/theuser/curio/extern/bellperson/src/groth16/prover/supraseal.rs ```

This message, appearing at index 3001 of a sprawling optimization conversation spanning dozens of segments and thousands of messages, represents a critical inflection point. The assistant has just received two OOM (Out of Memory) failures while attempting to increase synthesis parallelism from 10 to 12 and 15 workers. The user's cryptic observation — "It's not 400GB tho, something is leaking somewhere" (msg id=2993) — has reframed the problem. What initially appeared to be a simple capacity ceiling (not enough RAM for more concurrent work) is now suspected to be a memory leak. This message is the moment the assistant pivots from performance benchmarking to forensic memory analysis.

The Context: Phase 12's Split GPU Proving API

To understand why this message matters, we must understand what Phase 12 accomplished. The Phase 12 "split API" was a structural redesign of the Groth16 proof generation pipeline. In the previous Phase 11, the GPU worker performed proof generation monolithically: it would send work to the GPU, wait for all GPU operations (including the b_g2_msm — a multi-scalar multiplication on the G2 curve), and only then return results. The critical insight was that b_g2_msm runs on the GPU but doesn't need the GPU worker to block — it can run asynchronously while the GPU worker picks up the next partition. Phase 12 split the API into start_groth16_proof (which initiates GPU work and returns a PendingProofHandle) and finish_pending_proof (which joins the background b_g2_msm thread and completes finalization).

The performance win was modest but real: 37.1 seconds per proof versus the Phase 11 baseline of 38.0 seconds, a ~2.4% improvement. But the structural change introduced a new memory lifecycle. In the old monolithic path, the Rust-side synthesis data (the provers, input_assignments, and aux_assignments — collectively ~13 GiB per partition) was freed synchronously when gpu_prove returned. In the new split path, this data must remain alive until the background b_g2_msm thread finishes reading it. The PendingProofHandle holds these allocations, and they are only freed when finish_pending_proof runs — which happens on a spawned finalizer task after the GPU worker has already moved on.

The Reasoning: Why This Message Was Written

The assistant's reasoning in this message is a textbook example of systematic debugging under uncertainty. Three facts are in tension:

  1. The C++ side reports clean deallocation: The assistant had just confirmed (msg id=3000) that exactly 200 C++ async deallocations occurred for 200 partitions — a perfect 1:1 ratio. The C++ groth16_pending_proof struct's split_vectors and tails were being freed correctly.
  2. The memory pressure is real: Attempting partition_workers=12 (pw=12) caused OOM after 8 proofs, and pw=15 failed immediately. Yet the arithmetic didn't add up — adding just 2 more concurrent syntheses (~26 GiB extra) shouldn't exhaust a 755 GiB system.
  3. The user suspects a leak: The user's intervention — "It's not 400GB tho, something is leaking somewhere" — provides the crucial reframing. The assistant had been thinking in terms of capacity ceilings; the user is thinking in terms of unreclaimed memory. The message begins with "OK, 200 C++ deallocs for 200 partitions — that's fine. The C++ side is cleaning up." This is the assistant confirming that the C++ memory management is sound, then pivoting: "Let me now think about the Rust side." This pivot is the key reasoning step — the assistant is narrowing the search space to the Rust-side deallocation path, specifically the finish_pending_proof function.

The Thinking Process Visible in the Message

The assistant's thinking is laid bare in the transition between the first and second paragraphs. The first paragraph poses a question: "In finish_pending_proof, the provers/assignments are moved into a dealloc thread. But does that thread actually get to run timely?" This is a question about scheduling — not about whether deallocation happens, but whether it happens fast enough to prevent memory accumulation.

The second paragraph begins with "Actually, let me re-read the finish_pending_proof carefully" — the word "Actually" signals a self-correction. The assistant realizes that it needs to verify its mental model of the deallocation flow against the actual source code before reasoning further. This is a disciplined debugging practice: don't speculate about code behavior when you can read the code.

The message then issues a [read] tool call to examine finish_pending_proof in supraseal.rs. The assistant is about to discover something critical: the Rust-side deallocation uses a static DEALLOC_MTX (a global mutex) to serialize deallocation threads, preventing concurrent munmap() TLB shootdown storms. But this serialization creates a bottleneck — if deallocation threads queue up on the mutex, each holding ~13 GiB of data waiting to be freed, memory accumulates.

Input Knowledge Required to Understand This Message

To fully grasp what's happening here, one needs knowledge spanning several domains:

Groth16 proof generation: The message assumes familiarity with the structure of Groth16 proofs, particularly the multi-partition architecture where a single proof is split into 10 partitions, each requiring ~13 GiB of synthesis data. The b_g2_msm (multi-scalar multiplication on the G2 elliptic curve) is a specific GPU operation that Phase 12 offloaded to run asynchronously.

CUDA and GPU programming: The split API design relies on CUDA's ability to launch asynchronous operations. The prep_msm_thread runs on the CPU while GPU kernels execute, reading Rust-owned memory through raw pointers. This creates a lifetime constraint: the Rust data must outlive the background thread.

Rust's async runtime and memory model: The message deals with spawn_blocking (Tokio's mechanism for running blocking operations on a dedicated thread pool), std::thread::spawn (OS threads), and static Mutex synchronization. The assistant is reasoning about how these interact to delay memory reclamation.

Linux memory management: The malloc_trim(0) calls mentioned in surrounding messages (msg id=2997-2998) reflect an understanding of glibc's memory allocator behavior — specifically that malloc_trim is per-thread and that freed memory may not be returned to the OS without explicit trimming.

The Phase 12 codebase: The assistant is working with code it just wrote. The PendingProofHandle struct, the finish_pending_proof function, and the DEALLOC_MTX pattern were all introduced in Phase 12. Understanding the message requires knowing that these are new code paths with unproven memory behavior.

Output Knowledge Created by This Message

This message produces several forms of knowledge:

A confirmed hypothesis boundary: The assistant has ruled out a C++ memory leak (200 deallocs for 200 partitions) and narrowed the investigation to the Rust-side deallocation path. This is negative knowledge — knowing where the problem isn't — which is often more valuable than positive knowledge in debugging.

A specific question to investigate: "Does the dealloc thread actually get to run timely?" This frames the subsequent investigation. The assistant will go on to discover that the DEALLOC_MTX serialization causes deallocation threads to queue up, and that the PendingProofHandle holds synthesis data alive longer than necessary because the prep_msm_thread captures references to Rust-owned memory.

A methodology for memory debugging: The message demonstrates a pattern: verify one layer (C++ deallocs), then pivot to the next layer (Rust deallocs), reading source code rather than speculating. This methodology will pay off in the subsequent messages where the assistant builds a global buffer tracker with atomic counters to diagnose the real bottleneck.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message:

That 200 C++ deallocs for 200 partitions proves correctness: While the count matches, the assistant doesn't yet verify that each deallocation actually frees the expected amount of memory. A dealloc function could be called but only partially free memory. This assumption turns out to be correct, but it's worth noting.

That the Rust dealloc thread model is the same as pre-Phase 12: The assistant notes that the DEALLOC_MTX pattern is "the same as the pre-Phase 12 code." While structurally similar, the timing is different — in Phase 12, the dealloc happens later (after the finalizer task runs) and the data is held alive longer (in the PendingProofHandle). The assistant hasn't yet fully accounted for this timing shift.

That malloc_trim on any thread triggers a global trim: In msg id=3004 (the next message), the assistant will question this assumption, noting that "malloc_trim only affects the calling thread's arena in glibc." This turns out to be a nuanced point — modern glibc does perform a global trim from any thread, but the freed pages may not be visible until the dealloc thread actually completes the drop().

That the bottleneck is deallocation throughput: The assistant suspects that dealloc threads queueing on the mutex cause memory accumulation. While this is partially true, the deeper investigation in subsequent messages (msg id=3010-3012) will reveal that the real issue is more subtle: the prep_msm_thread captures a dangling reference to the stack-allocated provers array, creating a use-after-free bug that coincidentally keeps memory alive. The memory accumulation is actually caused by the partition semaphore releasing too early, allowing synthesized jobs to pile up while blocking on the single-slot GPU channel.

The Broader Significance

This message is a turning point in the Phase 12 optimization campaign. Before this message, the assistant was operating in "performance tuning" mode — adjusting parameters (pw=10, 12, 15) and measuring throughput. After this message, the assistant shifts to "memory debugging" mode — tracing allocations, building instrumentation, and eventually discovering both a critical use-after-free bug and a pipeline scheduling problem.

The message also illustrates a key dynamic in human-AI collaboration: the user's brief intervention ("something is leaking somewhere") provides the conceptual reframing that the assistant needed. The assistant had the data (OOM at pw=12) but was interpreting it through the wrong lens (capacity ceiling). The user's intuition about a leak redirected the investigation, leading to the discovery of real bugs.

In the subsequent messages, the assistant will: