The Dangling Pointer: How a Use-After-Free Bug Was Discovered in a GPU Proving Pipeline

Introduction

In the high-stakes world of zero-knowledge proof generation for Filecoin's Proof-of-Replication (PoRep), every microsecond counts and every memory byte matters. The SUPRASEAL_C2 Groth16 proving pipeline, a complex chain spanning Go, Rust, C++, and CUDA, pushes hardware to its limits — generating proofs that consume ~200 GiB of peak memory across CPU synthesis and GPU computation. When the Phase 12 "split API" optimization was introduced to offload the b_g2_msm (Miller loop scalar multiplication) computation from the GPU worker's critical path, it promised a clean throughput gain by decoupling GPU work from CPU post-processing. But as message [msg 3013] reveals, the optimization harbored a subtle and dangerous bug: a use-after-free error that could silently corrupt proofs or crash the system.

This article examines that single message — a moment of discovery where the assistant, while investigating a puzzling memory accumulation issue, traced the data flow and uncovered that the C++ background thread was reading from Rust-owned memory that had already been freed. The message is a masterclass in systems-level debugging: tracing pointer lifetimes across language boundaries, questioning assumptions about ownership, and connecting the dots between a performance regression and a latent memory safety violation.

The Context: Phase 12's Split Architecture

To understand the discovery in [msg 3013], we must first understand what Phase 12 changed. The original Groth16 proving pipeline was synchronous: for each partition of the circuit, the GPU worker would (1) synthesize the circuit on the CPU, (2) transfer data to the GPU, (3) run the GPU kernels (NTT, MSM, etc.), and (4) wait for the b_g2_msm to complete before returning. Phase 12 split this into two phases: start_groth16_proof would initiate the GPU work and return immediately with a PendingProofHandle, while a separate finish_pending_proof call would later join the background b_g2_msm thread and complete the proof. This allowed the GPU worker to pick up the next partition's synthesis without waiting for the previous partition's b_g2_msm to finish — a classic latency-hiding technique.

The Rust FFI boundary was the critical seam. On one side, Rust held the synthesized circuit data: provers, input_assignments, and aux_assignments — massive vectors of scalar field elements representing the circuit's wire assignments. On the other side, C++ CUDA code needed to read these values to prepare the MSM computation. The start_groth16_proof function in Rust would build a Vec<Assignment<S>> — a vector of structs containing raw pointers into the Rust-owned data — and pass it to the C++ function generate_groth16_proofs_start_c. The C++ function would then spawn a background thread (prep_msm_thread) to process these assignments while the Rust function returned immediately.

The Investigation Begins: Memory Accumulation Under Load

The immediate trigger for the investigation was a memory problem. Earlier benchmarking (see [msg 2994] and [msg 2997]) had shown that increasing partition workers from 10 to 12 caused Out-of-Memory (OOM) errors on a 755 GiB system. The assistant initially suspected a memory leak — perhaps the PendingProofHandle was holding onto Rust data longer than necessary, or the async deallocation threads were stacking up behind a mutex.

The assistant methodically traced the memory lifecycle. It checked whether malloc_trim was being called in the new finalizer path (it was). It counted C++ deallocations versus GPU completions (200 each — perfectly matched). It examined the finish_pending_proof function to verify that the PendingProofHandle was properly consumed and its data freed. Everything looked correct on paper. Yet RSS was growing monotonically during runs, peaking at 367 GiB with pw=10 and 15 concurrent jobs — far higher than expected.

The Critical Insight: Tracing Pointer Lifetimes

The turning point came when the assistant asked a specific question: "Does the C++ prep_msm_thread (b_g2_msm) still reference the Rust-owned prover data?" ([msg 2995]). This was the right question because the entire Phase 12 optimization depended on the assumption that the C++ background thread had copied all the data it needed before start_groth16_proof returned. If the thread was still referencing Rust-owned memory after the function returned, the data would be freed prematurely — a classic use-after-free.

The assistant examined the C++ code and found the smoking gun:

std::thread prep_msm_thread([&, num_circuits]
{
    // ...
    auto& prover = provers[c];
    // reads prover.inp_assignment_data, prover.aux_assignment_data, etc.
});

The [&, num_circuits] capture captures all local variables by reference. The provers parameter — const Assignment<fr_t> provers[] — is a pointer to the Rust Vec's buffer. The thread captures a reference to this pointer variable, which lives on the C++ function's stack. When generate_groth16_proofs_start_c returns, the stack frame is destroyed, and the reference becomes dangling. Worse, the Rust Vec that owns the underlying Assignment structs is dropped when start_groth16_proof returns, freeing the memory entirely.

The Use-After-Free Bug: A Detailed Walkthrough

The bug chain is worth tracing step by step, as the assistant did in [msg 3013]:

  1. Rust side: start_groth16_proof builds a local Vec<Assignment<S>> containing raw pointers to input_assignments, aux_assignments, and density bit vectors. This Vec is a stack local.
  2. FFI call: The Vec's buffer pointer is passed to C++ as const Assignment<fr_t> provers[].
  3. C++ side: generate_groth16_proofs_start_c spawns prep_msm_thread with [&, num_circuits] capture — the thread captures a reference to the provers parameter (the pointer itself, not the data it points to).
  4. Return: generate_groth16_proofs_start_c returns to Rust. The Rust function start_groth16_proof returns, dropping the Vec<Assignment<S>> and freeing the buffer.
  5. Background: The prep_msm_thread continues running, reading from provers[c].inp_assignment_data — but provers now points to freed memory. The assistant's exclamation — "This is a use-after-free bug! The prep_msm_thread is reading from freed memory!" — captures the moment of realization. But then the assistant immediately qualifies: "Actually wait, let me re-read the C++ side more carefully." This is the hallmark of rigorous debugging: even after a breakthrough insight, double-check the evidence. The assistant then compares the new split-path code with the old synchronous function generate_groth16_proofs_c, which did not have this bug because it completed all work (including b_g2_msm) before returning — the provers array was alive for the entire function duration.

Why This Bug Survived Testing

The use-after-free bug is particularly insidious because it would not necessarily cause immediate crashes. The freed memory might still contain valid data for some time before being reused by another allocation. The background thread might read the correct values by luck, producing valid proofs. Or it might read corrupted data, producing incorrect proofs that pass verification by coincidence. The bug is a time bomb: under different load patterns, different memory layouts, or different allocator behavior, it could manifest as silent data corruption, crashes, or security vulnerabilities.

The fact that Phase 12 benchmarking showed a 37.1s/proof throughput (a ~2.4% improvement) without obvious failures is concerning — it means the bug was likely benign in the tested configuration but could have caused catastrophic failures in production.

Assumptions and Their Failure

The message reveals several assumptions that turned out to be incorrect:

Assumption 1: The C++ background thread copies all needed data before the function returns. This is the fundamental assumption of the split API design, and it was wrong. The thread captured references to stack-local data.

Assumption 2: The Rust Vec<Assignment<S>> is held alive by the PendingProofHandle. In fact, the PendingProofHandle only holds the provers as an opaque pointer (*const c_void), not as an owned Vec. The Vec itself is dropped when start_groth16_proof returns.

Assumption 3: The memory accumulation was caused by a leak in the new code path. The assistant initially suspected the async deallocation mechanism or the DEALLOC_MTX contention. The real issue turned out to be more fundamental: the memory was accumulating because the system was working correctly, but the use-after-free meant that the memory being "freed" was still in use by a background thread.

The Thinking Process: A Model of Systems Debugging

What makes [msg 3013] remarkable is the thinking process it reveals. The assistant doesn't jump to conclusions. It follows a systematic chain:

  1. Observe the symptom: Memory grows unboundedly under load.
  2. Formulate hypotheses: Is there a leak? Is the dealloc mutex causing contention? Is malloc_trim missing?
  3. Test each hypothesis: Count deallocs vs completions, check malloc_trim calls, examine the Rust dealloc path.
  4. Find the contradiction: Everything looks correct, yet memory is still accumulating.
  5. Ask a deeper question: "Does the C++ b_g2_msm thread reference any Rust-owned memory?"
  6. Trace the data flow: Follow the pointer from Rust Vec through FFI to C++ thread capture.
  7. Identify the root cause: The [&, num_circuits] capture creates a dangling reference.
  8. Validate against the old code: Check how the synchronous path avoided this bug. This is textbook systems debugging: start with the symptom, rule out simpler explanations, then trace the data flow across abstraction boundaries until the contradiction is found.

Input Knowledge Required

To fully understand [msg 3013], the reader needs:

Output Knowledge Created

This message creates several pieces of valuable knowledge:

  1. A documented use-after-free bug in the Phase 12 split API implementation, with precise location and mechanism.
  2. A corrected understanding of the memory lifecycle: The Rust Vec<Assignment<S>> must be kept alive until the C++ background thread completes.
  3. A fix direction: Either copy the provers array into heap-allocated memory owned by the groth16_pending_proof struct, or change the Rust side to hold the Vec alive until finalization.
  4. A debugging methodology: How to trace pointer lifetimes across language boundaries when investigating memory issues.

Conclusion

Message [msg 3013] captures a pivotal moment in the development of a high-performance GPU proving system. What began as a routine investigation into memory pressure uncovered a subtle and dangerous concurrency bug that could have caused silent data corruption in production. The assistant's systematic approach — questioning assumptions, tracing data flows, comparing old and new code paths — exemplifies the rigor required when building systems that span multiple languages and run on expensive, memory-constrained hardware.

The use-after-free bug in the Phase 12 split API is a cautionary tale about the risks of asynchronous programming across FFI boundaries. When a background thread in C++ captures references to data owned by Rust, the lifetimes must be explicitly managed — the default assumption that "the data will still be there" is not safe. The fix, implemented in subsequent messages, involved copying the provers array into the heap-allocated groth16_pending_proof struct, ensuring the background thread always accesses stable memory regardless of when the Rust function returns.

In the broader narrative of the SUPRASEAL_C2 optimization effort, this discovery was a turning point. It transformed the investigation from "why is memory accumulating?" to "how do we safely manage cross-language pointer lifetimes?" — a much more fundamental question with implications for the entire split API design. The bug fix not only resolved the immediate memory issue but also made the system more robust against future changes.