The Perils of Shadowing: A C++ Compiler Error Reveals a Use-After-Free Bug in Phase 12's Split GPU Proving API

Message Overview

The subject message, <msg id=3022>, is a single bash command and its output: an attempted build of the cuzk-daemon binary that fails with a C++ compilation error. The error is deceptively simple:

cargo:warning=cuda/groth16_cuda.cu(386): error: "provers" has already been declared in the current scope
cargo:warning=      const auto& provers = pp->provers_own...

A name collision. A trivial mistake. But this trivial error sits at the culmination of an intense debugging session that uncovered a subtle and dangerous use-after-free concurrency bug in high-performance CUDA code. The message captures the exact moment when a carefully reasoned fix meets the unforgiving reality of the C++ compiler — and loses.

Context: The Phase 12 Split API and the Use-After-Free Bug

To understand why this message matters, we must understand the architecture it was trying to repair. The system under development is cuzk, a high-performance Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). Phase 12 introduced a "split API" design: instead of performing the full proof generation synchronously on the GPU worker, the API was split into start_groth16_proof (which launches GPU kernels and returns immediately) and a separate finalization step that runs later. The goal was to offload the b_g2_msm (a multi-scalar multiplication on the G2 curve) from the GPU worker's critical path, hiding its ~1.7s latency behind other work.

But this split introduced a critical concurrency hazard. The C++ function generate_groth16_proofs_start_c spawns a background thread (prep_msm_thread) that reads from the provers array — a stack-allocated parameter containing Assignment<fr_t> structs with pointers into Rust-owned memory. In the old synchronous API, this thread was joined before the function returned, so the provers array remained valid throughout. In the Phase 12 split API, however, generate_groth16_proofs_start_c returns immediately, moving the still-running thread into a heap-allocated groth16_pending_proof handle. The provers array — a C function parameter living on the stack — goes out of scope when the function returns. The background thread continues reading from dangling memory.

As the assistant realized in <msg id=3013>: "This is a use-after-free bug! The prep_msm_thread is reading from freed memory!" The thread's lambda captured [&, num_circuits], taking provers by reference — a reference to a stack variable that evaporates upon return.

The Fix: Copy to Heap

The assistant's proposed fix was architecturally sound. The groth16_pending_proof struct (a heap-allocated object that outlives the function) already owned copies of the r_s and s_s arrays. The same pattern should apply to provers: add a std::vector<Assignment<fr_t>> provers_owned field to the struct, copy the function parameter's data into it early in the function, and redirect all subsequent code — including the background thread — to use this heap-stable copy.

The Assignment<fr_t> struct is a trivially copyable POD (plain old data) containing only pointers and sizes. Copying the array is cheap: it duplicates the pointer values, not the underlying data they point to (which remains Rust-owned and alive via the PendingProofHandle).

The assistant edited the code to add provers_owned to the struct, then copied the provers array into it right after allocating the pending proof handle. To avoid rewriting every reference to provers[c] throughout the ~800-line function, the assistant introduced a local reference alias:

const auto& provers = pp->provers_owned;

This single line was supposed to shadow the function parameter const Assignment<fr_t> provers[], making all subsequent code transparently use the heap-allocated vector. The thinking, as articulated in <msg id=3021>, was careful and nuanced: the assistant analyzed C++ reference semantics, lambda capture rules, and the lifetime of the heap-allocated pp object, concluding that the reference would safely resolve to pp->provers_owned even after the function returned.

The Compiler Error: Assumptions Meet Reality

The build failed. The compiler rejected the shadowing attempt with a clear error: "provers" has already been declared in the current scope. In C++, a function parameter occupies the function body's scope, and a local variable cannot redeclare the same name. The assistant's assumption that const auto& provers = ... could shadow the parameter was incorrect — C++ does not allow this, even though the types differ (a const Assignment<fr_t>* parameter vs. a const std::vector<Assignment<fr_t>>& local reference).

This is a subtle but important language rule. In many contexts, C++ permits inner scopes to shadow outer declarations, but a function parameter is considered part of the outermost block scope of the function body, and a local variable in that same block cannot reuse the name. The assistant's reasoning about what should work semantically collided with what the language actually permits syntactically.

The Deeper Significance

This message is remarkable not for its content — a failed build — but for what it represents. It is the visible surface of a much deeper debugging process that spanned dozens of messages and involved:

  1. Memory pressure analysis: The assistant had been monitoring RSS growth during benchmarks, watching it climb from 70 GiB to 367 GiB, and diagnosing whether it was a leak or accumulation ([msg 3010]).
  2. Code archaeology: Tracing through C++ and Rust code to understand what the prep_msm_thread actually reads, and whether those reads happen after the function returns ([msg 3011], [msg 3014], [msg 3015]).
  3. Concurrency reasoning: Understanding the difference between the old synchronous API (where the thread was joined before return) and the new split API (where the thread outlives the function), and realizing that the [&, num_circuits] lambda capture was capturing a reference to a stack variable ([msg 3013]).
  4. Language-lawyering: Debating whether C++ reference capture semantics would save them — does [&] capture the referent or the reference variable? Does the pointer value survive on the stack? ([msg 3016]). The compiler error in <msg id=3022> is the moment when all this careful reasoning meets a concrete obstacle. It forces a trivial but necessary adjustment: rename the local alias. In the very next message (<msg id=3023>), the assistant recognizes the error and uses a different name, and the build proceeds.

Input and Output Knowledge

To understand this message, the reader needs: knowledge of C++ function parameter scoping rules and why a parameter cannot be shadowed by a local variable; understanding of the Phase 12 split API design and why the provers array must outlive the function; familiarity with lambda capture semantics ([&] captures by reference); and awareness of the use-after-free bug that motivated this fix.

The message creates new knowledge in the form of a concrete compiler error that validates the naming collision. It also implicitly documents that the assistant's first attempt at the fix was rejected, requiring a second attempt. The message serves as a checkpoint: the fix was conceptually correct but syntactically flawed, and the error provides clear guidance for the correction.

The Thinking Process

The assistant's reasoning, visible across the preceding messages, shows a methodical approach to concurrency debugging. The chain of reasoning moves from observation (RSS grows during benchmarks) to hypothesis (synthesis data held too long) to investigation (what does the background thread read?) to discovery (the provers array is a dangling reference) to design (copy to heap) to implementation (add provers_owned, create alias) to failure (compiler rejects shadowing). Each step builds on the previous one, and the assistant documents its reasoning extensively, including second-guessing itself about C++ reference semantics.

The error in <msg id=3022> is a reminder that even the most careful reasoning can stumble on a simple language rule. The fix is trivial — rename the alias — but the bug it was intended to fix is anything but trivial. A use-after-free in concurrent GPU code, where a background thread reads from freed stack memory, could cause silent corruption, incorrect proofs, or crashes that are nearly impossible to reproduce deterministically. The assistant's discovery and attempted fix of this bug, culminating in this failed build, represents a genuine improvement in the correctness of the Phase 12 split API.