Tracing the 375 GB Ghost: A Forensic Memory Investigation in the cuzk Proving Engine

In the high-stakes world of Filecoin proof generation, memory is the most precious resource. When the Phase 5 Pre-Compiled Constraint Evaluator (PCE) benchmark reported a staggering 375 GB peak RAM usage, it threatened to derail the entire optimization strategy. The user's reaction was immediate and pointed: "In previous run why was peak mem 375G? Did we copy PCE for each partition and not dedupe?" (see [msg 1463]). This question, sharp as a scalpel, cut directly to the central architectural promise of the PCE — that its overhead is a one-time static cost, amortized across all proofs and all GPUs. If the PCE were being duplicated per partition, the entire design would need rethinking.

The message at [msg 1465] is the second step in a forensic investigation that would ultimately exonerate the PCE architecture and reveal a much more mundane explanation. It is a single read tool call — the assistant reading lines 240–249 of pipeline.rs:

[assistant] [read] /home/theuser/curio/extern/cuzk/cuzk-core/src/pipeline.rs
<path>/home/theuser/curio/extern/cuzk/cuzk-core/src/pipeline.rs</path>
<type>file</type>
<content>240:         CircuitId::WinningPost32G => &WINNING_POST_PCE,
241:         CircuitId::WindowPost32G => &WINDOW_POST_PCE,
242:         CircuitId::SnapDeals32G | CircuitId::SnapDeals64G => &SNAP_DEALS_PCE,
243:     }
244: }
245: 
246: /// Extract a pre-compiled circuit from a circuit instance and cache it.
247: ///
248: /// This runs the circuit with `RecordingCS` to capture R1CS structure.
249: /// It...

On its surface, this message is unremarkable — a routine code read. But in context, it is a critical piece of evidence gathering. The assistant is not reading randomly; it is executing a deliberate investigation plan. The previous message ([msg 1464]) had already begun the inquiry: "Good catch. Let me trace exactly what happened in that run. The 375 GB peak is suspicious — it ran both the baseline (old path) and PCE path in the same process, plus the PCE extraction, plus the validation comparison. Let me check whether we're actually duplicating the PCE." That message had read the extract_and_cache_pce_from_c1 function. Now, in [msg 1465], the assistant reads the get_pce function — the code that returns a reference to the cached PCE for a given circuit type.

The Architecture Under Scrutiny

To understand why this read is significant, one must understand the PCE's memory model. The Pre-Compiled Constraint Evaluator works by capturing the R1CS structure of a circuit (the sparse matrices A, B, C that define the constraint system) into a static, serializable format. Instead of re-executing the full circuit synthesis — which involves allocating witness variables and enforcing constraints — the PCE path runs only the witness computation (the alloc() closures) and then performs a sparse matrix-vector multiply (SpMV) to produce the constraint outputs a, b, c.

The critical architectural decision was to store the PCE in a static OnceLock — a global singleton that is initialized once and shared by all threads and all pipelines. Lines 240–242 show the dispatch logic: each CircuitId maps to a static reference (&amp;WINNING_POST_PCE, &amp;WINDOW_POST_PCE, &amp;SNAP_DEALS_PCE). The OnceLock ensures that the first call to extract the PCE for a given circuit type runs the full RecordingCS synthesis, and all subsequent calls reuse the cached result.

The user's suspicion was that this deduplication might be failing — that each partition (the PoRep proof is split across 10 partitions, each with ~13 million constraints) was independently extracting and storing its own PCE, multiplying the 25.7 GiB static overhead by 10 to produce 257 GiB of duplicated matrix data. Combined with the working sets, this would plausibly explain the 375 GB peak.## The Investigation Unfolds

The assistant's reasoning in this message is subtle but important. By reading lines 240–249 of pipeline.rs, the assistant is verifying three things simultaneously:

  1. The PCE is stored in a static OnceLock: The &amp;WINNING_POST_PCE references point to static OnceLock&lt;PreCompiledCircuit&gt; variables. This confirms the singleton pattern — there is exactly one copy of the PCE per circuit type per process.
  2. The dispatch is correct: The get_pce function maps CircuitId to the appropriate static reference. There is no per-partition or per-pipeline instantiation happening here.
  3. No per-circuit duplication exists: If there were a bug where each circuit created its own PCE, it would appear in this function or in the synthesize_with_pce function that calls it. The fact that the function returns a &amp;&#39;static PreCompiledCircuit — a reference with static lifetime — confirms that all circuits share the same instance. However, this read alone does not complete the investigation. The assistant is building a chain of evidence. The previous read (in [msg 1464]) examined extract_and_cache_pce_from_c1 to confirm that extraction stores into the OnceLock. This read examines the retrieval side. The next reads (in [msg 1466] and [msg 1467]) will examine synthesize_with_pce and the benchmark code to trace the actual memory allocation pattern.

Assumptions and Knowledge Requirements

The assistant makes several assumptions in this message that are worth examining. First, it assumes that the OnceLock pattern is correctly implemented — that the static variables are truly shared and not somehow duplicated per thread or per call site. In Rust, static items are indeed process-global, so this is a safe assumption. Second, it assumes that the benchmark is the source of the 375 GB peak, not the production path. This assumption is based on the structure of the pce-bench subcommand, which the assistant had written earlier and knows runs both the old and new paths in sequence for validation.

To understand this message fully, the reader needs knowledge of several domains. They need to understand Rust's static OnceLock pattern and its memory semantics — that a OnceLock&lt;PreCompiledCircuit&gt; stores exactly one instance, initialized on first access, and all subsequent accesses return a reference to the same allocation. They need to understand the Filecoin PoRep proof structure: that a single sector proof is split into 10 partitions, each with ~13 million constraints, and that the PCE captures the constraint structure common to all partitions. They need to understand the CSR (Compressed Sparse Row) matrix format and why storing A, B, C matrices for 130 million constraints requires ~25.7 GiB. And they need to understand the benchmark methodology — that the pce-bench subcommand runs the old path first, then the PCE path, then compares results for correctness validation.

The Output Knowledge Created

This message produces a subtle but critical piece of knowledge: the PCE dispatch code is clean. By reading lines 240–249 and finding no evidence of per-circuit duplication, the assistant narrows the investigation. The bug, if it exists, is not in the PCE caching layer. The investigation must move to the benchmark code itself — specifically to whether the benchmark holds both old-path and PCE-path results simultaneously.

The assistant's subsequent messages confirm this hypothesis. In [msg 1466], the assistant reads synthesize_with_pce and finds a minor issue with density tracker duplication (~480 MiB) but nothing that explains 375 GB. In [msg 1467], the assistant reads the benchmark main.rs and discovers the truth: the benchmark keeps baseline_synth and pce_synth alive simultaneously for validation. The calculation follows: baseline's 10 ProvingAssignments at ~163 GiB, plus PCE path's 10 a/b/c vectors at ~125 GiB, plus input/aux Arc vectors at ~42 GiB, plus the PCE static at ~25.7 GiB, plus misc overhead — sums to exactly ~375 GiB.

The real production overhead is just 25.7 GiB static + 4.2 GiB temporary witness copy per pipeline. For an 8-GPU system with 2 pipelines per GPU (16 concurrent pipelines), the PCE adds only ~3.6% to total system memory (~738 GiB vs ~712 GiB without PCE). The 375 GB peak was a benchmark artifact — a ghost in the measurement apparatus, not a flaw in the architecture.

The Deeper Significance

This investigation exemplifies a pattern that recurs throughout engineering: a suspicious measurement threatens to invalidate an architectural decision, and the engineer must trace the measurement back to its source rather than accept it at face value. The user's question was not just about a number — it was about trust in the PCE design. If the PCE were being duplicated per partition, the entire Phase 5 optimization would need to be re-architected, potentially invalidating weeks of work.

The assistant's response demonstrates a methodical approach: instead of speculating or theorizing, it reads the actual code — first the extraction path, then the retrieval path, then the synthesis function, then the benchmark driver. Each read eliminates a hypothesis and narrows the search space. By the time the assistant reaches the benchmark code in [msg 1467], the explanation is almost inevitable: the benchmark holds both result sets simultaneously for validation, and the 375 GB peak is simply the sum of two complete proof pipelines plus the PCE static overhead.

This message, standing alone as a single read tool call, is a testament to the power of forensic code reading. It produces no new code, no optimization, no architectural change. It simply gathers evidence. But that evidence is the foundation upon which the entire memory validation — the pce-pipeline benchmark, the sequential and parallel memory measurements, the documentation update in cuzk-project.md — is built. Without this read, the investigation would have proceeded on guesswork. With it, the assistant can confidently state: the PCE is not duplicated. The ghost is in the benchmark.