The Memory Scaling Question: When a Single Number Becomes a Deployment Concern

"pce ram - is it static/amortised even with multiple GPUs? How much ram will an 8GPU system need with strong pipelining and maybe overprovision (2 pipelines per gpu vram permitting)"

This message, sent by the user at a critical juncture in the cuzk proving engine's Phase 5 development, is deceptively simple. On its surface, it is a straightforward technical question about memory characteristics of the newly implemented Pre-Compiled Constraint Evaluator (PCE). But beneath that surface lies a sophisticated systems-thinking mind probing the most important question in any optimization effort: does this actually work in production?

The Context: A Promising but Concerning Benchmark

The message arrives immediately after the assistant delivered the Phase 5 Wave 1 results summary ([msg 1459]). Those results were a mixed bag. The PCE had achieved a 1.42× speedup over the old synthesis path (35.5s vs 50.4s), and all 10 circuits × 130 million constraints validated bit-for-bit correct. But the assistant had also reported a troubling number: 375 GB peak RAM, up from 340 GB in the old path. The assistant's own explanation was terse: "10 parallel MatVec output buffers."

For a researcher or academic, a 1.42× speedup with a 10% memory increase might be an unqualified success. But the user is not an academic — they are someone thinking about deploying this system on real hardware, with real GPUs, in a real data center. The 375 GB number is a red flag. If the PCE multiplies memory usage by the number of GPUs, the economics of a multi-GPU deployment could collapse.

What the User Is Really Asking

The question contains three distinct layers of concern, each revealing a different facet of the user's mental model:

Layer 1: "Is it static/amortised even with multiple GPUs?" — This is the core architectural question. The user has understood that the PCE is a pre-compiled representation of the R1CS circuit, stored in a OnceLock as a static singleton. But they want confirmation: does this singleton truly remain a singleton when multiple GPU pipelines run concurrently? Or does each pipeline end up with its own copy? The word "amortised" is carefully chosen — it reveals that the user understands the PCE extraction cost (46.9s) is a one-time investment, but they need to know whether the memory cost is similarly amortized.

Layer 2: "How much ram will an 8GPU system need with strong pipelining?" — This is a concrete capacity planning question. The user is thinking about a real deployment: 8 GPUs, each running a pipeline. They need a number they can use to spec hardware. This is not idle curiosity; it determines whether the PCE approach is viable at all for their target infrastructure.

Layer 3: "maybe overprovision (2 pipelines per gpu vram permitting)" — This is the most sophisticated part of the question. The user is already thinking beyond the naive one-pipeline-per-GPU model. They want to know if they can run two pipelines per GPU, sharing GPU VRAM, to improve utilization. The parenthetical "vram permitting" shows they understand that GPU memory, not system RAM, may be the binding constraint. This reveals a user who is deeply familiar with GPU compute economics: GPUs are expensive, and underutilizing them is wasteful. Overprovisioning pipelines per GPU is a standard technique to keep compute resources saturated, but it only works if the memory budget allows it.

Assumptions Embedded in the Question

The user makes several assumptions, all of them reasonable and productive:

The PCE is fundamentally shareable. The question "is it static/amortised" assumes that the PCE could be shared. If the user believed it was inherently per-pipeline, they would have asked a different question entirely, perhaps "how can we deduplicate the PCE across GPUs?" The framing reveals trust in the architecture's potential.
System RAM, not just GPU VRAM, is the relevant constraint. The user asks about "ram" generically, then refines with "vram permitting." They understand that the PCE's CSR matrices (25.7 GiB) live in system memory, not on the GPU. The GPU only sees the witness vectors and the a/b/c output vectors during the MSM/NTT phases. This is a nuanced understanding of the cuzk architecture's memory hierarchy.
Pipelining is the right operational model. The user doesn't ask "can we run one proof at a time?" They assume pipelining — multiple proofs in flight concurrently to keep GPUs busy. This is the correct model for a proving system where CPU-bound synthesis feeds GPU-bound proof generation.
The 375 GB peak is a real concern, not a benchmark artifact. The user doesn't explicitly say this, but the question's existence implies they took the 375 GB number seriously. They are giving the assistant a chance to explain it away, but they're not assuming it's harmless.

What the User Might Have Missed

The user's question is sharp, but it does contain one implicit assumption that turns out to be incorrect: that the 375 GB peak is a property of the PCE design rather than an artifact of the benchmark. The assistant's subsequent investigation ([msg 1464]) reveals that the 375 GB came from the benchmark holding both the old-path results (~163 GiB) and the PCE-path results (~125 GiB) simultaneously for validation comparison. In production, only one path runs at a time, so the real peak is much lower.

This is a subtle point. The user was right to be alarmed by 375 GB, but the alarm was triggered by a measurement artifact, not a design flaw. The assistant's response — tracing the exact source code to verify the PCE is stored in a single OnceLock — is exactly the right approach. The user's suspicion that "we copy PCE for each partition and not dedupe" (their follow-up in [msg 1463]) was a reasonable hypothesis that the assistant was able to definitively rule out by reading the code.

Input Knowledge Required to Understand This Message

To fully grasp the user's question, a reader needs:

Knowledge of the PCE architecture: That the Pre-Compiled Constraint Evaluator replaces the enforce() phase of circuit synthesis with a pre-computed CSR sparse matrix, stored as static data. The 25.7 GiB figure for the CSR matrices (A: 11.6 GiB, B: 5.2 GiB, C: 10.7 GiB) comes from the assistant's earlier analysis.
Knowledge of the proving pipeline: That Filecoin PoRep C2 proof generation involves 10 partitions, each producing a separate circuit, and that synthesis is the CPU-bound phase that feeds the GPU-bound MSM/NTT computation. The pipeline is: C1 output → synthesis (witness generation + constraint evaluation) → GPU proof generation.
Knowledge of the benchmark methodology: That the pce-bench subcommand runs both the old path and the PCE path sequentially in the same process, holding all results for validation. This is the source of the 375 GB peak.
Knowledge of GPU deployment patterns: That "strong pipelining" means keeping multiple proofs in flight to maximize GPU utilization, and "overprovision" means running more pipeline instances than physical GPUs to absorb CPU/GPU latency mismatches.

Output Knowledge Created by This Question

The user's question catalyzes one of the most important investigations of the entire Phase 5 effort. In response, the assistant ([msg 1461]) produces:

A precise memory model breaking down PCE static costs (25.7 GiB for CSR matrices) vs per-pipeline working set (~21 GiB for witness + a/b/c vectors). This is the first time these numbers are rigorously enumerated.
A multi-GPU scaling projection: For 8 GPUs with 2 pipelines each (16 concurrent pipelines), total system RAM is ~738 GiB with PCE vs ~712 GiB without — only a 3.6% overhead. This directly answers the user's deployment question.
Identification of an optimization opportunity: The 4.2 GiB unified witness copy per pipeline could be eliminated with a split-lookup SpMV, saving 67 GiB system-wide on a 16-pipeline system.
A follow-up investigation ([msg 1464]) that traces the 375 GB peak to its source, confirming it is a benchmark artifact and not a design issue. The assistant reads the pipeline source code, finds the OnceLock, and verifies no duplication occurs.
A new benchmarking tool: The pce-pipeline subcommand with RSS tracking and malloc_trim calls, built to empirically validate the memory model. This produces clean sequential (155.7 GiB → 25.8 GiB → 181.6 GiB → 25.9 GiB) and parallel (310.9 GiB peak for 2 concurrent pipelines) results that confirm the model.

The Thinking Process Visible in the Question

The user's question reveals a particular mode of reasoning: deployment-first validation. Rather than asking "is the PCE fast?" (which the assistant had already answered), the user immediately jumps to "does this work at scale?" This is characteristic of someone who has been burned by optimizations that look great in benchmarks but fail in production.

The structure of the question — starting with a binary property ("is it static/amortised"), then asking for a concrete number ("how much ram"), then adding a complexity ("overprovision") — shows a mind iterating through levels of abstraction. First verify the fundamental property, then compute the budget, then optimize within that budget.

The phrase "vram permitting" is particularly telling. It shows the user is already running a mental capacity calculation: if each GPU has N GB of VRAM, and each pipeline needs M GB of VRAM for the GPU phase, then floor(N/M) pipelines per GPU is the theoretical max. But system RAM might be the real constraint if the PCE overhead is large. The user is testing both constraints simultaneously.

Why This Message Matters

This message is a turning point in the conversation. Before it, the assistant was focused on performance optimization — making the PCE faster, fixing correctness bugs, parallelizing the MatVec. The 375 GB number was noted but not deeply interrogated. The user's question forces a shift from performance optimization to deployment optimization.

The assistant's response — tracing code, building a memory model, creating a new benchmark to empirically validate it — is a direct consequence of the user's framing. Without this question, the assistant might have moved on to Wave 2 (specialized MatVec) or Wave 3 (SHA-256 SizedWitness) without ever resolving the memory concern. The user's intervention ensures that when those future optimizations land, they will land on a foundation with a well-understood memory model.

In the broader narrative of the cuzk proving engine development, this message represents the moment when a promising but untested optimization meets the reality of production deployment. The user's question is the crucible in which the PCE's viability is tested — and, as the subsequent investigation shows, it passes.