The Two-Front War and the Null Result: How Phase 4 Delivered 13.2% While Proving a Quarter-Terabyte of Waste Didn't Matter

Introduction

In the high-stakes world of Filecoin proof generation, where a single 32 GiB PoRep C2 proof demands synthesizing over 130 million constraints through a Groth16 zk-SNARK pipeline, performance optimization is never a single battle. It is a campaign fought on multiple fronts, each requiring different weapons, different intelligence, and different tactics. Segment 15 of the cuzk proving engine development captures two such fronts in vivid detail: a triumphant multi-front optimization campaign that delivered a 13.2% end-to-end improvement, followed immediately by a beautifully reasoned hypothesis about allocation overhead that was completely invalidated by rigorous measurement.

The first front—the culmination of Phase 4—combined a surgical code change to eliminate unnecessary allocations in the synthesis hot path with the discovery and elimination of a hidden 10-second destructor bottleneck that had been silently wasting time in the GPU proving pipeline. The results were dramatic: synthesis time dropped from ~55.4 seconds to ~50.9 seconds (an 8.3% improvement confirmed by 91 billion fewer instructions), and after fixing the GPU wrapper regression, the total end-to-end proof time fell from 88.9 seconds to 77.2 seconds—a 13.2% improvement.

The second front was a masterclass in the gap between theory and measurement. The user posed a simple, elegant question: if deallocation was costing 10 seconds, surely allocation must have a comparable cost? The investigation revealed that a SynthesisCapacityHint API for pre-allocation already existed but was never wired up. The theoretical waste was staggering—~265 GB of redundant memory copies across 10 parallel circuits. Yet when the assistant wired up all six call sites and benchmarked the result, the impact was zero. The synthesis bottleneck was confirmed to be purely computational, directly motivating the Phase 5 Pre-Compiled Constraint Evaluator (PCE) approach.

This article tells the full story of Segment 15, drawing on the detailed accounts in [1] and [2] to weave together the methodological lessons, the technical discoveries, and the broader implications for performance engineering.

Part I: The Two-Front War — Phase 4 Culmination

The Synthesis Front: Boolean::add_to_lc

The Phase 4 optimization campaign began with data. Using perf record to capture 229,000 samples of the synthesis phase, the assistant constructed a remarkably precise breakdown of where ~55 seconds of CPU time were being spent. The critical insight, highlighted in the analysis, was that LC construction (23.52%) was actually larger than enforce itself (22.13%). The overhead of building linear combinations—the core data structure for representing constraint terms—exceeded the cost of the constraint system's core logic. And within LC construction, a single function stood out: Boolean::lc() consumed 6.51% of total synthesis time all by itself.

The Boolean::lc() method is a seemingly innocuous utility: it creates a new LinearCombination (backed by a Vec allocation) wrapping a single Boolean variable with its coefficient. In most circuits, a Boolean term has only 1-2 coefficients, so the Vec allocation is tiny. But the function is called in tight loops like UInt32::addmany, which iterates 32 times per operand. When you have millions of constraints, each requiring multiple Boolean operations, those tiny allocations multiply into a significant overhead.

The assistant had already tried several approaches to address allocation overhead. A Vec recycling pool for ProvingAssignment::enforce yielded only ~0.7% improvement because the dominant allocations were not in the six enforce Vecs but in the dozens of temporary LinearCombination objects created inside circuit gadget closures. A SmallVec optimization—replacing Vec<LinearCombinationTerm> with a small-buffer-optimized alternative—actually regressed performance because the enum discriminant checks and larger stack frames defeated Zen4's out-of-order execution engine, dropping IPC from 2.60 to 2.38. Software prefetch intrinsics added to eval loops provided only marginal benefit. An interleaved A+B eval approach was reverted after it hurt IPC.

The Boolean::add_to_lc approach was different. Instead of trying to make allocations cheaper (SmallVec) or recycle them (Vec pool), the assistant asked a more fundamental question: why allocate at all? The key insight was that most callers of Boolean::lc() don't need a new LinearCombination—they just need to add the Boolean's term(s) to an existing linear combination. By adding two new methods to the Boolean type:

The GPU Wrapper Front: The 10-Second Ghost

With the synthesis microbenchmark confirming the win, the assistant proceeded to the crucial step: running a full end-to-end proof through the cuzk daemon to verify correctness and measure the total impact. The result was a puzzle. Synthesis had indeed improved to 51.2 seconds. However, the GPU time had regressed: 36.0 seconds versus the baseline of 34.0 seconds. The net improvement was only 1.4 seconds (1.6%), far below the 4.5 seconds the synthesis microbenchmark had promised.

The assistant's first instinct was to suspect the bellperson wrapper layer—the Rust code that orchestrates data transfer to the GPU, invokes the CUDA kernels, and assembles the final proof. But the CUDA internal timing, emitted via fprintf(stderr) instrumentation embedded in the C++ code, showed gpu_total_ms=25657—essentially identical to the baseline of 25.3–25.8 seconds. The GPU compute itself was unaffected. The regression lived entirely in the wrapper.

By cross-referencing timestamps from the daemon log, a precise timeline emerged. The gap of 10.2 seconds fell entirely between the bellperson GPU prove finishing and the pipeline reporting completion. An exploration agent performed a deep dive, mapping the entire execution path from the Rust Instant::now() call through the C++ FFI boundary and into the CUDA kernels. It identified the critical insight: the Rust-side timer wraps the call to prove_from_assignments(), which internally calls the C++ generate_groth16_proof function. After the C++ function returns, the Rust side still holds ownership of massive data structures.

The diagnosis, crystallized in the analysis, was precise: "The 10.2s gap is almost certainly heap deallocation of ~37 GB in implicit destructors at function exit." When prove_from_assignments() returns, the C++ function's local variables—vectors holding split MSM bases, tail MSM data, and intermediate results—go out of scope. Their destructors must free the underlying heap memory. On the Rust side, the ProvingAssignment structs, moved into gpu_prove() as part of the SynthesizedProof, are also dropped when the function returns.

The assistant performed a back-of-the-envelope calculation that quantified the problem. Each ProvingAssignment has vectors a, b, c (each 130M × 32 bytes = 4.17 GB) and aux_assignment (23M × 32 bytes = 0.74 GB). For 10 circuits: 132.5 GB of Rust Vec allocations that get dropped synchronously. Plus the C++ side had approximately 37 GB of split_vectors and tail_msm_bases. Total: ~170 GB of heap memory being freed through synchronous destructor calls, blocking the calling thread for 10 seconds while the operating system unmapped pages, invalidated TLBs, and coalesced free chunks.

The fix was elegantly simple: move ownership to detached threads on both the C++ and Rust sides, allowing destructors to run in the background while the function returns immediately. The result was dramatic. GPU wrapper time dropped from 36.0s to 26.2s—matching CUDA internal timing exactly. Total E2E time fell to 77.2 seconds, a 13.2% reduction from the 88.9s baseline. A second run confirmed consistency at 77.0 seconds. All three timers—CUDA internal, bellperson, and pipeline wrapper—now reported identical values within milliseconds. The 10-second ghost had been exorcised.

All Phase 4 changes were committed as 2da2a901, including Boolean::add_to_lc, async deallocation on both C++ and Rust sides, the Vec recycling pool, software prefetch intrinsics, and the A4/D4 CUDA changes.

Part II: The Allocation Hypothesis — When a Quarter-Terabyte of Waste Changed Nothing

The Question That Launched an Investigation

The deallocation victory prompted a natural question from the user: "If there was a win in dealloc, is it possible that alloc can have a similar one?" This question launched a chain of investigation, implementation, and ultimately, a humbling encounter with reality that would reshape the team's understanding of where the synthesis bottleneck truly lies.

The assistant's response immediately validated the user's intuition: "If freeing ~130 GB of Rust Vecs took ~10s, then allocating them during synthesis must have a similar cost—it's the same mmap/page-fault overhead in reverse." A subagent task was dispatched to trace exactly how ProvingAssignment Vecs grow during synthesis.

The investigation revealed a striking picture. All five main Vecs in ProvingAssignment—a, b, c, input_assignment, and aux_assignment—start at capacity zero and grow organically via push(). For the 130 million constraints in a PoRep C2 proof, each Vec undergoes approximately 27 reallocation cycles as Rust's geometric doubling strategy doubles capacity each time it fills. Each reallocation follows the same pattern: mmap a new buffer of twice the current size, memcpy all existing data into the new buffer, then munmap the old buffer.

The theoretical waste was staggering. Per circuit, the geometric doubling series produces approximately 25 GB of redundant memory copies across all five Vecs. For 10 parallel circuits (the standard batch size), that's ~250 GB of wasted copies and ~810 mmap/munmap syscalls. These numbers are large enough that any engineer would expect them to matter.

But then came the critical discovery: the infrastructure to fix this problem already existed but was never wired up. Bellperson's supraseal module defined a SynthesisCapacityHint struct with fields for num_constraints, num_aux, and num_inputs, and a function synthesize_circuits_batch_with_hint that would pre-allocate all Vecs to their final capacity before synthesis began. The documentation even boasted that "providing accurate hints eliminates ~27 reallocation cycles per vector and avoids ~32 GiB of redundant memory copies for 32 GiB PoRep C2." Yet every call site in pipeline.rs passed None as the hint, leaving the optimization dormant.

Wiring Up Six Call Sites

The assistant's implementation strategy was methodical and carefully designed. Rather than hardcoding known values per circuit type (which would be fragile if circuit parameters changed), the assistant designed a global hint cache: the first synthesis of any circuit type would grow organically and record the actual final capacities; subsequent syntheses would use those cached hints for pre-allocation.

The implementation required several components. A global hint cache was created as a Mutex<HashMap<CircuitId, SynthesisCapacityHint>> protected by once_cell::sync::Lazy for thread-safe initialization. A helper function synthesize_with_hint was written to look up the cache for the given CircuitId, call synthesize_circuits_batch_with_hint with the cached hint (or None if not yet available), and after the first synthesis, store the hint from the resulting ProvingAssignment's actual capacities.

All six synthesis call sites in pipeline.rs needed to be updated: synthesize_porep_c2_multi (the multi-sector PoRep batch), synthesize_porep_c2_partition (single-partition PoRep), synthesize_porep_c2_batch (batch-mode PoRep from Phase 3), synthesize_winning_post, synthesize_window_post, and synthesize_snap_deals.

The implementation was not without its hiccups. A mass-edit using find-and-replace accidentally matched the wrong call site, requiring careful debugging and repair over several subsequent messages. The assistant had to grep for all occurrences, verify each replacement manually, and fix a mistaken replacement where a SnapDeals32G label ended up on a PoRep function. A compilation error also revealed that the CircuitId enum had unhandled variants (Porep64G, SnapDeals64G) that needed match arms. Through systematic effort, all six call sites were converted, the code compiled successfully, and the infrastructure was committed as 41999e0b with 117 insertions across pipeline.rs.

The Benchmark That Changed Nothing

The assistant ran a rigorous two-phase benchmark. First, a single-partition synth-only microbenchmark ran four iterations of PoRep C2 synthesis. The results were unambiguous:

| Iteration | Hint Status | Synthesis Time | |-----------|-------------|----------------| | Iter 1 | No hint (organic growth) | 50.7s | | Iter 2 | With cached hint | 50.9s | | Iter 3 | With cached hint | 51.0s | | Iter 4 | With cached hint | 50.9s |

Zero measurable impact. The variation between iterations was within measurement noise.

To be thorough, the assistant then ran a full end-to-end test with the daemon, comparing two consecutive proofs: the first with organic growth, the second with pre-allocation from the cached hint. The result was identical: 50.65 seconds of synthesis time in both cases.

The assistant's reaction was telling: "This is surprising given the ~250 GB of wasted copies we estimated." But the analysis that followed was a masterclass in understanding why.

Why Allocation and Deallocation Are Not Symmetric

The assistant's post-mortem analysis revealed a fundamental asymmetry between allocation and deallocation in this workload. The reasoning is worth examining in detail because it teaches a general lesson about memory management in high-performance computing.

Deallocation was expensive because munmap() of large contiguous regions is a synchronous kernel operation that tears down page tables all at once. After GPU proving completed, all ~130 GB of Rust Vecs and ~37 GB of C++ vectors were freed synchronously in their destructors. The munmap syscall must update page table entries, flush TLB entries across all cores, and return physical pages to the kernel's free list. With 130+ GB of memory, this takes real time—approximately 10 seconds. There was no computation to overlap with, no parallelism to hide the cost. The deallocation was a discrete, concentrated event on the critical path.

Allocation during synthesis, by contrast, happens incrementally through push() with geometric doubling. Each reallocation copies only the data that has been written so far. Early reallocations (when Vecs are small) are trivially cheap—copying 1, 2, 4, 8, 16 elements takes nanoseconds. The first ~20 doublings copy less than 256 MB total. By the time Vecs reach multi-gigabyte sizes, they double infrequently—only 3-4 times from 2 GB to 4 GB. Each of those late-stage reallocations copies 2-4 GB at memory bandwidth speed (~100ms on Zen4 with ~40 GB/s bandwidth), but they happen concurrently across 10 circuits thanks to rayon's parallel iterator.

The assistant quantified this precisely. For a single circuit, the total wasted copy across all reallocations is approximately 26.5 GB. At ~40 GB/s memory bandwidth, that's about 0.66 seconds of pure copy time. Distributed across 50.7 seconds of synthesis and overlapped with field arithmetic (which dominates the CPU profile at 16.98%), the reallocation cost is hidden in the noise.

The key insight is that Rust's geometric doubling strategy amortizes the cost beautifully. The average cost per push() is O(1) with a tiny constant. The mmap/munmap syscalls for the ~27 reallocations per Vec are fast because the kernel's virtual memory system handles them lazily—new pages are not actually faulted in until they're first accessed. And crucially, the reallocations happen during synthesis, interleaved with computational work, not after it like the deallocation.

The Confirmed Bottleneck: Synthesis Is Purely Computational

The zero-impact result of the allocation hypothesis has a profound implication: the synthesis bottleneck at ~50.8 seconds is now confirmed to be purely computational, not memory-management overhead. The time is dominated by field arithmetic (16.98% of CPU cycles), LC evaluation (13.53%), and other gadget operations—not by mmap/munmap syscalls or memcpy during reallocation.

This conclusion directly motivates Phase 5 of the roadmap: the Pre-Compiled Constraint Evaluator (PCE). PCE replaces circuit synthesis entirely with sparse matrix-vector multiply. Instead of traversing the circuit graph and evaluating each constraint individually, PCE pre-computes the R1CS matrix structure once per circuit topology and then generates witnesses using only alloc() closures, skipping enforce() entirely. The a/b/c vectors are produced by sparse matrix-vector multiply: a = A·w, b = B·w, c = C·w.

The projected impact is 3-5x faster synthesis, which would bring synthesis time from ~50s down to ~10-16s. Combined with the GPU time of ~26s, total proof time would be ~36-42s—approaching the 2-3x throughput target that Phase 4's "quick wins" failed to deliver.

The allocation hypothesis investigation was not wasted effort. It ruled out a plausible bottleneck, confirmed the true nature of the synthesis phase, and strengthened the case for the deeper architectural changes planned in Phase 5. In the high-stakes world of Filecoin proof generation, knowing where not to optimize is just as valuable as knowing where to optimize.

Part III: Methodological Lessons from Segment 15

Layered Instrumentation Is Essential

The most striking methodological lesson from this segment is the importance of instrumenting timing at every layer of a complex system. Without the CUDA internal timers (CUZK_TIMING), the 10-second GPU wrapper regression would have been invisible—the assistant would have seen 36.0 seconds of GPU time and assumed the CUDA code was slower. Only by comparing the Rust wrapper timer against the CUDA internal timer was the destructor overhead revealed.

This layered approach creates a diagnostic framework where anomalies become visible as gaps between timers. When the CUDA internal timer says 26 seconds but the Rust wrapper says 36 seconds, you know exactly where to look: the code between those two measurement points. This is far more powerful than having a single "total time" measurement, which hides all internal structure.

Microbenchmarks Measure Potential; E2E Tests Measure Reality

The Boolean::add_to_lc optimization delivered 8.3% improvement in isolation but only 1.6% in the first E2E test—not because the optimization was wrong, but because the system had other bottlenecks that masked its impact. Only by fixing all the bottlenecks (synthesis allocations, GPU destructor overhead) could the full potential be realized.

This is a crucial lesson for performance engineering: a microbenchmark win is necessary but not sufficient. Every optimization must be validated at the system level, because interactions between components can amplify or diminish individual gains.

The Asymmetry of Allocation and Deallocation

The async deallocation fix reveals a fundamental asymmetry in memory management: allocation via push() in Rust's Vec is amortized O(1) through geometric growth, and it happens in parallel with computation (the synthesis work). Deallocation, by contrast, is synchronous and serialized—the thread that owns the memory must walk every page, coalesce every free chunk, and return every virtual memory region. When 170 GB of data is freed on the hot path, the thread is blocked for ten seconds doing nothing but bookkeeping.

The async deallocation pattern transforms this cost from synchronous blocking into background work that overlaps with the next computation. The proving pipeline can begin processing the next job while the previous job's memory is still being reclaimed. This is particularly valuable in a persistent proving daemon, where the GPU worker thread should never be blocked on memory management.

Measurement Over Intuition

The allocation hypothesis was compelling. It had a clear mechanism (27 reallocation cycles × 10 circuits = 810 syscalls + 250 GB of copies), a plausible magnitude (if dealloc was 10 seconds, alloc must be comparable), and an existing fix API (just wire up the hint). Any engineer reading the analysis would nod along and say, "Yes, that must be the problem."

But it wasn't. The measurement showed zero impact.

This is the central lesson of this segment: intuition is not a substitute for instrumentation. The SmallVec optimization (A1) was cancelled because it caused a regression on Zen4. The cudaHostRegister optimization (B1) was cancelled because the mlock overhead dwarfed the transfer benefit. And now the allocation hint optimization (A2) was found to have no effect at all.

Each of these "obvious" wins was invalidated by measurement. And each invalidation taught the team something deeper about the system's behavior—that Zen4's cache hierarchy doesn't benefit from small-vector optimizations, that pinning 130 GB of memory has a cost that exceeds the DMA bandwidth gain, and that allocation during synthesis is effectively free because it's hidden behind computation.

The assistant's response to this zero result was exemplary. The SynthesisCapacityHint infrastructure was committed as a defensive optimization—it prevents pathological fragmentation under memory pressure, costs nothing when not triggered, and documents the expected vector sizes for anyone reading the code. But more importantly, the benchmark results were accepted and used to update the team's mental model of where the synthesis bottleneck truly lies.

Failed Experiments Are Valuable Data

One of the most instructive aspects of this segment is its honest documentation of failed experiments. The SmallVec optimization was tested in three configurations, all slower, and was cancelled. The cudaHostRegister optimization added 5.7 seconds of overhead and was reverted. The interleaved eval hurt IPC and was reverted. The allocation hint had zero impact. In each case, the assistant invested only enough effort to test the hypothesis, then moved on when the evidence was clear.

This discipline—failing fast, learning thoroughly, and not over-investing in dead ends—is the hallmark of an effective optimization process. The 13.2% improvement was not achieved by any single optimization but by the cumulative effect of multiple small wins combined with the elimination of several regressions.

Conclusion: The Two Sides of Performance Engineering

Segment 15 tells two stories that are mirror images of each other. The first story is about finding and fixing a hidden bottleneck: the 10-second destructor overhead that was invisible until layered instrumentation revealed the gap between timers. The fix was elegant and the impact was dramatic—13.2% end-to-end improvement from a single change.

The second story is about a hypothesis that was perfectly reasonable but completely wrong. The allocation overhead that seemed so obvious—265 GB of wasted copies, 810 syscalls—turned out to have zero impact on wall-clock time. The investigation was thorough, the implementation was clean, the measurement was definitive, and the conclusion was accepted.

Both stories teach the same lesson: measure, don't assume. The first story shows what happens when you measure and find something unexpected. The second story shows what happens when you measure and find nothing at all. In both cases, the measurement was the decisive act.

The Phase 4 changes—committed as 2da2a901—capture the tangible output of this segment: Boolean::add_to_lc methods, async deallocation on both C++ and Rust sides, the Vec recycling pool, software prefetch intrinsics, and the A4/D4 CUDA changes. The SynthesisCapacityHint infrastructure—committed as 41999e0b—captures the defensive optimization that may never matter but costs nothing to maintain.

But the real output of Segment 15 is knowledge. The team now knows that the synthesis bottleneck at ~50.8 seconds is purely computational, not memory-management overhead. They know that allocation and deallocation are not symmetric in this workload. They know that the path to 2-3x throughput improvement lies through Phase 5's PCE approach, not through further memory-management optimizations. And they know—perhaps most importantly—that intuition is not a substitute for instrumentation.

Sometimes the most valuable result of an experiment is learning what doesn't work. The allocation hypothesis was wrong, but testing it was exactly the right thing to do. In the high-stakes world of Filecoin proof generation, knowing where not to optimize is just as valuable as knowing where to optimize.

References

[1] "The Two-Front War: How Surgical Synthesis Optimization and Async Deallocation Delivered 13.2% in Phase 4" — Chunk article covering the Boolean::add_to_lc optimization and the async deallocation fix.

[2] "The Allocation Hypothesis: When a Quarter-Terabyte of Wasted Copies Changed Nothing" — Chunk article covering the investigation of allocation overhead and the zero-impact benchmark result.