The Two-Front War: How Surgical Synthesis Optimization and Async Deallocation Delivered 13.2% in Phase 4

Introduction

In the high-stakes world of Filecoin proof generation, where a single 32 GiB PoRep C2 proof demands synthesizing over 130 million constraints through a Groth16 zk-SNARK pipeline, performance optimization is never a single battle. It is a campaign fought on multiple fronts, each requiring different weapons, different intelligence, and different tactics. The chunk of work analyzed here—spanning the culmination of Phase 4 of the cuzk proving engine optimization—is a masterclass in this multi-front approach. It combines a surgical code change to eliminate unnecessary allocations in the synthesis hot path with the discovery and elimination of a hidden 10-second destructor bottleneck that had been silently wasting time in the GPU proving pipeline.

The results speak for themselves: synthesis time dropped from ~55.4 seconds to ~50.9 seconds (an 8.3% improvement confirmed by 91 billion fewer instructions), and after fixing a mysterious GPU wrapper regression, the total end-to-end proof time fell from 88.9 seconds to 77.2 seconds—a 13.2% improvement. But the real story is not just the numbers; it is the methodology, the detective work, and the layered instrumentation that made these discoveries possible.

Part I: The Synthesis Front — Boolean::add_to_lc

The Profile That Changed Everything

The Phase 4 optimization campaign began with data. Using perf record to capture 229,000 samples of the synthesis phase, the assistant constructed a remarkably precise breakdown of where ~55 seconds of CPU time were being spent. The results were illuminating:

| Category | Self % | ~Seconds | Key Functions | |---|---|---|---| | enforce | 22.13% | ~12.2s | ProvingAssignment::enforce | | LC construction | 23.52% | ~12.9s | Boolean::lc, Indexer::insert_or_update, LC::add, LC::sub | | Field arithmetic | 16.98% | ~9.3s | Montgomery multiplication, modular addition | | Memory allocation | 17.44% | ~9.6s | libc malloc/free, Vec::from_iter | | eval | 2.00% | ~1.1s | LinearCombination::eval | | Circuit logic | 9.51% | ~5.2s | UInt32::addmany, Boolean::xor | | Iterators/drops | 8.48% | ~4.7s | Map::try_fold |

The critical insight, highlighted in the assistant's analysis, was that LC construction (23.52%) was actually larger than enforce itself (22.13%). The overhead of building linear combinations—the core data structure for representing constraint terms—exceeded the cost of the constraint system's core logic. And within LC construction, a single function stood out: Boolean::lc() consumed 6.51% of total synthesis time all by itself.

The Root Cause: A Tiny Allocation, Called Millions of Times

The Boolean::lc() method is a seemingly innocuous utility: it creates a new LinearCombination (backed by a Vec allocation) wrapping a single Boolean variable with its coefficient. In most circuits, a Boolean term has only 1-2 coefficients, so the Vec allocation is tiny. But the function is called in tight loops like UInt32::addmany, which iterates 32 times per operand. When you have millions of constraints, each requiring multiple Boolean operations, those tiny allocations multiply into a significant overhead.

The assistant had already tried several approaches to address allocation overhead. A Vec recycling pool for ProvingAssignment::enforce yielded only ~0.7% improvement because the dominant allocations were not in the six enforce Vecs but in the dozens of temporary LinearCombination objects created inside circuit gadget closures. A SmallVec optimization—replacing Vec<LinearCombinationTerm> with a small-buffer-optimized alternative—actually regressed performance because the enum discriminant checks and larger stack frames defeated Zen4's out-of-order execution engine, dropping IPC from 2.60 to 2.38. Software prefetch intrinsics added to eval loops provided only marginal benefit. An interleaved A+B eval approach was reverted after it hurt IPC.

The Surgical Fix: add_to_lc and sub_from_lc

The Boolean::add_to_lc approach was different. Instead of trying to make allocations cheaper (SmallVec) or recycle them (Vec pool), the assistant asked a more fundamental question: why allocate at all? The key insight was that most callers of Boolean::lc() don't need a new LinearCombination—they just need to add the Boolean's term(s) to an existing linear combination. By adding two new methods to the Boolean type:

Validation: The Microbenchmark

The microbenchmark results were unambiguous. Running the cuzk-bench synth-only tool with three iterations, synthesis time dropped from a baseline of ~55.4 seconds to ~50.7 seconds—an 8.3% improvement. But the assistant didn't stop at wall-clock time. A perf stat comparison provided the mechanistic proof:

Part II: The GPU Wrapper Front — The 10-Second Ghost

The E2E Test That Revealed a Hidden Regression

With the synthesis microbenchmark confirming the win, the assistant proceeded to the crucial step: running a full end-to-end proof through the cuzk daemon to verify correctness and measure the total impact. The result, shown in [msg 1222], was:

timings: total=87459 ms (queue=264 ms, srs=0 ms, synth=51234 ms, gpu=35961 ms)

The proof completed successfully—correctness confirmed. But the timing data told a more complex story. Synthesis had indeed improved to 51.2 seconds. However, the GPU time had regressed: 36.0 seconds versus the baseline of 34.0 seconds. The net improvement was only 1.4 seconds (1.6%), far below the 4.5 seconds the synthesis microbenchmark had promised.

The assistant's first instinct was to suspect the bellperson wrapper layer—the Rust code that orchestrates data transfer to the GPU, invokes the CUDA kernels, and assembles the final proof. But the CUDA internal timing, emitted via fprintf(stderr) instrumentation embedded in the C++ code, showed gpu_total_ms=25657—essentially identical to the baseline of 25.3–25.8 seconds. The GPU compute itself was unaffected. The regression lived entirely in the wrapper.

Tracing the 10.2-Second Gap

The assistant methodically narrowed down the location of the gap. By cross-referencing timestamps from the daemon log, a precise timeline emerged:

The Diagnosis: 167 GB of Synchronous Destructors

The diagnosis, crystallized in [msg 1235], was precise: "The 10.2s gap is almost certainly heap deallocation of ~37 GB in implicit destructors at function exit." When prove_from_assignments() returns, the C++ generate_groth16_proof function's local variables—vectors holding split MSM bases, tail MSM data, and intermediate results—go out of scope. Their destructors must free the underlying heap memory. On the Rust side, the ProvingAssignment structs, moved into gpu_prove() as part of the SynthesizedProof, are also dropped when the function returns.

But how much memory was actually being freed? The assistant performed a back-of-the-envelope calculation in [msg 1265] that quantified the problem:

Each ProvingAssignment has:

The Fix: Async Deallocation on Both Sides

The fix, articulated in [msg 1256], was elegant: "After the proof assembly loop, move all the large vectors into a detached cleanup thread so the function can return immediately. This avoids blocking on destructors."

The assistant implemented this on the C++ side by moving the large split_vectors and tail_msm_bases into a std::thread lambda capture with .detach(). When the lambda went out of scope (potentially long after the main function had returned), the destructors would run in the background.

But the first test showed no improvement—the GPU wrapper time was still ~36 seconds. The assistant realized that the Rust side had more than three times the memory to free (132.5 GB vs ~37 GB). The ProvingAssignment vectors (a, b, c for 10 circuits) were being dropped synchronously when prove_from_assignments returned.

The assistant applied the same async deallocation pattern on the Rust side ([msg 1267]), spawning a std::thread::spawn with a move closure to take ownership of provers, input_assignments, and aux_assignments and free them in the background.

Validation: The 77-Second Confirmation

The result, shown in [msg 1270], was dramatic:

Part III: The Methodological Lessons

Layered Instrumentation Is Essential

The most striking methodological lesson from this chunk is the importance of instrumenting timing at every layer of a complex system. Without the CUDA internal timers (CUZK_TIMING), the 10-second GPU wrapper regression would have been invisible—the assistant would have seen 36.0 seconds of GPU time and assumed the CUDA code was slower. Only by comparing the Rust wrapper timer against the CUDA internal timer was the destructor overhead revealed.

This layered approach creates a diagnostic framework where anomalies become visible as gaps between timers. When the CUDA internal timer says 26 seconds but the Rust wrapper says 36 seconds, you know exactly where to look: the code between those two measurement points. This is far more powerful than having a single "total time" measurement, which hides all internal structure.

Microbenchmarks Measure Potential; E2E Tests Measure Reality

The Boolean::add_to_lc optimization delivered 8.3% improvement in isolation but only 1.6% in the first E2E test—not because the optimization was wrong, but because the system had other bottlenecks that masked its impact. Only by fixing all the bottlenecks (synthesis allocations, GPU destructor overhead) could the full potential be realized.

This is a crucial lesson for performance engineering: a microbenchmark win is necessary but not sufficient. Every optimization must be validated at the system level, because interactions between components can amplify or diminish individual gains.

The Asymmetry of Allocation and Deallocation

The async deallocation fix reveals a fundamental asymmetry in memory management: allocation via push() in Rust's Vec is amortized O(1) through geometric growth, and it happens in parallel with computation (the synthesis work). Deallocation, by contrast, is synchronous and serialized—the thread that owns the memory must walk every page, coalesce every free chunk, and return every virtual memory region. When 170 GB of data is freed on the hot path, the thread is blocked for ten seconds doing nothing but bookkeeping.

The async deallocation pattern transforms this cost from synchronous blocking into background work that overlaps with the next computation. The proving pipeline can begin processing the next job while the previous job's memory is still being reclaimed. This is particularly valuable in a persistent proving daemon, where the GPU worker thread should never be blocked on memory management.

Failed Experiments Are Valuable Data

One of the most instructive aspects of this chunk is its honest documentation of failed experiments. The SmallVec optimization was tested in three configurations, all slower, and was cancelled. The cudaHostRegister optimization added 5.7 seconds of overhead and was reverted. The interleaved eval hurt IPC and was reverted. In each case, the assistant invested only enough effort to test the hypothesis, then moved on when the evidence was clear.

This discipline—failing fast, learning thoroughly, and not over-investing in dead ends—is the hallmark of an effective optimization process. The 13.2% improvement was not achieved by any single optimization but by the cumulative effect of multiple small wins combined with the elimination of several regressions.

Conclusion: The Commit That Captured 13.2%

All Phase 4 changes were committed as 2da2a901, including Boolean::add_to_lc, async deallocation on both C++ and Rust sides, the Vec recycling pool, software prefetch intrinsics, and the A4/D4 CUDA changes. The commit message itself is a mini-document, capturing the root causes, the fixes, and the measured improvements.

But the real output of this chunk is not the commit—it is the methodology. The layered instrumentation, the hypothesis-driven experimentation, the willingness to measure and reject failed ideas, and the relentless pursuit of hidden overheads. These are the tools that turned a 10-second ghost into a 13.2% victory, and they will serve the project well in Phase 5 and beyond.

The two-front war—synthesis optimization and destructor elimination—demonstrates that performance engineering is never about a single silver bullet. It is about systematically identifying every bottleneck, understanding its root cause, and applying the right fix for each layer. Sometimes the fix is a surgical code change that eliminates an unnecessary allocation. Sometimes it is a thread that cleans up in the background. But always, it begins with measurement.