From Micro-Optimization to Architectural Change: How Phase 4 of cuzk Forged the Path to Phase 5

Introduction

On February 18, 2026, the cuzk project—a pipelined SNARK proving engine for Filecoin's proof-of-replication protocol—reached an inflection point. Message [msg 1352] documented the conclusion of Phase 4, a phase dedicated to compute-level optimizations of the synthesis hot paths and GPU wrapper overhead. The message is many things at once: a post-mortem, a strategic handoff, a methodological manifesto, and a remarkably candid account of which optimization hypotheses survived contact with real hardware.

This article examines that message not as a standalone artifact, but as the bridge between two fundamentally different approaches to performance optimization. Phase 4 was about micro-optimization: finding every possible cycle to squeeze from the existing synthesis pipeline through better data structures, reduced allocations, and faster memory management. Phase 5, which the message explicitly paves the way for, is about architectural change: replacing circuit synthesis with a Pre-Compiled Constraint Evaluator (PCE) that uses sparse matrix-vector multiplication. The message's central strategic contribution is the empirical demonstration that micro-optimization has reached diminishing returns on this particular pipeline, and that the 2-3x throughput target requires a fundamentally different approach.

The Hardware Substrate: Why Architecture Matters

Any discussion of Phase 4's results must begin with the hardware on which every optimization was tested. The target platform was an AMD Ryzen Threadripper PRO 7995WX—a 96-core Zen4 monster—paired with an NVIDIA RTX 5070 Ti GPU (Blackwell architecture, sm_120, 16 GB VRAM). These specifics are not incidental details; they are the substrate that determined which optimizations worked and which failed.

Zen4 is a high-IPC architecture with a large, aggressive out-of-order execution engine. It thrives on simple, predictable instruction streams with minimal branching. This architectural characteristic directly determined the fate of one of Phase 4's most instructive experiments: the SmallVec cancellation. SmallVec, a Rust data structure that stores small collections inline on the stack before spilling to heap, introduced conditional branches for every access (checking whether data was inline or on the heap). On Zen4, this caused an 8.5% IPC regression (from 2.60 to 2.38). The simpler instruction stream of plain Vec—with its unconditional heap access pattern—achieved higher IPC, and that advantage outweighed any cache benefits from stack-local storage.

This finding is a concrete demonstration of a general principle: optimization is never context-free. A technique that works on one microarchitecture may fail on another. The SmallVec finding, documented with precise IPC numbers and a causal explanation, is a gift to anyone optimizing Rust code on Zen4. It also explains why the message's hardware details are not boilerplate—they are essential context for interpreting every result.

The Empirical Methodology: Letting Hardware Have the Final Word

The defining characteristic of Phase 4 was its rigorous empirical methodology. Every optimization was proposed, implemented, tested on real hardware using multiple measurement tools, and either accepted or rejected based on data. This may sound like standard practice, but the consistency with which the assistant applied this methodology—and the honesty with which it reported failures—is noteworthy.

Three tools provided convergent evidence for every decision:

1. CUDA fprintf(stderr) instrumentation embedded in the .cu file provided precise internal timings of GPU operations, distinguishing between the C++ function body and destructor overhead. This instrumentation was what revealed the 10-second gap that led to the async deallocation fix.
2. perf stat provided hardware-level metrics: instructions retired, cycles, IPC, cache misses, branch mispredicts. When SmallVec caused an IPC regression, perf stat captured it. When Boolean::add_to_lc eliminated 91 billion instructions, perf stat confirmed it.
3. perf record with 229K samples provided a detailed breakdown of where synthesis time was actually spent, producing the profile that became the strategic foundation for Phase 5. The message reports results from all three tools, often in the same paragraph, creating a multi-dimensional view of each optimization's impact. This triangulation of evidence is what makes the conclusions trustworthy.

The Validated Wins: What Survived Contact with Reality

Phase 4 produced two significant wins, both of which were validated by the full suite of measurement tools.

Boolean::add_to_lc: Eliminating Temporary Allocations

The Boolean::add_to_lc optimization targeted a pattern that was pervasive in the circuit gadget code. The original code would call Boolean::lc() to create a temporary LinearCombination object, then add it to an accumulator. This generated millions of short-lived allocations that were immediately discarded. The fix was to add add_to_lc() and sub_from_lc() methods that directly modify an existing LinearCombination, bypassing the temporary entirely.

The perf stat data is striking: 91 billion fewer instructions (-15.3%), 18.6 billion fewer branches (-26.7%). Synthesis time dropped from 55.4s to 50.9s—a 4.5-second improvement (8.3%). The optimization was applied across multiple gadget types: UInt32::addmany, Num::add_bool_with_coeff, Boolean::enforce_equal, sha256_ch, sha256_maj, and the lookup gadgets. Each call site was individually patched.

This win is particularly instructive because it targeted a genuine algorithmic inefficiency (creating and discarding temporaries) rather than a memory management detail. It improved both instruction count and branch count, and the improvement was large enough to be clearly visible in wall-clock time.

Async Deallocation: The 10-Second Gap

The discovery of synchronous destructor overhead was the most impactful finding of Phase 4, and arguably the most interesting from a systems perspective. The assistant noticed a discrepancy between two timing measurements of the same GPU operation: CUDA internal timing (measured via fprintf(stderr) in the .cu file) reported 26.1s for the C++ function body, while the Rust-side GPU wrapper timing reported 36.0s. The 10-second gap was traced to synchronous destruction of large vectors after GPU proving completed.

The vectors in question were substantial: ~37 GB of C++ vectors (split_vectors, tail_msm_bases) plus ~130 GB of Rust Vecs (ProvingAssignment a/b/c). When these were freed synchronously, each munmap() of a multi-GB region required synchronous kernel page table teardown. The fix was elegant: move ownership of these large vectors to detached threads (C++ std::thread::detach() and Rust std::thread::spawn(move || drop(...))), allowing the main thread to proceed while the kernel tears down page tables in the background. GPU wrapper time dropped from 36s to 26s, matching the CUDA internal timing exactly.

This fix is a masterclass in understanding the difference between user-space and kernel-space costs. The deallocation itself (the C++ delete or Rust drop) is fast—it's the kernel's munmap processing that takes time. By moving the deallocation to a background thread, the main thread is no longer blocked on kernel page table teardown. The total wall-clock time spent on deallocation is unchanged, but it no longer sits on the critical path.

The Disproven Hypotheses: What Didn't Work and Why

Phase 4's most valuable output may be its documentation of experiments that failed. These are not failures in any meaningful sense—they are learning events, and the message treats them with the same analytical rigor as the successes.

SmallVec: IPC Regression on Zen4

The SmallVec experiment was described above. The key numbers: IPC dropped from 2.60 to 2.38 (8.5% regression). The optimization was cancelled. The lesson: on high-IPC architectures like Zen4, the simplicity of the instruction stream can matter more than cache behavior. SmallVec's conditional branches (checking whether data is inline or on the heap) degraded the branch predictor's effectiveness, and the resulting IPC loss outweighed any cache benefits.

cudaHostRegister: mlock Overhead

The cudaHostRegister experiment tested whether pinning host memory pages would improve GPU transfer performance. The conventional wisdom is that pinned memory enables faster DMA transfers and reduces TLB misses. The assistant implemented it and benchmarked it. The result: for 10 circuits × 3 arrays × 4.17 GB = 125 GB of pinned memory, the mlock syscall overhead was 5.7 seconds. This was a net negative. The optimization was reverted.

The lesson: cudaHostRegister is not free. Pinning memory requires the kernel to lock pages in physical memory, preventing swapping and requiring page table updates. For workloads that already have good GPU transfer performance, the overhead of pinning may exceed the benefit. This is particularly true when the total pinned memory is large (125 GB in this case), as the kernel must process each page individually.

Vec Pre-allocation: Zero Impact

The most counterintuitive finding of Phase 4 was that Vec pre-allocation had zero measurable impact. The assistant implemented a SynthesisCapacityHint mechanism that cached the final capacity of each ProvingAssignment Vec from the first synthesis run and pre-allocated to that capacity in subsequent runs. The theoretical justification was compelling: each Vec grows from empty to ~4.17 GB through geometric doubling, which involves approximately 27 reallocation cycles. Each reallocation copies the existing data to a new allocation, then frees the old one. Across 10 circuits running in parallel, the total "wasted" copy volume was estimated at ~265 GB. Pre-allocation should eliminate all of this.

The benchmark result: 0.0 seconds improvement.

The message's analysis of this finding is a masterclass in understanding why theoretical models fail in practice. The key insight is that Vec's geometric doubling strategy amortizes the cost of reallocation remarkably well. The average cost per push() is O(1) with a small constant. The expensive reallocations—the ones that copy multiple gigabytes—happen only a few times per Vec, late in the growth process. And crucially, these reallocations are interleaved with computation. The 10 circuits are synthesized in parallel across 96 cores using rayon's parallel iterator. While one circuit's Vec is being reallocated, other circuits are performing field arithmetic. The memory operations overlap with computation, hiding their latency.

The message also explains the alloc/dealloc asymmetry: "Dealloc was costly because munmap() of large contiguous regions is a synchronous kernel operation that tears down page tables all at once. Alloc via push() spreads cost across millions of incremental operations, each touching only a page or two." This is a profound insight about the different cost structures of allocation and deallocation at scale, and it explains why the 10-second dealloc problem (fixed by async deallocation) had no alloc-side counterpart.

The Synthesis Bottleneck: Empirical Foundation for Phase 5

Perhaps the most important output of Phase 4—and the key reason the message was written—is the detailed characterization of the synthesis bottleneck. Using perf record (229K samples), the assistant produced a breakdown of where the ~50.8s of synthesis time is actually spent:

| Category | Self % | ~Seconds | |---|---|---| | enforce (all 9 inlined fragments) | 22.13% | ~12.2s | | LC construction | 23.52% | ~12.9s | | Field arithmetic | 16.98% | ~9.3s | | Memory allocation | 17.44% | ~9.6s | | Circuit logic | 9.51% | ~5.2s | | eval | 2.00% | ~1.1s |

This breakdown is the empirical foundation for the strategic pivot to Phase 5. The synthesis time is dominated by three roughly equal components: constraint enforcement (22%), LinearCombination construction (24%), and field arithmetic (17%). Memory allocation, at 17%, is significant but not dominant—and as the pre-allocation experiment showed, it is already well-amortized by the geometric doubling strategy.

The key insight is that none of these components can be dramatically improved by further micro-optimization. The field arithmetic is already using optimized elliptic curve operations. The LC construction is fundamentally about building linear combinations of variables, which is a computational requirement of the R1CS construction. The constraint enforcement is the core work of the constraint system.

The message's conclusion is explicit: "The synthesis bottleneck at ~50.8s is now dominated by pure computation (field arithmetic + LC evaluation), not memory management. Further gains require Phase 5 (PCE) to eliminate synthesis entirely." This is a strategic insight that could only be reached through the kind of detailed profiling documented in this message.

Phase 4 vs Projections: The Honest Accounting

The message does not shy away from the gap between projections and reality. The Phase 4 plan had projected a 2-3x throughput improvement. The actual result was 1.15x (88.9s → 77.0s). Two of the projected "easy wins" (SmallVec, cudaHostRegister) were invalidated by hardware testing. The remaining Phase 4 items are acknowledged to have "diminishing returns."

The performance summary table tells the story clearly:

| Config | Synthesis | GPU | Total | |---|---|---|---| | Phase 2/3 baseline (batch=1) | 54.7s | 34.0s | 88.9s | | Phase 4 final (add_to_lc + async dealloc) | 50.8s | 26.2s | 77.0s | | Delta | -3.9s (-7.1%) | -7.8s (-22.9%) | -11.9s (-13.4%) |

The GPU side saw a 22.9% improvement (driven by async deallocation), while the synthesis side saw only 7.1%. This asymmetry reinforces the message's central strategic conclusion: the next major gains must come from the synthesis side, and they will require a different approach.

The Handoff to Phase 5

The message is, above all, a handoff document. It ensures that no lesson from Phase 4 is lost when the team moves to Phase 5. The empirical methodology is now established. The performance baselines are known. The bottleneck is understood. And the team knows that the path to 2-3x throughput lies not in further micro-optimization of the existing synthesis pipeline, but in a fundamental architectural change: replacing circuit synthesis with a Pre-Compiled Constraint Evaluator that performs sparse matrix-vector multiplication.

The Phase 5 approach—as described in the project's roadmap—is to pre-compile the constraint system into a CSR (Compressed Sparse Row) matrix representation, then evaluate constraints via sparse matrix-vector multiply rather than by walking the circuit graph. This eliminates the per-constraint overhead of LC construction and field arithmetic, replacing it with a highly optimized linear algebra kernel that can leverage SIMD and GPU acceleration.

The message's detailed profiling data provides the baseline against which Phase 5's success will be measured. The synthesis time of 50.8s is the target to beat. If the PCE can reduce synthesis to, say, 10-15 seconds, the total proof time would drop to 36-41 seconds—a 2x improvement over the Phase 2/3 baseline. This is the path to the 2-3x target that Phase 4 could not achieve through micro-optimization alone.

Conclusion

Message [msg 1352] is a remarkable document of empirical software engineering. It tells the story of Phase 4 of the cuzk project—a phase that set ambitious targets, tested those targets against reality, and produced a nuanced understanding of where optimization effort should be directed. The message's willingness to document cancelled experiments alongside validated wins, its detailed analysis of why certain optimizations failed, and its clear-eyed assessment of the gap between projections and reality make it a model for how to conduct and communicate optimization work.

The message's central lesson is that hardware is the ultimate arbiter of optimization hypotheses. SmallVec "should" have been faster. Pre-allocation "should" have helped. cudaHostRegister "should" have improved GPU transfer performance. But on a specific AMD Zen4 CPU with a specific NVIDIA Blackwell GPU, running a specific Rust codebase with specific parallelization patterns, these assumptions were tested and found false. The assistant's willingness to let the hardware have the final word, and to document that verdict in meticulous detail, is what makes this message not just a status report but a contribution to the engineering knowledge base of the entire project.

As the cuzk project moves into Phase 5—the Pre-Compiled Constraint Evaluator that aims to replace circuit synthesis with sparse matrix-vector multiplication—the lessons of Phase 4 are carried forward. The empirical methodology is now established. The performance baselines are known. The bottleneck is understood. And the team knows that the path to 2-3x throughput lies not in further micro-optimization of the existing synthesis pipeline, but in a fundamental architectural change. This message ensures that no lesson from Phase 4 is lost in the transition.## References

[1] "The Empirical Crucible: How Phase 4 of the cuzk Proving Engine Tested Assumptions Against Silicon" — The comprehensive technical post-mortem and status report for Phase 4, documenting all optimizations attempted, validated wins, cancelled experiments, and the detailed performance analysis that paved the way for Phase 5. This article serves as the primary source for the analysis presented above.