The Commit That Captured 13.2%: Formalizing Phase 4's Performance Breakthroughs

On February 18, 2026, at message index 1286 in a lengthy optimization session, an AI assistant executed a single git commit command that crystallized days of profiling-driven performance engineering into a permanent record. The commit, 2da2a901 on the feat/cuzk branch, bears the subject line "Phase 4 — synthesis hot path + async dealloc optimizations" and summarizes a 13.2% end-to-end throughput improvement for Filecoin PoRep C2 proof generation — from 88.9 seconds down to 77.2 seconds on an RTX 5070 Ti GPU. But this commit is far more than a routine checkpoint. It is the culmination of a detective story that began with a mysterious 10-second gap between what the GPU reported and what the Rust wrapper measured, and it encodes a deep understanding of how memory management, allocation patterns, and destructor ordering can silently dominate performance in high-throughput cryptographic proving pipelines.

The Moment of the Commit

The message itself is deceptively simple. The assistant writes:

The bellpepper-core, supraseal-c2/cuda files are new additions (showing full file sizes). Let me commit:

Then executes a git commit with a 30-line commit message, followed by git status to confirm the result. The commit touches 16 files, inserts 4,997 lines, deletes 46, and creates five new files — three in extern/bellpepper-core/src/ (the local fork of the bellpepper constraint system library) and two in extern/supraseal-c2/cuda/ (CUDA kernel sources). The commit message is unusually detailed, reading less like a typical VCS log entry and more like a miniature engineering report: it identifies root causes, quantifies improvements, lists specific patches, and provides hardware-validated benchmark results.

This level of detail is intentional. The commit message serves as both documentation and validation — a record that future readers (including the assistant itself, in subsequent sessions) can consult to understand why each change was made and what it achieved. It is the output knowledge created by this message, and it will outlive the ephemeral conversation context.

The Detective Story Behind the Numbers

To understand why this commit exists, one must understand the investigation that preceded it. The Phase 4 optimization work had already produced several wins: Boolean::add_to_lc eliminated temporary LinearCombination allocations in circuit gadget hot paths, a Vec recycling pool reduced allocation churn in ProvingAssignment::enforce, and software prefetch intrinsics improved memory access patterns in evaluation loops. A synth-only microbenchmark confirmed synthesis time dropped from ~55.4s to ~50.9s — an 8.3% improvement, with perf stat showing 91 billion fewer instructions (-15.3%) and 18.6 billion fewer branches (-26.7%).

But when the assistant ran the first full end-to-end proof after these changes, something was wrong. The total time was 87.5 seconds — better than the 88.9s baseline, but not by as much as expected. Worse, the GPU timing showed a puzzling discrepancy. The CUDA internal instrumentation reported approximately 26 seconds for the actual GPU computation (NTT, MSM, batch addition, tail MSM). Yet the Rust-side wrapper in bellperson — the prove_from_assignments function — reported 36.0 seconds. Where was the missing 10 seconds?

This is the kind of question that separates superficial optimization from deep performance engineering. The assistant did not accept the numbers at face value. Instead, it instrumented the C++ code to add a pre_destructor_ms timer, capturing the time just before C++ destructors began running. The result was revealing: the C++ pre-destructor time matched the CUDA internal time at ~26 seconds. The 10-second gap was entirely in destructor overhead — the synchronous freeing of approximately 37 GB of C++ vectors (split_vectors, tail_msm bases) and approximately 130 GB of Rust Vec allocations (ProvingAssignment a/b/c fields) after GPU proving completed.

The Asymmetry of Allocation and Deallocation

The root cause, as the commit message explains, was that "~37 GB of C++ vectors (split_vectors, tail_msm_bases) and ~130 GB of Rust Vecs (ProvingAssignment a/b/c) [were] freed synchronously in destructors after GPU proving, blocking return for ~10s of munmap() calls." This is a classic performance pitfall in systems that allocate large buffers: allocation is amortized and often overlapped with computation, but deallocation — particularly munmap of huge memory regions — is synchronous and serialized. The Rust Vec::drop implementation calls the allocator's dealloc, which for large allocations backed by mmap translates to a munmap system call. Ten parallel circuits, each with 4.17 GB a/b/c vectors plus auxiliary assignments, meant 130 GB of memory needed to be unmapped — and this happened on the critical path, blocking the function's return to its caller.

The fix was elegant and minimal: move the deallocation into detached threads on both the C++ and Rust sides. On the C++ side, split_vectors and tail_msm bases were moved into a std::thread that outlives the function return. On the Rust side, a spawned thread takes ownership of the provers, input_assignments, and aux_assignments vectors and drops them in the background. The function returns immediately, and the munmap calls happen concurrently, invisible to the timing measurement.

The result was dramatic. After the fix, the GPU wrapper time dropped to 26.2 seconds — matching the CUDA internal time exactly. The total E2E time fell from 87.5s to 77.2s. Two subsequent runs confirmed consistency at 77.3s and 77.0s. The 10-second gap was eliminated entirely, and the deallocation thread reported only 4.2 seconds of background work — meaning the munmap calls themselves were not even the full 10 seconds; the remainder was likely TLB shootdown and other kernel overhead from synchronous deallocation on the critical path.

The Synthesis Optimizations: Micro-Architectural Surgery

The commit message also details the synthesis-side optimizations, which were equally methodical. The key insight was that Boolean::lc() — a method that creates a fresh LinearCombination (a Vec<(Variable, Coeff)>) on every call — was being invoked millions of times in circuit gadget hot paths. Each call allocated a new Vec, populated it, and returned it, only for the caller to immediately append its contents to another LinearCombination. The fix was to add add_to_lc and sub_from_lc methods that append directly to an existing LinearCombination, eliminating the intermediate allocation entirely.

This pattern — identifying allocation churn in hot loops and replacing it with in-place mutation — is a textbook optimization technique, but its application here required deep knowledge of the bellpepper constraint system's internals. The assistant patched six hot call sites: UInt32::addmany, Num::add_bool_with_coeff, Boolean::enforce_equal, Boolean::sha256_ch, Boolean::sha256_maj, lookup3_xy, and lookup3_xy_with_conditional_negation. Each patch eliminated a temporary Vec allocation in a loop that runs millions of times per proof.

The Vec recycling pool in ProvingAssignment::enforce for the six LinearCombination buffers (a, b, c, and their auxiliary variants) further reduced allocation pressure. Instead of creating fresh Vecs for each constraint enforcement, the pool reuses previously allocated buffers, clearing them between uses. This is a common pattern in high-performance systems — the Vec::clear() method preserves the backing allocation, avoiding the cost of free and subsequent malloc.

The software prefetch optimization — adding _mm_prefetch intrinsics to eval_with_trackers and LinearCombination::eval — hints at the assistant's willingness to reach for CPU-level tuning when allocation optimizations alone were insufficient. Prefetch instructions tell the CPU to load cache lines before they are accessed, reducing cache miss latency in pointer-chasing data structures like LinearCombination's sparse vector of (Variable, Coeff) pairs.

The Git Workflow: A Model of Disciplined Engineering

The commit itself is preceded by a meticulous git workflow visible in the surrounding messages. The assistant checks git status repeatedly, verifies which files are modified, examines git diff --stat to understand the scope of changes, checks git log --oneline to see the commit history, investigates whether bellpepper-core and supraseal-c2 are tracked or untracked, checks for .gitignore rules, and confirms they are not submodules. It then stages files incrementally — first the core changes, then notices that extern/bellperson/Cargo.toml and extern/bellperson/src/groth16/mod.rs were modified but not staged, and adds them explicitly.

This careful staging reflects an understanding that a good commit is a coherent unit of change. The assistant is not simply dumping every modified file into a single commit; it is curating the set of changes that belong together under the Phase 4 theme. The five new files — three in bellpepper-core and two in supraseal-c2 — represent local forks created to apply patches that could not be made directly to the upstream libraries. By creating these forks and committing them, the assistant establishes a reproducible build environment where the optimizations are permanently recorded.

Input Knowledge Required

To fully understand this commit, a reader needs substantial context about the broader system. The Filecoin PoRep (Proof of Replication) C2 circuit is a Groth16 proving pipeline that involves synthesizing 10 parallel circuits, each with approximately 130 million constraints, then computing multi-scalar multiplications (MSMs) and number-theoretic transforms (NTTs) on a GPU. The pipeline spans multiple layers: Curio (the Go-based Filecoin miner) → cuzk (the Rust proving engine) → bellperson (the Rust bellman fork) → supraseal-c2 (the C++/CUDA layer). Each layer has its own allocation patterns, timing instrumentation, and destructor chains.

The reader must also understand Rust's memory model: that Vec::drop calls the allocator, that large allocations may use mmap/munmap, that Arc reference counting determines when the underlying data is freed, and that std::thread::spawn can be used to offload destructor work. On the C++ side, understanding that std::thread with a lambda captures the vectors by move and outlives the calling function is essential.

Output Knowledge Created

This commit creates several forms of output knowledge. First, it establishes a permanent, versioned record of the Phase 4 optimizations, tagged with commit 2da2a901 on the feat/cuzk branch. Any future developer can git show 2da2a901 and see exactly what changed, why, and with what measured impact. Second, the commit message itself serves as a mini-design document, capturing root causes (synchronous destructor overhead, temporary LC allocations), fixes (async deallocation, add_to_lc methods), and validation (E2E benchmark on RTX 5070 Ti). Third, the five new files — the bellpepper-core and supraseal-c2 forks — create a reproducible baseline for future optimization work. Fourth, the commit establishes a precedent for the level of detail expected in optimization commits within this project, implicitly setting a standard for future Phase 5 and beyond.

The Thinking Process Visible in the Commit

The commit message reveals the assistant's analytical framework. It does not simply list changes; it categorizes them by optimization domain ("Synthesis optimizations" vs "GPU async deallocation"), provides before-and-after measurements (55.4s → 50.9s, 36s → 26s, 88.9s → 77.2s), identifies root causes ("blocking return for ~10s of munmap() calls"), and explains the mechanism of each fix ("move ... into detached std::thread", "append directly to an existing LC"). This structure mirrors the scientific method: observe a problem, hypothesize a root cause, implement a fix, measure the result.

The inclusion of "CUDA internal timing unchanged (~26s); overhead was pure deallocation" is particularly telling. It shows that the assistant verified that the fix did not alter the actual GPU computation — the 10-second improvement came entirely from eliminating overhead, not from making the GPU faster. This is a crucial distinction in performance engineering: reducing overhead is qualitatively different from accelerating computation, and the commit message explicitly documents this.

Conclusion

Message 1286 is, on its surface, a routine git commit. But it represents the culmination of a deep investigative arc: from noticing a suspicious timing gap, to instrumenting multiple layers of the stack, to calculating memory sizes, to implementing a cross-language async deallocation strategy, to validating with multiple E2E runs, to curating a coherent set of changes into a well-structured commit. The 13.2% improvement it captures is not the result of a single clever trick but of systematic, measurement-driven optimization that attacked two independent bottlenecks: allocation churn in circuit synthesis and synchronous destructor overhead in the GPU proving path. The commit stands as a permanent record of that work, and its detailed message ensures that the reasoning behind each change will survive beyond the conversation that produced it.