The 5.7-Second Mistake: Verifying a Reverted Optimization in the cuzk Proving Pipeline

The Message

[assistant] No pin_abc — B1 is reverted. Let me verify the remaining timing instrumentation is still there:
[bash] strings /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c/out/libgroth16_cuda.a | grep "CUZK_TIMING"
CUZK_TIMING: gpu_tid=%zu ntt_msm_h_ms=%ld
CUZK_TIMING: gpu_tid=%zu batch_add_ms=%ld
CUZK_TIMING: gpu_tid=%zu tail_msm_ms=%ld gpu_total_ms=%ld
CUZK_TIMING: prep_msm_ms=%ld
CUZK_TIMING: b_g2_msm_ms=%ld num_circuits=%zu

This message, at first glance, appears to be nothing more than a routine verification step in a software engineering workflow. The assistant checks whether a recently reverted optimization—labeled "B1" (the cudaHostRegister memory pinning optimization)—has been successfully removed from the compiled binary, and confirms that the remaining CUDA timing instrumentation (CUZK_TIMING printf's) is still intact. But this brief exchange sits at the climax of a much larger narrative: a disciplined, data-driven performance engineering effort to diagnose and recover from a significant regression in the cuzk SNARK proving pipeline for Filecoin's Proof-of-Replication (PoRep).

Context: The Phase 4 Regression

To understand why this message matters, one must understand the journey that led to it. The cuzk project had successfully completed Phases 0 through 3, establishing a strong baseline performance of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 aimed to introduce a suite of compute-level optimizations drawn from a detailed optimization proposal document. Five changes were implemented in parallel:

A1: Replace Vec with SmallVec for the LC Indexer to reduce heap allocations
A2: Pre-size vectors in the synthesis path to avoid reallocations
A4: Parallelize the B_G2 CPU multi-scalar multiplications (MSMs)
B1: Pin a/b/c vectors with cudaHostRegister to improve GPU DMA transfer bandwidth
D4: Per-MSM window tuning for GPU kernels When these five optimizations were applied together, the proof time regressed from 88.9s to 106 seconds—a 19% slowdown. This was the opposite of the intended outcome. The team now faced a classic performance engineering challenge: a suite of changes, some beneficial and some harmful, whose aggregate effect was negative. The only way forward was systematic diagnosis.

The Diagnostic Campaign

The assistant embarked on a multi-step diagnostic campaign. First, detailed CUDA timing instrumentation (CUZK_TIMING printf's) was added to the GPU code to obtain precise phase-level breakdowns. A subtle bug was discovered: when stdout was redirected to a file, the CUDA printf output was lost due to full buffering. The fix was to replace printf with fprintf(stderr, ...) followed by fflush(stderr) after each timing print.

With the instrumentation working, the first instrumented test (at <msg id=962-963>) produced a breakthrough. The timing breakdown revealed:

| Phase | Time | Notes | |-------|------|-------| | Synthesis | 61.0s | vs 54.7s baseline (+11.5%) | | pin_abc (B1) | 5,733 ms | 10 circuits × 125 GiB pinned | | prep_msm | 1,722 ms | Classification + popcount scan | | ntt_msm_h | 21,417 ms | NTT + H-polynomial MSMs | | b_g2_msm | 22,849 ms | CPU-side B_G2 MSMs | | Total proof | 101.3s | vs 88.9s baseline |

The data was unambiguous. The B1 optimization (cudaHostRegister) was adding 5.7 seconds of overhead—nearly 20 times the 150–300 milliseconds estimated in the optimization proposal. The root cause was that cudaHostRegister calls mlock internally, and pinning ~125 GiB of host memory forces the operating system to touch every single page, incurring massive page-fault overhead. This single change accounted for the majority of the GPU-phase regression.

The Decision to Revert

With the data in hand, the assistant made a clear decision: revert B1. The optimization's theoretical benefit—improved DMA transfer bandwidth between host and GPU—was being overwhelmed by the practical cost of pinning 125 GiB of memory. The assistant noted in [msg 968]:

"B1 (cudaHostRegister) is causing a 5.7s penalty to save maybe 0.3-0.5s in DMA bandwidth. This is a clear net negative."

This is a textbook example of the importance of measuring, not guessing. The optimization proposal had estimated the pinning cost at 150–300ms, but the actual cost was 19× higher. Without the instrumentation, this would have remained invisible—the regression would have been attributed to some other change, or worse, accepted as "just how long it takes."

The reversion itself was straightforward: remove the cudaHostRegister calls that pinned the a/b/c vectors before GPU launch, and remove the corresponding cudaHostUnregister calls at the end. These edits were applied in [msg 970] and [msg 973].

The Verification: Why This Message Matters

This brings us to the subject message ([msg 980]). After reverting B1 and rebuilding, the assistant must verify two things:

That B1 is truly gone: The strings command searches the compiled static library (libgroth16_cuda.a) for the string pin_abc—the timing label used in the B1 instrumentation. The absence of this string confirms that the cudaHostRegister code path has been eliminated from the binary.
That the remaining instrumentation is intact: The grep "CUZK_TIMING" shows that five timing markers remain: ntt_msm_h_ms, batch_add_ms, tail_msm_ms, prep_msm_ms, and b_g2_msm_ms. These are the phase-level GPU timings that will be essential for further diagnosis. The assistant's comment—"No pin_abc — B1 is reverted"—is a moment of quiet satisfaction. The harmful change has been surgically removed while preserving the diagnostic infrastructure. The binary is now ready for the next round of testing.

Assumptions and Mistakes

This message reveals several assumptions that were made during the Phase 4 process:

The B1 estimation error: The optimization proposal assumed that cudaHostRegister on 125 GiB would cost 150–300ms. This assumption was wildly optimistic. The actual cost was 5.7 seconds because mlock touches every page. This mistake stemmed from reasoning about the cost of the cudaHostRegister API call itself without accounting for the OS-level page fault storm it triggers at that scale.

The build system nuance: Earlier in the diagnostic process (see <msg id=945-954>), the assistant discovered that the CUDA build artifacts are managed by build.rs and live outside the standard cargo output directory. The cargo clean -p supraseal-c2 command removed 0 files because it only cleans Rust compilation artifacts, not the build script output. This required manual cleanup of the build directories. The subject message's strings command on the .a file implicitly confirms that the build system correctly recompiled the CUDA code after the reversion—the timestamp and file size changed.

The assumption that SmallVec would be faster: The synthesis regression (61.0s vs 54.7s baseline) persisted even after reverting A2 (pre-sizing). The assistant correctly attributed this to A1 (SmallVec) and would later build a synth-only microbenchmark to isolate it. The subject message's verification of remaining CUZK_TIMING instrumentation ensures that the next round of testing will have the diagnostic data needed.

Input and Output Knowledge

Input knowledge required to understand this message includes:

The Phase 4 optimization suite (A1, A2, A4, B1, D4) and what each change does
The baseline performance (88.9s) and the regression (106s → 101.3s after partial reversion)
The instrumented test results showing B1's 5.7s overhead
The CUDA build system architecture (build.rs, rerun-if-changed, separate .a file)
The strings command as a verification tool for compiled binaries
The CUZK_TIMING instrumentation framework Output knowledge created by this message:
Confirmation that B1 is successfully reverted from the compiled binary
Confirmation that the CUZK_TIMING diagnostic infrastructure remains operational
A clean binary ready for the next round of A/B testing
Documentation of the verification step in the performance engineering process

The Broader Significance

This message exemplifies a principle that is central to high-performance systems engineering: measure everything, trust nothing, and have the discipline to revert harmful changes. The B1 optimization was well-intentioned and theoretically sound—pinning memory should improve GPU transfer bandwidth. But theory collided with reality at the scale of 125 GiB, and measurement revealed the truth.

The verification step itself—checking the compiled binary with strings—reflects a healthy skepticism about the build process. The assistant had already been burned by the build system's caching behavior (the CUDA .cu files weren't being recompiled when expected). By verifying the binary directly rather than trusting the build output, the assistant ensures that the next test will produce valid data.

The subject message also marks a transition point in the diagnostic workflow. With B1 eliminated, the remaining regression (synthesis at 61.0s vs 54.7s baseline) is now the focus. The preserved CUZK_TIMING instrumentation will be essential for the next round of testing, and the synth-only microbenchmark that the assistant builds in subsequent chunks will isolate the A1 SmallVec change as the culprit.

Conclusion

Message 980 is a small but pivotal moment in a complex performance engineering campaign. It represents the verification step after a data-driven decision to revert a harmful optimization—a decision made possible only by the careful instrumentation and systematic testing that preceded it. The message's brevity belies its significance: it confirms that the diagnostic infrastructure is intact, the harmful change is gone, and the path forward is clear. In the world of high-performance computing, where milliseconds matter and 5.7-second mistakes can derail entire projects, this kind of disciplined verification is not just good practice—it is essential.