The 744-Byte Difference: How a File Size Check Confirmed a Critical Optimization Reversion

Subject Message: A single bash command listing the size of a compiled CUDA static library archive, revealing that a build system had correctly recompiled after reverting a harmful optimization.

[assistant] [bash] ls -la /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-*/out/libgroth16_cuda.a 2>/dev/null
-rw-r--r-- 1 theuser theuser 2790792 Feb 17 23:52 /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c/out/libgroth16_cuda.a

At first glance, this message appears trivial: a developer checks the file size of a compiled library. But in the context of a high-stakes performance engineering effort — where every second of proof generation time translates directly to operational costs in a Filecoin proving pipeline — this ls command represents a critical verification gate. The 744 bytes that disappeared from the binary between builds told a story: the cudaHostRegister optimization (known as "B1") had been successfully removed, and the system was ready for the next benchmark run.

The Backstory: A Performance Regression Crisis

To understand why this file size check matters, we must first understand the crisis that led to it. The cuzk project is a pipelined Groth16 proof generation engine for Filecoin's Proof-of-Replication (PoRep) protocol. After successfully completing Phases 0 through 3 — establishing a solid baseline of 88.9 seconds for a single 32 GiB PoRep proof — the team entered Phase 4, a wave of compute-level optimizations designed to squeeze out every possible millisecond.

Five optimizations were implemented simultaneously:

• A1 (SmallVec): Replace standard Vec with SmallVec to eliminate heap allocations for small vectors
• A2 (Pre-sizing): Pre-allocate vectors with known capacities to avoid repeated reallocations
• A4 (Parallel B_G2): Parallelize the CPU-side B_G2 MSM computations
• B1 (cudaHostRegister): Pin host memory with cudaHostRegister to improve DMA transfer bandwidth
• D4 (Per-MSM Window Tuning): Tune MSM window sizes per operation The result was catastrophic: instead of improving, total proof time regressed to 106 seconds — a 17-second slowdown. This triggered a systematic diagnosis effort using CUDA timing instrumentation (CUZK_TIMING printf's) that had been added to the GPU code.

The Diagnosis: B1 as the Primary Culprit

The first instrumented test (message [msg 962]) produced a phase-level breakdown that immediately identified B1 as the dominant problem. The cudaHostRegister call was pinning approximately 125 GiB of host memory across 10 circuits (each with three 4.17 GB a/b/c vectors). The mlock-equivalent operation took 5,733 milliseconds — nearly 19x worse than the 150–300 ms estimate from the optimization proposal. The overhead of touching every page in 125 GiB of memory dwarfed any potential DMA bandwidth benefit.

The decision was clear: revert B1. The assistant removed the cudaHostRegister and cudaHostUnregister blocks from groth16_cuda.cu (messages [msg 969] through [msg 973]) and triggered a rebuild.

The Build Verification Problem

This is where message 978 becomes crucial. After reverting B1, the assistant ran cargo build --release -p cuzk-daemon (message [msg 975]), but the build output did not show the expected "Compiling supraseal-c2" line. The CUDA compilation artifacts are managed by a custom build.rs script and live outside the standard cargo output directory. Earlier in the session (messages [msg 946]–[msg 954]), the assistant had struggled with this exact issue — the build system's fingerprinting didn't reliably detect changes to .cu files, requiring manual cleanup of build directories.

The assistant had attempted to force recompilation by removing the build artifacts:

rm -rf /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-*

But the subsequent cargo build output (message [msg 976]) showed only Rust crate recompilations, with no mention of supraseal-c2. This was ambiguous: did the build system skip recompilation because it didn't detect the .cu file change, or did it recompile silently?

What the File Size Revealed

Message 978 answers this question definitively. By listing the static library archive, the assistant discovered:

1. The file existed at the expected path, meaning the build hadn't failed
2. The timestamp was current (Feb 17 23:52), confirming a recent build
3. The size had changed from 2,791,536 bytes (message [msg 954], built at 23:44 with B1 still present) to 2,790,792 bytes (built at 23:52 with B1 removed) The difference of 744 bytes is consistent with removing the cudaHostRegister and cudaHostUnregister code paths. The CUDA compiler (nvcc) had recompiled the .cu file, and the linker had incorporated the new object code into the static library. The build system had worked correctly despite the silent output. This verification was essential before proceeding to the next benchmark. Running a test with stale binaries would produce misleading results, potentially causing the team to draw incorrect conclusions about which optimizations were effective. The 744-byte difference was the green light to proceed.

Input Knowledge Required

To fully understand this message, one needs:

• The build architecture: That supraseal-c2 is a CUDA/C++ library compiled via a custom build.rs script, not through standard Cargo crate compilation. Its build artifacts live in a hash-named directory under target/release/build/.
• The B1 optimization: That cudaHostRegister pins host memory for faster GPU transfers, and that it had been identified as causing a 5.7-second regression.
• The reversion history: That the B1 code had just been edited out of groth16_cuda.cu in the preceding messages.
• The file size baseline: That the previous build (with B1) produced a 2,791,536-byte archive, providing a reference for comparison.
• The build system quirk: That cargo's output doesn't always indicate whether CUDA recompilation occurred, necessitating manual verification.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

1. Confirmation of successful recompilation: The CUDA code changes were picked up by the build system, and the new binary is ready for testing.
2. A quantitative measure of B1's code footprint: The 744-byte reduction in the static library provides a rough measure of how much code the B1 optimization added — primarily the cudaHostRegister/cudaHostUnregister calls and associated logic.
3. A checkpoint for the build process: The assistant now knows that rm -rf of the build artifacts followed by cargo build does trigger CUDA recompilation, even if the build output is silent about it.
4. Confidence to proceed: The team can now run the next benchmark with certainty that B1 is absent, allowing them to isolate the remaining synthesis regression (suspected to be caused by A1/SmallVec).

The Thinking Process Visible

The reasoning behind this message is a model of disciplined performance engineering. The assistant is operating under a clear methodology:

1. Hypothesis formation: B1 is causing a 5.7s regression via page-touch overhead.
2. Intervention: Revert B1 by editing the CUDA source.
3. Build verification: Ensure the change is actually compiled into the binary.
4. Measurement: Run a benchmark to confirm the regression is eliminated.
5. Iteration: If the total time is still above baseline, isolate the next culprit. Step 3 — build verification — is where most engineers would cut corners. The cargo build "succeeded" with exit code 0. Many would assume the binary was correct and proceed directly to benchmarking. But the assistant noticed the absence of the "Compiling supraseal-c2" line and refused to trust the build output at face value. This skepticism — born from earlier experience with the build system's quirks — led to the manual verification in message 978. The choice of verification method is also telling. Rather than re-running the entire benchmark (which takes ~100 seconds), the assistant used a quick ls command that returned in milliseconds. This is a classic debugging technique: use the cheapest possible check to validate your assumption before committing to an expensive operation.

The Broader Methodology

This single message exemplifies the broader engineering philosophy visible throughout the cuzk project. The team is not making changes and hoping for the best; they are:

• Instrumenting first: Adding CUZK_TIMING printf's before diagnosing regressions
• Measuring precisely: Using wall-clock timers, GPU timers, and memory monitors
• Reverting decisively: When an optimization causes regression, removing it without hesitation
• Verifying thoroughly: Confirming that reverted code is actually gone from the binary
• Isolating variables: Testing one change at a time to identify root causes The 744-byte difference in a static library archive is a small thing — a rounding error in a 2.8 MB file. But it represents the difference between guesswork and certainty, between hoping the build worked and knowing it did. In performance engineering, where a single misattributed regression can send a team down a week-long rabbit hole, this kind of rigor is not pedantry — it's survival.

Conclusion

Message 978 is a masterclass in the forgotten art of build verification. In an era of "it compiles, ship it," the assistant took 30 seconds to confirm that a critical optimization reversion had actually taken effect. The 744 bytes that disappeared from libgroth16_cuda.a told a story of disciplined engineering: a harmful optimization identified, removed, and verified gone. The next benchmark would confirm whether this was enough to recover the baseline — but at this moment, the team knew they were measuring what they thought they were measuring, and that knowledge is priceless.