The Build That Didn't: A Microcosm of Disciplined Performance Engineering

Message 976: cargo build --release -p cuzk-daemon 2>&1 | tail -15

At first glance, message 976 appears to be the most mundane artifact imaginable in a software engineering conversation: a Rust build command and its output. The assistant types cargo build --release -p cuzk-daemon 2>&1 | tail -15, and the response shows a cascade of compilation lines — Compiling filecoin-hashers, Compiling storage-proofs-core, Compiling cuzk-core, Compiling cuzk-server — scrolling past like so many logs in a forest. Nothing seems to happen. No errors, no breakthroughs, no data.

Yet this message is a fulcrum. It sits at the precise hinge point between diagnosis and action in one of the most exacting performance engineering exercises in the entire opencode session: the systematic isolation of a 17-second regression in a GPU-accelerated SNARK proving pipeline. Understanding why this build command was issued, what it was supposed to accomplish, and what it actually accomplished reveals the depth of the assistant's methodology — and the subtle ways that build systems can thwart even well-laid plans.

The Context: A Regression Unearthed

To grasp the significance of this build, one must understand the journey that led to it. The cuzk project had spent several phases building a pipelined, cross-sector-batching SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol. By the end of Phase 3, the system achieved a solid baseline: 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 was supposed to improve on this with a suite of micro-optimizations drawn from a detailed optimization proposal document.

Five optimizations were implemented in Wave 1:

• A1 (SmallVec): Replace Vec with SmallVec in the linear combination indexer to reduce heap allocations
• A2 (Pre-sizing): Pre-allocate synthesis vectors with known capacity to avoid reallocation
• A4 (Parallel B_G2): Parallelize the B_G2 multi-scalar multiplications across CPU threads
• B1 (cudaHostRegister): Pin host memory with cudaHostRegister to accelerate GPU DMA transfers
• D4 (Per-MSM window tuning): Tune the window sizes for each MSM operation individually The result of applying all five? 106 seconds — a 17-second regression, not an improvement. The assistant immediately recognized this as a red flag and embarked on a systematic diagnosis, adding CUDA timing instrumentation (CUZK_TIMING printf's) to the GPU code to obtain phase-level breakdowns. The first instrumented test (message 962) produced a breakthrough. The CUZK_TIMING data showed that B1 (cudaHostRegister) was adding 5.7 seconds of overhead — not the estimated 150–300 milliseconds. Pinning approximately 125 GiB of host memory (10 circuits × 3 arrays × ~4.17 GiB each) forced the system to mlock every single page, touching all that memory upfront. The optimization intended to save a few hundred milliseconds of DMA transfer time was instead costing nearly six seconds.

The Decision: Revert and Rebuild

Message 976 is the direct consequence of that discovery. The assistant's analysis in messages 967–968 laid out the verdict clearly: "B1 (cudaHostRegister) is causing a 5.7s penalty to save maybe 0.3-0.5s in DMA bandwidth. This is a clear net negative." The decision was made to revert B1 while keeping the other optimizations (A1, A4, D4) and re-test.

The reversion itself involved editing the CUDA source file groth16_cuda.cu to remove the cudaHostRegister calls (around line 139) and the corresponding cudaHostUnregister calls (around line 738). These edits were applied in messages 970–973. Then, in message 975, the assistant prepared to rebuild:

# Kill daemon
kill $(pgrep -f 'target/release/cuzk-daemon') 2>/dev/null
sleep 1

# Remove old CUDA build artifacts and rebuild
rm -rf /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-*
cargo build --release -p cuzk-daemon 2>&1 | tail -15

The rm -rf command was intended to delete the cached CUDA compilation artifacts so that cargo build would recompile supraseal-c2 from scratch, picking up the edited .cu file. This is where message 976 enters.

What the Build Output Actually Says

The output in message 976 shows a standard incremental Rust build:

Compiling filecoin-hashers v14.0.1
warning: `bellperson` (lib) generated 10 warnings
Compiling storage-proofs-core v19.0.1
Compiling storage-proofs-porep v19.0.1
Compiling storage-proofs-post v19.0.1
Compiling storage-proofs-update v19.0.1
Compiling filecoin-proofs v19.0.1
Compiling filecoin-proofs-api v19.0.0
Compiling cuzk-core v0.1.0 (...)
Compiling cuzk-server v0.1.0 (...)

Notice what is not in this list: supraseal-c2. The CUDA wrapper library that contains the edited groth16_cuda.cu was not recompiled. The build system, detecting no changes to the Rust source files of supraseal-c2 (because the .cu file is tracked via a rerun-if-changed=cuda directive in build.rs), decided it was already up to date. But the cached .a archive still contained the old code with cudaHostRegister — unless the rm -rf had actually worked.

The Hidden Problem: Shell Globbing and Build Artifacts

The assistant discovered the problem in the very next message (977). The rm -rf command had not actually deleted the build artifacts:

find /home/theuser/curio/extern/cuzk/target/release/build -name "*supraseal*" -maxdepth 1
/home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-0ac55c4ac30a878b
/home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-14f6b99f1cbc5f79
/home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c

Three build directories still existed. The glob pattern supraseal-c2-* in the rm -rf command had not matched. Why? Because the command was issued inside a [bash] tool call where the shell interprets the command. The * should have been expanded by the shell. However, looking more carefully at message 975, the rm -rf command was part of a multi-line bash block that included kill and sleep commands on previous lines. The entire block was passed as a single string to the bash tool. The glob expansion should have worked... unless there was a subtle issue with how the tool invocation handled the command string.

This is a moment where the article's reader can appreciate the fragility of build system automation. The assistant assumed the build artifacts were cleaned, issued the build command, and only discovered the truth when the output lacked the expected Compiling supraseal-c2 line. The subsequent investigation (messages 977–980) revealed that the build had actually picked up the changes — the .a file timestamp was 23:52 (after the edit) and its size had changed from 2791536 to 2790792 bytes. The strings check confirmed that pin_abc was gone from the binary. The build system had detected the changed .cu file via its rerun-if-changed directive and recompiled after all — but the rm -rf had been unnecessary because cargo's fingerprinting already handled the dependency tracking.

Assumptions Made and Lessons Learned

This message and its aftermath reveal several assumptions — some correct, some not:

Assumption 1: The rm -rf would clean build artifacts. This was partially incorrect. The glob may not have expanded as expected, or the directories may have been recreated during the build. Either way, the assumption that manual cleanup was necessary proved unnecessary.

Assumption 2: Cargo would not detect the .cu file change without manual cleanup. This was incorrect. Cargo's rerun-if-changed=cuda directive in build.rs (discovered in message 950) was working correctly. The build system did detect the modification to groth16_cuda.cu and recompiled the CUDA code — the assistant just couldn't tell from the truncated tail -15 output because supraseal-c2 compilation happens early in the build order and scrolled off.

Assumption 3: The build output would include Compiling supraseal-c2 if recompilation occurred. This was correct in principle but misleading in practice. The tail -15 truncated the output, hiding the earlier compilation lines. The assistant's choice to use tail -15 to keep the output manageable inadvertently hid the evidence that the build had worked correctly.

Assumption 4: The B1 reversion was the only change needed to recover baseline performance. This turned out to be correct for the GPU phase (the 5.7s overhead was eliminated), but the synthesis phase remained 5.5 seconds above baseline — pointing to A1 (SmallVec) as the next suspect, which would be investigated in subsequent messages using a synth-only microbenchmark.

The Broader Significance

Message 976 is, on its surface, a build command that appears to fail at its stated purpose. But that framing misses the point. The build did succeed — the binary was correctly compiled without the B1 optimization. What the assistant perceived as a failure (the absence of Compiling supraseal-c2 in the output) was actually a success obscured by a display truncation choice.

This moment encapsulates the essence of disciplined performance engineering. The assistant did not:

• Guess at the cause of the regression
• Apply random changes hoping for improvement
• Accept the 106-second result as "good enough"
• Commit unvalidated optimizations to the repository Instead, the assistant: 1. Measured precisely: Added CUZK_TIMING instrumentation to get phase-level breakdowns 2. Identified the culprit: B1's 5.7s overhead was unequivocally harmful 3. Reverted decisively: Removed the offending code cleanly 4. Rebuilt and verified: Confirmed the change compiled (even if the verification was momentarily confused) 5. Continued the investigation: The remaining 5.5s regression in synthesis would be tackled next with a synth-only microbenchmark The message also highlights a meta-lesson about build systems: they are complex, stateful, and sometimes opaque. A developer's intuition about what "should" trigger a recompilation can be wrong. The rerun-if-changed directive in build.rs was doing its job, but the assistant's mental model of the build process didn't account for it. The subsequent verification steps — checking file timestamps, comparing .a file sizes, using strings to inspect the binary — demonstrate the rigorous approach needed to build trust in the build system's output.

Conclusion

Message 976 is a build command that tells a story far beyond its 15 lines of output. It represents the intersection of performance analysis, surgical code modification, build system navigation, and the relentless pursuit of a known baseline. The assistant's journey from 106 seconds back toward 88.9 seconds required not just identifying the right optimization to revert, but also navigating the subtle complexities of a mixed-language build pipeline where CUDA compilation artifacts live outside cargo's standard output directory.

The build succeeded. The binary was correct. The regression was partially healed. And the stage was set for the next round of investigation — the A1 SmallVec synthesis slowdown — which would require an entirely different diagnostic approach. Message 976, for all its apparent mundanity, is a quiet testament to the discipline of measuring before acting, reverting before optimizing, and verifying before moving on.