The Build Artifact That Wouldn't Die: A Lesson in CUDA Build System Debugging

In the middle of a high-stakes performance regression diagnosis, a single message captures a moment of quiet realization that could have derailed an entire benchmarking session. Message [msg 977] is deceptively brief — just two lines of text and a shell command — but it represents a critical inflection point in the Phase 4 optimization work for the cuzk project, a pipelined SNARK proving engine for Filecoin's Proof-of-Replication (PoRep). The message reads:

Hmm, I notice supraseal-c2 wasn't recompiled again. The rm -rf command might not have worked. Let me check:

>

`` find /home/theuser/curio/extern/cuzk/target/release/build -name "*supraseal*" -maxdepth 1 /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-0ac55c4ac30a878b /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-14f6b99f1cbc5f79 /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c ``

This article unpacks what makes this moment so significant: the interplay between build system mechanics, performance engineering rigor, and the quiet discipline of noticing when something is wrong.

The Context: A Regression Hunt in Full Swing

To understand message [msg 977], we must first understand the war being fought. The cuzk project had successfully implemented Phases 0 through 3 of a multi-phase optimization pipeline, establishing a strong baseline of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 introduced five optimizations (A1 SmallVec, A2 Pre-sizing, A4 Parallel B_G2, B1 cudaHostRegister, D4 Per-MSM window tuning), but the initial application caused a regression to 106 seconds — a 19% slowdown.

The assistant had been systematically working through this regression. By message [msg 968], the first instrumented test had identified B1 (cudaHostRegister) as the primary culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead, far exceeding the estimated 150–300 milliseconds. The decision was made to revert B1 and re-test.

Message [msg 975] shows the assistant executing the reversion: editing the CUDA source files to remove the cudaHostRegister and cudaHostUnregister calls, then running a rebuild command:

rm -rf /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-*
cargo build --release -p cuzk-daemon 2>&1 | tail -15

The build output in message [msg 976] shows various Rust packages being compiled — filecoin-hashers, storage-proofs-core, filecoin-proofs, cuzk-core, cuzk-server — but notably absent is any line saying "Compiling supraseal-c2." This absence is what triggers the investigation in message [msg 977].## What the Message Reveals: The Artifact Persistence Problem

The core observation in message [msg 977] is that three build directories for supraseal-c2 still exist under the target/release/build/ directory, despite the rm -rf command that was supposed to delete them. The assistant's phrasing — "Hmm, I notice supraseal-c2 wasn't recompiled again. The rm -rf command might not have worked" — is telling. It's not a statement of certainty, but a hypothesis being tested.

The find output confirms the suspicion: three directories remain:

• supraseal-c2-0ac55c4ac30a878b
• supraseal-c2-14f6b99f1cbc5f79
• supraseal-c2-bbf265913f050d8c The fact that there are three build directories is itself informative. Cargo uses hash-based directory names to distinguish different build configurations. The presence of multiple directories suggests that different feature flags or dependency versions have caused Cargo to rebuild the package multiple times, each with a different hash. The most recent one (bbf265913f050d8c) is the one that was last used — and it still contains the old, B1-included artifact. This is a classic build system gotcha. When a Rust package uses a build.rs script to compile external C/CUDA code, the build artifacts live in a subdirectory of the Cargo build output directory. The rm -rf command targeted the pattern supraseal-c2-*, but the shell's glob expansion may have failed silently — or, more likely, the directories were recreated by a concurrent or subsequent build process. The assistant's cargo build command in message [msg 976] ran after the rm -rf, and Cargo, seeing that the build script's output directory was missing, would have recreated it and potentially reused cached compilation results from nvcc's own cache (PTX caching), giving the illusion of a fresh build while actually linking against the old CUDA object code.

The Reasoning Process: What the Assistant Was Thinking

The message reveals a subtle but crucial reasoning chain. The assistant had just executed two operations in sequence: (1) editing the CUDA source files to remove B1's cudaHostRegister calls, and (2) running a rebuild. The build output showed Rust packages recompiling but no "Compiling supraseal-c2" line. This could mean:

1. The build was genuinely fresh — Cargo detected the .cu file changes and recompiled supraseal-c2, but the output was scrolled away by the tail -15 filter.
2. The build was stale — Cargo did not detect the changes and used the old artifact.
3. The build artifacts were deleted but recreated — The rm -rf deleted the directories, but Cargo recreated them from nvcc's cache, producing the same binary without recompilation. The assistant's intuition leans toward option 2 or 3. The phrase "wasn't recompiled again" suggests this is a recurring problem — earlier in the session (messages [msg 945] through [msg 954]), the assistant had struggled with the CUDA build system not detecting changes to .cu files, discovering that the build script's rerun-if-changed=cuda directive should have worked but the caching behavior of nvcc made it appear as though no recompilation occurred. The decision to run find rather than immediately re-executing the rebuild is itself a methodological choice. Instead of blindly retrying the operation, the assistant pauses to gather evidence. This is a hallmark of disciplined debugging: verify the state before acting, not after.

The Input Knowledge Required

To understand this message, one needs several layers of context:

Build system architecture: Cargo's build system uses a build.rs script for packages that need to compile non-Rust code. For supraseal-c2, this script invokes nvcc (the NVIDIA CUDA compiler) to compile .cu files into a static library (libgroth16_cuda.a). The output goes into a hash-named directory under target/release/build/supraseal-c2-*/out/. Cargo tracks file changes via fingerprints and the rerun-if-changed directive.

The B1 optimization: One of the Phase 4 Wave 1 optimizations was to use cudaHostRegister to pin host memory, enabling faster DMA transfers between CPU and GPU. The instrumented test revealed this added 5.7 seconds of overhead due to mlock-ing 125 GiB of memory, making it a net negative.

The regression diagnosis workflow: The assistant had built a CUDA timing instrumentation system (CUZK_TIMING printf's with fflush(stderr)) to get phase-level breakdowns. The first instrumented run (message [msg 962]) showed B1 as the primary problem, leading to the reversion decision.

Shell glob behavior: The rm -rf command used a wildcard pattern (supraseal-c2-*). In zsh (the shell in use, as shown by the zsh: no matches found error in message [msg 946]), if no files match a glob pattern, the shell may either error out or pass the literal pattern to the command. The assistant had encountered this earlier and used explicit paths instead.## Assumptions and Their Consequences

Message [msg 977] exposes several assumptions, some valid and some questionable:

Assumption 1: The rm -rf command executed as intended. The assistant assumed that the shell command rm -rf /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-* would delete all matching directories. However, the find output shows three directories still present. Why? One possibility is that the glob expansion happened before the directories existed (if a previous build had already been cleaned and Cargo hadn't recreated them yet), but this seems unlikely given the context. More likely, the rm -rf command ran but the directories were recreated by the subsequent cargo build invocation, which detected the missing build output and re-ran the build script. The build script, in turn, may have used nvcc's cached PTX files, producing the same artifact without recompiling the .cu source.

Assumption 2: The build output would show supraseal-c2 compilation. The assistant expected to see "Compiling supraseal-c2" in the build output. Its absence was the trigger for investigation. This assumption was reasonable — if Cargo re-runs the build script, it typically prints the package name. However, the tail -15 filter could have scrolled the relevant line away, especially if the build script's output is verbose.

Assumption 3: The CUDA source changes would trigger a rebuild. The build.rs file contained println!("cargo:rerun-if-changed=cuda");, which should cause Cargo to re-run the build script whenever any file in the cuda/ directory changes. This assumption was correct in principle, but the caching behavior of nvcc (which caches compiled PTX based on source content hashes) could mask the recompilation. The assistant had verified this earlier by checking the library size and strings.

The Output Knowledge Created

Despite its brevity, message [msg 977] produces valuable knowledge:

1. Confirmation of stale artifacts: The find output proves that the old build directories persist. This is actionable information — the assistant now knows that the next proof test would use the old, B1-included binary, producing invalid benchmark results.
2. Identification of the build system failure mode: The persistence of three build directories reveals that Cargo's build artifact management for CUDA packages is more complex than expected. The hash suffixes (0ac55c4ac30a878b, 14f6b99f1cbc5f79, bbf265913f050d8c) correspond to different build configurations, and the most recent one may not be the one being used.
3. A methodological lesson: The message demonstrates the importance of verifying build state before running benchmarks. A less rigorous engineer might have proceeded with the test, obtained a timing of ~101 seconds (the B1-included result), and concluded that B1 wasn't the problem or that other changes were also regressing. The assistant's willingness to pause and check saved an entire benchmarking iteration.

The Thinking Process Visible in the Reasoning

The assistant's internal monologue is implicit but discernible. The sequence of events leading to message [msg 977] reveals a chain of reasoning:

1. "I edited the CUDA source to remove B1." (message [msg 970] through [msg 973])
2. "I ran rm -rf to delete old build artifacts." (message [msg 975])
3. "I ran cargo build and saw Rust packages recompiling, but no supraseal-c2." (message [msg 976])
4. "That's suspicious — the CUDA code should have changed. Let me check if the artifacts were actually deleted." (message [msg 977]) The "Hmm" at the beginning of the message is a verbalization of doubt — a moment of cognitive dissonance where the expected outcome (a clean rebuild with B1 removed) doesn't match the observed outcome (no CUDA recompilation). This is the same pattern seen throughout the session: the assistant repeatedly checks its assumptions against reality, using find, ls, strings, and other diagnostic tools to verify state before proceeding.

The Broader Significance

Message [msg 977] is, on its surface, a mundane build system debugging moment. But it represents something deeper: the discipline of performance engineering at scale. When optimizing a system that generates Groth16 proofs for Filecoin — where each proof takes ~90 seconds and consumes ~200 GiB of memory — every benchmarking run is expensive. A single invalid test (using the wrong binary) wastes not just time but trust in the data. If the assistant had run the B1-reverted test with the old binary, it would have gotten the same ~101-second result and might have concluded that B1 wasn't the problem, leading down a false path.

The message also illustrates a universal truth about software optimization: the build system is part of the system being optimized. When you're trying to shave seconds off a 90-second pipeline, you cannot afford to waste a single benchmarking iteration on stale artifacts. The assistant's vigilance here — catching the stale build before running the test — is the difference between reliable data and noise.

In the subsequent messages ([msg 978] through [msg 987]), the assistant successfully forces a clean rebuild, runs the instrumented test, and obtains a clean 94.4-second result — confirming that B1 was indeed the primary culprit and that the remaining 5.5-second regression (synthesis at 60.3 seconds vs. 54.7 seconds baseline) must be caused by the A1 SmallVec optimization. This sets the stage for the next phase of diagnosis: building a synth-only microbenchmark to isolate the synthesis regression, which ultimately reveals that SmallVec causes a 5–6 second slowdown on the AMD Zen4 Threadripper PRO 7995WX system.

Conclusion

Message [msg 977] is a masterclass in the art of noticing. In a high-pressure debugging session with dozens of tool calls, timing measurements, and code edits, the assistant pauses to question whether a build artifact is fresh. That moment of skepticism — the "Hmm" that starts the message — is what separates rigorous performance engineering from cargo-cult optimization. It's a reminder that in complex systems, the build system is never just plumbing; it's a first-class component that demands the same scrutiny as the code it compiles.