The Build System Detective: Forcing CUDA Recompilation in a Complex Multi-Repo Rust Workspace

"It didn't rebuild supraseal-c2 — the CUDA .cu files didn't get recompiled. The build system caches the result. Let me force it."

This single message, spoken by the AI assistant in an opencode coding session, captures a moment of quiet but critical friction in a high-stakes performance engineering effort. The team is deep in Phase 4 of optimizing cuzk, a custom SNARK proving daemon for Filecoin's Proof-of-Replication (PoRep) protocol. They have just finished reverting a suspected harmful optimization (A2, pre-sizing vectors) from the pipeline code, and now they need to rebuild and test. But the build system refuses to cooperate. The CUDA kernel files—the very code they've instrumented with timing printfs to diagnose a 19% performance regression—are sitting stale, cached, untouched. This message is the assistant's attempt to break that stalemate.

Context: A Performance Regression Under the Microscope

To understand why this message matters, we must understand the stakes. The cuzk project had achieved a solid baseline of 88.9 seconds for a single 32 GiB PoRep Groth16 proof. Phase 4 introduced five optimizations targeting both CPU synthesis and GPU proving: A1 (SmallVec for the LC Indexer), A2 (pre-sizing vectors to avoid reallocation), A4 (parallelizing B_G2 MSMs on the CPU), B1 (pinning a/b/c vectors with cudaHostRegister), and D4 (per-MSM window tuning). But the first end-to-end test revealed a troubling regression: 106 seconds, a 19% slowdown from baseline.

The assistant and user have been systematically diagnosing this regression. A2 was partially reverted in the multi-sector synthesis path, and in the immediately preceding messages ([msg 891]–[msg 906]), the assistant finished reverting A2 from the remaining call site in pipeline.rs, replacing synthesize_circuits_batch_with_hint with the plain synthesize_circuits_batch, and cleaning up the now-unused imports. The build succeeded, but something was wrong: the CUDA source files—groth16_cuda.cu and groth16_srs.cuh—had not been recompiled.

The Message: What Actually Happened

The assistant's message at index 907 is deceptively simple. It begins with a diagnosis:

It didn't rebuild supraseal-c2 — the CUDA .cu files didn't get recompiled. The build system caches the result. Let me force it.

Then it executes a two-step command:

cargo clean -p supraseal-c2 2>&1 && cargo build --release -p cuzk-daemon -p cuzk-bench 2>&1 | tail -50

The output confirms the clean removed 114 files totaling 207.8 MiB. The subsequent build output shows only Rust-level warnings about dead code and unused functions—no CUDA compilation messages are visible in the tail, but the implication is that the clean forced a fresh build of the CUDA code.

This message is the culmination of a build-system debugging subplot that began two messages earlier ([msg 900]), when the assistant first noticed that supraseal-c2 hadn't recompiled. The assistant had tried touching the CUDA files and the build.rs file ([msg 901]), but that failed to trigger recompilation because the CUDA compilation artifacts are managed by a custom build.rs script and live outside the standard cargo output directory. The assistant then tried building with --features cuda-supraseal on the wrong package ([msg 902]), and had to figure out that cuzk-daemon (not cuzk-bench) was the correct target ([msg 903]–[msg 906]). Message 907 is the resolution: brute-force the issue by cleaning the entire supraseal-c2 package.

Input Knowledge: What You Need to Understand This Message

To grasp the significance of this message, a reader needs several layers of context:

  1. The build system architecture: The workspace contains multiple Rust crates (cuzk-core, cuzk-daemon, cuzk-bench, etc.) with a complex dependency graph. The supraseal-c2 crate is a CUDA-accelerated Groth16 prover whose compilation is orchestrated by a custom build.rs script that invokes nvcc (the NVIDIA CUDA compiler). This is not a simple cargo build scenario—the CUDA compilation artifacts live outside cargo's standard target/ directory, so cargo's file-change detection doesn't work on them. Touching the .cu files or build.rs does not trigger a rebuild because cargo only tracks changes to files listed in Cargo.toml and standard Rust source files.
  2. The feature flag system: The CUDA backend is gated behind a cuda-supraseal feature that propagates through the dependency chain. cuzk-core defines this feature and passes it to bellperson, storage-proofs-core, and supraseal-c2. The daemon binary (cuzk-daemon) depends on cuzk-core with default features (which include cuda-supraseal). The bench binary (cuzk-bench) is just a gRPC client and doesn't need CUDA at all.
  3. The performance stakes: The entire purpose of this build is to run an instrumented single-proof test that will produce a CUZK_TIMING: phase-level breakdown from CUDA printf statements. This data will determine which of the five Phase 4 optimizations are kept and which are reverted. Every minute spent wrestling the build system is time not spent collecting that data.
  4. The git discipline: All Phase 4 changes remain uncommitted. The team is maintaining strict discipline: only a validated, regression-free set of optimizations will be committed to the feat/cuzk branch. This means the working tree is dirty with uncommitted changes across multiple repositories (bellperson, bellpepper-core, supraseal-c2, cuzk).

Assumptions and Their Consequences

The assistant made several assumptions in the lead-up to this message, some of which proved incorrect:

Assumption 1: Touching source files triggers rebuild. The assistant assumed that touch extern/supraseal-c2/cuda/groth16_cuda.cu would cause cargo to recompile the package. This assumption failed because cargo tracks file hashes for Rust source files listed in Cargo.toml, but the CUDA .cu files are compiled by a build.rs script. Cargo does not monitor files touched by build.rs—it only re-runs the build script if the build.rs file itself changes or if any file listed in Cargo.toml's build metadata changes. The .cu files are invisible to cargo's change detection.

Assumption 2: cargo build -p cuzk-bench --features cuda-supraseal would work. The assistant assumed that cuzk-bench had the cuda-supraseal feature. It didn't. This was a quick error-and-correct cycle ([msg 902]–[msg 903]), but it reveals the complexity of the feature propagation system across a multi-crate workspace.

Assumption 3: cargo clean -p supraseal-c2 would be sufficient. This assumption proved correct—the clean removed 114 files and 207.8 MiB of artifacts, and the subsequent rebuild presumably compiled the CUDA code fresh. But the clean is a heavy hammer: it invalidates the entire build cache for that package, not just the CUDA artifacts. A more targeted approach would be preferable in a production setting.

Output Knowledge: What This Message Created

This message produced several forms of knowledge:

  1. A working binary with instrumented CUDA code. The immediate output is a freshly compiled cuzk-daemon binary that includes the CUZK_TIMING printf instrumentation. This binary is the tool needed to collect the phase-level timing breakdown that will drive the optimization decisions.
  2. Build system knowledge. The message (and the surrounding messages) document a subtle build system behavior: CUDA files compiled by build.rs in a Rust workspace are not tracked by cargo's change detection. The only reliable way to force a rebuild is cargo clean -p <package>. This is a piece of institutional knowledge that would be valuable for anyone else working on this project.
  3. Confirmation of the clean's scale. The output "Removed 114 files, 207.8MiB total" provides a data point about the size of the supraseal-c2 build artifacts. This is useful for estimating build times and cache sizes.
  4. A decision point. With the build now containing the instrumented CUDA code, the team can proceed to the critical test: running a single proof and collecting the CUZK_TIMING breakdown. The next messages in the conversation will reveal whether the timing data confirms the suspected culprits (B1's cudaHostRegister overhead and A2's page-fault storm) or reveals new surprises.

The Thinking Process: What the Assistant's Reasoning Reveals

The assistant's reasoning in this message is implicit but clear. The message opens with a diagnosis: "It didn't rebuild supraseal-c2 — the CUDA .cu files didn't get recompiled. The build system caches the result." This shows the assistant has identified the root cause of the stale build: caching. The solution proposed is equally direct: "Let me force it."

The choice of cargo clean -p supraseal-c2 over other approaches (like modifying timestamps more aggressively, or deleting specific output files) reveals a pragmatic mindset. The assistant has already tried the gentler approaches (touching files, rebuilding with feature flags) and they failed. Now it's time for the nuclear option. The && chaining ensures that if the clean fails, the build won't attempt to run on a dirty state.

The use of tail -50 to capture the build output shows the assistant is being selective about what it displays—it knows the build will produce many warnings (the dead_code warnings from bellperson's metric_cs.rs are a recurring theme), and it only needs to see the tail end to confirm success or catch errors.

Why This Message Matters

In the grand narrative of the cuzk optimization project, message 907 is a small but necessary pivot point. It represents the moment when the team stops trying to work around the build system and instead works with it. The assistant could have spent more time trying to understand why touching files didn't work, or trying to configure cargo to track the CUDA files. Instead, it recognized that the fastest path to the instrumented binary was a brute-force clean and rebuild.

This is a lesson in practical performance engineering: the measurement tool (the instrumented binary) must exist before any data can be collected, and sometimes the most efficient way to build the tool is to accept a 200 MiB cache invalidation and wait for the compiler. The discipline shown here—finishing the A2 revert, cleaning up imports, verifying the build, and only then forcing the CUDA recompilation—is the same discipline that will later allow the team to confidently commit a validated set of optimizations.

The message also highlights a recurring challenge in heterogeneous computing projects that mix Rust and CUDA: the build systems for the two languages are not well-integrated. Cargo's excellent incremental compilation and change detection, one of Rust's greatest strengths, breaks down when the build process escapes into a custom build.rs script. This is a known pain point in the Rust+HPC ecosystem, and this message is a concrete example of that friction in action.