The Build That Wasn't: A Lesson in CUDA Compilation Verification

In the middle of a high-stakes performance regression hunt, a single message from an AI assistant captures a moment that could have derailed an entire investigation. The message, indexed as <msg id=900> in a multi-session coding conversation, is deceptively brief — just a few lines of text and a truncated build output. But within this short exchange lies a critical insight about the fragility of heterogeneous build systems, the importance of verification discipline, and the kind of systematic thinking that separates effective debugging from wasted effort.

The Context: A Regression Under the Microscope

To understand why this message matters, we must first understand the situation that produced it. The conversation centers on the cuzk project — a CUDA-accelerated zero-knowledge proving daemon for Filecoin's Proof-of-Replication (PoRep) protocol. The team had successfully completed Phases 0 through 3 of the project, establishing a solid baseline performance of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4, however, introduced a suite of five optimizations (codenamed A1, A2, A4, B1, and D4) aimed at squeezing more performance out of the system. When these changes were first tested end-to-end, the result was alarming: the proof time had regressed to 106 seconds — a 19% slowdown.

The investigation that followed was a masterclass in disciplined performance engineering. The assistant systematically began reverting suspected harmful changes, starting with A2 (pre-sizing vectors) and B1 (pinning host memory with cudaHostRegister). In the messages immediately preceding <msg id=900>, the assistant had located and reverted the last remaining call site of A2 in pipeline.rs, replacing synthesize_circuits_batch_with_hint with the plain synthesize_circuits_batch and cleaning up the now-unused imports. The todo list was updated, and the build command was issued.

The Moment of Discovery

The subject message begins with a simple observation that carries enormous weight:

Build succeeded. But wait — it didn't recompile supraseal-c2 with CUDA. Let me check if the CUDA changes are actually being picked up:

This is the voice of experience. The build succeeded — Cargo reported no errors, no warnings that would stop compilation. On the surface, everything looked fine. But the assistant recognized a subtle problem: the CUDA .cu source files, which contain the GPU kernel code for the proving pipeline, had not been recompiled. The build system had used cached artifacts, meaning the binary now running might not include the critical changes — including the CUDA timing instrumentation (CUZK_TIMING printf's) that was essential for diagnosing the regression.

The assistant then runs a targeted verification command:

cargo build --release -p cuzk-core --features cuda-supraseal 2>&1 | tail -30

This command builds only the cuzk-core package with the cuda-supraseal feature flag. The output shown is mostly warnings about unused fields in bellperson — a dependency crate — which are harmless but noisy. The critical observation is that the build completed almost instantly, indicating that Cargo's incremental compilation cache was serving up pre-built artifacts rather than recompiling from source.

Why the Build System Deceived Them

The root cause of this deception lies in how the project's CUDA compilation is structured. The supraseal-c2 library uses a custom build.rs script to invoke nvcc (the NVIDIA CUDA compiler) directly, rather than going through Cargo's standard compilation pipeline. This means:

  1. Cargo's dependency tracking doesn't fully cover CUDA files. Cargo knows about .rs files and can detect changes to them, but changes to .cu and .cuh files are only detected by the build.rs script's own logic.
  2. The compiled CUDA artifacts live outside Cargo's standard output directory. As later messages reveal, the compiled .a archive files reside in paths like extern/cuzk/target/release/build/supraseal-c2-*/out/libgroth16_cuda.a — a directory managed by build.rs, not by Cargo's normal artifact tracking.
  3. Cargo's incremental compilation cache can mask rebuilds. When the Rust wrapper code hasn't changed, Cargo may decide that no rebuild is necessary, even if the underlying CUDA source has been modified. The build.rs script is only re-run if Cargo detects that the build script itself or its inputs have changed — and this detection is not always reliable for files outside the normal Rust source tree. This is a common pitfall in mixed-language projects. The Rust/Cargo ecosystem is designed primarily for Rust code, and while build.rs provides an escape hatch for integrating foreign code (C, C++, CUDA, etc.), the integration is not seamless. The build script must implement its own change detection, and if that detection is imperfect — or if Cargo decides to skip re-running the build script entirely — the developer ends up with a stale binary.

The Assumptions at Play

Several assumptions are embedded in this moment, some correct and some that would need to be challenged:

Correct assumption: The assistant correctly assumes that a successful Cargo build does not guarantee that CUDA code has been recompiled. This is a crucial piece of domain knowledge about the project's build architecture.

Correct assumption: The assistant assumes that the CUDA timing instrumentation is essential for diagnosing the regression. Without the CUZK_TIMING printf's in the GPU code, the team would be flying blind — unable to distinguish between time spent in NTT (Number Theoretic Transform), MSM (Multi-Scalar Multiplication), memory pinning, or other GPU operations.

Implicit assumption being tested: The assistant implicitly assumes that the cuda-supraseal feature flag is the correct mechanism to trigger CUDA compilation. This turns out to be correct — cuzk-core has cuda-supraseal as its default feature, and enabling it propagates the feature flag down to bellperson, storage-proofs-core, and ultimately supraseal-c2.

Assumption about caching: The assistant assumes that touching the CUDA source files or cleaning the supraseal-c2 package will force a rebuild. In subsequent messages (outside the subject message), this assumption is tested and partially fails — cargo clean -p supraseal-c2 only removes Rust-side artifacts, not the CUDA-compiled .a files, which must be manually located and deleted.

The Knowledge Required to Understand This Message

To fully grasp what's happening in <msg id=900>, a reader needs:

  1. Understanding of Cargo's build model — that Cargo uses incremental compilation, caches artifacts, and tracks dependencies through file hashes.
  2. Knowledge of build.rs scripts — that Rust projects can include custom build scripts that run arbitrary code during compilation, often used to compile foreign language sources.
  3. Awareness of CUDA compilation workflows — that .cu files are compiled by nvcc, not by rustc, and that the resulting object files must be linked into the final binary via a static archive or shared library.
  4. Context about the regression investigation — that five optimizations were applied, the overall result was a slowdown, and the team is systematically reverting changes and measuring impact.
  5. Knowledge of the project's feature flag system — that cuda-supraseal is a Cargo feature that gates the inclusion of CUDA backend code, and that it must be enabled for the GPU kernels to be compiled and linked.

The Knowledge Created by This Message

This message produces several important pieces of knowledge:

  1. A verified gap in the build process: The assistant has confirmed that the CUDA code is not being automatically recompiled when source files change. This is a finding about the project's build infrastructure that could affect future development velocity.
  2. A documented verification step: The message establishes a pattern of verifying that builds actually contain expected changes, rather than trusting Cargo's "Finished" message. This becomes part of the team's operational knowledge.
  3. A decision point: The assistant now knows that further investigation is needed to force CUDA recompilation. The subsequent messages show the assistant searching for the compiled artifacts, checking timestamps, verifying that the instrumented code is present via strings on the binary, and ultimately confirming that the daemon does contain the CUZK_TIMING strings.
  4. Evidence for the build system's behavior: The fact that cargo build --release -p cuzk-core --features cuda-supraseal returns immediately with cached artifacts tells us that Cargo considers the Rust-side inputs to be unchanged, even though the CUDA sources have been modified. This is valuable diagnostic information about the limits of Cargo's change detection.

The Thinking Process Revealed

The reasoning visible in this message follows a pattern that experienced engineers will recognize:

Step 1 — Observe the anomaly: The build "succeeded" but something feels wrong. The assistant notices the absence of CUDA recompilation, likely because the build completed too quickly or because the expected nvcc output was missing from the build log.

Step 2 — Formulate a hypothesis: "The CUDA changes aren't being picked up." This is a hypothesis about the build system's caching behavior.

Step 3 — Design a test: Run a targeted build with the explicit feature flag and examine the output. The --features cuda-supraseal flag is deliberately chosen to ensure that the CUDA backend is activated, and -p cuzk-core focuses the build on the package that directly depends on supraseal-c2.

Step 4 — Interpret the results: The truncated output shows only Rust warnings — no CUDA compilation messages. The assistant interprets this as confirmation that the CUDA code was not recompiled.

Step 5 — Plan next steps: The message ends with the assistant poised to investigate further. In the subsequent messages, we see the assistant touching source files, checking timestamps, locating compiled artifacts, and verifying the presence of instrumentation strings in the final binary.

This is not the work of someone who blindly trusts tools. It is the work of someone who understands that every build system has blind spots, and that verification is not optional — especially when debugging a performance regression where a 5-second difference in a 90-second benchmark is significant.

The Broader Lesson

The message <msg id=900> is a microcosm of the entire regression investigation. It demonstrates that performance engineering is not just about writing faster code — it is about building trust in your measurements. If the assistant had accepted the build at face value and run the instrumented test, the timing data might have been subtly wrong, or the CUZK_TIMING printf's might not have appeared, leading to wasted debugging time and confusion.

The message also highlights a recurring theme in the conversation: the tension between the convenience of build systems and the reality of heterogeneous codebases. Cargo is excellent at managing Rust dependencies, but when you step outside the Rust ecosystem — into CUDA, into custom build scripts, into foreign code compilation — you inherit complexity that the build system cannot fully abstract away.

For the cuzk project, this moment of verification was not a distraction from the real work of diagnosing the regression. It was an essential part of it. The assistant's refusal to accept a "successful" build at face value is what allowed the investigation to proceed on solid ground, with confidence that the binaries being tested actually contained the changes being measured.

In the end, the regression was traced to the SmallVec optimization (A1) and the cudaHostRegister overhead (B1), not to build system issues. But the discipline of verification ensured that when the timing data came in, it could be trusted. That trust is the foundation upon which all performance analysis rests.