The Stale Artifact: A Two-Line Revelation in Performance Debugging

In the middle of a grueling performance regression diagnosis, a seemingly trivial two-line message from the assistant reveals the hidden complexity of mixed-language build systems and the discipline required for rigorous performance engineering. Message [msg 909] consists of just a brief observation — "Two copies — one might be stale. Let me check timestamps and clean properly" — followed by a single ls command and its output. Yet this moment crystallizes a critical insight about the gap between what developers think their build system is doing and what it actually does, a gap that can silently invalidate hours of benchmarking work.

The Context: Diagnosing a 19% Regression

To understand why this message matters, we must step back into the broader narrative. The cuzk project had successfully completed Phases 0 through 3, building a pipelined, batched SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) that achieved a baseline of 88.9 seconds per 32 GiB proof. Phase 4 Wave 1 introduced five optimizations (A1 SmallVec, A2 Pre-sizing, A4 Parallel B_G2, B1 cudaHostRegister, D4 Per-MSM window tuning) intended to improve throughput. Instead, the first end-to-end test showed a regression to 106 seconds — a 19% slowdown.

The assistant had been systematically working through a diagnostic plan. It had already reverted the A2 optimization (pre-sizing vectors) from the remaining call site in pipeline.rs ([msg 894]) and cleaned up unused imports ([msg 896]). The build succeeded ([msg 900]), but crucially, the CUDA source files — which contained newly added timing instrumentation (CUZK_TIMING printf's) — were not recompiled. The build system's caching layer had decided the CUDA artifacts were up-to-date when they were not, or rather, when the assistant could not be certain they were.

The Discovery: Two Copies, One Stale

The assistant's journey to this moment began with a series of attempts to force CUDA recompilation. Touching the source files and build.rs ([msg 901]) had no effect. Running cargo clean -p supraseal-c2 ([msg 907]) removed 114 Rust-side files but left the CUDA compiled objects untouched. A subsequent find command ([msg 908]) revealed the artifacts lived at extern/cuzk/target/release/build/supraseal-c2-*/out/libgroth16_cuda.a — and there were two copies.

This is the moment captured in message [msg 909]. The assistant's reasoning is visible in the terse observation: "Two copies — one might be stale." The hash suffixes in the directory names (a0c4f246f264bbc3 and bbf265913f050d8c) suggest two different build configurations or compiler fingerprint hashes. One copy was timestamped Feb 17 16:22 (2780312 bytes), the other Feb 17 23:33 (2791224 bytes). The 23:33 copy was 9,112 bytes larger and 7 hours newer — almost certainly the one containing the timing instrumentation. But which one was the build system actually linking?

The Build System Trap

This message exposes a fundamental challenge in heterogeneous build systems. Rust's cargo manages Rust compilation with precise dependency tracking and incremental rebuilds. But CUDA code compiled via a build.rs script using nvcc operates outside this framework. The build.rs script compiles .cu files into a static library (.a), which is then linked into the Rust binary. Cargo tracks the build.rs script itself as a dependency, but it does not deeply inspect the script's inputs and outputs. If the build.rs script does not properly signal to cargo that its outputs have changed, cargo will consider the artifact up-to-date even when the underlying CUDA source has been modified.

The presence of two copies with different hashes suggests that at some point, the build configuration changed (perhaps a compiler flag, a feature toggle, or a dependency version), causing cargo to create a new build directory. The old directory remained, orphaned but not cleaned. This is a well-known pain point: cargo clean only removes artifacts that cargo knows about, and the CUDA .a files produced by build.rs live outside cargo's artifact tracking.

Assumptions and Their Consequences

The assistant made a reasonable assumption: that cargo clean -p supraseal-c2 would remove all build artifacts for that package, including those produced by build.rs. This assumption proved incorrect. The Rust build system and the CUDA build system have different mental models of what constitutes a "build artifact," and neither fully accounts for the other.

The assistant also assumed that touching the source files (groth16_cuda.cu, groth16_srs.cuh, build.rs) would trigger a rebuild. This failed because cargo's fingerprinting checks the build.rs script's modification time against its previous output, but the script itself must be the one to detect changes in its inputs. If build.rs uses a cached timestamp file or simply doesn't recheck, touching files has no effect.

A subtler assumption was that the build system is deterministic and transparent — that the artifacts you see are the artifacts you get. The discovery of two copies undermines this trust. Which one is being linked? The linker searches library paths in a defined order, and if both directories are on the search path, the first match wins. Without understanding the link order, the assistant cannot know whether the instrumented or non-instrumented code is actually running.

Input Knowledge Required

To fully understand this message, one must know:

  1. How Rust's build system works: Cargo compiles packages in dependency order, caching results in target/. Each build configuration (profile, features, compiler version) gets a unique hash directory.
  2. How build.rs scripts work: A build.rs script is compiled and executed by cargo to perform custom build steps. It can invoke external compilers (like nvcc), produce arbitrary output files, and link them into the final binary. Cargo tracks the build.rs binary itself but has limited visibility into its runtime behavior.
  3. How CUDA compilation works in mixed-language projects: CUDA source files (.cu) are compiled by nvcc into object files, then archived into a static library (.a). This .a file is linked into the Rust binary. The build.rs script must manually track dependencies and trigger recompilation when inputs change.
  4. The concept of build artifact staleness: A stale artifact is one that was produced from older source code but is not rebuilt because the build system doesn't detect the change. This can lead to silently incorrect benchmarks.
  5. The Phase 4 regression context: The team is trying to measure the effect of specific optimizations. If the instrumented CUDA code isn't actually running, the timing data will be wrong or missing, potentially leading to incorrect conclusions about which optimizations help or hurt.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. Two build directories exist: The presence of a0c4f246f264bbc3 and bbf265913f050d8c indicates two separate build configurations or compiler fingerprints.
  2. Timestamps differ by ~7 hours: The older copy (16:22) predates the CUDA changes; the newer copy (23:33) likely includes them.
  3. File sizes differ: The newer copy is ~9 KB larger (2791224 vs 2780312), consistent with added instrumentation code.
  4. The build system is not cleaning CUDA artifacts: cargo clean and cargo clean -p supraseal-c2 do not remove these .a files, confirming they are outside cargo's artifact management.
  5. A manual cleanup strategy is needed: The assistant will need to delete the stale directories or use a more aggressive cleanup to ensure only the correct artifact is linked.

The Thinking Process

The assistant's thinking, though compressed into just two lines, reveals a structured diagnostic approach:

  1. Observation: The find command revealed two copies of libgroth16_cuda.a. This is unusual — normally there should be one per build configuration.
  2. Hypothesis: "One might be stale." The assistant suspects that an older build artifact survived a rebuild, potentially due to the build system's caching behavior.
  3. Verification plan: "Let me check timestamps and clean properly." The assistant needs to determine which copy is current and which is stale, then remove the stale one to ensure the next build produces a clean, known-good artifact.
  4. Execution: The ls command with the glob pattern supraseal-c2-*/out/libgroth16_cuda.a lists both files with their timestamps and sizes, providing the data needed to make a judgment. The brevity of the message belies its significance. The assistant is not just looking at files — it is building a mental model of the build system's behavior, identifying a failure mode (stale artifacts persisting through rebuilds), and planning a remediation. This is the essence of disciplined performance engineering: when measurements don't match expectations, you don't just tweak parameters — you verify the entire chain from source code to running binary.

The Broader Lesson

Message [msg 909] exemplifies a principle that extends far beyond this specific project: trust the measurement, not the build. In performance engineering, it is tempting to assume that a successful build produces a binary containing exactly the code you wrote. But build systems are complex, caching is aggressive, and stale artifacts can silently persist. The only way to be certain is to verify — check timestamps, compare file sizes, force clean rebuilds, and instrument the output to confirm the instrumentation is actually present.

This message also highlights the value of visible thinking in debugging. The assistant's brief statement — "Two copies — one might be stale" — is not just a status update. It is a hypothesis, a plan, and a rationale, all compressed into eight words. For anyone reading the conversation log, this provides a window into the assistant's reasoning process and builds confidence that the subsequent actions are grounded in careful diagnosis rather than guesswork.

In the end, the assistant's ability to identify and articulate this build system subtlety — in just two lines — is what separates systematic debugging from trial-and-error. The stale artifact problem, once recognized, can be addressed. But it must first be seen.