The Phantom Rebuild: A Micro-Drama in CUDA Build System Debugging

In the course of a deep performance engineering session on the cuzk SNARK proving engine, a single, deceptively simple message at index 953 captures a moment of quiet revelation. The message reads:

New ones were created. Let me check: ``bash ls -la /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-*/out/libgroth16_cuda.a 2>/dev/null -rw-r--r-- 1 theuser theuser 2791536 Feb 17 23:44 /home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c/out/libgroth16_cuda.a ``

This is not a dramatic message. There are no complex code changes, no bold declarations, no architectural decisions. Yet it represents the climax of a frustrating subplot within a larger performance regression hunt — the struggle to understand whether a C++/CUDA source file had actually been recompiled after being edited. The message is the moment the assistant realizes that the build system had been quietly doing its job all along, despite every indication to the contrary.

The Context: Chasing a Performance Regression

To understand why this message matters, we must step back into the broader narrative. The cuzk project is a high-performance SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol. After successfully completing Phases 0 through 3 — which established a strong baseline of 88.9 seconds for a single 32 GiB PoRep proof — the team had moved to Phase 4, implementing a suite of compute-level optimizations. Five changes were made: A1 (SmallVec optimization for the LC Indexer), A2 (pre-sizing vectors), A4 (parallelizing B_G2 CPU MSMs), B1 (pinning a/b/c vectors with cudaHostRegister), and D4 (per-MSM window tuning).

When the combined changes regressed performance from 88.9 seconds to 106 seconds, a systematic diagnosis began. The assistant had added detailed CUDA timing instrumentation — CUZK_TIMING printf calls — throughout the GPU code to enable precise phase-level breakdowns. But when the first instrumented test ran, the timing output was nowhere to be found. The printf output from the C++ host code was being fully buffered because stdout was redirected to a file, and the buffer was never flushed before the process exited.

The fix was straightforward: add fflush(stderr) after each printf("CUZK_TIMING: ...") call in groth16_cuda.cu. The assistant made these edits across six locations in the file and then attempted to rebuild.

The Build System Puzzle

What followed was a comedy of errors with the cargo build system. The assistant's first attempt — cargo clean -p supraseal-c2 — removed zero files because cargo clean only cleans Rust compilation artifacts, not the output of build.rs scripts. The CUDA compilation is managed by a custom build.rs that invokes nvcc, and its outputs live in a separate directory tree under target/release/build/supraseal-c2-*.

The next attempt used a zsh glob: rm -rf extern/cuzk/target/release/build/supraseal-c2-*. This failed with "no matches found" because zsh's globbing behavior differs from bash — the * wasn't expanding as expected. The assistant then fell back to using find to locate the build directories and manually deleted them with explicit paths.

After deleting the build artifacts and rerunning cargo build --release -p cuzk-daemon, the output showed no "Compiling supraseal-c2" line. The assistant concluded, "It still didn't recompile supraseal-c2." This seemed to confirm that the build system wasn't detecting changes to .cu files, despite the build.rs containing println!("cargo:rerun-if-changed=cuda"). The assistant even checked the directory listing to confirm the .cu file had been modified (timestamp Feb 17 23:42).

The Moment of Discovery

Message 953 is the pivot point. The assistant, having given up on reading cargo's output as a reliable signal, decides to check the actual artifact directly. The phrase "New ones were created" is the first acknowledgment that the build system had been working. Despite the misleading cargo output — no "Compiling" line, no visible recompilation — new build directories were created, and a fresh libgroth16_cuda.a existed with a timestamp of 23:44, two minutes after the edit.

The file size, 2,791,536 bytes, is itself a subtle confirmation. In the following message ([msg 954]), the assistant notes that this size differs from the previous build artifact (2,791,224 bytes). The change is tiny — just 312 bytes — but it proves that the CUDA code was recompiled. The fflush(stderr) calls were baked into the static library.

Why the Build System Was Misleading

The cargo build system's behavior, while correct, was deeply misleading in this context. The build.rs script for supraseal-c2 compiles CUDA files using nvcc, which has its own caching layer. When the .cu file was edited and build.rs was re-run, nvcc detected that only a small portion of the file had changed (adding fflush calls after existing printf lines) and likely reused cached PTX or object files for the unchanged portions. This made the rebuild nearly instantaneous — fast enough that cargo didn't print the "Compiling supraseal-c2" progress line, which typically only appears for compilation units that take noticeable time.

Furthermore, cargo's incremental compilation fingerprinting tracks the build.rs script itself and the files in the directories listed by rerun-if-changed. Since the .cu file's modification time had changed, build.rs was re-executed. But cargo only prints "Compiling" for Rust source files, not for the output of build scripts. The assistant's assumption that "no 'Compiling supraseal-c2' line means no recompilation" was incorrect — a reasonable inference that happened to be wrong.

Assumptions, Mistakes, and Lessons

This episode reveals several assumptions that, while individually reasonable, collectively led to a dead end:

Assumption 1: cargo clean -p supraseal-c2 would remove all build artifacts for that package. This is true for Rust compilation artifacts but false for build.rs outputs, which are treated differently by cargo's clean system.

Assumption 2: The zsh glob supraseal-c2-* would match directories. In zsh, if no match is found, the shell reports an error rather than passing the literal glob string. The assistant was running in zsh (as evidenced by the error message) but expected bash-like behavior.

Assumption 3: The absence of a "Compiling" line in cargo output meant the package wasn't rebuilt. This conflates cargo's progress reporting with actual build execution. Build script execution is silent by default.

Assumption 4: The rerun-if-changed=cuda directive might not be working. In fact, it was working perfectly — the build script was re-executed, nvcc was invoked, and the new library was produced. The only thing missing was visible feedback.

The most significant mistake was trusting the absence of a signal (no "Compiling" line) over the presence of contradictory evidence (the file modification time of the .cu file had changed, and the build directories had been deleted). The assistant's debugging process was thorough — checking build.rs, checking file timestamps, checking directory contents — but the final inference was wrong until the direct artifact check in message 953.

Input and Output Knowledge

To fully understand this message, one needs knowledge of: the cargo build system's incremental compilation model, the distinction between Rust compilation and build.rs script execution, nvcc's caching behavior, the layout of cargo's target directory, and the specific architecture of the supraseal-c2 package where CUDA code is compiled via a build script rather than through Rust's standard compilation pipeline.

The message creates new knowledge: confirmation that the CUDA timing instrumentation is compiled into the daemon binary. This unlocks the next phase of the regression diagnosis — running the instrumented test to collect phase-level timing data. Without this confirmation, the assistant might have wasted time on alternative instrumentation approaches or incorrectly concluded that the build system was fundamentally broken.

The Deeper Significance

On the surface, message 953 is a trivial check — a single ls command confirming a file exists. But in the context of the broader performance engineering effort, it represents the resolution of a frustrating impedance mismatch between the developer's mental model of the build system and its actual behavior. The assistant had spent several messages fruitlessly trying to force a rebuild, deleting directories, checking fingerprints, and questioning the build.rs configuration. The real problem was never the build system — it was the lack of visible feedback from a build step that was working correctly but silently.

This is a common pattern in systems debugging: the most perplexing bugs often arise not from broken components, but from mismatches between expected and actual behavior of correctly functioning components. The CUDA code was being recompiled all along; the assistant just couldn't see it happening. Message 953 is the moment the veil lifts — the artifact is there, the timestamp is fresh, the size has changed. The build system was never broken. It was just quiet.

The lesson extends beyond this specific session: when debugging build issues, check the artifacts directly rather than relying on log output. File existence, modification times, and sizes are more reliable signals than the presence or absence of progress messages. And when a tool's behavior contradicts your expectations, consider that the tool might be working correctly but communicating differently than you assume.