The Vanishing Build Artifacts: A Case Study in CUDA Build System Debugging

Introduction

In the midst of a high-stakes performance regression diagnosis, a single bash command can reveal the gap between what a developer thinks the build system is doing and what it actually does. Message <msg id=947> captures one such moment in the cuzk project's Phase 4 optimization effort — a moment where the assistant, having just attempted to force a recompilation of CUDA source files, runs a find command to locate the compiled static libraries, only to discover that the build artifacts remain stubbornly unchanged. This article examines that message in depth, unpacking the reasoning, assumptions, and technical context that make this seemingly trivial command a pivotal moment in a larger debugging narrative.

The Message

The subject message is deceptively brief. The assistant executes:

find /home/theuser/curio/extern/cuzk/target -path "*supraseal*" -name "*.a" 2>/dev/null

And receives the output:

/home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-a0c4f246f264bbc3/out/libgroth16_cuda.a
/home/theuser/curio/extern/cuzk/target/release/build/supraseal-c2-bbf265913f050d8c/out/libgroth16_cuda.a

Two static library files, each containing the compiled CUDA kernels for the Groth16 prover. The presence of both files — with different hash suffixes (a0c4f246f264bbc3 vs bbf265913f050d8c) — tells a story of incremental builds, stale artifacts, and a build system that does not behave the way the assistant expected.

Why This Message Was Written

To understand why this find command was executed, we must trace the chain of reasoning that led to it. The context begins with a performance regression: the Phase 4 optimizations, intended to improve upon an 88.9-second baseline for a single 32 GiB PoRep proof, had instead caused a slowdown to over 100 seconds. The assistant had added CUDA timing instrumentation — CUZK_TIMING printf statements — to the GPU code to obtain precise phase-level breakdowns. The first test run ([msg 928]) produced overall timing (102.7 seconds total, with 61.7 seconds in synthesis and 40.8 seconds in GPU proving), but the detailed CUDA timing output was mysteriously absent from the log file.

The assistant diagnosed this as a C printf buffering issue ([msg 936]): when stdout is redirected to a file, the C runtime uses full buffering rather than line buffering, so the printf output was sitting in an unflushed buffer. The fix was to add fflush(stderr) after each timing print (<msg id=937-944>). But then came the build problem.

The assistant attempted to rebuild the CUDA code by running cargo clean -p supraseal-c2 ([msg 945]), which reported "Removed 0 files" — a clear signal that the package name didn't match what cargo expected for cleaning CUDA build artifacts. Then came an attempt to delete the build directories directly with rm -rf extern/cuzk/target/release/build/supraseal-c2-* ([msg 946]), which failed because zsh's globbing couldn't match the pattern (the asterisk wasn't quoted, and zsh's no_match behavior caused an error).

At this point, the assistant was in an uncertain state: had the build artifacts been deleted? Had the CUDA source files been recompiled? The find command in <msg id=947> was the natural next step — a reconnaissance mission to determine the actual state of the build directory before deciding how to proceed.

The Build System Challenge

The core issue revealed by this message is the complexity of hybrid Rust/CUDA build systems. The supraseal-c2 package uses a build.rs script to invoke nvcc (the NVIDIA CUDA compiler) during the Cargo build process. The compiled output — libgroth16_cuda.a — is placed in a hash-named directory under target/release/build/supraseal-c2-<hash>/out/. This directory structure is opaque to standard Cargo commands: cargo clean -p supraseal-c2 only cleans the Rust compilation artifacts, not the output of the build script. The CUDA .a files persist across clean commands, creating a disconnect between the developer's mental model and the build system's behavior.

The two .a files found by the find command tell an additional story. The first (a0c4f246f264bbc3) has a timestamp of February 17 at 16:22, while the second (bbf265913f050d8c) is from 23:33 on the same day (as revealed in <msg id=909>). The newer file is 1,112 bytes larger (2,791,224 vs 2,780,312), confirming that it includes the CUDA timing instrumentation. But the critical question — which one is the active build? — is not answered by the find command alone. The build system may be linking against either one, depending on which hash directory corresponds to the current build configuration.

Assumptions and Their Consequences

This message reveals several assumptions that the assistant made, some of which proved incorrect:

Assumption 1: cargo clean -p supraseal-c2 would remove CUDA build artifacts. This assumption stemmed from a mental model in which all compilation outputs are managed uniformly by Cargo. In reality, build.rs scripts have full control over their output directory, and Cargo's clean command only removes files it knows about. The "Removed 0 files" message was the first hint that this assumption was wrong.

Assumption 2: The zsh glob supraseal-c2-* would match the build directories. This failed because zsh's default behavior for unmatched globs is to print an error, not to pass the literal string to the command. The assistant learned this the hard way when the rm -rf command produced "no matches found."

Assumption 3: The CUDA source files would be recompiled automatically when the .cu files changed. This assumption is partially correct — Cargo's build script dependency tracking should detect changes to source files listed in build.rs. However, the assistant's edit to add fflush(stderr) calls (<msg id=937-944>) modified the .cu file, and the subsequent build ([msg 945]) did not trigger recompilation. This suggests either that the build script's dependency tracking is incomplete, or that the cargo invocation didn't actually reach the supraseal-c2 package (perhaps because the daemon binary was already up-to-date from Cargo's perspective, even though the CUDA archive was stale).

Knowledge Required and Created

To fully understand this message, the reader needs knowledge of: Cargo's build system architecture, particularly the distinction between Rust compilation and build.rs script outputs; the CUDA compilation workflow using nvcc and static library archives; the behavior of Unix stdio buffering (full buffering vs line buffering when stdout is a file); and the zsh shell's globbing semantics for unmatched patterns.

The message creates new knowledge about the state of the build directory: both .a files still exist, meaning the attempted clean operations were ineffective. The CUDA code has not been recompiled with the fflush fix. The assistant must find an alternative approach — perhaps touching the CUDA source files to force a rebuild, or manually deleting the build directories, or using cargo build with a specific flag to force recompilation of the supraseal-c2 package.

The Thinking Process

The assistant's reasoning in this message is visible through the sequence of actions across the preceding messages. The thinking process follows a classic debugging loop:

1. Observe symptom: CUDA timing output is missing from the log file.
2. Form hypothesis: The printf output is buffered and not flushed.
3. Test hypothesis: Check if the CUDA code is compiled into the binary (confirmed in <msg id=911-913>).
4. Implement fix: Add fflush(stderr) after each printf call.
5. Attempt rebuild: Use cargo clean -p supraseal-c2 to force recompilation.
6. Observe failure: "Removed 0 files" — the clean didn't work.
7. Attempt alternative: Delete build directories directly with rm -rf.
8. Observe second failure: zsh globbing error.
9. Reconnaissance: Run find to determine the actual state. The find command in <msg id=947> is the reconnaissance step — a pause in the action to gather information before deciding the next move. It reveals that both .a files still exist, confirming that neither the cargo clean nor the rm -rf achieved their intended effect. The assistant must now choose a new approach: perhaps using touch on the CUDA source files to update their timestamps, or manually deleting the build directories with an explicit path, or modifying the build.rs script to force recompilation.

Significance in the Larger Narrative

This message, though brief, represents a critical inflection point in the Phase 4 regression diagnosis. The assistant has identified the likely culprit (B1: cudaHostRegister pinning overhead, as confirmed in the subsequent chunk analysis) and has added instrumentation to verify it, but is now stuck on a build system obstacle. The inability to force CUDA recompilation delays the collection of timing data, which in turn delays the decision about which optimizations to keep and which to revert.

The broader lesson is about the hidden complexity of modern build pipelines. In a project that spans Go, Rust, C++, and CUDA — with build scripts that invoke compilers, archive tools, and linkers across multiple languages — the simple act of adding a printf and recompiling can become a multi-step ordeal. The developer must understand not just their own code, but the entire chain of tools that transforms source files into binaries.

Conclusion

Message <msg id=947> is a testament to the importance of understanding one's build system. A single find command, executed in frustration after two failed attempts to force recompilation, reveals the gap between intention and reality in a hybrid Rust/CUDA project. The two static library files, sitting undisturbed in their hash-named directories, represent a build system that operates on its own terms — indifferent to cargo clean and resistant to shell globs. For the assistant, the path forward requires a deeper understanding of how build.rs manages its outputs, and a more targeted approach to forcing recompilation. For the reader, this message offers a window into the meticulous, methodical work of performance engineering — where even a simple printf can become a journey through buffering semantics, build system architecture, and shell behavior.