The Final Audit: Verifying Git Tracking Completeness for the cuzk Proving Engine

Introduction

In the closing stages of a months-long engineering effort to build a high-performance, memory-efficient Groth16 proof generation pipeline for Filecoin's Curio storage proving system, one seemingly mundane task stood between the team and a finished pull request: ensuring that every file required for a clean git clone && make curio cuzk build was properly tracked in version control. Message [msg 3542] captures a single, focused moment in this upstreaming process — the assistant running a diff between what Git tracks and what exists on disk for the extern/bellpepper-core/ directory, following a similar check on extern/cuzk/. While the message appears brief and technical on the surface, it represents the culmination of a rigorous auditing methodology and reveals deep assumptions about reproducible builds, vendored dependencies, and the boundary between development artifacts and source code.

The Message in Full

The assistant writes:

Good — cuzk is fully tracked too. Now let me check bellpepper-core and supraseal-c2 in detail:

>

``bash cd /home/theuser/curio && diff <(git ls-tree -r HEAD --name-only extern/bellpepper-core/ | sort) <(find extern/bellpepper-core/ -type f | sed 's|^|./|;s|^\./||' | sort) ``

>

`` 0a1,9 > extern/bellpepper-core/benches/lc.rs > extern/bellpepper-core/.cargo-ok > extern/bellpepper-core/Cargo.toml > extern/bellpepper-core/Cargo.toml.orig > extern/bellpepper-core/.cargo_vcs_info.json > extern/bellpepper-core/LICENSE-APACHE > extern/bellpepper-core/LICENSE-MIT > extern/bellpepper-core/README.md > extern/bellpepper-core/src/constraint_system.rs 1a11 > extern/bellpepper-core/src/gadgets/mod.rs 3a14,16 > extern/bellpepper-core/src/lib.rs > extern/bellpepper-core/src/util_cs/mod.r... ``

The output is truncated at the end (the diff continues with additional untracked files), but the pattern is clear: the assistant is systematically comparing the set of files Git knows about against the set of files physically present on disk.

Context and Motivation

To understand why this message exists, one must appreciate the broader engineering context. The cuzk proving engine ([msg 3505] through [msg 3541]) is a complex system spanning multiple languages and build toolchains: Go for the Curio orchestrator, Rust for the proving daemon, C++ and CUDA for GPU kernels, and Protocol Buffers for gRPC communication. The team had chosen to vendor three forked Rust crates — bellperson, bellpepper-core, and supraseal-c2 — directly inside the Curio repository under extern/, rather than pushing branches to upstream repositories or waiting for crate releases. This "vendor-in-repo" approach (Option B, as the chunk summary notes) ensures a self-contained, reproducible build without external coordination, but it imposes a strict requirement: every file needed by cargo build must be present in the repository at commit time.

The user's instruction at [msg 3533] was explicit: "Add all cuzk code such that git clone -> make curio cuzk; builds both correctly from a fresh clone." This is a stronger requirement than simply having the code work on the developer's machine. It demands that a brand-new developer or storage provider, cloning the repository for the first time on a machine with no prior setup beyond the Rust toolchain and CUDA toolkit, can run a single make command and get a working binary. Any missing file — a Cargo.toml, a license file, a build.rs — would cause the build to fail, producing a frustrating "works on my machine" scenario.

The message at [msg 3542] is the third in a sequence of tracking audits. At [msg 3541], the assistant verified that extern/bellperson/ is fully tracked (70 tracked files matching 70 on-disk files) and that extern/cuzk/ is also fully tracked (35/35). Now the assistant turns to extern/bellpepper-core/ and, by implication, extern/supraseal-c2/, which were known from [msg 3536] to have untracked files.

The Methodology: A Systematic Diff

The assistant's choice of tool is telling. Rather than relying on git status — which would show untracked files but not distinguish between source files and build artifacts — the assistant constructs a precise diff between two sorted lists:

1. git ls-tree -r HEAD --name-only extern/bellpepper-core/ | sort — This queries Git's internal tree object for the committed state of the repository. It lists every file that Git knows about in the extern/bellpepper-core/ directory at the current HEAD commit, sorted alphabetically.
2. find extern/bellpepper-core/ -type f | sed 's|^|./|;s|^\./||' | sort — This lists every file on disk in the same directory, excluding directories (-type f), then normalizing the paths to match Git's format (removing the leading ./ that find produces), and sorting identically. The diff command then compares these two sorted lists line by line. The output format uses standard diff notation: lines prefixed with > appear only in the second list (on-disk files not tracked by Git). The 0a1,9 notation means "after line 0 of the first file, append lines 1-9 of the second file" — in other words, the first nine untracked files appear at the beginning of the on-disk listing. This approach is more rigorous than git status --porcelain because it catches a subtle class of errors: files that are tracked but have been deleted from disk, or files that are present on disk but excluded by .gitignore patterns. The assistant had already verified at [msg 3538] that the root .gitignore's **/*.a pattern does not accidentally block any source files (all .a files live inside target/ directories, which are separately gitignored). The diff-based audit is the final, definitive check.

What the Diff Reveals

The diff output for extern/bellpepper-core/ reveals a significant gap. The tracked files (from the original commit that only included the modified source files) are a subset of what's needed for a build. The untracked files include:

• Build configuration: Cargo.toml, Cargo.toml.orig — Without Cargo.toml, cargo build cannot resolve the crate name, version, dependencies, or metadata. Cargo.toml.orig is a backup created by cargo package and is harmless but not strictly necessary.
• Version control metadata: .cargo_vcs_info.json — Generated by cargo package to record the VCS commit from which the crate was published. Not needed for building, but harmless.
• Cargo registry markers: .cargo-ok — An empty marker file used by Cargo's registry extraction to indicate successful unpacking. Not needed for building from source (it's only used when vendoring from a registry cache), but including it doesn't hurt.
• Licenses: LICENSE-APACHE, LICENSE-MIT — The bellpepper-core crate is dual-licensed under Apache 2.0 and MIT. These files are legally required for distribution and should absolutely be tracked.
• Documentation: README.md — Not strictly required for building, but essential for anyone who clones the repository and wants to understand what the crate does.
• Source files: benches/lc.rs, src/constraint_system.rs, src/gadgets/mod.rs, src/lib.rs, src/util_cs/mod.rs — These are the actual Rust source files. Without src/lib.rs, the crate has no library entry point. Without src/constraint_system.rs, the core constraint system abstraction is missing. These are not optional. The critical observation is that the original commit had only tracked a subset of source files — presumably the ones that were modified relative to the upstream crate. But for a vendored build, the entire crate must be present. The Cargo.toml is the most essential missing piece: without it, Cargo doesn't even know this is a crate.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message, most of which are well-founded but worth examining:

Assumption 1: The diff is symmetric. The diff command only shows lines that differ. By running diff <(git ls-tree ...) <(find ...), the assistant only sees files on disk that aren't tracked. The reverse — files tracked but missing from disk — would also appear (as lines prefixed with <), but the output shows only > lines, confirming no tracked files have been deleted. This is a correct interpretation.

Assumption 2: All files in extern/bellpepper-core/ are source files. The find -type f command excludes directories but includes everything else. Could there be build artifacts or generated files in this directory that shouldn't be tracked? The assistant had already verified at [msg 3538] that no .a or .pc files exist outside target/ directories. The extern/bellpepper-core/ directory is a vendored crate, not a build workspace, so it should contain only source files, configuration, and metadata. This assumption is safe.

Assumption 3: The sed path normalization is correct. The sed 's|^|./|;s|^\./||' pipeline first prepends ./ to every path (so extern/bellpepper-core/Cargo.toml becomes ./extern/bellpepper-core/Cargo.toml), then removes the leading ./ again. This is a no-op on the surface, but it's actually a clever trick: find without a starting path prefix produces paths like ./extern/bellpepper-core/Cargo.toml only when the starting directory is .. By explicitly prepending and then stripping ./, the assistant ensures consistent formatting regardless of how find behaves on different systems. This is a portable approach.

Potential mistake: The diff doesn't check for .gitignore interference. The assistant had already checked this at [msg 3538], so the concern is addressed, but a reader of this message alone might wonder whether .gitignore patterns could be hiding files from git ls-tree (they don't — ls-tree reads the Git object database directly, ignoring .gitignore).

Potential mistake: The output is truncated. The message ends with extern/bellpepper-core/src/util_cs/mod.r..., suggesting the diff output was longer than what fits in the conversation window. The assistant doesn't comment on the truncation or confirm that the full list was reviewed. In practice, the assistant had already seen the full untracked list at [msg 3536] via git status, so this is a cross-check rather than the primary discovery.

Input Knowledge Required

To fully understand this message, a reader needs:

1. Git internals: Understanding that git ls-tree -r HEAD reads the committed tree object, not the working directory, and that --name-only produces a list of file paths suitable for comparison.
2. Cargo build system: Knowing that a Rust crate requires at minimum a Cargo.toml and a src/lib.rs (or src/main.rs) to build, and that vendored crates need their complete source trees.
3. The project structure: Awareness that extern/ is where vendored third-party code lives, that bellpepper-core is a forked Rust crate, and that the goal is a self-contained build from a fresh clone.
4. The preceding context: Understanding that the assistant has already verified bellperson and cuzk are fully tracked ([msg 3541]), checked .gitignore patterns ([msg 3538]), and confirmed Cargo.lock is tracked ([msg 3539]).

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. A definitive list of untracked files in extern/bellpepper-core/, which the assistant will need to git add before committing.
2. Confirmation that the diff methodology works for identifying tracking gaps, establishing a pattern that can be applied to extern/supraseal-c2/ (which the assistant mentions checking "in detail" but doesn't show the output for in this message).
3. Validation that no tracked files are missing from disk — the diff shows only > lines (files on disk not in Git), not < lines (files in Git not on disk), meaning nothing has been accidentally deleted.
4. A baseline for the commit: Once these files are staged, the assistant can proceed to write documentation, update SUMMARY.md, and create the final commit that the user requested at [msg 3533].

The Thinking Process

The assistant's reasoning in this message is methodical and defensive. The sequence of thought can be reconstructed as follows:

1. "Good — cuzk is fully tracked too." — This refers to the immediately preceding message ([msg 3541]) where the assistant verified that extern/cuzk/ has 35 tracked files matching 35 on-disk files. The word "too" connects back to the bellperson verification in the same message.
2. "Now let me check bellpepper-core and supraseal-c2 in detail" — The assistant knows from [msg 3536] that these directories have untracked files. The phrase "in detail" signals a shift from the summary-level git status check to a precise file-by-file comparison.
3. The command construction shows deliberate care: the diff is symmetric (both inputs sorted), the find output is normalized to match Git's path format, and the working directory is explicitly set with cd /home/theuser/curio && to avoid path mismatches.
4. The interpretation of the diff output is implicit — the assistant doesn't comment on each file, but the act of running the command and displaying the output demonstrates that the assistant is building a mental inventory of what needs to be staged. What's notable is what the assistant does not do: it doesn't immediately run git add on the untracked files. This is because the user's instruction at [msg 3533] also requires documentation, and the assistant is still in the auditing phase. The commit will come after all pieces are assembled.

Conclusion

Message [msg 3542] is a small but critical piece of a larger engineering workflow: the final verification before committing a complex, multi-language system to version control. The assistant's use of a diff between Git's tree object and the filesystem demonstrates a rigorous approach to ensuring build reproducibility. While the message appears to be a simple command-and-output pair, it encapsulates the discipline required to ship production infrastructure — checking assumptions, verifying completeness, and leaving no file behind. For anyone who has ever cloned a repository only to find that make fails with a missing Cargo.toml or a mysteriously absent source file, this message represents the preventative care that makes the difference between a smooth onboarding and a debugging session.