The Build That Revealed the Invisible Dependency: Verifying Phase 8's Dual-Worker GPU Interlock

The Message

[assistant] [bash] cd /home/theuser/curio/extern/cuzk && cargo build --release -p cuzk-daemon 2>&1 | tail -80
  --> /home/theuser/curio/extern/bellperson/src/gadgets/test/mod.rs:16:16
   |
16 |     Constraint(usize),
   |     ---------- ^^^^^
   |     |
   |     field in this variant
   |
   = note: `NamedObject` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
   = note: `#[warn(dead_code)]` on by default
help: consider changing the field to be of unit type to suppress this warning while preserving the field numbering, or remove the field
   |
16 -  ...

At first glance, this message appears unremarkable — a routine build command, its output truncated to the last 80 lines, showing only a pre-existing dead-code warning from an unrelated test module. But this message is a fulcrum point in the Phase 8 implementation of the cuzk SNARK proving engine. It represents the moment when nearly 200 lines of carefully crafted changes across seven files — spanning C++ CUDA kernels, Rust FFI bindings, bellperson's Groth16 prover, and the engine's worker dispatch logic — faced their first real test: the compiler.

Context: What Phase 8 Was Trying to Achieve

To understand why this build command matters, we must first understand what Phase 8 was designed to do. The cuzk project is a pipelined SNARK proving engine for Filecoin's Curio node, and its central challenge is maximizing GPU utilization during Groth16 proof generation. Phase 7 had revealed a critical bottleneck: a static std::mutex inside the C++ function generate_groth16_proofs_c (at line 133 of groth16_cuda.cu) that locked the entire function — including approximately 1.3 seconds of pure CPU preprocessing work — while only about 2.1 seconds of that 3.5-second window actually needed GPU exclusive access. This coarse locking meant that when a single GPU worker held the mutex, no other worker could dispatch CUDA kernels, creating idle gaps that reduced overall GPU efficiency to 64.3%.

Phase 8's insight was elegantly simple: narrow the mutex to cover only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs), allowing CPU preprocessing and the b_g2_msm computation to run outside the lock. With this narrowed scope, two GPU workers per device could interleave their work — one worker would run CPU preprocessing while the other held the mutex for CUDA kernel execution, and vice versa. The design, documented in c2-optimization-proposal-8.md, required coordinated changes across the entire call chain: the C++ kernel had to accept a passed-in mutex pointer, the FFI layer had to thread that pointer through supraseal-c2 into bellperson, and the Rust engine had to spawn multiple workers per GPU sharing a per-device mutex allocated via new create_gpu_mutex/destroy_gpu_mutex helpers.

What the Build Command Actually Tests

The message at index 2201 is Step 7 in the Phase 8 implementation plan: "Build and verify." The assistant had just completed six prior steps:

C++ mutex refactor in groth16_cuda.cu — removing the static mutex, adding a mutex pointer parameter, and narrowing the lock scope to the CUDA-only region.
FFI plumbing in supraseal-c2/src/lib.rs — adding the mutex pointer parameter to the extern declaration and wrapper function.
Rust bellperson changes — adding gpu_mutex parameter to prove_from_assignments and create_proof_batch_priority_inner.
Pipeline changes in pipeline.rs — adding GpuMutexPtr parameter to gpu_prove and updating all callers.
Engine changes in engine.rs — the largest change, spawning gpu_workers_per_device workers per GPU (default 2), creating per-GPU C++ mutexes, and passing them through the call chain.
Config changes in config.rs and cuzk.example.toml — adding the gpu_workers_per_device configuration field. Before this build command, the assistant had also forced a rebuild of the C++ code by deleting the cached build artifacts (rm -rf extern/cuzk/target/release/build/supraseal-c2-*). This was necessary because modifying a .cu file does not automatically trigger recompilation of the CUDA code — the build system needs to detect that the source has changed, and deleting the cached artifacts is the most reliable way to force that detection.

What the Output Reveals — and What It Hides

The output shown in the message is deliberately truncated to 80 lines via tail -80. What we see is a dead-code warning from bellperson/src/gadgets/test/mod.rs — a file completely unrelated to the Phase 8 changes. The warning notes that the Constraint(usize) variant has a field that is never read, and suggests changing it to a unit type. This is a pre-existing warning, not introduced by the Phase 8 modifications.

The absence of compilation errors in the visible output is deceptive. What we don't see — because it was scrolled off by the tail truncation — is the actual error that this build revealed. The very next message ([msg 2202]) shows the assistant discovering a critical issue: "The engine can't directly access supraseal_c2 — it needs to go through bellperson or pipeline." The engine's code was calling supraseal_c2::alloc_gpu_mutex() and supraseal_c2::free_gpu_mutex(), but supraseal_c2 was not a direct dependency of the cuzk-core crate. The dependency chain was cuzk-core → bellperson → supraseal-c2, not cuzk-core → supraseal-c2 directly. The Rust compiler, enforcing its strict module visibility rules, rejected the call.

This is a classic example of how a build serves not merely as a compilation step but as a discovery mechanism for architectural assumptions. The assistant had assumed that because supraseal_c2 was available in the dependency graph (through bellperson), it would be directly importable from the engine. But Rust's dependency model does not work that way — a crate can only use crates it directly declares in its Cargo.toml. The cuzk-core crate depended on bellperson, which depended on supraseal-c2, but that transitive relationship did not grant cuzk-core access to supraseal-c2's public API.

The Assumption Error and Its Correction

The mistaken assumption was subtle but significant. Throughout the Phase 7 implementation, the assistant had used supraseal_c2 directly in pipeline.rs (which is part of cuzk-core). Let me check — actually, looking at the grep results from [msg 2202], supraseal_c2 is only referenced in bellperson's files, not in cuzk-core's pipeline.rs. So the assistant was introducing a new dependency on supraseal_c2 from the engine module, which hadn't existed before.

The correction was to add wrapper functions in bellperson's supraseal module (src/groth16/prover/supraseal.rs) that expose alloc_gpu_mutex and free_gpu_mutex through bellperson's public API, and then re-export them from bellperson::groth16::mod.rs. The engine then calls bellperson::groth16::alloc_gpu_mutex() instead of supraseal_c2::alloc_gpu_mutex(). This respects the existing dependency hierarchy and keeps the FFI boundary properly encapsulated.

Why This Message Matters

This single build command encapsulates several important aspects of systems programming in a polyglot codebase:

The build as verification. In a project spanning C++, CUDA, Rust FFI, and async Rust orchestration, the compiler is the first and most unforgiving reviewer. The assistant could not simply reason about correctness in the abstract — it had to submit the code to the compiler and interpret the results. The build command is the moment of truth where design meets reality.

The information asymmetry of truncated output. The tail -80 is a practical necessity — cargo build --release on a project of this size produces thousands of lines of output. But it also creates an information hazard: the assistant sees only the last 80 lines, which in this case showed a harmless warning while the actual error scrolled off. The assistant had to infer the error from the context (the next message shows the diagnosis), but the reader of this message alone would not know what went wrong.

The invisible dependency graph. The error revealed by this build was not a type mismatch or a logic error — it was a visibility error. The Rust module system enforces strict boundaries about which crates can see which APIs, and these boundaries are not always obvious when designing cross-cutting changes. The assistant's assumption that supraseal_c2 would be accessible from the engine was reasonable given that the engine already used types from the supraseal pipeline, but it was wrong because Rust's use statements require direct dependency declarations.

The fragility of cross-layer changes. Phase 8's changes spanned C++ (the mutex refactor), C FFI (the alloc/free helpers), Rust FFI wrappers (supraseal-c2), the bellperson abstraction layer, and the engine orchestration. Each layer had its own visibility rules, type system, and compilation model. A change that is correct at one layer can fail at another for reasons that are invisible from the original layer's perspective. The build is where all these layers meet and must agree.

Input Knowledge Required

To fully understand this message, one needs to know:

The architecture of the cuzk proving engine: that it uses a C++ CUDA backend (groth16_cuda.cu) exposed through a Rust FFI layer (supraseal-c2), wrapped by bellperson's Groth16 prover, and orchestrated by the async engine in cuzk-core.
The Phase 8 design: that it narrows a static mutex to allow dual GPU workers per device, requiring a mutex pointer to be threaded through the entire call chain.
The dependency graph: cuzk-core → bellperson → supraseal-c2, with supraseal-c2 not being a direct dependency of cuzk-core.
The build process: that modifying .cu files requires forcing a rebuild by deleting cached artifacts, and that cargo build --release -p cuzk-daemon compiles only the daemon binary and its dependencies.
The Rust module system: that use statements require direct dependency declarations, and transitive dependencies are not automatically visible.

Output Knowledge Created

This message produces several forms of knowledge:

Build status: The Phase 8 changes compile through the C++ and FFI layers (no CUDA compilation errors), but the Rust engine code has a visibility error.
Warning baseline: The dead-code warning in bellperson's test module is pre-existing and unrelated to Phase 8.
Dependency gap: The engine cannot directly access supraseal_c2 — this is a concrete architectural constraint that must be addressed.
Next steps: The assistant must either add supraseal_c2 as a direct dependency of cuzk-core, or re-export the mutex functions through bellperson.

The Thinking Process

The assistant's reasoning is visible in the sequence of actions. After completing the code changes across all seven files, the assistant explicitly labels this as "Step 7: Build and verify." The decision to force a rebuild by deleting cached artifacts shows an understanding of the CUDA build system's quirks — without this step, the C++ changes might not be recompiled, leading to confusing linker errors or stale behavior.

The choice of tail -80 is pragmatic: the full build output for a project of this size can run to thousands of lines, and the assistant is looking specifically for errors and warnings. The 80-line window is a heuristic — enough to see any final error messages or the "Compilation successful" banner, but not so much that the output becomes unmanageable.

What's particularly interesting is what happens after this message. The assistant does not immediately notice the error from the truncated output. Instead, the next message ([msg 2202]) shows the assistant discovering the issue: "The engine can't directly access supraseal_c2 — it needs to go through bellperson or pipeline." This suggests that the assistant either re-ran the build with a wider view, or examined the full build log, or was already aware of the potential issue and verified it. The dead-code warning shown in the message was a red herring — the real error was elsewhere in the build output.

Conclusion

Message 2201 is a deceptively simple build command that serves as the verification gateway for a complex, multi-layer optimization. It reveals how the compiler acts as an architectural enforcer, catching assumptions about dependency visibility that escape human review. The truncated output shows a harmless warning while the actual error — a Rust module visibility violation — lurks unseen. This message is a microcosm of the entire Phase 8 implementation: a carefully designed change that must navigate the strict boundaries between C++, CUDA, FFI, and Rust to achieve its goal of eliminating GPU idle gaps and improving proof throughput by 13-17%.