The Preparatory Read: How Six Parallel File Reads Launched Phase 8 of the cuzk SNARK Proving Engine

Introduction

In any complex software engineering effort, the moment before the first line of new code is written often contains as much insight as the implementation itself. This is especially true when the change involves threading a synchronization primitive through multiple abstraction layers spanning two programming languages, a foreign function interface (FFI), and a distributed proving pipeline. Message [msg 2147] captures precisely that moment: the assistant, having just finished reading the Phase 8 design specification, issues six parallel read tool calls to examine every file that will need modification. On its surface, the message is mundane — a batch of file reads. But beneath that simplicity lies a rich story about architectural understanding, multi-layer synchronization design, and the disciplined engineering process that characterizes high-performance systems work.

The Context: Why Phase 8 Existed

To understand the significance of this message, one must first understand what Phase 8 aimed to solve. The cuzk SNARK proving engine, purpose-built for Filecoin's Proof-of-Replication (PoRep) protocol, had been through seven previous optimization phases. By Phase 7, the engine had achieved per-partition dispatch — each partition of a Groth16 proof could be dispatched independently to the GPU, allowing overlap between CPU synthesis and GPU compute. However, benchmarking revealed a persistent problem: GPU utilization gaps.

The root cause was traced to a C++ static mutex inside generate_groth16_proofs_c, the CUDA kernel entry point in the supraseal-c2 library. This mutex guarded the entire proof-generation function, meaning that when one GPU worker held the lock to execute CUDA kernels (NTT, MSM, batch additions), no other worker could even begin its CPU-side preprocessing — the very work that should have overlapped with GPU execution. The result was a structural idle pattern: the GPU would finish its work and sit idle while the next worker's CPU preprocessing ran under the lock. Phase 8's design, documented in the spec the assistant had just read in [msg 2146], proposed a surgical fix: narrow the mutex to cover only the CUDA kernel region itself, allowing CPU preprocessing (and the b_g2_msm computation) to proceed in parallel outside the lock. With two GPU workers per device, one could run CUDA kernels while the other prepared data — a classic producer-consumer interlock pattern applied at the GPU-work granularity.

What the Message Actually Does

Message [msg 2147] is, at its core, an information-gathering operation. The assistant issues six read tool calls in parallel, each targeting a file that will need modification:

  1. groth16_cuda.cu — The C++ CUDA kernel file containing the static mutex that is the target of the refactor. This is the heart of the change: the mutex must be converted from a static global to a passed-in pointer, and its lock scope must be narrowed to cover only the CUDA kernel operations (NTT+MSM, batch additions, tail MSMs).
  2. lib.rs — The Rust FFI bindings in supraseal-c2 that expose the C++ functions to Rust. This file must be updated to accept a mutex pointer parameter in the extern declaration and wrapper function.
  3. supraseal.rs (in bellperson) — The Bellperson prover implementation that calls into supraseal-c2. This is the Rust-side bridge that must thread the mutex from the engine through to the FFI call.
  4. pipeline.rs — The cuzk pipeline that orchestrates proof generation. This file must be updated to pass the mutex through the proving pipeline.
  5. config.rs — The configuration structs. A new gpu_workers_per_device field must be added to allow the operator to control how many GPU workers share each device's mutex.
  6. engine.rs — The central engine coordinator that spawns GPU workers and manages SRS. This is where the per-GPU mutex must be allocated and the workers spawned with shared access to it. The parallelism of these reads is itself significant. The assistant does not read files sequentially — it reads them all at once, treating the codebase as a system to be understood holistically rather than a linear document. This reflects a deep understanding of the architecture: the change touches every layer from CUDA to configuration, and the assistant needs to see all layers simultaneously to plan the implementation coherently.

The Architecture Revealed

The set of files being read reveals the full call chain of the cuzk proving engine, from top to bottom:

Layer 1: Configuration (config.rs) — The outermost layer, where operators specify parameters like partition count, concurrency, and now GPU worker count. This is the user-facing interface to the engine's behavior.

Layer 2: Engine (engine.rs) — The central coordinator that owns the scheduler, GPU workers, and SRS manager. This is where threads are spawned and resources are allocated. For Phase 8, this layer must create per-GPU mutexes and spawn multiple workers per device.

Layer 3: Pipeline (pipeline.rs) — The orchestration layer that manages the flow of proofs through synthesis and GPU compute. The pipeline must be updated to carry the mutex reference through to the proving calls.

Layer 4: Bellperson Prover (supraseal.rs) — The Rust-side prover implementation that bridges between cuzk's pipeline and supraseal-c2's FFI. This layer must accept a mutex parameter and pass it through to the C++ call.

Layer 5: FFI Bindings (lib.rs) — The Rust extern declarations that define the C ABI boundary. A mutex pointer parameter must be added to the function signature, and the Rust wrapper must handle the raw pointer safely.

Layer 6: CUDA Kernel (groth16_cuda.cu) — The C++ implementation that runs on the GPU. The static mutex must be removed, replaced by a passed-in pointer, and the lock scope narrowed to the CUDA kernel region only.

This six-layer architecture is typical of high-performance proving systems: a Rust application (cuzk) calls into a Rust prover library (bellperson), which calls into a Rust FFI wrapper (supraseal-c2's lib.rs), which calls into a C++ library (supraseal-c2's CUDA code), which launches GPU kernels. Each layer adds its own concerns — safety, ergonomics, performance — and threading a new parameter through all of them requires understanding each layer's conventions and constraints.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of background knowledge:

Groth16 Proof Structure: The message assumes familiarity with Groth16, the zero-knowledge proof system used by Filecoin. In particular, the distinction between the a, b, and c components of a proof matters: b_g2_msm (the G2 multi-scalar multiplication for the b component) is computed on the CPU and can be moved outside the mutex, while NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) for a and c run on the GPU.

CUDA Kernel Execution Model: The message assumes understanding that GPU kernel launches are asynchronous from the CPU's perspective — the CPU submits work to the GPU and can continue processing while the GPU computes. This is precisely why CPU preprocessing can overlap with GPU execution.

Static Mutex Contention: The core insight that a static mutex (a single global lock) becomes a bottleneck when multiple threads compete for GPU access. The message assumes the reader understands that narrowing the lock scope allows true parallelism.

FFI Mechanics: The message involves threading a C++ std::mutex pointer through Rust FFI. This requires understanding that Rust can hold a raw pointer to a C++ mutex and pass it across the FFI boundary, but must manage safety carefully (no lifetime guarantees, no automatic destruction).

The cuzk Architecture: The message assumes familiarity with the cuzk proving engine's architecture, including its partition-based proof generation, its SRS (Structured Reference String) management, and its worker-thread model.

Output Knowledge Created

By reading these six files, the assistant creates a comprehensive mental model of the current state of each layer. This knowledge is immediately actionable: the assistant can now plan the exact changes needed, knowing the existing function signatures, type definitions, and synchronization patterns.

The reads also reveal specific implementation details that will shape the Phase 8 changes:

Assumptions and Decisions

Several assumptions underpin this message and the Phase 8 design it serves:

Assumption: The static mutex is the sole cause of GPU idle gaps. The Phase 7 benchmarks had identified mutex contention as the primary bottleneck, but the assistant implicitly assumes that narrowing the mutex will fully eliminate GPU idle time. This assumption proved correct — Phase 8 benchmarks later showed 100% GPU efficiency for single-proof runs.

Assumption: Two workers per device is the right default. The design spec proposed gpu_workers_per_device = 2 as the default, based on the reasoning that two workers can perfectly interleave (one pre-processing while the other runs CUDA kernels). This assumption was validated by the benchmarks, but the configurable parameter allows adjustment if the optimal number differs.

Assumption: The mutex pointer can be safely passed through FFI. Passing a C++ std::mutex* through Rust FFI requires careful lifetime management. The assistant assumes that allocating the mutex on the C++ side (via create_gpu_mutex/destroy_gpu_mutex helpers) and passing the raw pointer through Rust is safe, provided the Rust side never dereferences the pointer — it only passes it back to C++ functions.

Decision: Read all files in parallel. Rather than reading files one by one, the assistant reads all six simultaneously. This decision reflects an understanding that the files form an interconnected system and that understanding any one file requires understanding its context in the larger architecture.

Decision: Start with the current state, not the design spec. The assistant had already read the Phase 8 design spec in [msg 2146]. The decision to then read the actual source files shows a disciplined approach: the design spec describes the what and why, but the source files reveal the how — the exact function signatures, type definitions, and code patterns that the implementation must work with.

The Thinking Process

The reasoning visible in this message is primarily structural. The assistant does not explain its reasoning explicitly — there is no "thinking" block or commentary — but the choice of which files to read reveals the reasoning process:

  1. Identify the core change target: The static mutex in groth16_cuda.cu is the root cause, so that file must be read first.
  2. Trace the call chain upward: The mutex must be passed from the engine down to the CUDA kernel. Each layer in the chain must be read to understand its current interface.
  3. Identify configuration impact: The number of GPU workers per device is a tunable parameter, so the config file must be read to understand the existing configuration pattern.
  4. Identify the engine's worker model: The engine spawns workers and manages resources, so it must be read to understand how to allocate per-GPU mutexes and spawn multiple workers. The parallelism of the reads suggests that the assistant is building a holistic mental model rather than a linear one. It wants to see all the pieces simultaneously to understand how they fit together.

Mistakes and Potential Pitfalls

While the message itself contains no errors — it is simply reading files — the approach carries inherent risks:

Risk of incomplete understanding: Reading files in parallel means the assistant processes all of them at once, potentially missing subtle interactions between layers. For example, the Bellperson prover's closure-based callback pattern might interact unexpectedly with the mutex threading.

Risk of FFI safety issues: Passing a C++ mutex pointer through Rust FFI is inherently unsafe. If the Rust side accidentally dereferences the pointer or if the C++ side destroys the mutex while Rust still holds the pointer, undefined behavior results.

Risk of over-narrowing: If the mutex scope is narrowed too aggressively (e.g., excluding operations that must be serialized), correctness could be compromised. The Phase 8 design carefully identified which operations must remain under the lock (CUDA kernel launches) and which can be moved outside (CPU preprocessing, b_g2_msm), but this analysis depends on deep understanding of the GPU execution model.

Conclusion

Message [msg 2147] is a study in disciplined engineering practice. Before writing a single line of new code, the assistant reads every file that will be touched, building a comprehensive understanding of the current state. The six parallel reads reveal the full six-layer architecture of the cuzk proving engine, from TOML configuration to CUDA kernels. They demonstrate an understanding that threading a synchronization primitive through multiple abstraction layers requires seeing all layers simultaneously.

In the broader narrative of the cuzk optimization project, this message is the turning point between design and implementation. The Phase 8 design spec ([msg 2146]) described what to build and why. Message [msg 2147] begins the how — by understanding the current state deeply enough to plan the exact changes needed. The subsequent messages in the session would go on to implement those changes across all six files, achieving 13-17% throughput improvement and 100% GPU efficiency. But none of that would have been possible without the foundational understanding built in this single, seemingly simple message of parallel file reads.