The Commit That Made cuzk a Full Filecoin Proving Engine

Introduction

In the development of complex distributed systems, a single commit message can encapsulate weeks of research, dozens of architectural decisions, and hundreds of lines of carefully crafted code. Message 339 in this coding session is exactly such a moment: the commit that transforms the cuzk proving daemon from a single-purpose PoRep (Proof-of-Replication) accelerator into a full-fledged Filecoin proving engine capable of handling all four proof types required by storage providers. This article examines that commit in depth—the reasoning behind it, the research that enabled it, the decisions it embodies, and the knowledge it both consumed and produced.

The Message

The subject message is a git commit command and its accompanying commit message:

[bash] cd /home/theuser/curio && git add extern/cuzk/cuzk-bench/src/main.rs extern/cuzk/cuzk-core/src/engine.rs extern/cuzk/cuzk-core/src/prover.rs extern/cuzk/cuzk-core/src/types.rs extern/cuzk/cuzk-proto/proto/cuzk/v1/proving.proto extern/cuzk/cuzk-server/src/service.rs && git commit -m "feat(cuzk): Phase 1 — all proof types, multi-GPU workers, priority scheduling

Wire up WinningPoSt, WindowPoSt, and SnapDeals provers via filecoin-proofs-api:
- prove_winning_post: generate_winning_post_with_vanilla
- prove_window_post: generate_single_window_post_with_vanilla (per-partition)
- prove_snap_deals: generate_empty_sector_update_proof_with_vanilla

Multi-GPU worker pool:
- Auto-detect GPUs via nvidia-smi or use config gpus.devices list
- Spawn one async worker loop per GPU with CUDA_VISIBLE_DEVICES isolation
- Per-worker SRS affinity tracking (last_circuit_id for future routing)

Proto/API updates:
- Add repeated bytes vanilla_proofs field for PoSt/SnapDeals multi-proof inputs
- Rename SnapDeals fields to comm_r_old/comm_r_new/comm_d_new (raw 32-byte)
- Registered proof type enum conversion (FFI V1_1 ↔ proofs-api V1_2 mapping)

Bench tool updated:
- Supports all proof types with --vanilla (JSON array of base64 proofs)
- New flags: --registered-proof, --randomness, --comm-r-old/new, --comm-d-new

8 unit tests pass, 0 warnings, clean cargo check --no-default-features."
[feat/cuzk d8aa4f1d] feat(cuzk): Phase 1 — all proof types, multi-GPU workers, priority scheduling
 6 files changed, 778 insertions(+), 245 deletions(-)

On its surface, this is a routine git operation. But the commit message tells a story of deep systems research, careful API mapping across language boundaries, and thoughtful architectural evolution. Let us unpack what this commit represents and why it was written.

Context: The cuzk Proving Daemon

To understand this commit, one must understand what cuzk is and why it exists. The cuzk project is a pipelined SNARK proving daemon designed for Filecoin storage providers. Filecoin's proof system requires storage providers to periodically generate cryptographic proofs—Groth16 zk-SNARKs—to demonstrate they are storing data correctly. These proofs are computationally expensive, requiring ~200 GiB of peak memory and significant GPU compute time for a single PoRep (Proof-of-Replication) proof.

The earlier segments of this session (Segments 0-5) had established the foundation. The team had conducted an exhaustive investigation of the entire C2 proof generation pipeline, identifying 18 micro-optimizations, designing five optimization proposals, and ultimately architecting the cuzk proving daemon. Phase 0 had implemented a working PoRep-only daemon with SRS (Structured Reference String) residency, achieving a 20.5% speedup by keeping the proving parameters loaded in GPU memory across proof generations.

But PoRep is only one of four proof types that Filecoin storage providers must generate. The others are:

  1. WinningPoSt — Generated when a storage provider wins a block election, proving they have the sectors they claimed
  2. WindowPoSt — Generated periodically to prove sectors are still being stored (one per partition)
  3. SnapDeals — Generated when replacing data in a sector with new data (sector update)
  4. PoRep C2 — The original proof-of-replication (already implemented in Phase 0) Phase 1's mission was to expand cuzk from a PoRep-only tool to a universal proving engine supporting all four proof types. This commit marks the completion of that mission.

Why This Message Was Written: The Culmination of Deep Research

The commit message is not merely a summary of code changes; it is the artifact of an intensive research process visible in the preceding messages (301-338). The assistant did not simply "know" how to implement these proof types. Each one required tracing through multiple layers of abstraction:

Tracing the FFI Enum Mapping

One of the most critical discoveries was the mapping between the Go-side RegisteredPoStProof enum values and the Rust filecoin-proofs-api enum values. The Go FFI layer uses a #[repr(i32)] C-compatible enum where variants are auto-incremented from zero: StackedDrgWinning2KiBV1 = 0, StackedDrgWinning8MiBV1 = 1, and so on up to StackedDrgWindow64GiBV1_1 = 14. However, the filecoin-proofs-api crate uses a different enum with V1_2 variants (e.g., StackedDrgWinning32GiBV1_2). The Go/FFI layer maps its V1_1 variants to the API's V1_2 variants.

This mapping was not documented anywhere obvious. The assistant had to:

Understanding Multi-Proof Serialization

Another key insight was that PoSt proofs require multiple vanilla proofs per request — one per sector being proven. The existing PoRep implementation used a single vanilla_proof: bytes field, but PoSt and SnapDeals need vanilla_proofs: repeated bytes. This distinction was discovered by examining the filecoin-proofs-api function signatures:

SnapDeals Commitment Fields

For SnapDeals (sector updates), the proof requires three commitment CIDs: comm_r_old (previous replica commitment), comm_r_new (new replica commitment), and comm_d_new (new data commitment). The assistant discovered that the existing protobuf had named these fields differently and had to rename them to match the API's expectations. The commit message notes "Rename SnapDeals fields to comm_r_old/comm_r_new/comm_d_new (raw 32-byte)" — a seemingly small change that required understanding the Filecoin sector update protocol.

Architectural Decisions Embodied in This Commit

Multi-GPU Worker Pool Design

The most significant architectural change in this commit is the multi-GPU worker pool. Phase 0 had a single GPU worker. Phase 1 needed to support multiple GPUs, as production storage providers typically have 4-8 GPUs per machine.

The design decisions visible in the commit message are:

  1. Auto-detection vs. configuration: GPUs are detected either by running nvidia-smi or from a configuration list (config gpus.devices). This provides flexibility for both automated and manually configured deployments.
  2. CUDA_VISIBLE_DEVICES isolation: Each worker process gets its own CUDA_VISIBLE_DEVICES environment variable, restricting it to a single GPU. This is a well-known pattern for GPU isolation in multi-GPU systems, preventing accidental cross-GPU memory access.
  3. Per-worker async loops: Each GPU gets its own async worker loop, allowing independent operation. If one GPU is busy, others can still accept work.
  4. Per-worker SRS affinity tracking: The commit mentions "Per-worker SRS affinity tracking (last_circuit_id for future routing)." This is a forward-looking design: by tracking which circuit each worker last loaded into its GROTH_PARAM_MEMORY_CACHE, the scheduler can preferentially route similar proof types to the same GPU, avoiding SRS reload overhead. However, this is noted as "for future routing" — the actual affinity-based scheduling was deferred to Phase 2, as the current process-global cache makes per-GPU tracking unnecessary for Phase 1.

Priority Scheduling Retention

The commit title includes "priority scheduling," but this is actually a retention of the Phase 0 design rather than a new addition. The shared priority queue using BinaryHeap was already implemented. The decision to keep it unchanged was deliberate: the assistant had reviewed the project plan and determined that the current design was correct for Phase 1, with more sophisticated scheduling deferred to Phase 2.

The Bench Tool as a Testing Vehicle

The commit also updates the bench tool to support all proof types with new flags (--vanilla, --registered-proof, --randomness, --comm-r-old/new, --comm-d-new). This is not merely a convenience feature; it is essential for testing. Without the bench tool, there would be no way to verify that the new proof types work end-to-end without integrating with the full Curio stack. The bench tool serves as both a development debugging aid and a production benchmarking utility.

Input Knowledge Required

To understand and produce this commit, the assistant needed knowledge spanning multiple domains:

  1. Filecoin proof types: Understanding the differences between PoRep, WinningPoSt, WindowPoSt, and SnapDeals, and when each is used in the Filecoin protocol.
  2. Groth16 zk-SNARKs: Understanding the proving pipeline, including the role of vanilla proofs (witness generation) and C2 proofs (final SNARK compression), and the SRS/CRS parameter loading.
  3. Rust async patterns: The engine uses tokio::sync::Mutex, watch channels, oneshot channels, and async worker loops. Understanding these patterns is essential for the multi-GPU design.
  4. gRPC and protobuf: The service definition uses protobuf with tonic (Rust gRPC framework). The commit modifies the .proto file and the generated Rust types.
  5. GPU isolation techniques: The CUDA_VISIBLE_DEVICES pattern for multi-GPU isolation is a specific piece of systems knowledge.
  6. Cross-language FFI mapping: The most subtle knowledge required was understanding how Go's abi.RegisteredPoStProof enum values map through CGO to Rust's #[repr(i32)] FFI enum, and then to the filecoin-proofs-api enum. This required reading code in three languages.
  7. The existing codebase: The assistant had to understand the Phase 0 architecture (engine, scheduler, prover, service) to extend it correctly.

Output Knowledge Created

This commit produces several forms of knowledge:

  1. Working code: Six files modified, 778 lines added, 245 removed. The code compiles cleanly and passes 8 unit tests.
  2. A documented API: The protobuf definition now explicitly supports all proof types, serving as a contract between the daemon and its clients.
  3. A tested pattern for enum conversion: The manual match statements for converting between FFI enum values and API enum values are documented in the prover module, serving as a reference for future proof type additions.
  4. A benchmarkable system: The updated bench tool allows anyone to test and measure the performance of any proof type, creating empirical data for future optimization.
  5. A committed reference point: The git commit d8aa4f1d serves as a milestone, allowing the team to reference this exact state of the codebase for debugging, deployment, or rollback.

Assumptions and Potential Issues

The commit makes several assumptions worth examining:

  1. nvidia-smi availability: The GPU auto-detection relies on nvidia-smi being installed and in PATH. This is reasonable for production GPU servers but may fail in containerized environments without the NVIDIA toolkit.
  2. CUDA_VISIBLE_DEVICES correctness: The assumption that setting CUDA_VISIBLE_DEVICES to a single GPU index provides proper isolation is well-founded but depends on the NVIDIA driver version and configuration.
  3. Process-global SRS cache: The deferral of GPU affinity-based scheduling to Phase 2 assumes that the process-global GROTH_PARAM_MEMORY_CACHE makes per-GPU tracking unnecessary. This is correct for the current implementation but may need revisiting if the cache is later made per-GPU.
  4. Enum value stability: The manual enum conversion assumes that the numeric values of the FFI #[repr(i32)] enum will not change between versions. This is a reasonable assumption for a stable API but should be verified during upgrades.

The Thinking Process Visible in the Preceding Messages

The messages leading up to this commit (301-338) reveal a systematic, research-driven approach. The assistant:

  1. Read the source code of filecoin-proofs-api to understand function signatures (messages 301-302)
  2. Searched for type definitions using grep to find EmptySectorUpdateProof and understand its structure (messages 303-307)
  3. Traced the enum mapping across Go, C, and Rust layers (messages 308-317)
  4. Implemented incrementally: first the prover functions, then the engine, then the service, then the bench tool (messages 318-326)
  5. Verified compilation at each step (messages 333-337)
  6. Added tests for the enum conversion functions (message 335)
  7. Committed only after all tests passed and the build was clean (message 339) This is not a "code and pray" approach. Each implementation decision is grounded in evidence from reading the actual API code, not assumptions about how it should work.

Conclusion

Message 339 is a commit that marks a significant milestone in the cuzk project. It transforms a single-purpose PoRep accelerator into a universal Filecoin proving engine, capable of handling all four proof types across multiple GPUs with priority-aware scheduling. But more than that, it is the artifact of a rigorous research process—one that traced enum values across three programming languages, decoded protobuf serialization patterns from API signatures, and made careful architectural decisions about GPU isolation and scheduler design.

For someone reading this commit message in isolation, it appears as a straightforward summary of code changes. But when examined in the context of the preceding research, it reveals the depth of systems thinking required to build production-grade cryptographic proving infrastructure. The commit message's bullet points are not just a changelog; they are the crystallization of knowledge gained through tracing, reading, and understanding the entire Filecoin proof generation pipeline from Curio's Go task layer through Rust FFI into C++/CUDA kernels.