From Enum to Engine: The Architecture of Phase 1 in the cuzk Proving Engine

Introduction

In the span of a single chunk of development activity—34 messages, six files modified, 778 lines of code changed—the cuzk proving engine transformed from a single-purpose PoRep daemon into a multi-proof, multi-GPU proving system capable of handling all four Filecoin proof types. But this transformation was not merely a matter of writing code. It was the culmination of a systematic, research-driven engineering process that began with tracing FFI enum discriminants across three programming languages and ended with an architectural decision to defer a planned feature because the underlying infrastructure didn't yet support it.

This article synthesizes the work of that chunk, examining how the assistant navigated the boundary between research and implementation, how architectural decisions were made and validated, and how the project transitioned from Phase 1 delivery to Phase 2 planning. The story of this chunk is the story of how a distributed proving system was built—not through heroic coding sprints, but through careful investigation, deliberate decision-making, and a disciplined commitment to building the right thing at the right time.

The Enum at the Boundary: Where Integration Begins

Every complex integration project has a Rosetta Stone—a single artifact that, once understood, unlocks the entire system. For the cuzk proving engine's Phase 1 expansion, that artifact was the #[repr(i32)] enum RegisteredUpdateProof in the filecoin-ffi Rust layer [1]. This enum, with its five variants mapping sector sizes to integer discriminants (0 for 2KiB, 1 for 8MiB, 2 for 512MiB, 3 for 32GiB, 4 for 64GiB), was the key to understanding how SnapDeals proof requests would arrive over gRPC and how they should be dispatched to the filecoin-proofs-api proving functions.

The assistant's investigation of this enum (see [msg 317]) was the final piece of a larger research puzzle that spanned the entire chunk. Earlier messages had mapped the RegisteredPoStProof enum for WinningPoSt and WindowPoSt, tracing the V1_1 to V1_2 mapping that accounts for the grindability fix in the WindowPoSt circuit. The RegisteredUpdateProof investigation completed the picture, giving the assistant the complete mapping from the numeric proof_type values arriving over gRPC to the correct Rust enum variants for all four proof categories.

What makes this investigation noteworthy is not just what was discovered, but how it was discovered. The assistant did not rely on documentation or assumptions. It read the actual source files—the Go constants in Curio's FFI layer, the C #[repr(i32)] enum definitions, and the Rust filecoin-proofs-api type signatures—tracing the numeric values across all three language boundaries. This research-first approach ensured that the proving functions would be called with the correct enum values, preventing a class of bugs that would have been extremely difficult to diagnose (an incorrect enum discriminant produces an invalid proof, not a crash).

The Moment of Synthesis: From Research to Implementation

With the enum mappings fully understood, the assistant moved from research to implementation in a concentrated burst of activity spanning messages 318 through 326. The implementation touched every layer of the system:

The protobuf definition was extended with repeated bytes vanilla_proofs to support the multi-proof nature of PoSt and SnapDeals requests [5]. Unlike PoRep C2, which requires a single vanilla proof, WinningPoSt and WindowPoSt require one vanilla proof per challenged sector, and SnapDeals requires partition proofs. The protobuf schema had to accommodate this variability, and the field naming was updated to match the API conventions (comm_r_old, comm_r_new, comm_d_new for SnapDeals commitments).

The prover functions in cuzk-core/src/prover.rs were rewritten with real implementations for prove_winning_post(), prove_window_post(), and prove_snap_deals(). Each function called the corresponding filecoin-proofs-api function with the correct parameters, including the manual enum conversion from the gRPC numeric values to the Rust enum types. The WindowPoSt implementation included the critical V1_1 to V1_2 mapping, ensuring that the grindability fix was applied correctly.

The engine in cuzk-core/src/engine.rs was refactored to support a multi-GPU worker pool [13]. This was the architectural centerpiece of Phase 1: instead of a single worker loop, the engine now spawns N async workers, one per GPU, with each worker setting CUDA_VISIBLE_DEVICES to its assigned GPU ordinal before proving. The running field became a map from worker_id to (JobId, ProofKind), enabling per-worker state tracking and laying the foundation for future GPU affinity scheduling.

The service layer in cuzk-server/src/service.rs was updated to convert the new protobuf fields into the core types, and the GPU status reporting was extended to show per-worker state with running job information [6].

The bench tool in cuzk-bench/src/main.rs gained full support for all four proof types, with new flags for --vanilla, --registered-proof, --randomness, and the SnapDeals commitment fields [10]. The tool was refactored into a ProofParams struct for clean batch, single, and concurrent modes.

The Multi-GPU Architecture: Design Decisions and Trade-offs

The multi-GPU worker pool was the most consequential architectural decision of Phase 1. The assistant's design, articulated in the engine rewrite at [msg 329], made several deliberate choices:

One worker per GPU: Rather than a dynamic pool where workers claim GPUs as needed, the engine spawns one worker per GPU at startup and maintains that mapping for the daemon's lifetime. This avoids the overhead of CUDA context initialization per proof and aligns with the SRS residency strategy—keeping parameters loaded in GPU memory across proof requests requires dedicated GPU ownership.

Process-level GPU isolation via CUDA_VISIBLE_DEVICES: Each worker sets this environment variable before making any CUDA calls, ensuring it can only see its assigned GPU. This is a stronger isolation mechanism than CUDA streams or contexts within a single process—a crash in one worker doesn't affect others, and resource accounting is simpler. The trade-off is higher memory overhead (each worker loads its own CUDA runtime) and more complex inter-worker communication.

Shared priority queue with BinaryHeap: All workers compete for jobs from a single priority queue, with the scheduler using a BinaryHeap to maintain priority ordering (CRITICAL > HIGH > NORMAL). This is a classic work-stealing pattern that maximizes throughput—all workers stay busy as long as there are N+ jobs pending—and ensures that the highest-priority job is always served next regardless of which worker picks it up.

The shared queue design was later validated in the post-commit review (see [msg 347]), where the assistant considered whether to implement GPU affinity-based scheduling and concluded it should be deferred. The reasoning was precise: the GROTH_PARAM_MEMORY_CACHE is a process-global lazy_static HashMap, meaning the SRS lives in host memory, not in per-GPU VRAM. All workers benefit from the same cache regardless of which GPU they control. True GPU affinity—routing jobs based on which GPU has the SRS in VRAM—requires the custom SRS manager planned for Phase 2. In Phase 1, the shared queue is not just simpler; it is architecturally correct [31].

The Moment of Verification: Eight Tests and a Clean Compile

The implementation phase culminated in a validation checkpoint at [msg 337], where the assistant confirmed that all 8 unit tests passed and the workspace compiled cleanly with zero warnings [21]. This was not a single event but the end of a deliberate verification sequence:

  1. cargo check --workspace --no-default-features at [msg 333] confirmed that the entire workspace compiled without errors or warnings [17].
  2. cargo test --workspace at [msg 334] ran the existing test suite, which passed 5 tests [18].
  3. New enum conversion tests were added at [msg 335] to validate the critical FFI-to-proofs-api mapping logic [19].
  4. Final test run at [msg 336] confirmed all 8 tests passed, including the new tests [20]. The commit at d8aa4f1d (documented at [msg 341]) marked the formal delivery of Phase 1 [25]. The commit message summarized the changes across six files: prover implementations for all four proof types, multi-GPU worker pool with priority scheduling, protobuf API updates with repeated bytes vanilla_proofs, gRPC service updates for the new fields, and bench tool support for all proof types.

The Pause Between Phases: Plan Review and Architectural Reflection

What distinguishes this chunk from a simple implementation sprint is what happened after the commit. Rather than immediately moving to the next feature, the assistant paused to review the project plan (cuzk-project.md) and assess remaining Phase 1 deliverables [27][28]. This post-commit review revealed several important insights:

GPU affinity scheduling was deferred to Phase 2: The project plan listed this as a Phase 1 deliverable, but the assistant's analysis showed it was premature. The process-global SRS cache means there is no per-GPU state to optimize for. The shared priority queue with BinaryHeap is the correct design for Phase 1, and GPU affinity should wait for the custom SRS manager in Phase 2 [31].

The gen-vanilla command was identified as the next critical deliverable: Without a way to generate vanilla proofs for PoSt and SnapDeals, the new proof types could not be tested end-to-end. The assistant recognized that testability was the bottleneck, not additional features, and pivoted to building the gen-vanilla command [32][33].

Research into vanilla proof generation APIs was completed: The assistant launched a structured research task to discover the exact function signatures for generate_winning_post_sector_challenge, generate_fallback_sector_challenges, generate_single_vanilla_proof, and generate_partition_proofs, along with the type constructors for PrivateReplicaInfo, SectorId, and ProverId [32]. This research set the stage for the next implementation push.

The Empty Message: Trust and Autonomy in Collaborative Development

One of the most remarkable artifacts in this chunk is the empty user message at [msg 350] [34]. After the user's instruction at [msg 342] to "read through the -commit-.md docs to understand plans," the assistant spent messages 343 through 349 executing that instruction—reading the project plan, evaluating architectural decisions, researching APIs. Then came an empty message: just the conversation_data wrapper with no content.

This empty message functioned as a continuation signal—a nod from the user saying "I'm still here, proceed." The assistant interpreted it correctly, producing a comprehensive synthesis document at [msg 351] that consolidated everything learned across the entire project. The empty message works because of the dense web of shared context and trust that had been built over the course of the session. It is a testament to the depth of the collaborative relationship that silence can be so communicative.

Conclusion: The Architecture of a Phase

The cuzk Phase 1 chunk is a case study in systematic engineering. It demonstrates a pattern that recurs throughout the project: research first, implement second, validate third, reflect fourth. The assistant did not guess at enum values—it traced them across three language boundaries. It did not assume GPU affinity was needed—it evaluated the actual SRS caching model and deferred the feature. It did not declare Phase 1 complete and move on—it paused to review the plan, assess remaining deliverables, and set the agenda for what comes next.

This pattern is not accidental. It is the architecture of a disciplined engineering process, applied to a complex distributed system where the cost of a mistake is high (an invalid proof can lose Filecoin mining rewards) and the cost of premature optimization is wasted effort. The assistant's systematic approach—read the source, understand the types, verify the mappings, implement with confidence, validate with tests, reflect on the architecture—is a model for how to build reliable systems on top of existing, multi-language codebases.

The result is a proving engine that can handle all four Filecoin proof types across multiple GPUs with priority scheduling, committed as d8aa4f1d. And the path forward is clear: build the gen-vanilla command, test the new proof types end-to-end, and then, in Phase 2, implement the custom SRS manager that will make GPU affinity scheduling meaningful. The architecture is honest—it builds what the system needs now, not what the plan says it might need later. That is the mark of engineering maturity.