Building the cuzk SNARK Proving Engine: Phase 0 from Blueprint to Validated Pipeline

Introduction

In the span of a single sustained engineering session spanning dozens of messages, hundreds of commands, and thousands of lines of code, the cuzk pipelined SNARK proving engine was born. What began as a design document—an architectural response to the nine structural bottlenecks identified in Filecoin's SUPRASEAL_C2 Groth16 proof generation pipeline—transformed into a running, end-to-end validated distributed system. A six-crate Rust workspace was created from scratch. A full gRPC protobuf API was defined. A priority scheduler was implemented. Real filecoin-proofs-api calls were wired in. And on the first attempt, a 51 MB proof request flowed through the entire pipeline, was dispatched to the proving library, and returned a cleanly propagated error back to the client.

This article traces the full arc of that transformation. It is a story of disciplined engineering: of synthesis before construction, of incremental validation at every layer, of systematic debugging when the build system fought back, and of the messy but necessary collision between clean architecture and real-world infrastructure. The Phase 0 construction of the cuzk proving engine is a case study in how to build a complex distributed system from nothing—and how to prove that it works before all the dependencies are in place.

The Blueprint: From Bottleneck Analysis to Unified Architecture

The cuzk project did not begin with a mkdir command. It began with an exhaustive investigation of the Filecoin SUPRASEAL_C2 proof generation pipeline—a system responsible for producing the cryptographic proofs that underpin Filecoin's storage verification protocol. That investigation, spanning multiple prior segments of the conversation, had mapped the full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, accounting for the pipeline's ~200 GiB peak memory footprint and identifying nine structural bottlenecks. These included the per-proof loading of 32 GiB SRS parameters, the lack of pipelining between CPU synthesis and GPU proving, and the absence of a persistent daemon to amortize initialization costs across multiple proof requests.

The response to these bottlenecks was a series of five optimization proposals, each documented in exhaustive detail. Proposal 1 addressed micro-optimizations across GPU kernels and CPU synthesis. Proposal 2 exploited known constraint shapes (SHA-256 dominance, boolean witnesses) for precomputation. Proposal 3 designed a pre-compiled constraint evaluator with specialized MatVec operations. Proposal 4 introduced pre-computed split MSM topology. And Proposal 5 synthesized all of these into a total impact assessment with an implementation roadmap.

But proposals alone do not ship software. The team recognized that these optimizations needed to be woven into a coherent, production-ready system. That system was named cuzk—a pipelined SNARK proving daemon designed from the ground up to address the identified bottlenecks. The name itself encoded the mission: a CUDA-accelerated zk-SNARK proving kernel that could run persistently, accept requests over gRPC, and manage its own memory and scheduling.

The synthesis point arrived in a single dense assistant response that wove the seven prior documents into a unified architectural blueprint. This message defined the cuzk architecture for the first time: a gRPC server that accepts proof requests, a priority scheduler that orders them, a tiered SRS memory manager that keeps parameters resident across proofs, and a phased 18-week implementation roadmap. It established the key constraint that would shape all subsequent work: "Phase 0 requires ZERO upstream library modifications"—a masterstroke of pragmatic engineering that ensured the first working version could be built without waiting for changes to bellperson, filecoin-proofs, or supraseal-c2.

The blueprint was comprehensive. It specified the crate architecture (six crates with clear responsibilities), the gRPC API (eight RPCs covering proof submission, daemon management, metrics, and SRS lifecycle), the scheduler design (priority queue with configurable worker pools), and the prover interface (wrapping filecoin-proofs-api calls). It even anticipated the monitoring infrastructure, specifying Prometheus metrics for proofs completed, proofs failed, pinned memory, and scheduler queue depth.

The Scaffold Takes Shape

With the blueprint in hand, the assistant executed a final reconnaissance before building. It verified that protoc was available (v33.1), that cargo and rustc were installed (v1.82.0), and that the extern/ directory contained the existing filecoin-ffi, supra_seal, and supraseal crates but no cuzk directory yet. It launched a detailed exploration task to read extern/supra_seal/c2/Cargo.toml and understand the dependency structure for linking against the Filecoin proving stack. With these final pieces of information in hand, the assistant declared that the implementation was ready to begin.

The first tangible act of creation was a single mkdir -p command creating seven empty directories. This unassuming command encoded the entire crate architecture: cuzk-proto for protobuf definitions, cuzk-core for the engine and scheduler, cuzk-server for the gRPC service layer, cuzk-daemon for the binary entry point, cuzk-bench for the benchmarking client, and cuzk-ffi for future FFI bindings. The scaffolding moment was the moment when possibility became reality, when design became implementation.

From there, the construction proceeded in a textbook bottom-up order. The workspace Cargo.toml was written first, establishing the member crates and shared dependencies. Then came the protobuf definitions—a full gRPC API with eight RPCs covering proof submission, daemon management, metrics, and SRS preload/evict. The build.rs script for tonic code generation was configured, and the individual Cargo.toml files for each member crate were created.

Then came the moment of creation: a single burst of six source files that formed the core of the proving engine. The config.rs defined the daemon's TOML-based configuration. The types.rs established the shared vocabulary—JobId, ProofKind, Priority, ProofRequest. The scheduler.rs implemented a priority queue using Rust's BinaryHeap. The prover.rs defined the interface for invoking proof generation. And the engine.rs tied everything together, exposing the public API that the gRPC service layer would consume. These six files were the bones of the cuzk proving engine, and every subsequent message depended on them.

The Build System Struggle

The workspace compiled on the first attempt—but only after a grueling debugging session that revealed the hidden complexity of building against a deep dependency tree. The first cargo check triggered a download of 329 packages and then failed with a cryptic error. The root cause: a transitive dependency (blake2b_simd v1.0.4) required Rust edition 2024, which was not supported by the active toolchain (Rust 1.82.0).

The assistant's diagnostic process was methodical. It checked which Rust toolchains were installed, discovered that none were new enough, and then looked to the sibling project filecoin-ffi for guidance. The rust-toolchain.toml in that project specified channel = "1.86.0". The assistant created a matching rust-toolchain.toml for the cuzk workspace, pinning the build to Rust 1.86.0. This single file—a one-line configuration change—saved the entire project from a dead end.

But the toolchain fix was only the beginning. The subsequent compilation attempts revealed a cascade of issues: a version incompatibility with the home crate, missing dependencies in cuzk-bench, a missing base64 dependency in cuzk-core, and an inconsistent acronym in the protobuf file where PreloadSRSRequest conflicted with PreloadSrsRequest. Each issue was identified, diagnosed, and fixed in turn.

The assistant's approach to this debugging process reveals a key engineering philosophy: never let a warning stand. The assistant systematically eliminated every compiler warning—unused imports, unnecessary qualifications—until the workspace compiled with zero warnings. This discipline was not pedantry; it was risk management. Every warning was a potential bug in disguise, and the assistant was determined to start from a clean baseline.

One particularly instructive episode was the PreloadSRSRequest vs PreloadSrsRequest inconsistency. The protobuf file defined a message called PreloadSrsRequest (lowercase 'rs') but the Rust code referenced PreloadSRSRequest (uppercase 'RS'). This inconsistency would have caused a compilation error that would have been difficult to diagnose because the error message would reference the generated Rust type name, not the protobuf field name. The assistant caught this during a cargo check run and fixed it by making the protobuf definition consistent with the Rust code. This is the kind of bug that can easily slip through review in a handwritten protobuf file, and catching it early saved hours of debugging later.## Wiring Real Proving

With the workspace compiling cleanly, the assistant turned to the critical task of wiring the prover module to the real filecoin-proofs-api calls. This required tracing the serialization chain from Curio's Go code through the FFI layer to the Rust side—understanding how the C1 JSON output from seal_commit_phase1 was structured, how it encoded the SealCommitPhase1Output as base64-wrapped bincode, and how seal_commit_phase2 consumed it.

The assistant explored the API surface of filecoin-proofs-api, read the source code of seal_commit_phase2, verified the ProverId construction, and finally replaced the stub prover with real calls to the Filecoin proving library. A single cargo check confirmed that the wiring was correct—the workspace compiled with the real proving calls integrated.

The assistant then ran the five unit tests in the workspace, all of which passed. It verified that both binaries (cuzk-daemon and cuzk-bench) produced correct help output. The todo list was systematically updated, marking each milestone as completed. The scaffold was solid.

But the assistant was not satisfied with compilation and unit tests alone. The true test of a distributed system is whether the components can communicate with each other over the network. The next step was to start the daemon, connect the bench client, and submit a real proof request.

The Moment of Truth: End-to-End Validation

The pivot from construction to validation came when the assistant declared: "Both binaries run and produce correct help output. Now let me do an end-to-end test—start the daemon on TCP and submit a proof via bench."

But before submitting a 51 MB proof, the assistant first validated the communication layer with the simplest possible test: the Status RPC. The daemon was started, and cuzk-bench status returned a clean response:

=== cuzk daemon status ===
uptime:           9s
proofs completed: 0
proofs failed:    0
pinned memory:    0 / 0 bytes

This was the first handshake—the moment when the client and server, built from scratch over the preceding hours, finally spoke to each other. The gRPC pipeline was alive.

Then came the critical test. The assistant checked for the 32 GiB Groth16 parameters and found them missing. The parameter cache was empty. A full proof execution was impossible. But the assistant made a strategic decision: run the test anyway. The goal was not to produce a valid proof but to validate the pipeline—the serialization, the scheduler dispatch, the prover invocation, and the error propagation path.

The test was executed. The daemon started, the 51 MB C1 JSON was transmitted over gRPC, the server deserialized it, the scheduler dispatched it, the prover called seal_commit_phase2, and the error from the missing parameters propagated cleanly back to the client. The Prometheus metrics recorded proofs failed: 1. Every layer of the pipeline worked except the final proof generation, which failed only because of a well-understood environmental dependency.

The Debugging Spiral: When Shell Environment Fights Back

The end-to-end test was not without its complications. The assistant encountered a series of shell environment issues that threatened to derail the validation effort. A stale daemon process was still bound to port 9820, producing an "Address already in use" error when the assistant tried to start a fresh instance. Yet the test still ran—against the old daemon.

The assistant's analysis of this accidental test is a masterclass in separating signal from noise. Despite the port conflict, the assistant identified six discrete steps that had been successfully validated: the bench tool connected via gRPC, loaded the 51 MB C1 JSON file, sent it over gRPC within the 128 MiB message limit, the daemon deserialized the C1 wrapper and decoded the base64, dispatched to the GPU worker, and called seal_commit_phase2. The proof failed because the 32 GiB parameters were missing, but the error propagated cleanly back to the client.

This accidental validation was valuable, but the assistant recognized that it was not definitive. The old daemon might have had residual state, cached connections, or partial data from previous runs. A clean test was needed. The assistant orchestrated a carefully designed test sequence: a fresh daemon on a fresh port (9822), output redirected to a log file, and a structured sequence of commands—status, submit, post-proof status, metrics, log tail—designed to produce a complete picture of the system's behavior.

What followed was an unexpected debugging spiral. Despite careful redirection, the daemon log file was never created—or was created and immediately consumed by the shell environment. The assistant tried multiple approaches: using nohup, writing a shell script, checking process state via /proc, and running the daemon in a subshell. Each approach failed in a different way.

This debugging spiral reveals a critical lesson about distributed systems development: the testing infrastructure—shell scripts, process management, log redirection—is itself a system that must be debugged before it can be used to validate the target system. The assistant's persistence through these tooling difficulties, without ever questioning whether the cuzk code itself was correct, demonstrates a clear separation in the mental model between "the system I built" and "the environment I'm testing it in."

The breakthrough came when the assistant abandoned the script approach and ran the daemon directly in the foreground with &, then verified it was listening with ss -tlnp. This produced visible daemon startup logs confirming the engine was running on port 9827. The gRPC status endpoint confirmed the daemon was operational: proofs completed: 0, proofs failed: 0. The stage was set for the actual proof submission.

A Successful Failure

The proof submission returned the result that the assistant had been working toward for over a dozen messages:

=== Proof Result ===
status:    FAILED
job_id:    083c6109-b2fe-4033-8ce1-01a2952f5b26
error:     seal_commit_phase2 failed

This was a successful failure. The assistant had deliberately designed the test knowing that the 32 GiB Groth16 parameters were not present on the machine. The goal was never to produce a valid proof—it was to validate the entire communication pipeline: that the client could load a 51 MB JSON file, serialize it into a gRPC message, transmit it over TCP, have the daemon deserialize it, decode the base64-encoded C1 wrapper, parse the SealCommitPhase1Output JSON, dispatch the job through the priority scheduler to a GPU worker, invoke seal_commit_phase2, catch the inevitable failure, wrap it in a proper error response, and transmit it back to the client.

Every single one of those steps succeeded. The error propagated cleanly. The client displayed the result with proper status and error fields. The daemon's Prometheus metrics incremented cuzk_proofs_failed_total to 1. The monitoring infrastructure was alive.

The assistant enumerated seven discrete steps that were successfully validated: loading the 51 MB C1 JSON, sending it over gRPC within the message size limit, deserializing the C1 wrapper and decoding base64, dispatching to the GPU worker via the priority scheduler, calling seal_commit_phase2, catching the failure and wrapping it in a proper error response, and returning the result to the bench client. The metrics check—proofs failed: 1—confirmed that the observability infrastructure was accurate.

Resolving the Parameter Dependency

With the software pipeline validated, the focus shifted to the environmental dependency that had caused the expected failure: the 32 GiB Groth16 parameters. The user ran curio fetch-params 32GiB with FIL_PROOFS_PARAMETER_CACHE=/data/zk/params, but a path resolution bug in the tool caused all files to be downloaded to ~/scrot//data/zk/params/—a path that prepended the current working directory to the absolute path.

The output of this command is a chronicle of cascading failure. The download itself succeeded—files were written to the wrong path—but the tool's sanity check then tried to open them at the intended path (/data/zk/params/), which didn't exist or was empty. The tool then attempted to remove and retry each file, but the removal also failed because the files weren't at the expected path. The result was a massive aggregated error message listing dozens of failed file removals, each with the same "no such file or directory" error.

The assistant's diagnostic work across subsequent messages is a textbook example of data-driven debugging. First, it checked /data/zk/params/ and found it empty. Then it checked the default Filecoin parameters location at /var/tmp/filecoin-proof-parameters/ and found only 4 small .params files, none matching the 8-8-2 pattern that identifies 32 GiB sector parameters. Finally, it discovered the files in ~/scrot/data/zk/params/ and confirmed their contents.

The critical discovery: the downloaded files included the 45 GiB v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-...params file—the main PoRep circuit SRS. The assistant noted that these were the 8-8-0 variants (not 8-8-2), which are the correct parameters for PoRep 32G V1.1. The assistant then copied the files to the correct location using cp -n, verified that 29 files were now present, and confirmed the three largest .params files (45 GiB, 33 GiB, and 57 GiB) were in place.

The Path Resolution Bug: A Deeper Analysis

The curio fetch-params path resolution bug deserves closer examination because it illustrates a class of infrastructure failures that are surprisingly common in complex toolchains. The tool was given an absolute target path (/data/zk/params/) via the FIL_PROOFS_PARAMETER_CACHE environment variable. However, the download logic appears to have constructed the output path by concatenating the current working directory with the parameter cache path, producing ~/scrot//data/zk/params/—note the double slash, a telltale sign of string concatenation gone wrong.

This bug has several interesting properties. First, it only manifests when the tool is run from a non-root working directory. If the user had been in / when running the command, the concatenation would have produced ///data/zk/params/ which, while ugly, would have resolved to the same absolute path. Second, the bug is invisible in the tool's INFO-level log messages, which report the intended path (/data/zk/params/) rather than the actual download path. Only the [NOTICE] messages from the download backend reveal the true path. Third, the tool's error handling is self-defeating: the sanity check opens files at the intended path, fails, then tries to remove them at that path (which also fails), and presumably retries the download to the same wrong path. This creates an infinite loop of download → verify-fail → remove-fail → retry.

The assistant's ability to diagnose this bug from the log output alone—without access to the tool's source code—demonstrates a valuable skill in systems engineering: reading error messages for structural patterns rather than just surface content. The double slash in the download paths, combined with the mismatch between download paths and sanity check paths, immediately identified the root cause.

The Knowledge Created by Phase 0

This phase of work produced several forms of lasting knowledge. First and most importantly, it established that the Phase 0 gRPC pipeline is functional end-to-end. The 51 MB payload can be serialized, sent over gRPC, deserialized, and dispatched to the proving logic. Error handling works correctly—when seal_commit_phase2 fails, the error propagates cleanly through the scheduler, the gRPC service, and back to the client. Prometheus metrics are accurate: proofs failed: 1 correctly tracks the outcome. The daemon survives a failed proof without crashing, deadlocking, or entering an inconsistent state.

Second, the phase documented the complete parameter dependency graph for 32 GiB PoRep proving. The log output from curio fetch-params lists every VK and params file required, including the specific IPFS CIDs for each file. The three largest .params files—45 GiB for PoRep, 33 GiB for empty-sector-update, and 57 GiB for PoSt—confirm the ~200 GiB peak memory footprint identified in earlier analysis.

Third, the phase identified a path resolution bug in curio fetch-params that can cause parameter downloads to fail when run from a non-root working directory. The diagnostic pattern—double slashes in download paths combined with sanity check failures at the intended path—provides a template for identifying similar issues in other tools.

Fourth, the phase established a validation methodology that can be reused for future integration testing. The pattern of "submit real input, observe real failure, verify metrics" is a template for testing distributed systems where environmental dependencies are not yet in place.

Conclusion: The Architecture of Systematic Construction

The Phase 0 implementation of the cuzk proving engine is a case study in how to build a complex distributed system from scratch. The assistant's approach reveals several principles that generalize beyond this specific project:

Synthesis before construction. The assistant did not begin coding until the architecture was fully understood and documented. The seven prior documents, the cuzk-project.md design, and the final reconnaissance all preceded the first mkdir command. This investment in understanding paid dividends when build issues arose—the assistant could distinguish between fundamental design problems and surface-level configuration issues.

Incremental validation at every layer. The assistant never tested a complex operation through an unverified channel. The build was validated crate-by-crate. The gRPC channel was validated with the simplest possible RPC before the proof submission was attempted. The proof submission was attempted even though it was expected to fail, because the failure itself validated the pipeline. Each layer of the system was proven before the next layer was tested.

Systematic debugging. When the build failed, the assistant did not guess at fixes. It diagnosed the root cause (Rust edition mismatch), gathered environmental data (installed toolchains, sibling project's configuration), applied the fix (rust-toolchain.toml), and verified the result. This pattern repeated for every issue: observe, diagnose, gather data, fix, verify.

Attention to detail. The assistant's insistence on zero warnings, its catch of the PreloadSRSRequest vs PreloadSrsRequest inconsistency, and its methodical cleanup of unused imports all reflect a philosophy that small inconsistencies matter. In a system with 329 transitive dependencies and six interdependent crates, discipline at the micro scale prevents chaos at the macro scale.

Expectation-driven testing. The assistant's decision to run the proof test knowing it would fail was a deliberate choice to validate the pipeline under realistic conditions. The failure was expected; the question was whether it would fail in the right way. This approach—designing tests to validate specific properties of the system rather than to produce a pass/fail outcome—is a hallmark of mature engineering practice.

The cuzk Phase 0 scaffold is now complete. It is a working skeleton—a daemon that can accept proof requests, schedule them with priority, dispatch them to the real Filecoin proving library, and return results. The immediate next steps are to generate a fully successful proof using the now-resolved parameter dependencies, then to extend the system with multi-proof-type support, SRS management, and batching in subsequent phases. But the foundation is solid, and the architecture is proven. The pipelined SNARK proving engine is no longer a design document—it is a running system.

References

[1] "From Blueprint to Working Pipeline: The Phase 0 Construction of the cuzk SNARK Proving Engine" — chunk article covering messages 95-162

[2] "From Pipeline Validation to Environmental Reality: The Cuzk Phase 0 Chunk" — chunk article covering messages 163-191