The Critical Bridge: How One Research Message Unlocked Filecoin Proof Serialization for the cuzk Proving Engine

"Now let me understand the existing test data and what's available. Let me also check Curio's Go FFI layer to see how it serializes vanilla proofs for PoSt."

This single message, dispatched at a pivotal moment in the development of the cuzk distributed proving daemon, represents far more than a routine research task. It is the architectural keystone that bridges two fundamentally different worlds: the Go-based Curio mining infrastructure and the Rust/CUDA-powered proof generation pipeline. Without the knowledge produced by this message, the entire Phase 1 implementation of cuzk—supporting all four Filecoin proof types across multiple GPUs—would have been built on guesswork and broken serialization formats. This article examines the reasoning, context, assumptions, and knowledge flows embedded in this deceptively simple research dispatch.

The Message in Full

The message, as recorded in the conversation, reads:

[assistant] Now let me understand the existing test data and what's available. Let me also check Curio's Go FFI layer to see how it serializes vanilla proofs for PoSt. [task] {"description":"Check Curio PoSt/Snap data flow","prompt":"I need to understand how Curio's Go code sends vanilla proofs to the FFI layer for each proof type. This will tell us the serialization format cuzk should expect over gRPC.\n\nRead these files:\n1. tasks/window/compute_do.go - Look for how... task_id: ses_393a091cdffeCT4NFOW8Y8zlpo

>

<task_result> Now I have all the pieces. Here is a comprehensive summary. --- ## Vanilla Proof Serialization for Each Proof Type in Curio ### 1. Window PoSt (WindowPoStWithVanilla) Vanilla proof generation: - File: /home/theuser/curio/tasks/window/compute_do.go (lines 463-568) - File: /home/theuser/curio/lib/paths/local.go (lines 1151-1252) Each sector's vanilla proof is generated individually vi...

The message is structured as a task dispatch: the assistant declares an intent to investigate, then launches an automated research task with a specific prompt, and receives back a structured result. This pattern—declare, dispatch, receive—is the operational heartbeat of the entire cuzk development effort.

The Strategic Context: Why This Message Exists

To understand why this message was written, we must understand the predicament the assistant faced at message index 288. The cuzk project had already completed Phase 0, which implemented a PoRep-only proving daemon with gRPC API, priority scheduling, and SRS residency caching. Phase 1 demanded expanding the engine to support all four Filecoin proof types: WinningPoSt, WindowPoSt, SnapDeals, and the existing PoRep. The assistant had already accomplished two critical research tasks:

  1. Codebase reconnaissance ([msg 286]): Every source file in the extern/cuzk/ workspace had been read and catalogued, revealing the existing protobuf definitions, engine architecture, scheduler design, and prover stubs.
  2. API signature discovery ([msg 287]): The filecoin-proofs-api 19.0.0 crate had been exhaustively explored, yielding the exact Rust function signatures for generate_winning_post_with_vanilla, generate_single_window_post_with_vanilla, and generate_empty_sector_update_proof_with_vanilla, along with their parameter types and the critical #[repr(i32)] enum mapping for RegisteredSealProof. But a gap remained. The Rust API signatures told the assistant what functions to call and what types they expected, but not how the data would arrive over the gRPC wire. The cuzk daemon receives proof requests from Curio—a Go application—through a protobuf-defined gRPC API. The Go side serializes vanilla proofs in a particular format before sending them. If the Rust side deserialized them incorrectly, the proofs would fail silently or, worse, produce invalid Groth16 proofs that waste GPU time and return garbage. This is the precise motivation for message 288: the serialization contract between Go and Rust was unknown, and without it, the entire Phase 1 implementation would be built on sand.## The Research Target: Curio's Go FFI Layer The assistant's prompt specifies reading files in Curio's Go source tree, particularly tasks/window/compute_do.go and lib/paths/local.go. These are not arbitrary files—they are the precise locations where Curio, the Filecoin mining orchestration layer, constructs and serializes proof requests before handing them to the FFI boundary. The choice of targets reveals a sophisticated understanding of the system architecture: - tasks/window/compute_do.go contains the Window PoSt computation logic, where each sector's vanilla proof is generated individually. Understanding how Curio iterates over sectors and serializes their proofs into the FFI call tells the assistant whether the gRPC message should carry a single proof or a repeated list. - lib/paths/local.go handles local storage paths and proof caching. The assistant needed to understand whether Curio passes raw proof bytes or wraps them in some envelope format. The prompt also reveals the assistant's core question: "This will tell us the serialization format cuzk should expect over gRPC." This framing is critical—it shows the assistant is thinking about the system end-to-end, from Curio's Go code through the gRPC boundary into the Rust daemon and finally into the filecoin-proofs-api C FFI calls.

The Knowledge That Was Produced

The task result delivered exactly what the assistant needed. The comprehensive summary revealed several critical details that directly shaped the Phase 1 implementation:

Multi-proof semantics for PoSt: Window PoSt and Winning PoSt both require multiple vanilla proofs per request—one per sector being proven. This meant the protobuf ProveRequest message needed a repeated bytes vanilla_proofs field rather than a single bytes vanilla_proof. The assistant had already defined vanilla_proof: Vec&lt;u8&gt; in the Phase 0 types; this research confirmed it needed to become vanilla_proofs: Vec&lt;Vec&lt;u8&gt;&gt;.

SnapDeals commitment structure: SnapDeals (sector updates) require three commitment CIDs: the old sector's commitment, the new sector's commitment, and the replica commitment. The protobuf definition needed three distinct bytes fields, not a single commitment. The assistant had already anticipated this in [msg 286] but the Curio research confirmed the exact field names and ordering.

The registered_proof enum mapping: Curio's Go code maps proof types to numeric values through a C FFI enum. The assistant had discovered in [msg 287] that RegisteredSealProof uses #[repr(i32)] with specific numeric values. The Curio research confirmed which numeric values correspond to which proof types in actual production use, eliminating any guesswork.

Assumptions and Their Validity

The message rests on several assumptions, most of which proved correct:

Assumption 1: Curio's serialization format is the authoritative specification. The assistant assumes that how Curio serializes proofs for its own FFI calls is exactly how cuzk should expect to receive them over gRPC. This is a reasonable assumption—Curio is the production Filecoin mining software, and cuzk is designed to replace its proving backend. If cuzk's gRPC contract diverges from Curio's internal format, either Curio would need to be modified (a non-goal) or cuzk would need a translation layer. The research confirmed that the formats align, validating this assumption.

Assumption 2: The Go source code is accessible and readable. The assistant assumes it can navigate Curio's Go codebase to find the relevant serialization logic. This was validated—the task successfully located and parsed the files.

Assumption 3: Vanilla proof serialization is consistent across proof types. The assistant implicitly assumes that WinningPoSt, WindowPoSt, and SnapDeals all serialize their vanilla proofs in a similar fashion, differing only in the number of proofs and additional metadata fields. The research confirmed this: all three use the same underlying proof format, just with different cardinalities and commitment structures.

One potential blind spot: The assistant does not explicitly investigate whether Curio applies any transformation or encoding (e.g., base64, length-prefixing, compression) to proof bytes before passing them to FFI. The research focused on the logical structure (which proofs, which commitments) rather than the wire format. This gap was later addressed during implementation, where the assistant confirmed that raw bytes are passed directly.

The Thinking Process Visible in the Message

The message's structure reveals a deliberate, methodical thinking process:

  1. State awareness: The assistant begins by acknowledging what it already knows ("the existing test data and what's available") and what it still needs ("check Curio's Go FFI layer").
  2. Hypothesis formation: The assistant hypothesizes that the serialization format used by Curio's Go code is the format cuzk should expect. This is not stated explicitly but is embedded in the research prompt's phrasing: "This will tell us the serialization format cuzk should expect over gRPC."
  3. Targeted investigation: Rather than broadly searching the Curio codebase, the assistant targets specific files known to contain proof serialization logic. This precision reflects the knowledge gained from prior research tasks about where in the Curio codebase proof generation occurs.
  4. Result integration: The task result is not treated as final output but as input to the next phase of implementation. The assistant immediately uses the discovered serialization formats to update the protobuf definitions, the types.rs structures, and the prover.rs implementation.

The Broader Significance

This message exemplifies a pattern that recurs throughout the cuzk development: research-driven implementation. Every significant code change is preceded by a targeted investigation that resolves specific unknowns. The assistant never guesses at serialization formats, never assumes API behavior, and never writes code that depends on unverified assumptions. Instead, it systematically eliminates uncertainty through directed reading of source code, API documentation, and production systems.

For the cuzk project as a whole, message 288 was the moment when the Phase 1 implementation shifted from "we know what functions to call" to "we know what data to send them." The protobuf definitions, type structures, and prover logic that emerged in the subsequent messages ([msg 289] through [msg 316]) all trace their correctness back to this single research task. Without it, the multi-GPU worker pool, the priority scheduler, and the elegant gRPC API would have been wired to the wrong serialization contracts, producing proofs that either failed at the FFI boundary or, worse, silently produced invalid results.

In engineering terms, this message represents the difference between a system that works by coincidence and one that works by design. The assistant's insistence on understanding the exact serialization contract before writing a single line of implementation code is what separates professional systems engineering from ad-hoc prototyping. It is a lesson worth studying: in distributed systems, the interfaces between components are where complexity hides, and the only way to tame that complexity is to trace the data flow end-to-end before committing to a design.