The VanillaProof Threshold: Narrowing the Search Space in a Cross-Language PoRep Bug

Introduction

In distributed systems debugging, the most valuable moments are often not the dramatic "aha" discoveries but the quiet, methodical eliminations of plausible hypotheses. Message 1639 of this opencode session represents precisely such a moment: a brief but consequential checkpoint in a deep-dive investigation into a production bug where PoRep (Proof of Replication) PSProve tasks fail with the error "porep failed to validate" when processed through the CuZK GPU proving engine, despite Snap PSProve tasks and normal CuZK PoRep C2 tasks working correctly. In this message, the assistant performs a critical narrowing of the search space by examining the C1OutputWrapper struct definitions across the Go and Rust codebases, ruling out one potential source of mismatch and pivoting toward the next investigative target.

The Context: A Cross-Language Serialization Bug Hunt

To understand why this message was written, one must appreciate the architecture of the PSProve system. PSProve is a proof-sharing mechanism where one party (the "market") performs the computationally intensive SealCommitPhase1 (C1) computation and distributes the C1 output to proving providers who then compute SealCommitPhase2 (C2) to produce the final SNARK proof. The C1 output is a complex JSON structure produced by Rust's filecoin_proofs_api library, serialized via serde_json, and then transmitted to Go-based workers through a JSON-based API. When the CuZK GPU proving engine is involved, this data takes an additional hop: the Go worker re-marshals the C1 data into a C1OutputWrapper struct and sends it via gRPC to the CuZK Rust daemon, which deserializes it and runs the GPU-accelerated proving pipeline.

The production bug manifested as a failure in exactly this CuZK path for PoRep proofs, but only when the C1 data arrived through the PSProve flow. The normal CuZK path (where the C1 data was produced locally by the same process) worked flawlessly. The non-CuZK PSProve path (where the Go worker calls ffi.SealCommitPhase2 directly without CuZK) also worked. This pattern—working in paths A and B but failing in path C—strongly suggested a data plumbing issue specific to how the PSProve path packaged data for the CuZK gRPC call.

The Reasoning: Systematic Code Path Comparison

In the messages immediately preceding the subject message, the assistant had laid the groundwork for a meticulous side-by-side comparison of the two CuZK code paths. Message 1636 presented a detailed comparison showing that both paths construct nearly identical SubmitProofRequest protobuf messages, differing only in the RequestId prefix (cosmetic) and crucially in how the VanillaProof field was populated. In the normal path, raw Rust JSON bytes from ffi.SealCommitPhase1() were base64-encoded into the wrapper. In the PSProve path, the C1 data had been deserialized into a Go Commit1OutRaw struct and then re-marshaled to JSON before being base64-encoded into the wrapper.

This re-marshaling step was the prime suspect. If the Go JSON serialization produced structurally different JSON from the original Rust JSON, the CuZK Rust daemon might deserialize it incorrectly, leading to a subtly corrupted SealCommitPhase1Output and ultimately an invalid SNARK proof. However, message 1637 introduced a crucial counter-observation: the non-CuZK PSProve path also used this same re-marshaled JSON, and it worked fine. If the JSON round-trip were broken, the non-CuZK path would also fail. This eliminated the most obvious hypothesis and forced the investigation deeper into the CuZK-specific data packaging.

The Subject Message: Ruling Out SectorNum

Message 1639 begins with a focused observation: the Rust C1OutputWrapper struct (defined in cuzk-core/src/prover.rs) declares sector_num as u64, while both Go-side wrapper definitions (in cuzk_funcs.go and task_prove.go) declare SectorNum as int64. This might appear to be a type mismatch—a classic source of cross-language bugs. However, the assistant correctly reasons that since sector numbers are always positive (they are unsigned integers in the Filecoin protocol), the int64 representation in Go can hold any valid u64 value without loss. The json.Marshal of an int64 with a positive value produces the same JSON number as a u64. The Rust serde_json deserializer, when targeting a u64 field, will accept a positive integer from JSON without complaint. Therefore, this is a non-issue.

This ruling-out is significant not because it reveals a bug, but because it doesn't. In debugging, every eliminated hypothesis tightens the remaining search space. The assistant has now confirmed that:

The Pivot: Following the VanillaProof Field

Having cleared these hypotheses, the assistant pivots to the next logical target: the VanillaProof field in the protobuf SubmitProofRequest. The message ends with a grep command searching for vanilla_proof (without trailing 's') to distinguish the singular VanillaProof field (used for PoRep) from the plural VanillaProofs field (used for Snap and PoSt proofs). This is a crucial distinction because the protobuf schema uses different fields for different proof types, and a mismatch in how the data is placed into or extracted from these fields could explain the PoRep-specific failure.

The assistant's thinking here demonstrates a methodical approach to cross-language debugging: trace the data through every transformation boundary. The data flow for the PSProve CuZK path is:

  1. Rust filecoin_proofs_api produces SealCommitPhase1Output as JSON bytes
  2. Market serializes these bytes as a JSON string in the proof-data API
  3. Go worker deserializes into Commit1OutRaw struct
  4. Go worker re-marshals Commit1OutRaw to JSON bytes
  5. Go worker wraps these bytes in C1OutputWrapper (base64-encoded)
  6. Go worker marshals C1OutputWrapper to JSON bytes → VanillaProof protobuf field
  7. gRPC transmits to CuZK Rust daemon
  8. CuZK deserializes C1OutputWrapper from JSON
  9. CuZK base64-decodes the Phase1Out field
  10. CuZK deserializes SealCommitPhase1Output from JSON
  11. CuZK calls seal::seal_commit_phase2(c1_output, prover_id, sector_id) Each of these 11 steps is a potential failure point. The assistant has now eliminated steps 3-4 (the JSON round-trip, proven by the working non-CuZK path), steps 5-6 (the wrapper struct definitions are identical), and step 2 (the SectorNum type is benign). The remaining suspect is step 6—specifically, how the VanillaProof protobuf field is populated and whether the CuZK Rust daemon interprets it differently than expected.

Assumptions and Their Validity

The assistant makes several implicit assumptions in this message. First, it assumes that the VanillaProof protobuf field is the correct place to look next. This is a reasonable assumption given the elimination of other hypotheses, but it carries the risk that the actual bug lies elsewhere—perhaps in the RegisteredProof enum mapping (which the user had specifically suggested investigating) or in some other protobuf field like SectorNumber or MinerId.

Second, the assistant assumes that the structural identity of the two Go c1OutputWrapper definitions means they produce identical JSON. While the struct fields are the same, the Go json.Marshal function could theoretically produce different output if the struct types are defined in different packages with different marshaler configurations. In practice, Go's encoding/json package produces consistent output for identical struct definitions, so this assumption is safe.

Third, the assistant assumes that the int64/u64 mismatch is benign. This is correct for positive values, but if a sector number somehow exceeded the int64 maximum (9.2 × 10^18), it would overflow. In practice, Filecoin sector numbers are 64-bit unsigned integers but rarely exceed a few million, so this is safe.

The Broader Significance

This message exemplifies a debugging methodology that is essential in distributed, polyglot systems: the systematic enumeration and elimination of data transformation boundaries. Each transformation—JSON serialization, base64 encoding, protobuf marshaling, gRPC transmission—is a potential site for subtle corruption. The assistant's approach is to work through these boundaries one by one, using working code paths as control experiments to isolate the failing path.

The message also highlights the importance of understanding the semantic meaning of data types across language boundaries. A u64 in Rust and an int64 in Go are not the same type, but for the specific values in question, they are semantically equivalent. The assistant correctly recognizes this and avoids wasting time on a red herring.

Conclusion

Message 1639 is a brief but pivotal step in a complex debugging journey. By confirming that the C1OutputWrapper struct definitions are consistent and the SectorNum type difference is benign, the assistant narrows the investigation to the VanillaProof protobuf field and the CuZK Rust daemon's handling of it. This sets the stage for the next phase of the investigation: examining how the CuZK Rust engine deserializes the VanillaProof bytes in its pipeline and whether the re-marshaled Go JSON introduces subtle structural differences—such as field ordering, whitespace, or null field handling—that affect the Rust serde_json deserialization of SealCommitPhase1Output. In a debugging process that spans Go, Rust, protobuf, and gRPC boundaries, each eliminated hypothesis is a step closer to the root cause.