The Fork That Changed Everything: Tracing cuzk's Bellperson Divergence

Introduction

In the middle of an intensive debugging session spanning dozens of messages, a single observation crystallized the investigation into an intermittent proof verification failure in the cuzk proving system. The message at index 1780 represents a pivotal moment of synthesis—where the assistant, after systematically ruling out serialization issues, input mismatches, and concurrency bugs, identified what it called "the key difference" between two code paths that should have been identical. That difference lay in the bellperson dependency: cuzk used a forked version with a cuda-supraseal backend, while the standard Filecoin FFI used the upstream bellperson/cuda. This article examines that message in depth, exploring the reasoning that led to it, the knowledge it produced, and its significance in the broader investigation.

The Message in Full

The assistant wrote:

The key difference: bellperson/cuda-supraseal in cuzk vs bellperson/cuda in FFI. Let me check the bellperson fork:

>

``bash cat /tmp/czk/extern/cuzk/Cargo.toml | grep -A 5 bellperson ``

>

Output: ``` # Forked bellperson with split synthesis/GPU API (Phase 2). # The [patch.crates-io] below ensures ALL transitive deps use the fork. bellperson = { path = "../bellperson", default-features = false }

>

# BLS12-381 types (already transitive via bellperson/blstrs) blstrs = "0.7" ff = "0.13" rayon = "1.10" rand_core = "0.6" bincode = "1.3" -- bellperson = { path = "../bellperson" } bellpepper-core = { path = "../bellpepper-core" } supraseal-c2 = { path = "../supraseal-c2" }

>

[profile.release] opt-leve... ```

This is a short message—barely a paragraph of analysis followed by a command and its output—but it carries enormous weight in the investigative narrative. It is the moment the assistant shifted from asking "what is different about the inputs?" to asking "what is different about the proving engine itself?"

The Reasoning: Why This Message Was Written

To understand why this message was written, one must trace the investigative arc that preceded it. The session had been deep in the weeds of a production bug: cuzk's PoRep (Proof of Replication) proofs were intermittently failing verification on the Go side. The error message was "porep failed to validate"—a Go-side VerifySeal call was returning false after cuzk had successfully generated a proof.

The assistant had already eliminated several plausible causes. It had traced the Go JSON serialization roundtrip and proven that the Go-to-Rust-to-Go data path preserved all fields correctly. It had examined the RegisteredSealProof enum mappings across Go, C, and Rust and confirmed structural parity. It had investigated whether fr32 seed masking (the seed[31] &= 0x3f pattern) could corrupt the interactive randomness, and ruled that out by tracing the seed's usage—it was consumed as raw bytes in SHA256 for challenge derivation, never converted to a BLS12-381 scalar field element. It had even discovered that the standard FFI's SealCommitPhase2 was itself intermittently unreliable for 2KiB test sectors, suggesting a deeper instability in bellperson's GPU proving path.

But the production bug had a specific pattern: cuzk's path failed, while the FFI path succeeded on the same inputs. If both paths used the same bellperson library with the same GPU backend, this pattern would be inexplicable—both should fail or both should succeed. The fact that they diverged meant something fundamental was different between the two environments.

The assistant's reasoning, visible in the preceding messages, was methodical. It had checked the cuzk engine's dispatch logic, examined the prove_porep_c2 function, traced the verification inputs on both the Go and Rust sides, and confirmed that cuzk's internal self-check (the verify_seal call inside seal_commit_phase2) was passing. If the proof passed Rust's internal verification but failed Go's VerifySeal, the inputs to verification must differ. But the assistant had already checked those inputs—CommR, CommD, proverID, randomness, seed, sector number—and they matched. This contradiction forced a new hypothesis: perhaps the proof itself was structurally different, generated by a different version of the proving library.

The Discovery: A Forked Bellperson with Split Synthesis

The command the assistant ran—cat /tmp/czk/extern/cuzk/Cargo.toml | grep -A 5 bellperson—was a targeted probe into the dependency configuration. The output revealed three critical facts.

First, cuzk used a forked bellperson located at path = "../bellperson". This was not the upstream filecoin-project/bellperson but a local fork with modifications. The comment in the Cargo.toml explained the fork's purpose: "Forked bellperson with split synthesis/GPU API (Phase 2)." This meant the fork restructured bellperson's internal pipeline, separating the CPU-bound circuit synthesis phase from the GPU-bound proving phase. This architectural change was the entire point of cuzk's existence—it allowed overlapping synthesis and GPU work for higher throughput.

Second, the fork was configured with default-features = false. This is a Cargo feature flag that disables all default feature sets, meaning GPU acceleration features (like cuda) were not automatically enabled. The cuzk codebase would need to explicitly opt into whatever GPU backend it required. This was a significant departure from the standard FFI build, which typically enables cuda by default through its dependency chain.

Third, the workspace included supraseal-c2 = { path = "../supraseal-c2" }—a custom GPU proving backend. The name "supraseal" strongly suggested this was an alternative to bellperson's built-in GPU support, possibly a more aggressively optimized C++ CUDA backend. The existence of this custom backend meant cuzk's proof generation pipeline was fundamentally different from the standard FFI's, even when both were nominally performing the same Groth16 proving operation.

The Assumptions Embedded in the Message

The assistant's statement—"The key difference"—carries an implicit assumption: that the bellperson fork divergence is the root cause of the intermittent verification failures. This is a hypothesis, not a proven conclusion, but it is a well-motivated one. The assistant had already eliminated all other plausible explanations. The fork difference was the remaining variable that could explain why cuzk's proofs sometimes differed from FFI's proofs.

A second assumption is that the fork difference manifests as a proof-quality issue rather than a performance or interface issue. The assistant is implicitly reasoning that a modified proving pipeline—with a split synthesis/GPU API and a custom CUDA backend—might produce proofs that are structurally valid (pass internal verification) but subtly different from standard proofs in ways that cause external verification to fail. This is a plausible mechanism: if the fork modifies the constraint system layout, the proof encoding, or the random challenge generation, the resulting proof could be valid within the fork's own verification logic but incompatible with the standard verifier.

A third assumption, visible in the investigative methodology, is that the dependency configuration is the right place to look. The assistant chose to inspect Cargo.toml rather than, say, running a diff between the two bellperson source trees or examining the compiled binary's symbol table. This choice reflects a software engineering intuition: architectural differences are encoded in dependency graphs, and the Cargo.toml is the canonical source of truth for Rust project structure.

Input Knowledge Required

To fully understand this message, a reader needs substantial domain knowledge. They must understand that bellperson is a zk-SNARK proving library implementing the Groth16 protocol, and that it supports GPU acceleration through a cuda feature flag. They need to know that cuzk is a GPU-accelerated proving daemon for Filecoin's proof-of-replication (PoRep) protocol, designed to improve throughput by pipelining synthesis and proving. They must understand the role of the Filecoin FFI (foreign function interface) as the Go-to-Rust bridge that standard proof operations pass through. And they need to grasp the investigative context: that an intermittent verification failure was occurring specifically on the cuzk path, that the FFI path worked reliably on the same inputs, and that all input-level explanations had been exhausted.

The reader also needs familiarity with Rust's Cargo build system, particularly the [patch.crates-io] mechanism for overriding transitive dependencies and the default-features = false flag for controlling feature sets. The path = "../bellperson" syntax indicates a local path dependency, meaning the codebase includes a modified copy of bellperson rather than using the crates.io published version.

Output Knowledge Created

This message creates several pieces of new knowledge. First, it establishes that cuzk's bellperson fork is not merely a version bump or a minor patch—it is a structural fork with a "split synthesis/GPU API" design, a custom supraseal-c2 backend, and explicit feature control. Second, it reveals that the fork is configured with default-features = false, meaning GPU features must be explicitly enabled rather than inherited from upstream defaults. Third, it documents the [patch.crates-io] mechanism that forces all transitive dependencies to use the fork, ensuring consistency across the dependency tree.

This knowledge reframes the investigation. Previously, the assistant was looking for data-level differences—wrong inputs, corrupted serialization, mismatched enums. Now, the investigation must consider algorithm-level differences: does the fork's modified pipeline produce proofs that are structurally incompatible with the standard verifier? Does the supraseal-c2 backend use a different proof encoding or a different random oracle? Is the split synthesis phase introducing a subtle constraint ordering change that affects the proof's validity?

The Thinking Process Visible in the Message

The assistant's thinking process is visible in the structure of the message itself. The opening statement—"The key difference"—is a conclusion drawn from the preceding investigation. It is not hedged with qualifiers like "possibly" or "might be"; the assistant has reached a point of sufficient confidence to declare this the key variable. The bold formatting of bellperson/cuda-supraseal and bellperson/cuda emphasizes the contrast and signals that this is the central finding.

The command that follows is not exploratory but confirmatory. The assistant already knows that cuzk uses a forked bellperson—it has seen the source tree at /tmp/czk/extern/bellperson/ and has likely read the fork's README or commit history. The grep -A 5 bellperson command is designed to extract the specific dependency declaration and its surrounding context, including the explanatory comment and the workspace-level overrides. The assistant is looking for the exact configuration details to understand how the fork is integrated.

The output is presented without additional commentary, which is itself a rhetorical choice. The assistant lets the Cargo.toml content speak for itself. The comment "Forked bellperson with split synthesis/GPU API (Phase 2)" is the key revelation, and the assistant trusts the reader to recognize its significance. The default-features = false and supraseal-c2 entries are supporting evidence that confirms the fork is not a trivial modification.

Broader Significance

This message represents a turning point in the investigation. Before it, the assistant was operating under the assumption that cuzk and FFI used the same proving library with the same algorithms, differing only in deployment architecture (gRPC service vs direct FFI call) and pipeline structure (partitioned vs monolithic). The discovery that cuzk uses a fundamentally different bellperson fork with a custom GPU backend invalidates that assumption. It opens up an entirely new category of possible root causes: algorithmic divergence, proof encoding differences, and compatibility issues between the fork's proving path and the standard verification path.

The message also demonstrates a key principle of debugging complex systems: when all data-level explanations are exhausted, examine the build configuration. The dependency graph encodes architectural decisions that may not be visible in the source code itself. The Cargo.toml, with its path dependencies, feature flags, and workspace overrides, is a map of the system's hidden structure. Reading that map is often the key to understanding emergent behavior that cannot be explained by examining individual components in isolation.

In the broader arc of the session, this message set the stage for the eventual fix. The assistant would go on to discover that the real issue was not the bellperson fork itself but a missing self-check in cuzk's pipeline assembly code—the fork's split synthesis/GPU API had introduced a code path where partition proofs were assembled and returned without the internal verification that the monolithic path performed. But that discovery was only possible because the assistant first understood that cuzk's pipeline was structurally different from the standard FFI's. The bellperson fork was the clue that led to the real bug.