The Bellperson Fork: A Pivotal Probe in the PSProve Debugging Session

Introduction

In the midst of a sprawling, multi-hour debugging session spanning Go, Rust, C, and GPU-accelerated proving pipelines, a single ls command often marks the transition from hypothesis to evidence. Message 1786 in this opencode session is precisely such a moment. The assistant, having traced an intermittent "porep failed to validate" error through JSON serialization roundtrips, enum mappings, and concurrency hazards, arrives at a critical fork in the investigation: the vendored bellperson library. The command ls /tmp/czk/extern/bellperson/src/groth16/ is not merely a file listing—it is a deliberate probe into the heart of the proof generation machinery, seeking structural differences that could explain why proofs produced by the cuzk GPU proving daemon sometimes fail Go-side verification while the FFI path succeeds.

The Message

The message itself is deceptively simple:

[assistant] [bash] ls /tmp/czk/extern/bellperson/src/groth16/
aggregate
ext.rs
ext_supraseal.rs
generator.rs
mapped_params.rs
mod.rs
multiscalar.rs
params.rs
proof.rs
prover
supraseal_params.rs
verifier.rs
verifying_key.rs

A directory listing of 14 entries. But in the context of the investigation, each filename carries weight.

Context and Motivation

To understand why this ls command was issued at this precise moment, we must trace the investigation's trajectory through the preceding messages. The session had been deep in the weeds of a production bug: cuzk-generated PoRep (Proof of Replication) proofs were intermittently failing the Go-side VerifySeal call, producing the error "porep failed to validate". The assistant had systematically ruled out several potential causes:

1. Go JSON roundtrip corruption ([msg 1764]): A 2KiB sector test proved that even raw Rust JSON serialization produced intermittent failures, eliminating the Go marshaling layer as the culprit.
2. Concurrency hazards ([msg 1774]): The assistant discovered that cuzk's monolithic proof path sets CUDA_VISIBLE_DEVICES as a process-level environment variable inside spawn_blocking tasks—a classic race condition. But for the sequential PSProve case, this was unlikely to be the trigger.
3. Verification input mismatches (<msg id=1775-1776>): A detailed comparison of Go's VerifySeal inputs against Rust's internal verification in seal_commit_phase2 showed they used identical parameters (CommR, CommD, proverID, randomness, seed, sector number). The inputs were not the issue.
4. The 2KiB flakiness (<msg id=1777-1778>): The assistant realized that the 2KiB test sector failures revealed a separate, foundational unreliability in ffi.SealCommitPhase2 itself—a bellperson-level issue for small circuits that might not affect production 32GiB sectors. With these avenues narrowing, the assistant's attention turned to the bellperson fork itself. In [msg 1780], the assistant noted a critical difference in dependency configuration: cuzk uses bellperson/cuda-supraseal while the FFI uses bellperson/cuda. The supraseal feature flag implies a custom GPU proving backend. In [msg 1784], the assistant discovered that the bellperson fork is vendored directly into the curio repository tree, not pulled from crates.io or a separate git repo. This means any modifications to bellperson are local and potentially divergent from the upstream version used by the FFI. The ls command at [msg 1786] is the natural next step: having identified that the bellperson fork is vendored and uses a different feature flag, the assistant now needs to understand what structural changes exist in the groth16 module—the core of SNARK proof generation.## What the Directory Listing Reveals The presence of ext_supraseal.rs and supraseal_params.rs in the groth16 directory is the most telling finding. These files do not exist in the upstream bellperson repository—they are custom additions for the cuzk fork. The supraseal naming convention points to a GPU-accelerated proving backend that bypasses or replaces the standard bellperson prover. The ext.rs file (likely a base extension module) and ext_supraseal.rs (the GPU-specific variant) suggest a modular architecture where the standard proving path can be swapped for a GPU-optimized one. The prover subdirectory and files like mapped_params.rs, multiscalar.rs, and verifier.rs are standard bellperson components. But the presence of both ext.rs and ext_supraseal.rs implies that cuzk maintains two parallel proving paths: one for standard CPU proving and one for the GPU supraseal backend. This structural duality is exactly the kind of divergence that could introduce subtle behavioral differences.

Assumptions and Reasoning

The assistant's decision to inspect the bellperson fork rests on several key assumptions:

1. The bug is in the proof generation, not the verification: Since cuzk's seal_commit_phase2 internally verifies the proof before returning (as established in [msg 1765]), a proof that passes cuzk's self-check but fails Go's VerifySeal must be structurally valid but somehow different from what the Go side expects. The most plausible explanation is that the cuzk fork of bellperson produces proofs with a different format, encoding, or internal structure.
2. The cuda-supraseal feature flag is the differentiator: The assistant had already noted in [msg 1780] that cuzk uses bellperson = { path = "../bellperson", default-features = false } with the cuda-supraseal feature, while the FFI uses the standard cuda feature. This feature flag discrepancy is the strongest lead.
3. Structural differences in the groth16 module would be visible at the file level: By listing the directory, the assistant expects to see files that indicate how the supraseal backend integrates—whether it replaces the prover entirely, adds wrapper types, or modifies the proof format. These assumptions are reasonable but carry risk. The bug could still be elsewhere—in the Go-side proof deserialization, in the cuzk engine's job dispatch logic, or in the GPU hardware itself. The directory listing is a reconnaissance step, not a diagnostic one.

Input Knowledge Required

To interpret this message meaningfully, one must understand:

• The role of bellperson: Bellperson is the core SNARK proving library for Filecoin, implementing the Groth16 proving system over BLS12-381. It handles constraint synthesis, proof generation, and verification.
• The groth16 module structure: Standard bellperson groth16 contains files for the prover, verifier, generator, proof types, parameters, and multi-scalar multiplication. Custom additions like ext_supraseal.rs signal a forked or extended implementation.
• The cuzk architecture: cuzk is a GPU-accelerated proving daemon that replaces the standard CPU-based proof generation with a pipeline that can use NVIDIA GPUs via CUDA. It forks bellperson to add the supraseal backend.
• The investigation history: The preceding 25+ messages had systematically eliminated other potential causes, narrowing the field to the bellperson fork itself.

Output Knowledge Created

This message creates actionable knowledge:

1. Confirmation of structural divergence: The presence of ext_supraseal.rs and supraseal_params.rs confirms that the cuzk bellperson fork has significant custom code in the groth16 module. This validates the hypothesis that proof generation differs between cuzk and the standard FFI path.
2. A map for further investigation: The file listing provides a roadmap for the next phase of debugging. Each file is a potential entry point for code comparison: ext_supraseal.rs for the GPU prover integration, proof.rs for the proof type definition, params.rs and supraseal_params.rs for parameter handling.
3. Evidence for the feature flag theory: The existence of supraseal-specific files alongside the standard ones supports the theory that the cuda-supraseal feature flag activates an alternative proving path that may produce subtly different proofs.

The Thinking Process

The assistant's reasoning in this message is visible through the sequence of commands that led here. In [msg 1784], the assistant attempted to diff the bellperson Cargo.toml against its original, finding only normalization differences. In [msg 1785], the assistant listed the top-level src/ directory to understand the fork's scope. Now, in [msg 1786], the assistant drills into the groth16/ subdirectory—the most relevant module for the proof format question.

The progression shows a systematic narrowing strategy: from dependency configuration (Cargo.toml) to module structure (src/) to the specific proving subsystem (groth16/). Each step filters out irrelevant code and focuses on the most likely location of the divergence.

The assistant is also implicitly comparing against knowledge of the standard bellperson structure. A developer familiar with bellperson would know that ext_supraseal.rs and supraseal_params.rs are not standard files, making them immediate targets for code review.

Conclusion

Message 1786 is a quiet but pivotal moment in a complex debugging session. A simple ls command, informed by hours of systematic elimination, transforms a hypothesis about feature flag divergence into a confirmed structural difference. The directory listing of the bellperson groth16 module provides the roadmap for the next phase of the investigation: comparing the supraseal-specific code paths against the standard implementation to find the exact proof format discrepancy that causes intermittent verification failures. In the broader narrative of the session, this message marks the transition from elimination-based debugging to targeted code comparison—a shift that ultimately leads to the discovery and fix of the self-check bypass bug in the cuzk engine.