The Diagnostic That Wasn't: How a Silent Self-Check Allowed Invalid Proofs to Reach Production

The Message

In the middle of a deep investigation into intermittent PoRep proof failures, the assistant wrote:

Now let me fix the engine. The self-check currently logs warnings but still returns the invalid proof. I need to make it return JobStatus::Failed when the self-check fails: [read] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs

This is message <msg id=1833> in the conversation. On its surface, it appears to be a simple statement of intent followed by a file read. But this message represents the culmination of an exhaustive debugging session that spanned dozens of messages, multiple codebases (Go, Rust, C++), and a deep understanding of the Filecoin proving pipeline. It is the precise moment when the assistant identified the root cause of a production bug and committed to fixing it.

The Investigation That Led Here

To understand why this message was written, one must trace the investigation that preceded it. The user had reported that PoRep proofs generated by the cuzk proving engine were intermittently failing Go's VerifySeal with the error "porep failed to validate". Some challenges succeeded, others failed — there was no obvious pattern.

The assistant's investigation began with a systematic exploration of possible causes. It first examined whether the Go JSON serialization round-trip (used to pass proof data between Go and Rust) was corrupting the proof bytes. This required tracing the C1OutputWrapper struct through Go's JSON marshaling, through the FFI boundary, and into Rust's deserialization logic. The assistant wrote extended roundtrip tests with 2KiB sectors to capture byte-level discrepancies, and ruled out JSON as the source of the problem.

Next, the assistant investigated whether the RegisteredSealProof enum mappings were consistent across Go, C, and Rust. A mismatch in sector size constants or proof type identifiers could cause the wrong verification parameters to be used. The assistant traced enum values through all three languages and confirmed structural parity.

The assistant then examined the possibility that fr32 seed masking was the cause. In Filecoin, some randomness values require the high bits to be masked (the seed[31] &= 0x3f pattern used for PoSt). The assistant traced the complete seed flow through Rust proof crates, cusvc challenge generation, and Filecoin chain actors, ultimately proving that PoRep seeds are used exclusively as raw bytes in SHA256 for challenge derivation — they are never converted to BLS12-381 scalar field elements, so masking is unnecessary and cannot be the cause.

The Critical Discovery

The breakthrough came when the assistant examined the cuzk engine's code paths for PoRep proof generation. It discovered that cuzk has three distinct modes for generating PoRep proofs:

  1. Monolithic mode (fallback): Calls prover::prove_porep_c2 which internally calls seal::seal_commit_phase2. This path includes a self-check that verifies the proof before returning it. If verification fails, an error is returned to the caller.
  2. Phase 6 slot-based pipeline (slot_size > 0): Calls pipeline::prove_porep_c2_partitioned. This path processes proofs in slots but lacks the internal self-verification.
  3. Phase 7 partition_workers pipeline (partition_workers > 0): Dispatches each of the 10 partitions independently through a synthesis→GPU pipeline, then assembles the results. The production configuration used PARTITION_WORKERS=16 (set in run.sh), which meant cuzk was running in Phase 7 mode. The assistant traced the code in engine.rs and found that the Phase 7 path does have a self-check — it calls verify_porep_proof() after assembling all partition proofs. But critically, the code at lines 293-304 returned JobStatus::Completed with the proof bytes regardless of whether the self-check passed or failed. The self-check was diagnostic only: it logged warnings when verification failed, but still delivered the invalid proof to the caller. This was the root cause. The intermittent GPU proving instability (likely originating from the supraseal C++ backend) would occasionally produce an invalid partition proof. The self-check would detect this, log a warning, and then... return the invalid proof anyway. Go would receive it, call VerifySeal, and correctly return "porep failed to validate".

The Reasoning Behind the Fix

The subject message represents the moment the assistant recognized that the fix was trivial in concept but critical in impact. The reasoning is captured in the preceding message ([msg 1832]):

"But critically, look at lines 293-304: it returns the proof regardless of whether the self-check passed or failed. The self-check is diagnostic only — it still sends the proof to the caller even if it's invalid!"

The fix required changing a single control flow decision: when verify_porep_proof() returns Ok(false) or Err(...), the code should return JobStatus::Failed instead of JobStatus::Completed. This is a one-line conceptual change, but it transforms a diagnostic warning into a hard error, preventing invalid proofs from ever reaching the Go caller.

The assistant's decision to read the file at this point was deliberate. It needed to see the exact code structure — the variable names, the control flow, the error handling patterns — before making the edit. The read was not exploratory; it was preparatory. The assistant already knew what the bug was; it needed to confirm the precise location and structure of the code to apply the fix correctly.

Assumptions and Their Validity

The assistant made several assumptions during this investigation, most of which proved correct:

Assumption 1: The pipeline path lacks self-verification. Initially, the assistant assumed the Phase 7 path had no self-check at all ([msg 1802]: "confirmed: the pipeline path does NOT have a self-check"). This was partially incorrect — the path did have a self-check, but it was non-functional (diagnostic only). The assistant corrected this assumption upon reading the full code in <msg id=1832>.

Assumption 2: The monolithic path's self-check works correctly. This was a necessary assumption to narrow the investigation. If the monolithic self-check were also buggy, the problem would be more fundamental. The assistant implicitly trusted the existing seal::seal_commit_phase2 verification logic.

Assumption 3: The supraseal C++ backend is the source of intermittent invalid proofs. The assistant noted that the unsafe uninitialized vector pattern in supraseal.rs (lines 93-98) could produce garbage proof bytes if the C++ code didn't fill all fields correctly. This remains an open hypothesis — the fix prevents bad proofs from propagating, but the underlying GPU proving instability is not resolved.

Assumption 4: The production configuration uses partition_workers > 0. The assistant confirmed this by reading run.sh, which set PARTITION_WORKERS=16 by default. This was critical — if production used the monolithic path, the bug would manifest differently (as a Rust-side error, not a Go-side VerifySeal failure).

Input Knowledge Required

To understand this message, one needs knowledge of:

Output Knowledge Created

This message and the fix it initiated produced:

  1. A corrected control flow: The self-check now gates proof return. When verification fails, JobStatus::Failed is returned instead of JobStatus::Completed, preventing invalid proofs from reaching the caller.
  2. A hardened production system: The fix was deployed to the production cuzk worker by building a minimal binary and hot-swapping the daemon. This ensured that all pipeline modes (Phase 6, Phase 7, batched multi-sector, and single-sector) enforce the self-check.
  3. A documented class of bugs: The investigation revealed a pattern where diagnostic code exists but has no effect on the program's behavior. The self-check was logging failures but not acting on them — a "warning-only" pattern that can silently mask critical errors.
  4. A remaining open issue: The underlying GPU proving instability (intermittent invalid partition proofs from supraseal C++) remains unresolved. The fix prevents bad proofs from propagating, but the root cause in the GPU proving pipeline may require retry logic or further investigation.

The Thinking Process

The assistant's thinking process in this message is visible in the surrounding context. It moved through several phases:

Phase 1 — Hypothesis generation: The assistant generated multiple hypotheses for the intermittent failures: JSON serialization corruption, enum mapping mismatches, seed masking, and pipeline mode differences.

Phase 2 — Systematic elimination: Each hypothesis was tested through code reading, test execution, and static analysis. The JSON roundtrip was tested with 2KiB sectors. The enum mappings were traced across three languages. The seed masking was ruled out by tracing the seed's usage path.

Phase 3 — Code path analysis: The assistant identified the three pipeline modes and traced the control flow for each. It discovered that production used Phase 7 (partition_workers > 0) and that this path had a diagnostic-only self-check.

Phase 4 — Root cause confirmation: By reading the engine.rs code, the assistant confirmed that the self-check at lines 190-286 logged warnings but the code at lines 293-304 unconditionally returned JobStatus::Completed. This was the bug.

Phase 5 — Fix planning: The assistant recognized that the fix was straightforward — gate proof return on the self-check result — and prepared to apply it. This is where message [msg 1833] sits: at the transition between root cause identification and fix implementation.

The assistant's thinking shows a disciplined approach to debugging: generate hypotheses, test each one, trace the actual code paths, and only apply a fix when the root cause is confirmed. It resisted the temptation to apply speculative fixes and instead invested the time to understand the full system architecture before making changes.

Conclusion

Message [msg 1833] is a pivotal moment in the conversation. It represents the transition from investigation to remediation — the point where the assistant had gathered sufficient evidence to identify the root cause and was ready to apply the fix. The message itself is brief, but it carries the weight of the exhaustive analysis that preceded it. The fix it initiated — making the self-check mandatory rather than diagnostic — was a one-line conceptual change that prevented intermittent invalid proofs from reaching the Filecoin ProofShare challenge protocol, protecting the production system from a subtle and difficult-to-diagnose failure mode.