The Three Modes: Tracing a Production Bug Through cuzk's Pipeline Architecture

In the course of investigating an intermittent "porep failed to validate" error in a Filecoin Proof-of-Replication (PoRep) proving system, an AI assistant reached a critical turning point. Message <msg id=1819> represents a moment of synthesis — where scattered observations about code paths, configuration defaults, and self-verification logic crystallize into a clear hypothesis about the root cause of a production bug. This message is the bridge between diagnosis and remediation, and understanding it requires tracing the architecture of the cuzk GPU proving engine through the lens of a single, pivotal insight.

The Message

The assistant writes:

So there are THREE modes: 1. partition_workers > 0 (Phase 7): Parse C1 once, dispatch partitions individually through synth→GPU pipeline 2. slot_size > 0, partition_workers == 0 (Phase 6): prove_porep_c2_partitioned (self-contained pipelined) 3. Monolithic (fallback): prover::prove_porep_c2 → seal::seal_commit_phase2 (with self-check)

>

Both pipeline modes (1 and 2) lack the internal self-verification. The monolithic mode has it.

>

Now here's the important question for production: which mode is the user's cuzk running in? Let me check the Vast worker config: ``bash rg -n "slot_size\|partition_workers\|pipeline" /tmp/czk/docker/cuzk/entrypoint.sh 2>/dev/null | head -20 ``

This short message is deceptively dense. In just a few lines, it encapsulates the outcome of an extensive multi-hour investigation, identifies a structural vulnerability in the proving pipeline, and pivots to the decisive question that will determine whether the fix is simple configuration change or a deeper code repair.

The Reasoning Behind the Message

The assistant arrived at this message after a long chain of analysis. The investigation began with a user report that some PoRep challenges succeeded while others failed with "porep failed to validate" — an error originating from Go's VerifySeal function. The intermittent nature of the failure suggested either a race condition, a data corruption issue, or a code path divergence.

Earlier in the session, the assistant had systematically ruled out several potential causes:

  1. JSON serialization round-trip bugs — The Go-side code serializes the C1 output (vanilla proof) as JSON and passes it to cuzk. The assistant tested this path extensively with 2KiB sector round-trips and found that while there were some serialization quirks (custom marshalers, field name differences), they were consistent and not intermittent.
  2. Seed masking — Filecoin uses "fr32" masking for some randomness values (masking the high bits of the last byte). The assistant traced the complete seed flow through Rust proof crates, cusvc challenge generation, and Filecoin chain actors, confirming that PoRep seeds are used exclusively as raw bytes in SHA256 for challenge derivation and never converted to a BLS12-381 scalar field element. The masking was unnecessary for this path.
  3. Enum mapping mismatches — The assistant traced RegisteredSealProof enum mappings across Go, C, and Rust codebases, confirming structural parity.
  4. Supraseal C++ backend instability — The 2KiB test suite revealed that even the monolithic path had intermittent failures in the FFI layer, but these appeared to be a separate bellperson issue for small sectors. The breakthrough came when the assistant examined the pipeline code paths in /tmp/czk/extern/cuzk/cuzk-core/src/pipeline.rs and compared them with the monolithic path in engine.rs. At <msg id=1800>, the assistant made the crucial discovery: "There's no self-verification in the pipeline path!" The monolithic path calls seal::seal_commit_phase2 which internally verifies the proof before returning it. But the pipeline path directly calls gpu_prove per partition and assembles the bytes without any verification. This was the structural flaw: if any partition proof generated by the GPU is invalid (due to transient hardware issues, memory corruption, or supraseal instability), the pipeline path would silently return the bad proof to the caller. The Go side would then correctly reject it with "porep failed to validate".

The Three Modes Architecture

Message <msg id=1819> formalizes this discovery into a taxonomy of three modes. Understanding this taxonomy is essential to grasping why the bug exists and how it must be fixed.

Mode 1: Phase 7 (partition_workers > 0). This is the most sophisticated mode, designed for maximum throughput. The engine parses the C1 output (vanilla proof JSON) once, then dispatches each partition as an independent work item through a synthesis→GPU pipeline. Partitions are synthesized by spawn_blocking workers gated by a semaphore, then sent through a channel to GPU workers. This mode overlaps synthesis and GPU proving for maximum hardware utilization. Critically, each partition's proof is produced independently and assembled by a ProofAssembler struct that simply concatenates them in order. There is no verification step after assembly.

Mode 2: Phase 6 (slot_size > 0, partition_workers == 0). This mode uses prove_porep_c2_partitioned, a self-contained pipelined function that handles all partitions internally. It also lacks self-verification. The slot_size parameter controls how many partitions are batched together in each "slot" of the pipeline.

Mode 3: Monolithic (fallback). This is the original, simple path. It calls prover::prove_porep_c2 which internally calls seal::seal_commit_phase2. This function generates all partition proofs in a single batch call to the supraseal prover, then runs the proof through verify_porep_proof() — a self-check that validates the proof before returning it. If verification fails, the monolithic path returns an error, preventing invalid proofs from reaching the caller.

The key insight in message <msg id=1819> is that both pipeline modes lack the self-check that the monolithic mode has. This means that any intermittent GPU proving failure — which the supraseal C++ backend is known to produce occasionally, especially for certain proof configurations — would be silently propagated to the caller in pipeline mode, but caught and rejected in monolithic mode.

Assumptions and Decisions

The assistant makes a critical assumption in this message: that the production deployment is using one of the pipeline modes. This assumption is based on the observation that partition_workers defaults to a positive value (the code at line 962 reads self.config.synthesis.partition_workers as usize, suggesting a non-zero default) and that pipeline mode is described as the "default for production" in earlier analysis.

However, the assistant does not assume — it verifies. The message ends with a bash command to check the Vast worker configuration files (entrypoint.sh) for slot_size, partition_workers, or pipeline settings. This is the decisive pivot: the fix strategy depends entirely on which mode is active.

If the production system is running in monolithic mode (Mode 3), the bug must be elsewhere — perhaps in the supraseal backend itself, or in data corruption between Go and Rust. But if it's running in a pipeline mode (Mode 1 or 2), the fix is clear: add self-verification to the pipeline assembly path, mirroring what the monolithic path already does.

Input Knowledge Required

To fully understand this message, one needs:

  1. Understanding of Filecoin PoRep proving architecture. PoRep (Proof of Replication) is a Filecoin consensus mechanism that proves a miner is storing a unique copy of data. It involves a two-phase proving process (C1 and C2), where C2 is the GPU-intensive Groth16 proof generation. For large sectors (32 GiB), the proof is split into 10 partitions, each proven independently and then assembled.
  2. Knowledge of Groth16 zk-SNARKs and batch proving. The assistant references create_random_proof_batch, MultiProof, and Proof::write — all concepts from the bellperson library (a fork of Bellman for Filecoin's BLS12-381 curve). Understanding that proofs are serialized as concatenated curve points (G1/G2 affine coordinates) is necessary to follow the serialization analysis.
  3. Familiarity with the cuzk engine architecture. The assistant has been reading /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs and pipeline.rs extensively. The engine is a GPU job dispatcher that manages synthesis workers, GPU workers, and proof assembly. The slot_size and partition_workers configuration parameters control which pipeline mode is active.
  4. Context from the ongoing investigation. The assistant has spent several hours tracing code paths, running 2KiB sector tests, comparing serialization formats, and ruling out other potential causes. Message <msg id=1819> is the culmination of this work.

Output Knowledge Created

This message creates several important outputs:

  1. A clear taxonomy of the three proving modes, with their distinguishing characteristics. This taxonomy is the mental model that will guide the fix.
  2. A specific hypothesis about the root cause: pipeline modes lack self-verification, so intermittent GPU failures propagate to the caller.
  3. A verification step: checking the production configuration to determine which mode is active. This is the critical next action.
  4. Implicitly, a fix strategy: add self-verification to the pipeline assembly path. If the proof fails verification, return JobStatus::Failed instead of JobStatus::Completed.

The Thinking Process

The assistant's reasoning in this message is a model of systematic debugging. The progression is:

  1. Synthesize the three modes from scattered code observations.
  2. Identify the critical difference between modes: self-verification presence vs. absence.
  3. Connect to the symptom: intermittent failures are consistent with missing verification in pipeline mode.
  4. Pivot to verification: check which mode is actually running in production. The message is structured as a conclusion followed by an action. The "So there are THREE modes" line is a summary of the assistant's current understanding, delivered with the confidence of someone who has just connected the dots. The bold emphasis on "which mode is the user's cuzk running in?" shows the assistant prioritizing the decisive question. Notably, the assistant does not jump to a fix yet. It resists the temptation to immediately patch the code. Instead, it first verifies the hypothesis by checking the production configuration. This is disciplined debugging: confirm the theory before applying the remedy.

Mistakes and Correct Assumptions

The assistant's analysis in this message is accurate based on the code examined. However, there is one subtlety worth noting: the assistant assumes that the monolithic path's self-check is comprehensive. While seal::seal_commit_phase2 does call verify_porep_proof(), this verification itself could have false negatives (rejecting valid proofs) or false positives (accepting invalid proofs) if the verification logic has bugs. The assistant does not examine the verification implementation in detail, trusting that it correctly validates Groth16 proofs.

Additionally, the assistant's assumption that pipeline mode is "the default for production" may not hold for all deployments. The slot_size defaults to 0, which would select the monolithic path. But partition_workers may default to a positive value, selecting Phase 7 mode. The assistant's next action — checking the config — will resolve this ambiguity.

Conclusion

Message <msg id=1819> is the turning point in a complex debugging session. It transforms a confusing, intermittent failure pattern into a clear structural explanation: the proving engine has three modes, two of which lack a critical safety check. The message demonstrates how systematic code analysis, combined with a clear mental model of the system architecture, can isolate a bug even when the root cause is not a single line of incorrect code but a missing feature in an alternative code path. The fix that follows — adding mandatory self-verification to all pipeline paths — is a direct consequence of the insight captured in this message.