When the PCE Doesn't Fit: Debugging a 196-Input Mismatch in WindowPoSt Proving

Introduction

In the high-stakes world of zero-knowledge proof generation for Filecoin, every microsecond counts. The CuZK proving engine, a GPU-accelerated prover, had recently gained a powerful optimization: Pre-Compiled Constraint Evaluator (PCE) extraction, which captures the fixed R1CS structure of a circuit once and reuses it across all subsequent proofs of the same type, eliminating redundant constraint system construction. After successfully implementing PCE extraction for WinningPoSt, WindowPoSt, and SnapDeals proof types, and adding a partitioned pipeline for SnapDeals that promised ~43% wall-clock improvement, the system was deployed for testing. What came back was a crash — and a debugging journey that would reveal a subtle but critical mismatch between two constraint system implementations.

The subject message, message index 79 in the conversation, is a user-submitted log dump showing exactly this crash. It is the raw evidence of a bug: the Pre-Compiled Constraint Evaluator extracted from a WindowPoSt proof had 25,840 inputs, but the witness produced during fast synthesis had 26,036 inputs — a difference of exactly 196. The assertion in eval.rs:131 failed, the thread panicked, and the job was retried four times, crashing identically each time. This article dissects that single message: why it was written, what it reveals, the assumptions it challenged, and the thinking process it triggered.

The Message: Raw Evidence of a Crash

The message is a log transcript spanning approximately four minutes of real time. It begins with a successful WindowPoSt proof:

2026-03-01T18:40:48.264659Z  INFO cuzk_core::scheduler: job enqueued job_id=wdpost-180965-0-[80 241 31 80 18 235 210 8] proof_kind=window-post priority=High queue_position=0

This first proof, with job ID wdpost-180965-0-[80 241 31 80 18 235 210 8], proceeds through synthesis using the standard path (no PCE yet — it's the first proof of this type, so no cached PCE exists). The log shows num_sectors=102, synthesis taking 5.6 seconds, and the GPU proving taking 64 seconds. After the proof completes, a background thread begins PCE extraction:

2026-03-01T18:41:13.042042Z  INFO cuzk_core::engine: background PCE extraction starting for WindowPoSt

Three minutes later, the PCE extraction completes successfully:

2026-03-01T18:44:26.233056Z  INFO cuzk_pce::recording_cs: RecordingCS: extracted pre-compiled circuit num_inputs=25840 num_aux=126902376 num_constraints=125305057 a_nnz=862416559 b_nnz=127398015 c_nnz=157401792

The PCE is saved to disk at /data/zk/params/pce-window-post.bin — a 39.9 GiB file representing the complete R1CS structure of the WindowPoSt circuit for 32 GiB sectors with 102 sectors.

Then a second request arrives:

2026-03-01T18:45:17.565460Z  INFO cuzk_server::service: Prove request_id=wdpost-180965-0-[25 25 84 229 182 218 78 102] proof_kind=3 input_size=0

This time, the synthesis path detects the cached PCE and takes the fast path:

2026-03-01T18:45:17.615040Z  INFO synthesize_window_post{job_id="wdpost-180965-0-[25 25 84 229 182 218 78 102]" partition=0}: cuzk_core::pipeline: using PCE fast path for synthesis circuit_id=wpost-32g

Witness generation completes in 3.1 seconds, and then the crash:

thread 'tokio-runtime-worker' panicked at /tmp/czk/extern/cuzk/cuzk-pce/src/eval.rs:131:5:
assertion `left == right` failed: input_assignment length mismatch: got 26036, expected 25840
  left: 26036
 right: 25840

The job is retried three more times, each time crashing with the exact same assertion failure. The error propagates up as a panicked task, logged as synthesis task panicked. The system retries automatically, but the bug is deterministic — every attempt produces 26,036 witness inputs against a PCE expecting 25,840.

Why This Message Was Written

This message was written by the user — the person testing the CuZK deployment — to report a critical failure. The assistant had just completed a substantial implementation effort spanning multiple rounds: adding PCE extraction functions for all three proof types (WinningPoSt, WindowPoSt, SnapDeals), wiring them into the engine's monolithic path, and implementing a partitioned pipeline for SnapDeals to overlap synthesis with GPU proving. The changes compiled cleanly and were deployed.

The user's test was straightforward: send a WindowPoSt proof request, observe whether PCE acceleration works. The first request succeeded and produced a PCE. The second request, which should have used the cached PCE for a ~3-5x faster synthesis, crashed. The user captured the logs and submitted them as-is — a raw, unfiltered view of the failure.

The message serves multiple purposes. First, it provides undeniable evidence of a bug: the assertion failure is explicit, the numbers are concrete (26,036 vs 25,840), and the reproducibility (four identical crashes) rules out transient issues. Second, it contains rich contextual information: the job IDs, the synthesis path taken ("using PCE fast path"), the timing data, and the PCE extraction summary. Third, it implicitly defines the scope of the investigation: the crash occurs in evaluate_pce() at the input assignment length check, meaning the witness produced by WitnessCS has more inputs than the PCE recorded.

Context and Motivation

To understand this message fully, one must understand what preceded it. The assistant had been working on extending the CuZK proving engine's PCE extraction — a technique where the constraint system structure (the A, B, C sparse matrices of the R1CS) is recorded once using a RecordingCS implementation and then reused for all subsequent proofs of the same circuit type. This avoids the overhead of rebuilding the constraint structure during every synthesis, which is especially valuable for GPU-resident proving where the matrices must be transferred to the device.

The assistant had already implemented PCE extraction for PoRep (the original proof type). The task was to extend this to WinningPoSt, WindowPoSt, and SnapDeals. Additionally, the user had requested a partitioned pipeline for SnapDeals, mirroring the PoRep architecture where synthesis of partition N+1 overlaps with GPU proving of partition N, reducing wall-clock time.

The implementation involved:

1. Adding extract_and_cache_pce_from_winning_post(), extract_and_cache_pce_from_window_post(), and extract_and_cache_pce_from_snap_deals() functions in pipeline.rs.
2. Wiring all four proof kinds into the engine's monolithic path with background PCE extraction threads.
3. Implementing a ParsedSnapDealsInput struct and associated functions for the partitioned SnapDeals pipeline.
4. Generalizing PartitionWorkItem with a ParsedProofInput enum to support both PoRep and SnapDeals partitioned data. The changes compiled cleanly and were deployed. The user then tested WindowPoSt, and this message is the result.

The Numbers: 25,840 vs 26,036

The difference of 196 inputs is the central mystery of this message. The PCE was extracted from a WindowPoSt proof with 102 sectors and recorded 25,840 inputs. The subsequent proof, also with 102 sectors (same partition), produced a witness with 26,036 inputs. The circuit dimensions for a given proof type should be fixed — the R1CS structure is determined by the circuit topology, not by the specific input values. So why the discrepancy?

The assistant's initial hypothesis, visible in the subsequent message (message 80), was that WindowPoSt circuit dimensions might vary based on the number of sectors in the partition:

"This is a real bug. The PCE was extracted from a WindowPoSt proof with 102 sectors (25,840 inputs), but the next proof has a different number of sectors (26,036 inputs). Unlike PoRep where the circuit structure is truly fixed, WindowPoSt's R1CS dimensions vary depending on how many sectors are in the partition."

This assumption was reasonable at first glance. The WindowPoSt circuit is built from sector proofs — each sector contributes a set of constraints and inputs. If the number of sectors changes, the circuit size changes. But the user quickly corrected this assumption (message 85):

"Note: this was same partition, no change expected"

And again (message 120):

"It's not different count of sectors, r1cs can't just morph shape so inputs have to be the same"

This correction was crucial. The R1CS structure is indeed fixed for a given proof type and partition configuration. The sector_count for WindowPoSt 32 GiB is a constant 2,349 (defined in WINDOW_POST_SECTOR_COUNT), and the circuit always pads to this number regardless of how many actual sectors are in the request. The circuit's synthesize() method always makes the same number of alloc_input() calls. So the 196-input difference could not come from variable sector counts — it had to come from something else.

The Root Cause: A Tale of Two Constraint Systems

The assistant's investigation, spanning messages 80 through 133, traced the root cause to a fundamental mismatch between two implementations of the ConstraintSystem trait: RecordingCS (used for PCE extraction) and WitnessCS (used for fast witness generation during PCE-accelerated synthesis).

The ConstraintSystem trait in bellpepper-core defines a method is_extensible():

fn is_extensible() -> bool {
    false  // default
}

WitnessCS overrides this to return true:

fn is_extensible() -> bool {
    true
}

RecordingCS does not override it, so it inherits the default false.

The FallbackPoSt circuit (the circuit used for WindowPoSt) checks this flag and dispatches to different synthesis paths:

• synthesize_default (when is_extensible() is false): Runs sequentially, processing all sectors in a single thread. Each sector's inputs are allocated directly into the parent constraint system.
• synthesize_extendable (when is_extensible() is true): Splits sectors into parallel chunks, each running on a separate thread. Each chunk creates its own child constraint system via CS::new(), allocates a "temp ONE" input (one extra input per chunk), synthesizes its sectors, and then the child is merged into the parent via extend(). The extend() method in WitnessCS skips the first input (the built-in ONE variable) of the child and appends the rest, including the "temp ONE". This means each parallel chunk adds exactly one extra public input to the final witness — the "temp ONE" that was allocated at index 1 of the child CS. The number of chunks is determined by SETTINGS.window_post_synthesis_num_cpus. With 196 chunks (the configured number of synthesis CPUs), the synthesize_extendable path produces 196 more inputs than the synthesize_default path. So the chain of causation is: 1. RecordingCS::is_extensible() returns false → circuit takes synthesize_default path → PCE records 25,840 inputs. 2. WitnessCS::is_extensible() returns true → circuit takes synthesize_extendable path → witness has 25,840 + 196 = 26,036 inputs. 3. evaluate_pce() asserts that the witness input count matches the PCE input count → assertion fails → crash. The difference of 196 is not a coincidence — it's exactly the number of parallel chunks, each contributing one "temp ONE" input.

Input Knowledge Required

To fully understand this message and the subsequent investigation, several pieces of domain knowledge are essential:

R1CS Structure: Rank-1 Constraint Satisfaction Systems have three components: public inputs (known to both prover and verifier), private auxiliary inputs (known only to the prover), and constraints (equations relating inputs and aux variables). The number of inputs is determined by the circuit topology, not the specific values.

PCE (Pre-Compiled Constraint Evaluator): An optimization that records the sparse matrix structure (A, B, C) of an R1CS once and reuses it. During synthesis, only the witness values need to be computed; the matrix structure is loaded from the cached PCE. This avoids rebuilding the CSR matrices every time.

RecordingCS vs WitnessCS: Two implementations of the ConstraintSystem trait. RecordingCS captures the constraint structure into CSR matrices for later reuse. WitnessCS only tracks witness values (input and auxiliary assignments) and ignores constraint structure (its enforce() is a no-op). Both implement alloc_input() and alloc_aux() to track variable counts.

The is_extensible() Flag: A trait method that signals whether the constraint system supports parallel synthesis via chunking and merging. When true, the circuit dispatches to synthesize_extendable, which creates child CS instances for each chunk, allocates a "temp ONE" input in each, and merges them back via extend(). The "temp ONE" is necessary because each child CS needs a ONE variable for linear combination construction, but the parent already has one — so the child's ONE is skipped during merge, but the "temp ONE" (allocated explicitly) becomes a real input.

FallbackPoSt Circuit: The WindowPoSt circuit implementation in storage-proofs-post. It iterates over sectors, each of which allocates inputs for comm_r, comm_c, comm_r_last, and Merkle paths. The number of sectors is fixed by WINDOW_POST_SECTOR_COUNT (2,349 for 32 GiB), but the actual number of vanilla proofs provided may be smaller — the circuit pads by repeating the last sector.

CSR Matrix Encoding: RecordingCS stores constraints in Compressed Sparse Row format. Column indices use tagged encoding where bit 31 distinguishes input variables from aux variables. During extend(), column indices from the child CS must be remapped to account for the parent's existing variable ranges.

Output Knowledge Created

This message, combined with the subsequent investigation, produced several important pieces of knowledge:

Bug Confirmation: The PCE optimization for WindowPoSt is broken due to a structural mismatch between the extraction path and the fast synthesis path. The bug is deterministic and reproducible.

Root Cause: The is_extensible() flag divergence between RecordingCS (false) and WitnessCS (true) causes the FallbackPoSt circuit to take different synthesis paths, producing different input counts.

Quantitative Impact: The difference is exactly 196 inputs, matching the configured number of synthesis CPUs. Each parallel chunk adds one "temp ONE" input.

Fix Strategy: Make RecordingCS extensible by implementing is_extensible() (returning true) and extend() (with proper variable index remapping). Additionally, align RecordingCS::new() with WitnessCS::new() by pre-allocating a ONE input, and update extract_precompiled_circuit() to avoid double-allocation.

Architectural Insight: The PCE optimization is more fragile than initially assumed. It requires strict structural parity between the constraint system used for extraction and the one used for fast proving. Any divergence in synthesis paths — even one as subtle as a parallelization flag — can cause silent corruption that only manifests as an assertion failure at evaluation time.

The Thinking Process

The assistant's reasoning, visible in the messages following the bug report, demonstrates a methodical debugging approach. The initial hypothesis (variable sector counts) was reasonable but quickly disproven by the user's correction. The assistant then pivoted to examining the actual code paths.

The key insight came from reading the WitnessCS implementation and discovering is_extensible() returns true. This led to checking RecordingCS — which doesn't implement the method, getting the default false. The assistant then traced through the FallbackPoSt circuit's synthesize() to confirm the dispatch logic, and verified that synthesize_extendable creates num_chunks child CS instances, each allocating a "temp ONE" input.

The number 196 was the smoking gun. It matched the configured CPU count for parallel synthesis, confirming that the extra inputs came from the parallelization chunks rather than from any circuit variability.

The fix involved several subtle considerations:

1. Variable Index Remapping: When extending a child RecordingCS into the parent, input indices must be remapped: the child's input 0 (built-in ONE) maps to parent input 0, child input 1 ("temp ONE") maps to parent.num_inputs, child input 2 maps to parent.num_inputs + 1, etc. Aux indices are offset by parent.num_aux.
2. Initialization Parity: WitnessCS::new() pre-allocates input_assignment = vec![Scalar::ONE], making index 0 the built-in ONE. RecordingCS::new() originally started with num_inputs = 0. This caused a mismatch in the child CS: in WitnessCS, "temp ONE" is at index 1; in RecordingCS, it would be at index 0. The fix was to make RecordingCS::new() also pre-allocate a ONE input.
3. Extraction Code Update: The existing extract_precompiled_circuit() function manually called cs.alloc_input(|| "one", || Ok(Scalar::ONE)) after new(). With new() now pre-allocating ONE, this manual call would double-count. The extraction code needed to be updated to remove the redundant allocation. The assistant implemented these changes, compiled cleanly, and confirmed the fix restored parity between the extraction and proving paths.

Broader Implications

This bug and its resolution highlight several important lessons for high-performance zero-knowledge proving systems:

Structural Invariants: When implementing optimizations like PCE that separate circuit structure extraction from witness generation, maintaining strict structural invariants between the two paths is critical. Any divergence — even one as seemingly innocuous as a parallelization flag — can cause hard-to-diagnose failures.

Trait Design: The is_extensible() default of false is a design choice that prioritizes safety for new implementations (they won't accidentally take a parallel path they don't support), but it creates a subtle trap: a constraint system that doesn't implement is_extensible() silently takes a different synthesis path than one that does. Better documentation or a required override could prevent this class of bug.

Testing Strategy: The bug was caught in integration testing (running actual proofs against the deployed system), not in unit tests. The PCE extraction test only verified that extraction succeeded, not that the extracted structure matched what the fast synthesis path would produce. Adding a cross-validation test — extract with RecordingCS, synthesize with WitnessCS, compare input/aux counts — would catch this class of regression.

The Value of Assertions: The assertion in eval.rs:131 that caught this bug (assert_eq!(input_assignment.len(), num_inputs)) is a textbook example of a good defensive check. It caught a structural mismatch that could have caused silent data corruption (wrong witness values mapped to wrong matrix columns) and turned it into a clean, debuggable crash.

Conclusion

Message 79 in this conversation is far more than a simple bug report. It is a snapshot of a complex system at the moment of failure, rich with diagnostic information. The 196-input mismatch between the PCE and the witness was the visible symptom of a deeper structural divergence between two constraint system implementations — a divergence caused by a single boolean flag defaulting to false instead of true.

The debugging journey that followed — from initial hypothesis of variable sector counts, through code inspection of the FallbackPoSt circuit, to the final discovery of the is_extensible() mismatch — demonstrates the importance of understanding the full architecture of a proving system when diagnosing optimization failures. The fix, making RecordingCS fully extensible with proper variable remapping and initialization parity, restored the invariant that the extraction path and the fast synthesis path produce identical circuit structures.

For anyone working on zero-knowledge proving systems, this case study offers a cautionary tale: when you separate circuit structure extraction from witness generation, you must ensure that every trait method, every flag, and every initialization path produces identical behavior. The PCE optimization is powerful — it can reduce synthesis time by 3-5x — but it is only as reliable as the structural parity between the two constraint system implementations that underpin it.