The Phantom Input: Tracing a 196-Input Mismatch Through Three Constraint Systems

Introduction

In the high-stakes world of zero-knowledge proof generation, where circuits routinely span hundreds of millions of constraints and gigabytes of parameter data, the smallest structural inconsistency can bring the entire proving pipeline to a halt. This article examines a single user message from an opencode coding session—message index 185—that captures a pivotal moment in a debugging odyssey spanning multiple constraint system implementations. The message is deceptively simple: a user reports "Same still, did rebuild; At least no crash" and pastes the logs from a run of the CuZK proving daemon. But beneath this laconic report lies a rich story of a subtle bug, a partial fix, and the revelation of a deeper inconsistency that would require harmonizing three separate constraint system types across two codebases.

The message arrives at a critical juncture. The assistant has just implemented Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types—WinningPoSt, WindowPoSt, and SnapDeals—in the CuZK proving engine. The PCE is an optimization that pre-computes the R1CS constraint matrices (A, B, C) for a circuit once, then reuses them across many proofs, skipping the expensive constraint-building phase during synthesis. However, enabling PCE for WindowPoSt caused a crash due to a mismatch in input counts between the witness and the PCE. The assistant traced the root cause to the is_extensible() flag: RecordingCS (used for PCE extraction) returned false while WitnessCS (used for witness generation) returned true, causing different synthesis paths. The fix made RecordingCS extensible by implementing is_extensible() and extend() methods, and corrected its initialization to pre-allocate a ONE input, ensuring structural parity.

But as the user's message reveals, this fix was only half the story.

What the Message Shows

The user runs the CuZK daemon with CUZK_VALIDATE_PARTITIONS=1 environment variable set, pointing it at a Unix socket and a configuration file. The daemon starts up, preloads the SRS (Structured Reference String) for the PoRep-32G circuit from disk—a 44 GiB file that takes about 15 seconds to load. It then attempts to load PCE files from disk for all four circuit types: PoRep-32G, WinningPoSt-32G, WindowPoSt-32G, and SnapDeals-32G. Only the PoRep PCE is found and loaded successfully (a 25.7 GiB file with 328 inputs, 130 million aux variables, and 130 million constraints). The other three PCE files are not found, which is expected—they haven't been extracted yet.

The daemon then initializes GPU workers (1 GPU, 2 workers per device), starts the pipeline with 16 partition workers, and begins listening on the Unix socket. Everything looks normal.

Then a WindowPoSt prove request arrives: wdpost-180965-0-[156 40 145 6 58 29 183 77]. The daemon processes it through the standard synthesis path (no PCE yet, since the WindowPoSt PCE hasn't been extracted). The synthesis takes 5.45 seconds, producing a circuit with 25,840 inputs, 126,902,376 aux variables, and 125,305,057 constraints. The proof proceeds to the GPU, taking about 66 seconds total. Crucially, the daemon kicks off background PCE extraction for WindowPoSt after this first proof completes. The PCE extraction runs in parallel with GPU proving and completes successfully, producing a PCE with the same dimensions: 25,840 inputs, 126,902,376 aux, 125,305,057 constraints. This PCE is saved to disk as /data/zk/params/pce-window-post.bin—a 39.9 GiB file.

The first proof completes successfully. So far, so good.

Then a second WindowPoSt prove request arrives: wdpost-180965-0-[41 19 58 78 78 105 1 35]. This time, the daemon detects that the PCE is available on disk and takes the PCE fast path for synthesis. The log line reads: using PCE fast path for synthesis circuit_id=wpost-32g. The PCE witness generation completes in about 3 seconds, using WitnessCS to run only the allocation closures (skipping constraint enforcement). But then disaster strikes:

thread 'tokio-runtime-worker' panicked at /tmp/czk/extern/cuzk/cuzk-pce/src/eval.rs:131:5:
assertion `left == right` failed: input_assignment length mismatch: got 26036, expected 25840
  left: 26036
 right: 25840

The witness generation produced 26,036 inputs, but the PCE expects 25,840. The difference is exactly 196—the number of parallel chunks used in WindowPoSt's synthesize_extendable path. The daemon retries the proof (the log shows a second attempt with the same panic), but it fails again identically.

Why the First Proof Succeeds but the Second Fails

This is the central puzzle of the message. The first proof goes through the standard synthesis path using ProvingAssignment, which produces 25,840 inputs. The background PCE extraction uses RecordingCS, which (after the first fix) also produces 25,840 inputs. Everything matches.

The second proof tries to use the PCE fast path. This path uses WitnessCS to generate the witness assignments (the actual field element values for each variable) by running only the alloc() closures from the circuit, skipping the expensive enforce() calls. The PCE provides the pre-computed constraint matrices, so the witness just needs to fill in the variable assignments. But WitnessCS produces 26,036 inputs—196 more than the PCE expects.

The key insight: the first fix only addressed RecordingCS (the PCE extraction side). It made RecordingCS::new() start with zero inputs and had extract_precompiled_circuit explicitly allocate the ONE input before synthesis, matching ProvingAssignment's behavior. But the witness generation path uses WitnessCS, which was not fixed. WitnessCS::new() still pre-allocated the ONE input at index 0, setting input_assignment = [ONE] with length 1. This structural difference between WitnessCS and ProvingAssignment caused the 196-input mismatch in the synthesize_extendable parallel chunk path.

The Deeper Root Cause Revealed

The message reveals that the problem is not in the PCE extraction at all—it's in the witness generation path. The PCE now correctly captures the circuit structure with 25,840 inputs, matching the standard prover. But the witness generator (WitnessCS) produces a different number of inputs because its new() constructor behaves differently from ProvingAssignment::new().

To understand why, we need to trace through the synthesize_extendable path that WindowPoSt uses. WindowPoSt processes 102 sectors in parallel across 196 chunks (the exact number comes from the parallel decomposition of the FallbackPoSt circuit). Each chunk is synthesized by a child constraint system created via CS::new(). The flow is:

  1. Parent CS is created (either ProvingAssignment for the standard prover, or WitnessCS for the PCE witness path).
  2. The parent calls alloc_input("", ONE) to allocate the constant ONE input at index 0.
  3. The parent calls circuit.synthesize(), which detects is_extensible() = true and takes the synthesize_extendable path.
  4. For each of 196 chunks, a child CS is created via CS::new().
  5. Each child calls alloc_input("temp ONE", ONE) to allocate its own ONE input.
  6. Each child synthesizes its sector, adding more inputs.
  7. The parent calls extend(child), which appends the child's inputs (skipping index 0, which is the child's ONE that maps to the parent's ONE at index 0). The critical difference is in step 4. For ProvingAssignment::new(), the child starts with 0 inputs. After step 5, it has 1 input (the temp ONE at index 0). After step 6, it has 1 + N inputs (temp ONE + sector inputs). In step 7, extend() skips index 0, so only the N sector inputs are appended. The temp ONE at index 0 is correctly mapped to the parent's ONE. For WitnessCS::new() (before the fix), the child starts with 1 input (the pre-allocated ONE at index 0). After step 5, it has 2 inputs (pre-allocated ONE at index 0, temp ONE at index 1). After step 6, it has 2 + N inputs. In step 7, extend() skips index 0 (the pre-allocated ONE), but the temp ONE at index 1 survives as an extra input. With 196 chunks, this produces 196 extra inputs—exactly the mismatch seen in the logs.

Assumptions Made and Mistakes

The first fix operated under a key assumption: that making RecordingCS behave like ProvingAssignment would be sufficient. The assistant assumed that RecordingCS was the only constraint system type that needed fixing, because the crash occurred when the PCE (extracted by RecordingCS) was compared against the witness (generated by... well, that's the question). The crash in eval.rs:131 compared the PCE's expected input count against the witness's actual input count. Since the PCE was the new component, it seemed natural that the PCE extraction was the culprit.

But the message reveals a more nuanced picture. The first proof (standard path) and the PCE extraction both produce 25,840 inputs. The mismatch is between the PCE (25,840) and the witness generation (26,036). The witness generation uses WitnessCS, which was never modified—it still had the old pre-allocating new() constructor.

The assistant's mistake was a form of premature localization: assuming that because the crash message mentioned the PCE (eval.rs is in the cuzk-pce crate), the bug must be in the PCE extraction code. In reality, the crash was in code that compares the PCE against the witness, and either side could be wrong. The first fix corrected the PCE side, but the witness side remained broken.

There was also an assumption about the relationship between RecordingCS and WitnessCS. The assistant initially thought that making RecordingCS extensible and matching its new() behavior to WitnessCS would fix the problem. But the actual goal should have been to match ProvingAssignment, not WitnessCS. The standard prover uses ProvingAssignment, and both the PCE extraction and the witness generation need to produce the same structure as the standard prover. WitnessCS itself was inconsistent with ProvingAssignment.

Input Knowledge Required to Understand This Message

To fully grasp this message, one needs:

  1. Understanding of the CuZK architecture: The daemon uses a pipeline with synthesis (circuit building) and GPU proving (MSM/NTT operations). PCE is a caching layer that pre-computes constraint matrices.
  2. Knowledge of bellperson's constraint system hierarchy: ProvingAssignment (used by the standard prover), WitnessCS (used for fast witness generation), and RecordingCS (used for PCE extraction) all implement the ConstraintSystem trait but have different new() constructors.
  3. Understanding of extensible synthesis: Circuits that implement synthesize_extendable create child constraint systems for parallel processing, then merge them via extend(). The extend() method skips the first input (the constant ONE) from each child, assuming it maps to the parent's ONE.
  4. Familiarity with the FallbackPoSt circuit structure: WindowPoSt uses 196 parallel chunks to process 102 sectors, which is why the mismatch is exactly 196.
  5. Knowledge of R1CS conventions: The constant ONE is always allocated at input index 0 in Groth16 proving systems. This is a universal convention across all constraint system implementations.
  6. Understanding of the PCE fast path: The PCE witness generation uses WitnessCS to run only alloc() closures (skipping enforce()), then evaluates the pre-computed constraint matrices via sparse MatVec multiplication. This is the "fast path" that avoids expensive LinearCombination evaluation.

Output Knowledge Created by This Message

This message creates several critical pieces of knowledge:

  1. The first fix was incomplete: The RecordingCS fix corrected the PCE extraction side, but the witness generation side (WitnessCS) remained broken. The symptom shifted from a PCE extraction crash to a witness generation assertion failure.
  2. The bug is symmetric: Both RecordingCS and WitnessCS had the same pre-allocation issue. The fix needs to be applied to both, plus the caller code that creates the initial CS before synthesis.
  3. The correct reference is ProvingAssignment: All three CS types (ProvingAssignment, WitnessCS, RecordingCS) need to have consistent new() behavior. The standard prover's ProvingAssignment is the reference implementation—it starts with zero inputs and relies on the caller to explicitly allocate ONE.
  4. The mismatch is deterministic and predictable: The 196-input difference is exactly the number of parallel chunks, confirming the root cause analysis. Each chunk contributes exactly one extra input due to the pre-allocated ONE surviving the extend() call.
  5. The PCE extraction itself is now correct: The background PCE extraction produced 25,840 inputs, matching the standard prover. The PCE file saved to disk has the correct dimensions.
  6. The fix requires changes in three places: RecordingCS::new() (already fixed), WitnessCS::new() (needs fixing), and the caller code in the PCE witness path (needs to explicitly allocate ONE before synthesis).

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, shows a systematic debugging approach. After the user reports the crash, the assistant:

  1. Verifies the PCE extraction is correct: The log shows RecordingCS: extracted pre-compiled circuit num_inputs=25840, confirming the PCE side produces the right count.
  2. Identifies the witness side as the problem: The assertion failure shows got 26036, expected 25840. The "got" value (26,036) is from WitnessCS, while the "expected" value (25,840) is from the PCE.
  3. Traces through the synthesize_extendable path: The assistant mentally simulates the child CS creation and extension process for both ProvingAssignment and WitnessCS, identifying the exact point where the extra input is introduced.
  4. Discovers the root cause in WitnessCS::new(): The assistant finds that WitnessCS::new() pre-allocates [ONE] in input_assignment, while ProvingAssignment::new() starts empty. This is the source of the extra input per chunk.
  5. Considers multiple fix options: The assistant evaluates several approaches—adding a new_empty() method, changing WitnessCS::new() directly, or using ProvingAssignment instead of WitnessCS for witness generation—and chooses the most principled one: make WitnessCS::new() match ProvingAssignment::new().
  6. Checks for side effects: The assistant verifies that WitnessCS::new() is only called from the PCE witness path (not from the standard prover), and that the generate_witness_into_cs method is unaffected.
  7. Verifies the fix compiles: The assistant runs cargo check and confirms clean compilation. The reasoning shows a deep understanding of the constraint system architecture and the ability to trace through complex parallel execution paths mentally. The assistant doesn't just look at the error message—it reconstructs the entire execution flow from first principles, comparing the behavior of three different constraint system implementations.

The Broader Significance

This message is a textbook example of a partial fix revealing a deeper bug. The first fix (making RecordingCS extensible and matching its new() to ProvingAssignment) was necessary but not sufficient. It fixed the PCE extraction side, but the witness generation side had the same underlying issue. The crash simply moved from one location to another—from the PCE extraction code to the witness comparison code.

The pattern is common in complex systems: a bug manifests at a boundary between two components, and fixing one component shifts the symptom to the other component. The true fix requires harmonizing both sides of the boundary.

The message also illustrates the importance of having a canonical reference implementation. In this case, ProvingAssignment is the reference—it's what the standard prover uses, and it has the correct behavior. Both RecordingCS and WitnessCS need to match it. The assistant's initial mistake was trying to make RecordingCS match WitnessCS (since WitnessCS was the existing extensible CS), when it should have been matching ProvingAssignment all along.

Conclusion

The user's message at index 185 captures a pivotal debugging moment. The first WindowPoSt proof succeeds, the PCE is extracted correctly with 25,840 inputs, and everything looks promising. But the second proof crashes with a 196-input mismatch, revealing that the witness generation path (WitnessCS) has the same pre-allocation bug that was just fixed in the PCE extraction path (RecordingCS).

The message is a testament to the complexity of modern zero-knowledge proving systems, where a single design decision—whether to pre-allocate the ONE input in a constructor—can ripple through hundreds of millions of constraints and manifest as a hard-to-diagnose assertion failure. The fix requires harmonizing three constraint system types across two codebases, ensuring that ProvingAssignment, WitnessCS, and RecordingCS all agree on the fundamental question: who is responsible for allocating the ONE input?

The answer, as the assistant ultimately determines, is that the caller is responsible. The CS constructor should start with zero inputs, and the caller (whether it's the bellperson prover, the PCE extractor, or the PCE witness generator) should explicitly allocate ONE before synthesis. This simple convention, consistently applied across all three implementations, resolves the 196-input mismatch and restores correct proving for WindowPoSt with PCE enabled.

The message also serves as a cautionary tale about premature localization and the dangers of assuming that the component where a crash occurs is the component that needs fixing. In complex systems, the crash site is often just the tip of the iceberg—the place where an inconsistency between two components becomes visible. The real work is tracing back through the execution flow to find the root cause, which may be far from the crash location and may require changes in multiple components.