The 196-Input Mystery: Tracing a WindowPoSt PCE Crash Through Filecoin's Proof Architecture

In the middle of a complex debugging session spanning a high-performance zero-knowledge proving engine, message [msg 119] marks a critical pivot point. The assistant, having just implemented Pre-Compiled Constraint Evaluator (PCE) extraction for all proof types in the CuZK system, is now confronting an unexpected crash: WindowPoSt proving fails because the witness has 26036 inputs while the PCE expects only 25840. The difference is exactly 196 inputs, and this message captures the assistant's methodical effort to understand why.

The Context of the Crash

To understand the stakes, we need to step back. The CuZK proving engine is a GPU-accelerated zero-knowledge proof system for Filecoin, designed to dramatically reduce proving time by pre-compiling circuit constraints. The PCE approach works by extracting the fixed circuit topology (the "skeleton" of constraints) during an initial extraction phase, then reusing that skeleton across many proofs with different witness values. This avoids re-synthesizing the entire circuit for every proof, saving precious GPU time.

The assistant had successfully implemented PCE extraction for WinningPoSt, WindowPoSt, and SnapDeals proof types, and even added a partitioned pipeline for SnapDeals to overlap synthesis and GPU proving. Everything compiled cleanly. But when the user tested WindowPoSt with PCE enabled, the system crashed with a stark error: the witness had 26036 inputs, but the PCE expected 25840. The proving engine could not reconcile the two numbers.

The Message: A Detective's Notebook

Message [msg 119] reads like a detective's notebook mid-investigation. The assistant opens with a definitive fact:

WINDOW_POST_SECTOR_COUNT for 32 GiB = 2349. That's the constant sector_count for the circuit.

This is the first major piece of evidence. The assistant has traced through the Filecoin proof registry code to discover that WindowPoSt for 32 GiB sectors uses exactly 2349 sectors per circuit. This constant comes from filecoin-proofs-api-19.0.0/src/registry.rs, where WINDOW_POST_SECTOR_COUNT is stored in a RwLock<HashMap<u64, usize>> — meaning it's configurable at runtime, but for the 32 GiB variant it defaults to 2349.

The assistant then reasons through the implications:

So for WindowPoSt 32G, the circuit always has 2349 sectors. Both proofs should produce exactly the same dimensions. But the first one had num_inputs=25840 and the second had num_inputs=26036.

This is the core contradiction. If the circuit is structurally fixed at 2349 sectors, then every proof for the same proof type should have the same number of inputs. Yet the logs show two different counts. The assistant is wrestling with the tension between what the code should produce and what the runtime actually produces.

Tracing the Extraction Path

The assistant then walks through the extraction function's logic in detail. The extraction receives num_sectors=102 vanilla proofs — these are the actual sectors being proven in this particular WindowPoSt window. It builds pub_inputs.sectors with 102 entries. Then CompoundProof::circuit() calls pub_inputs.sectors.chunks(2349).nth(0), which returns... all 102 sectors, since 102 is less than 2349. The circuit then pads internally to reach the full 2349 sector count.

The critical observation here is that the extraction path and the synthesis path both use the same CompoundProof::circuit() method, so they should produce identical circuit structures. The assistant has already verified this in earlier messages ([msg 99]) by comparing the extraction and synthesis functions side by side.

The 196-Input Gap

The assistant then focuses on the numerical difference:

But what if the second request has a different number of sectors? Maybe 103? Let me check: 26036 - 25840 = 196. Per the analysis, each sector contributes 1 comm_r input + challenge_count * packed_inputs. What's challenge_count?

The assistant executes a bash grep to find WINDOW_POST_CHALLENGE_COUNT and discovers it equals 10. This is a crucial piece of data: each sector in the WindowPoSt circuit involves 10 challenges.

The hypothesis forming here is that the second request might have a different number of sectors (perhaps 103 instead of 102), which would change the number of alloc_input() calls during circuit synthesis. But the assistant has already established that the circuit pads to 2349 sectors regardless of how many actual sectors are provided — the padding logic repeats the last sector's data to fill the fixed circuit size. So the number of sectors in the request shouldn't affect the input count at all.

What the Message Reveals About the Debugging Process

This message is a beautiful example of systematic debugging under uncertainty. The assistant is working through multiple layers of abstraction:

  1. Log analysis: The crash logs show num_inputs=25840 for the PCE and num_inputs=26036 for the witness. The difference is 196.
  2. Code archaeology: The assistant traces through extract_and_cache_pce_from_window_post, synthesize_window_post, FallbackPoStCompound::setup, window_post_setup_params, and the RegisteredPoStProof::sector_count() method — spanning four different crates and dozens of files.
  3. Constant discovery: The assistant uncovers that WINDOW_POST_SECTOR_COUNT is stored in a runtime-configurable RwLock<HashMap<u64, usize>>, meaning the sector count could theoretically change between requests if the global state is modified.
  4. Hypothesis testing: The assistant considers whether different randomness seeds could lead to different challenged sectors, whether the number of vanilla proofs could vary between requests, and whether the circuit padding logic might behave differently in different scenarios.

Assumptions and Their Limits

The assistant makes several assumptions in this message, some of which will later prove incorrect:

That the circuit dimensions are truly fixed. The assistant states "the circuit always has 2349 sectors" and "both proofs should produce exactly the same dimensions." This is correct in theory but misses the real issue: the circuit structure is fixed, but the synthesis path can diverge based on the is_extensible() flag of the constraint system being used.

That the sector count is the likely culprit. The assistant spends considerable energy exploring whether different requests could have different sector counts. This turns out to be a red herring — the real cause is a mismatch in how RecordingCS and WitnessCS handle the is_extensible() trait, which causes the FallbackPoSt circuit to take different synthesis paths (one allocating extra "temp ONE" inputs).

That both requests use the same partition index. The assistant notes that both request IDs share the pattern wdpost-180965-0-, suggesting the same partition. This is correct and helps rule out partition-related differences.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message creates several valuable pieces of knowledge:

  1. The specific constants for WindowPoSt 32 GiB: WINDOW_POST_SECTOR_COUNT = 2349 and WINDOW_POST_CHALLENGE_COUNT = 10. These are the building blocks for understanding the circuit dimensions.
  2. The 196-input gap: The difference between the two input counts is exactly 196, which will later be traced to the number of synthesis CPU threads (each allocating a "temp ONE" input in the extensible path).
  3. The runtime-configurable nature of WINDOW_POST_SECTOR_COUNT: This is stored in a global RwLock<HashMap>, meaning it's not a compile-time constant but can be modified at runtime. This is an important architectural detail for anyone working with the Filecoin proof system.
  4. Confirmation that the extraction and synthesis paths use the same circuit construction logic: The assistant verifies that both paths call the same CompoundProof::circuit() method, ruling out a simple code-path divergence as the cause.

The Thinking Process

The assistant's reasoning in this message is notably transparent and iterative. It moves from certainty ("the circuit always has 2349 sectors") to doubt ("Wait — the logs say num_sectors=102") to refined analysis ("Actually, look at the extraction function more carefully"). This back-and-forth between what the code says and what the runtime produces is the essence of debugging.

The assistant also demonstrates excellent use of the available tools: reading files, grepping for constants, tracing function calls across crate boundaries. Each piece of evidence is checked against the running hypothesis, and when evidence contradicts the hypothesis (as when the assistant realizes both extraction and synthesis use the same chunking logic), the hypothesis is revised.

Why This Message Matters

Message [msg 119] is the turning point in the debugging session. It's the moment when the assistant has gathered enough data to realize that the simple explanations (different sector counts, different partitions, different randomness) don't fit. The 196-input gap remains unexplained by the code paths examined so far.

This frustration — the gap between what should happen and what does happen — is what drives the assistant to look deeper. In the subsequent messages, the assistant will discover that the real cause is the is_extensible() flag mismatch between RecordingCS and WitnessCS, a subtle trait implementation difference that causes the FallbackPoSt circuit to take entirely different synthesis paths depending on which constraint system is used.

The 196 inputs, it turns out, correspond exactly to the number of synthesis CPU threads (configured to 196), each allocating a "temp ONE" input in the extensible path. The assistant's careful documentation of the numerical difference in this message — 26036 - 25840 = 196 — will prove to be the exact fingerprint of the bug.

In the broader narrative of the coding session, this message represents the transition from implementation to debugging. The assistant has built something new (PCE extraction for all proof types) and is now confronting the inevitable consequences of extending a complex system. The message captures the intellectual honesty required for this work: stating what you know, acknowledging what you don't, and methodically gathering the evidence that will lead to the truth.