First Light: Validating the cuzk Proving Pipeline with a Real Groth16 Proof

The Message That Changed Everything

At message index 230 in the coding session, the assistant delivered a brief but momentous announcement:

THE PROOF COMPLETED SUCCESSFULLY!

The message then presented a compact table of results:

| Metric | Value | |---|---| | Proof length | 1920 bytes (correct for Groth16 BLS12-381) | | Total prove time | 116.8 seconds (~1 min 57 sec) | | seal_commit_phase2 time | 116.6 seconds | | Queue wait | 28 ms | | SRS load | ~15 seconds (included in prove time) | | Verification | PASSED (seal_commit_phase2 includes verify_seal internally) |

Beneath the table, the assistant noted that the daemon had cached the verifying key and then immediately issued a shell command to check the bench client's output.

On its surface, this is a status update. But in the context of the multi-month cuzk project — a ground-up redesign of Filecoin's proof infrastructure — this message represents the single most important milestone of the entire endeavor: the first real, end-to-end, GPU-accelerated Groth16 proof produced by the new proving daemon, validated against real Filecoin parameters on real hardware.

The Journey to This Point

To understand why this message carries such weight, one must appreciate the scope of what cuzk aims to accomplish. The project, designed across five optimization proposals and a comprehensive architecture document ([msg 203] through [msg 239] trace the immediate context), seeks to replace the existing ad-hoc proof generation pipeline with a persistent, memory-efficient, multi-GPU proving daemon. The existing pipeline, SUPRASEAL_C2, consumed approximately 200 GiB of peak memory for a single 32 GiB sector proof, loaded 45 GiB SRS parameters from disk on every invocation, and had no mechanism for batching, prioritization, or observability.

Phase 0 of cuzk, implemented in segment 4 and validated in segment 5, was deliberately minimal: a gRPC server accepting proof requests, a priority scheduler, a prover module wrapping filecoin-proofs-api calls, and Prometheus metrics. The goal was not performance — it was correctness. Could the new architecture produce a valid Filecoin proof at all?

The answer, delivered in message 230, was a definitive yes.

Anatomy of the Message

The message is structured as a triumphant announcement followed by a data table and a concluding shell command. This structure reveals the assistant's priorities:

  1. Celebrate the milestone — "THE PROOF COMPLETED SUCCESSFULLY!" in bold signals that this is the culmination of a long effort.
  2. Present objective evidence — The table provides six metrics that together prove the pipeline works correctly.
  3. Acknowledge internal verification — Noting that seal_commit_phase2 includes verify_seal internally means the proof was self-validated, not just produced.
  4. Check the client — The shell command jobs -l; wait; fg; ps aux shows the assistant is being thorough, verifying that the client-side also completed cleanly. The table itself is carefully chosen. Each metric addresses a specific risk or question that had been open: - Proof length (1920 bytes): Confirms the Groth16 proof format is correct for BLS12-381. A wrong proof length would indicate a serialization mismatch or wrong curve. - Total prove time (116.8s): Establishes a baseline for future optimization work. This is the "cold start" time that all subsequent improvements will be measured against. - seal_commit_phase2 time (116.6s): Shows that virtually all the time is in the actual proving, not in gRPC overhead or queue management (28 ms queue wait). - SRS load (~15s): Explicitly calls out the parameter loading cost, setting up the comparison for the second proof. - Verification (PASSED): The most critical metric. Without this, none of the other numbers matter.

The Technical Achievement

What makes this message significant is not just that a proof was produced, but how it was produced. The pipeline that succeeded involved:

  1. gRPC transport of 51 MB of C1 data — The C1 output is a JSON structure containing the circuit's public inputs and the proof from the first phase. Transmitting 51 MB over a protobuf-based RPC and deserializing it correctly is non-trivial.
  2. Base64 decoding and nested JSON deserialization — The C1 data is base64-encoded within the outer JSON. The prover had to decode this, then deserialize the inner structure into Filecoin's internal types.
  3. SRS parameter loading from disk — The 45 GiB parameter file was read into the SupraSeal cache in approximately 15 seconds. This required the FIL_PROOFS_PARAMETER_CACHE environment variable to be correctly set and the storage-proofs-core settings system to pick it up via lazy initialization.
  4. Bellperson circuit synthesis with SupraSeal CUDA backend — The message confirms "Bellperson 0.26.0 with SupraSeal is being used!" meaning the GPU-accelerated path was active, running on an RTX 5070 Ti (Blackwell architecture, sm_120, CUDA 13.1).
  5. Internal verification — The proof was verified internally by seal_commit_phase2 before being returned, ensuring that the daemon never returns an invalid proof. The assistant's earlier monitoring (message 227) showed the daemon using 14,253% CPU (approximately 142 cores) and 203 GiB RSS during synthesis. The GPU utilization remained low during this phase, confirming that circuit synthesis is the CPU-bound bottleneck — a finding that directly informed later optimization work.

The SRS Residency Discovery

Although message 230 only reports the first proof, the assistant immediately recognized the significance of the SRS cache hit. The log line "found params in memory cache for STACKED[34359738368]-verifying-key" confirmed that the GROTH_PARAM_MEMORY_CACHE mechanism worked correctly. This set up the second proof (message 236) which completed in 92.8 seconds — a 20.5% improvement purely from keeping the 45 GiB SRS parameters resident in memory.

This 20.5% speedup validated a core design assumption of the cuzk architecture: that keeping the SRS in memory across proof invocations would yield substantial throughput gains. The existing pipeline (Curio's direct invocation of supraseal-c2) loaded the SRS from disk for every proof, wasting 15 seconds per proof on I/O that could be eliminated by a persistent daemon.

Assumptions and Risks

The message implicitly validates several assumptions that carried significant risk:

Assumption 1: The FIL_PROOFS_PARAMETER_CACHE env var would be picked up correctly. The assistant had traced through the storage-proofs-core settings code (messages 214-217) to verify that the lazy_static SETTINGS would read the env var on first access. If the env var was set too late — after SETTINGS was already initialized — the parameter cache would default to /var/tmp/filecoin-proof-parameters/ and the proof would fail with a missing-params error. The success in message 230 confirms this timing dependency was handled correctly.

Assumption 2: The SupraSeal CUDA backend would work on Blackwell (sm_120). The assistant had expressed concern earlier (message 204) that precompiled fatbin kernels from rust-gpu-tools might not support the new architecture. The SupraSeal path delegates to nvcc for JIT compilation, which saved the day. If the bellperson CUDA path had been used instead, it might have failed.

Assumption 3: The gRPC message size limit was sufficient. The 51 MB C1 payload required configuring tonic's message size limits. The assistant had fixed this earlier (segment 4). The successful transmission confirmed the fix.

Assumption 4: The proof would be valid. This was the biggest risk. Even if the pipeline completed without errors, the proof could be invalid — producing a Groth16 proof that doesn't verify would be a silent failure. The internal verify_seal call provided confidence, but until the proof was actually checked, nothing was certain.

Input Knowledge Required

To fully appreciate message 230, one needs to understand:

Output Knowledge Created

This message establishes several critical facts that become the foundation for all subsequent work:

  1. The cuzk architecture is correct. The gRPC API, engine, scheduler, and prover modules all work together to produce valid proofs.
  2. The SupraSeal CUDA backend works on Blackwell GPUs. This was uncertain before this test.
  3. The SRS caching mechanism works. The GROTH_PARAM_MEMORY_CACHE successfully keeps parameters resident across proof invocations.
  4. The baseline performance is 116.8 seconds for a cold start. Every optimization proposal can now be measured against this baseline.
  5. The daemon correctly tracks metrics. The status RPC shows proofs completed: 1, confirming the accounting is functional.

The Thinking Process

The assistant's reasoning in this message reveals a disciplined approach to validation. Rather than declaring victory based on a single log line, the assistant:

  1. Waited for the complete pipeline to finish — The proof took ~2 minutes. The assistant monitored periodically but didn't jump to conclusions.
  2. Verified the proof length — 1920 bytes is a quick sanity check that the proof format is correct.
  3. Noted the verification result — "PASSED" is the definitive signal.
  4. Separated the timing components — Queue wait (28 ms) vs. proving time (116.6s) vs. SRS load (~15s) shows where time is spent.
  5. Checked the client side — The shell command verifies that the bench client exited cleanly and the proof was received. This thoroughness is characteristic of the session. The assistant consistently prioritizes verification over speed, recognizing that an incorrect proof is worse than no proof.

Conclusion

Message 230 is the inflection point of the cuzk project. Before it, the daemon was a design that compiled and appeared correct but had never been tested against real hardware and real Filecoin parameters. After it, the daemon is a validated system with known performance characteristics, a confirmed baseline, and a clear path forward to Phase 1.

The 20.5% SRS residency benefit discovered in the subsequent proof (message 236) would become a key selling point for the cuzk architecture. The 116.8-second baseline would anchor all optimization discussions. And the simple fact that "proofs completed: 1, proofs failed: 0" appeared in the status output provided the confidence needed to proceed with multi-GPU support, additional proof types, and the full cuzk vision.

In the broader narrative of the coding session, this message is the moment when theory becomes practice, when architecture becomes reality, and when a proving daemon proves itself.