The Serialization Detective: Mapping the C1-to-C2 Data Flow in Filecoin's SUPRASEAL_C2 Pipeline

Introduction

In the intricate machinery of Filecoin's proof-of-replication (PoRep) system, few details matter more than the precise format in which data passes between computational phases. The SUPRASEAL_C2 Groth16 proof generation pipeline, which consumes approximately 200 GiB of peak memory, spans three programming languages (Go, Rust, C++), two serialization formats (JSON and bincode), and a complex FFI boundary that shuttles tens of megabytes of cryptographic proof data between orchestration and computation layers. Understanding exactly how the output of Commit Phase 1 (C1) is serialized for consumption by Commit Phase 2 (C2) is not merely an academic exercise—it is a prerequisite for any optimization that aims to reduce memory footprint, improve throughput, or enable new proving architectures.

This article synthesizes a deep-dive investigation into that serialization boundary, tracing the data flow from Curio's Go task layer through CGO bindings into Rust FFI and ultimately into C++/CUDA kernels. The investigation produced four comprehensive documents, identified nine structural bottlenecks, and developed three composable optimization proposals that together project a 5–6× reduction in cost per proof. At the heart of this work lies a single question: what is the wire format of the SealCommitPhase1Output as it travels from C1 to C2?

The Investigation: A Forensic Approach

The investigation began with a user's precise five-part request: trace the serialization format of SealCommitPhase1Output from Curio's Go orchestration code through the FFI boundary into Rust, and validate against a concrete serialized artifact—the 51 MB c1.json file sitting at /data/32gbench/c1.json. What followed was a systematic forensic examination that exemplifies how to reverse-engineer complex cross-language data pipelines.

Phase 1: Reconnaissance by Grep

The first response was a deliberate choice to not attempt immediate synthesis. Instead, the assistant launched parallel grep searches across the codebase, targeting three distinct layers simultaneously: the Curio-specific orchestration layer (sdr_funcs.go), the filecoin-ffi Go bindings (proofs.go, workflows.go), and the Rust FFI implementation (api.rs). A bash command confirmed the existence and size of the c1.json file [11].

This parallel reconnaissance strategy revealed several critical clues in a single turn:

Phase 2: Tracing the Serialization Chain

The assistant then executed a coordinated set of file reads targeting specific line ranges [10]. The reads spanned five layers: the Go orchestration layer (sdr_funcs.go), the Go FFI wrapper (proofs.go), the Rust FFI implementation (api.rs), the CGO bridge (cgo/proofs.go), and the cached output (c1.json).

The critical discovery came from reading the Rust seal_commit_phase2 function: it takes a c_slice::Ref<'_, u8> (a C-compatible byte slice reference) and deserializes it with serde_json::from_slice. This confirmed that the Rust side expects JSON-formatted bytes. But the c1.json file's outer structure—{"SectorNum":1,"Phase1Out":"eyJ..."}—showed a base64-encoded Phase1Out field. What was inside that base64?

The answer came from a single bash command piped through Python [1]:

head -c 200 /data/32gbench/c1.json | python3 -c "
import json, base64, sys
data = sys.stdin.read()
idx = data.find('\"Phase1Out\":\"') + len('\"Phase1Out\":\"')
b64_start = data[idx:idx+100]
decoded = base64.b64decode(b64_start + '==')
print('Decoded start:', decoded[:200])
"

The output revealed the inner payload: {"registered_proof":"StackedDrg32GiBV1_1","vanilla_proofs":{...}}. This confirmed that the serialization chain is a JSON-inside-base64-inside-JSON sandwich: the Rust SealCommitPhase1Output is serialized as JSON by serde_json, which Go receives as []byte bytes, and when Go writes these bytes to a JSON file using encoding/json, the []byte field is automatically base64-encoded.

The Dual-Path Architecture

The investigation revealed that Curio supports two distinct serialization paths, each with its own format and translation requirements [9]:

Path A: Standard Rust FFI (JSON)

In the standard path, the Go code calls ffi.SealCommitPhase1(...) through CGO into the Rust FFI. The Rust function serializes the SealCommitPhase1Output struct using serde_json::to_vec and returns JSON bytes. These bytes pass transparently through the CGO boundary—no additional encoding occurs—and are consumed by seal_commit_phase2 via serde_json::from_slice. This is the clean, direct path.

Path B: SupraSeal C++ (Bincode)

The SupraSeal C++ implementation takes a different approach. It writes a bincode-serialized file to the cache directory (named commit-phase1-output). The Go code then reads this file and decodes it using a custom decoder (DecodeCommit1OutRaw) defined in porep_vproof_bin_decode.go [2]. After decoding from bincode into a Go Commit1OutRaw struct, the code re-encodes it as JSON using json.Marshal() before passing it to the Rust C2 FFI.

This translation layer exists because the C++ SupraSeal implementation was developed independently from the standard Rust implementation. Rather than modifying the C++ code to output JSON, the Curio developers added a Go-side translation layer—a decision with implications for performance, memory, and correctness.

The Bincode Discovery

The discovery of the bincode decoder was a pivotal moment [8]. A grep for DecodeCommit1OutRaw and SealCommitPhase1Output led to porep_vproof_bin_decode.go, whose file header explicitly states: "This file contains a bincode decoder for Commit1OutRaw. This is the format output by the C++ supraseal C1 implementation."

The companion test file (porep_vproof_bin_test.go) contains an embedded Rust program that validates the round-trip: it reads bincode from disk, deserializes it into SealCommitPhase1Output<SectorShape32GiB> using bincode::deserialize, converts it to the newer SealCommitPhase1OutputOut type, and serializes it to JSON using serde_json::to_string [7]. This test program is the Rosetta Stone of the entire investigation—it demonstrates the exact conversion pipeline that bridges the C++ and Rust worlds.

The Serialization Stack: A Summary

The complete serialization chain, as documented in [9], can be summarized as:

| Stage | Format | Details | |-------|--------|---------| | Rust C1 output (standard path) | JSON (serde_json::to_vec) | Rust seal_commit_phase1 serializes SealCommitPhase1Output to JSON bytes | | SupraSeal C1 output (file on disk) | bincode | C++ writes raw bincode to commit-phase1-output file | | SupraSeal conversion | bincode → Go struct → JSON | DecodeCommit1OutRaw() reads bincode, json.Marshal() produces JSON | | Go to Rust C2 boundary | JSON bytes passed as raw []byte / c_slice::Ref<u8> | No additional encoding at the CGO boundary | | Rust C2 input | JSON (serde_json::from_slice) | Rust seal_commit_phase2 deserializes from JSON | | c1.json file | Outer: JSON, Phase1Out field: base64-encoded JSON | Go's encoding/json encodes []byte as base64; the underlying bytes are the JSON SealCommitPhase1Output |

This table is more than a reference—it is a decision record that documents the architectural choices made across the pipeline. JSON is the canonical format, bincode is a SupraSeal-specific exception, base64 encoding is a Go JSON serialization artifact, and the translation layer is the bridge between the two.

From Understanding to Optimization

The serialization investigation was not an end in itself—it was a prerequisite for the broader optimization effort. With the full pipeline mapped, the analysis expanded to include the Curio orchestration model, circuit value distribution statistics (~99% boolean aux_assignment), and computational hotpath characterization at the instruction level (SHA-256 bit manipulation, Fr field arithmetic, NTT memory bandwidth, GPU kernel occupancy).

This expanded analysis produced three composable optimization proposals:

Proposal 1: Sequential Partition Synthesis

The current architecture runs all 10 partitions in parallel, each requiring ~16 GiB of memory, contributing to the ~200 GiB peak footprint. Sequential Partition Synthesis breaks this model by streaming partitions one-at-a-time through the GPU, reducing peak memory from ~200 GiB to approximately 64–103 GiB. This is achieved by reusing the same GPU memory allocation across partitions rather than allocating for all 10 simultaneously.

Proposal 2: Persistent Prover Daemon

Each proof generation currently incurs a ~60-second overhead for loading the Structured Reference String (SRS) parameters from disk. The Persistent Prover Daemon eliminates this overhead by keeping the proving process alive across multiple proof requests. The SRS (~48 GiB in pinned memory) is loaded once and reused, and the daemon architecture allows for efficient request queuing and prioritization.

Proposal 3: Cross-Sector Batching

The freed memory headroom from Sequential Partition Synthesis enables a new capability: batching multiple sectors' circuits into single GPU invocations. This exploits the fact that GPU kernels for NTT and MSM operations are most efficient when processing large batches. The proposal projects 2–3× throughput per GPU and, when combined with the other proposals, a 5–6× reduction in cost per proof.

Micro-Optimization Analysis

A final round of micro-optimization analysis examined CPU synthesis hotpaths (the enforce() loop, SHA-256 bit manipulation, Fr addition), GPU NTT/MSM compute characteristics, H-to-D transfer patterns, and the feasibility of recomputing a/b/c vectors on-the-fly to avoid materialization entirely. This analysis identified specific instruction-level opportunities for improvement, particularly in the Fr field arithmetic operations that dominate the CPU synthesis phase.

The Broader Significance

The overarching achievement of this investigation is a shift in perspective: from optimizing individual proof generation toward architecting a continuous, memory-efficient proving pipeline optimized for heterogeneous cloud rental markets where RAM cost dominates. The serialization format at the C1→C2 boundary is a critical design constraint for this new architecture. Any optimization that modifies how C1 output is produced or consumed must account for both the JSON and bincode paths, and must handle the base64 decoding step efficiently without loading the entire 51 MB file into memory.

The dual-path architecture also has implications for the Persistent Prover Daemon proposal. If the daemon is to accept C1 outputs over gRPC, it must support both JSON and bincode input formats, or normalize them to a common representation before processing. The serialization investigation provides the foundational knowledge needed to make these architectural decisions.

Conclusion

The investigation into the SUPRASEAL_C2 serialization format is a case study in systematic reverse-engineering of complex cross-language data pipelines. Starting from a user's five-part question, the investigation traced the SealCommitPhase1Output from Go orchestration code through CGO bindings into Rust FFI, discovered the dual JSON/bincode architecture, validated the format against a concrete 51 MB artifact, and produced a comprehensive map of the serialization stack.

This map is not merely documentation—it is the foundation for the optimization proposals that follow. The Sequential Partition Synthesis, Persistent Prover Daemon, and Cross-Sector Batching proposals all depend on understanding exactly how data flows between C1 and C2. By establishing that JSON is the canonical wire format (with a bincode exception for SupraSeal), the investigation provides the architectural clarity needed to design a continuous, memory-efficient proving pipeline that can reduce peak memory by 50–70% and cut cost per proof by 5–6×.

In the world of Filecoin storage proving, where every GiB of memory and every second of latency translates directly to operational cost, understanding the serialization format is not an academic exercise—it is the first step toward building a more efficient, more scalable proving infrastructure.## References

[1] "Decoding the C1 Serialization: A Forensic Examination of Curio's Proof Pipeline" — Analysis of the base64 decoding command that confirmed the JSON-inside-base64-inside-JSON format.

[2] "Unearthing the Bincode Serialization Layer: How Curio's C1 Output Bridges C++ and Go" — Discovery of the bincode decoder/encoder pair in Curio's proof library.

[3] "The Serialization Detective: Tracing the SealCommitPhase1Output Across Go, CGO, and Rust FFI" — The user's original five-part investigative request and its motivation.

[4] "Decoding the Serialization Chain: How Curio Bridges Go, Rust, and C++ in Filecoin PoRep Proof Generation" — Analysis of the test file and types file that confirmed the bincode hypothesis.

[5] "Tracing the Serialization Chain: A Deep Dive into Curio's C1 Output Format" — Examination of the makePhase1Out function and the Go-side serialization logic.

[6] "The Moment of Synthesis: Confirming the C1 Serialization Format in Curio's PoRep Pipeline" — The verification step that confirmed the complete serialization chain.

[7] "Tracing the Serialization Boundary: How Curio's C1 Output Crosses from C++ to Go" — Analysis of the bincode utility functions and the Rust test program.

[8] "Tracing the Serialization Boundary: How a Single Grep Revealed the Rust-Go Data Contract in Curio's PoRep Pipeline" — The grep that discovered the bincode decoder and type definitions.

[9] "The Serialization Bridge: How SealCommitPhase1Output Travels from C1 to C2 in Filecoin PoRep" — The definitive synthesis documenting the complete serialization architecture.

[10] "Reading the Serialization Chain: How an AI Agent Traces the C1-to-C2 Data Flow in Filecoin Proof Generation" — Analysis of the parallel file reading strategy.

[11] "Reconnaissance by Grep: How an AI Agent Systematically Explored a Complex Codebase" — The initial reconnaissance phase that mapped the codebase terrain.

[12] "Reading the Serialization Trail: How an Assistant Navigates a Complex Codebase to Uncover the C1/C2 Wire Format" — The data-gathering pivot that located critical source files.