The Pivot Point: Replacing Bincode with a Raw Binary Format for 25.7 GiB of Pre-Computed Circuit Data

Introduction

In the course of optimizing the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), a critical bottleneck emerged not in computation, but in serialization. The Pre-Compiled Constraint Evaluator (PCE) — a 25.7 GiB data structure containing the R1CS constraint matrices for the C2 circuit — needed to be saved to and loaded from disk. The initial implementation used bincode, a generic binary serialization library for Rust. The result was a sobering benchmark: loading the PCE from a tmpfs filesystem took 49.9 seconds, which was actually slower than the 47 seconds required to extract the PCE from scratch. This meant disk persistence was providing negative value — it was faster to recompute than to load.

The message at index 1642 in the conversation is the turning point where the assistant commits to replacing bincode with a custom raw binary format. Though the message itself is brief — "Good. The density struct uses Vec<u64> which is also safe to raw-cast. Let me rewrite the save/load functions with a raw format:" followed by a file write — it represents a decisive architectural shift born from careful measurement and analysis.

The Discovery of the Bottleneck

The story begins with the first successful PCE disk serialization benchmark ([msg 1637]). The assistant had wired up the --save-pce option in the cuzk-bench tool to call cuzk_pce::save_to_disk using bincode serialization. The results were revealing:

| Operation | Time | Throughput | |---|---|---| | Save to tmpfs | 16.8s | 1.6 GB/s | | Save to NVMe | 30.6s | 0.9 GB/s | | Load from tmpfs | 49.9s | 0.6 GB/s |

The load time of 49.9 seconds was the alarm bell. The assistant immediately recognized the root cause: "bincode deserialization of 722M scalars involves per-element parsing. The bottleneck is bincode's deserialize, not I/O" ([msg 1637]). This insight is crucial — it distinguishes between a storage bottleneck (which could be addressed with faster hardware or caching) and a CPU-bound deserialization bottleneck (which requires a fundamentally different data format).

The PCE contains 722,388,891 non-zero entries across three CSR (Compressed Sparse Row) matrices (A, B, C). Each entry has a 32-byte scalar value (vals) and a 4-byte column index (cols), plus row pointers and density bitmaps. Bincode, being a generic serializer, processes each scalar individually: it reads the length prefix, then iterates element-by-element, deserializing each Scalar through its serde::Deserialize implementation. For 722 million elements, this per-element overhead dominates.

Evaluating the Alternatives

The assistant considered two options ([msg 1637]):

  1. Use a zero-copy format — write the vectors as flat binary dumps with known layout, enabling load via bulk read() calls and direct memory reinterpretation.
  2. Use bincode with serde_bytes — a crate that allows byte slices to be serialized as bulk blobs rather than element-by-element, potentially improving throughput while keeping bincode. The assistant chose option 1. This decision was informed by a deeper investigation into the memory layout of blstrs::Scalar ([msg 1638]). A Python calculation estimated that raw format load would take approximately 3–5 seconds (at DDR5 or NVMe speeds) versus the 50 seconds of bincode — a 10–16× improvement. The assistant then confirmed ([msg 1639]) that blstrs::Scalar is #[repr(transparent)] over blst_fr, which is [u64; 4] — a 32-byte value with a stable, predictable memory layout. This means a &[Scalar] slice can be safely reinterpreted as &[u8], and vice versa, using unsafe pointer casts.

The Subject Message: Committing to the Raw Format

The message at index 1642 is the moment of commitment. After reading the density bitmap module and confirming that DensityBitmap uses Vec<u64> (also safe to raw-cast), the assistant states:

"Good. The density struct uses Vec<u64> which is also safe to raw-cast. Let me rewrite the save/load functions with a raw format:"

Then writes the file. The brevity of the message belies the weight of the decision. The assistant is choosing to:

Assumptions and Risks

The raw binary format rests on several assumptions:

  1. Memory layout stability: The Scalar type's #[repr(transparent)] guarantee means its layout matches the inner [u64; 4] exactly. This is a compiler guarantee in Rust, but it assumes the underlying blst_fr type never changes its representation.
  2. Endianness: Writing raw u64 values assumes the reader and writer share the same endianness. Since the PCE is both written and read on the same machine (or same architecture), this is safe in practice, but it precludes cross-architecture portability.
  3. Alignment: The raw byte slices must be properly aligned for u32 and Scalar types when reinterpreted. The read_exact into a Vec<u8> followed by pointer cast must respect alignment requirements. The standard Vec<u8> allocation is only 1-aligned, which could cause misalignment for Scalar (which requires 8-byte alignment for u64). This is a potential correctness bug that the assistant did not explicitly address in the reasoning visible in the conversation.
  4. No schema evolution: Unlike bincode, which can handle optional fields and versioned schemas, the raw format is rigid. Any change to the PreCompiledCircuit structure requires a corresponding format version bump and migration logic. The assistant implicitly accepts these risks in exchange for the 5.4× load speedup that the raw format ultimately delivers ([msg 1648]).

The Implementation

The raw format design was outlined in [msg 1640]:

  1. 32-byte header: magic bytes (4B), version (4B), circuit dimensions (20B), reserved (4B) — same as before.
  2. Per CSR matrix (A, B, C): length-prefixed raw byte dumps of row_ptrs (Vec<u32>), cols (Vec<u32>), and vals (Vec<Scalar>).
  3. Density bitmaps: raw bytes.
  4. All vectors: written as u64 length followed by raw bytes. The save function iterates over each vector, writes a 64-bit length in bytes, then writes the raw byte representation of the entire vector. The load function reads the length, allocates a Vec&lt;u8&gt; of that size, reads the bytes, then uses pointer casts to reinterpret the byte vector as the appropriate typed vector. This is essentially a memcpy-based serialization — the data on disk is a direct image of the data in memory. The results were dramatic ([msg 1648]): | Operation | Bincode (v1) | Raw (v2) | Speedup | |---|---|---|---| | Save (tmpfs) | 16.8s (1.6 GB/s) | 7.7s (3.6 GB/s) | 2.2× | | Load (tmpfs) | 49.9s (0.6 GB/s) | 9.2s (3.0 GB/s) | 5.4× | | Save (NVMe) | 30.6s (0.9 GB/s) | 22.3s (1.2 GB/s) | 1.4× | The load time dropped from 49.9s to 9.2s — a 5.4× improvement that brought disk loading well below the 47s extraction time, making persistence genuinely useful. The assistant noted that the 9.2s load was now bottlenecked by memory allocation (allocating 25.7 GiB of Vecs) rather than deserialization logic.

Broader Significance

This decision had far-reaching consequences for the project. The PCE disk persistence enabled the daemon to preload the PCE at startup ([msg 1648]), eliminating the first-proof penalty entirely. Without the raw format, the 49.9s load time would have been a non-starter — the daemon would have been better off extracting the PCE on demand. The raw format made PCE preloading practical, which in turn enabled the daemon architecture that the project was building toward.

The message at index 1642 is a textbook example of data-driven optimization: measure first, identify the true bottleneck (CPU-bound deserialization, not I/O), evaluate alternatives against concrete performance targets, and commit decisively. The assistant's willingness to abandon a standard library in favor of a custom solution — with full awareness of the tradeoffs — reflects a mature engineering judgment shaped by the specific constraints of the problem: 25.7 GiB of data, 722 million elements, and the need for sub-10-second load times to make disk persistence viable.