When the API Is Private: A Case Study in Rust Visibility Boundaries During Pipeline Development

Introduction

In the course of building a high-performance SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol, an AI assistant encountered a seemingly mundane but architecturally significant Rust compilation error. The error—error[E0603]: module 'api' is private—blocked the expansion of a pipelined proving system to support PoSt (Proof-of-Spacetime) and SnapDeals proof types. The assistant's response in [msg 572] is a compact but revealing artifact: a diagnostic decision point where the assistant identified the root cause, analyzed the upstream API's visibility boundaries, and chose a pragmatic path forward. This message, though only a few lines of reasoning followed by an edit command, encapsulates a recurring tension in systems programming: the boundary between reusing existing library code and maintaining architectural independence.

Context: The Pipelined Proving Engine

The message occurs within a larger effort to build cuzk, a Rust-based proving daemon that replaces the monolithic Groth16 proof generation pipeline in Filecoin's supraseal implementation. Phase 1 had established a working monolithic prover that called filecoin-proofs-api functions directly. Phase 2 aimed to split proof generation into two stages: synthesis (CPU-bound circuit construction using bellperson) and GPU proving (the computationally intensive Groth16 proof creation on CUDA hardware). This split enables pipelining—overlapping synthesis of one proof with GPU proving of another—to improve throughput on continuous proof streams.

By [msg 554], the assistant had successfully implemented per-partition pipelining for PoRep C2 (the most memory-intensive proof type, requiring ~200 GiB for a 32 GiB sector with 10 partitions). However, end-to-end GPU testing revealed a critical performance regression: sequential per-partition proving took ~611 seconds versus ~93 seconds for the monolithic baseline. The assistant identified the root cause and pivoted to implement a batch-all-partitions synthesis mode that matches the monolithic approach for single-proof latency, while preserving the pipeline architecture for future throughput optimization.

The next logical step was to expand pipeline support to the remaining proof types: WinningPoSt, WindowPoSt, and SnapDeals. This required implementing synthesis functions that mirror the circuit construction logic used by the upstream filecoin-proofs crate. The assistant had already researched these APIs in [msg 555] and [msg 556], identifying the specific utility functions needed for vanilla proof partitioning.

The Error: A Visibility Wall

In [msg 569], the assistant ran cargo check --workspace --no-default-features after writing a comprehensive new pipeline.rs that included synthesize_post() and synthesize_snap_deals() functions. The compilation succeeded with only unused-import warnings—promising. But in [msg 570], a more targeted check revealed two errors:

error[E0603]: module `api` is private
   --> cuzk-core/src/pipeline.rs:628:47
    |
628 |     let partitioned_proofs = filecoin_proofs::api::post_util::partition_vanilla_proofs(
    |                                               ^^^ private module

A second identical error at line 817 pointed to single_partition_vanilla_proofs. Both functions lived in filecoin_proofs::api::post_util, a module that the upstream crate had declared as pub(crate)—accessible only within filecoin-proofs itself, not to external consumers like cuzk-core.

The Diagnostic Response

The subject message ([msg 572]) is the assistant's immediate diagnostic response. It opens with a clear statement of the problem:

The api module in filecoin-proofs is private. I can't call partition_vanilla_proofs or single_partition_vanilla_proofs directly. I need to replicate that logic in cuzk-core.

This is not merely a restatement of the compiler error. The assistant has already performed the critical analysis: it recognizes that the pub(crate) visibility modifier is the barrier, and it understands that the solution is not to fight the upstream crate's API design but to replicate the essential logic within cuzk-core. This is a design decision masquerading as a bug fix.

The message then provides a brief architectural breakdown of what each function does:

For WinningPoSt, the key is reshaping the vanilla proofs into the partition format expected by CompoundProof::circuit(). For WindowPoSt single partition, it's finding sectors by ID and padding.

This reveals the assistant's understanding of the underlying algorithms. The partition_vanilla_proofs function for WinningPoSt takes an array of vanilla proofs (one per sector) and reshapes them into the multi-partition format required by the circuit's CompoundProof::circuit() method. The single_partition_vanilla_proofs function for WindowPoSt handles the single-partition case, which involves locating the correct sector by its ID and padding the proof to the expected length. These are not complex cryptographic operations—they are data layout transformations—but they are essential for bridging between the vanilla proof format and the circuit synthesis API.

The Decision: Inline vs. Upstream Patch

The assistant faced a fork in the road. Three options were available:

  1. Patch the upstream crate: Modify filecoin-proofs to make post_util public, then depend on the patched version. This would require maintaining a fork or vendoring the crate.
  2. Use re-exported APIs: Check if filecoin-proofs re-exports these functions through any public path. (The assistant had already verified this was not the case.)
  3. Inline the logic: Replicate the partitioning logic directly in cuzk-core/src/pipeline.rs, accepting some code duplication but maintaining zero coupling to internal APIs. The assistant chose option 3, and the reasoning is implicit in the message's structure. The assistant had already created a bellperson fork in [msg 570] of the earlier segment to expose synthesis/GPU split APIs—so it was not categorically opposed to forking. However, forking a crate as foundational as filecoin-proofs (which underpins the entire Filecoin proof pipeline) would be a heavy maintenance burden. The partitioning logic, by contrast, is relatively simple data reshaping. Inlining it is the pragmatic choice: it keeps cuzk-core self-contained, avoids version synchronization headaches, and preserves the ability to evolve the pipeline independently of upstream changes. The message ends with the application of an edit to pipeline.rs, implementing the inlined partitioning logic.

Assumptions and Their Validity

The assistant made several assumptions in this message, most of which are reasonable:

Assumption 1: The upstream crate cannot be modified. This is sound for a project that consumes filecoin-proofs as a published dependency. Even though the workspace includes a local checkout of filecoin-proofs (via the filecoin-ffi Go binding), modifying its visibility would create a fork divergence that complicates upgrades.

Assumption 2: The partitioning logic is simple enough to inline. This assumption proved correct—the subsequent compilation succeeded, and the end-to-end GPU test in [msg 604] produced a valid proof in 91.2 seconds, matching the monolithic baseline.

Assumption 3: The pub(crate) visibility is intentional. This is almost certainly true. The api module in filecoin-proofs contains internal implementation details that the crate authors chose not to expose. Respecting this boundary is good software citizenship.

One could argue that the assistant made an initial incorrect assumption in [msg 564] when writing the pipeline code: it assumed these functions would be publicly accessible without checking their visibility. This is a common pitfall in Rust development, where the distinction between pub, pub(crate), and private is a compile-time boundary that can only be discovered by attempting to use the API. The assistant's iterative approach—write, compile, diagnose, fix—is the natural workflow for navigating such boundaries.

Input and Output Knowledge

Input knowledge required to understand this message includes:

The Thinking Process

The assistant's reasoning, visible in the message text, follows a clear diagnostic pattern:

  1. Observe the error: The compiler says api is private.
  2. Identify the specific barrier: pub(crate) visibility on post_util module.
  3. Understand the consequence: Cannot call partition_vanilla_proofs or single_partition_vanilla_proofs.
  4. Determine the solution strategy: Replicate the logic in cuzk-core.
  5. Analyze the algorithms: WinningPoSt needs reshaping; WindowPoSt needs sector-finding and padding.
  6. Execute: Apply the edit to pipeline.rs. This is classic systems debugging: from symptom to root cause to fix. What makes this message interesting is the intermediate step where the assistant articulates what each function does conceptually before implementing the replacement. This conceptual understanding is what enables correct inlining—without it, the assistant would be blindly copying code without understanding its purpose.

Broader Significance

This message, while brief, illustrates a fundamental tension in software engineering: the boundary between reuse and independence. Every external dependency draws a line between what is public (the API contract) and what is private (implementation details). When a project needs functionality that falls on the private side of that line, it faces a choice: fork the dependency, lobby for API changes, or replicate the logic. The assistant's choice to replicate reflects a pragmatic assessment of the tradeoffs: the partitioning logic is simple, stable, and unlikely to change; the cost of forking or depending on internal APIs outweighs the benefit of reuse.

In the context of the larger cuzk project, this decision also reflects a design philosophy. The pipeline architecture is explicitly about decoupling from the monolithic proving functions. By inlining the partitioning logic, cuzk-core becomes more self-contained and less dependent on the internal structure of filecoin-proofs. This independence will pay dividends when the pipeline evolves to support true async overlap across multiple proof jobs—the next architectural goal identified in [msg 604].

Conclusion

Message [msg 572] is a small but revealing moment in the development of a complex proving engine. A Rust visibility error, encountered while expanding pipeline support to new proof types, triggered a diagnostic response that balanced technical analysis with architectural judgment. The assistant correctly identified the pub(crate) boundary, understood the algorithms behind the inaccessible functions, and chose to replicate rather than fork or lobby for API changes. The decision preserved the project's independence from upstream internal APIs while keeping the implementation path clear for the next phase of development. In doing so, it demonstrated a pattern of pragmatic engineering that values working software over abstract purity—a lesson that applies far beyond the Filecoin proving ecosystem.