The Configuration That Enables a Pipeline: Wiring Phase 2 into cuzk's Config Struct

Message Summary

The subject message, <msg id=501>, is deceptively brief:

Now add the pipeline field to the top-level Config struct: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/config.rs Edit applied successfully.

This single edit — adding a pipeline field to the configuration struct — is the keystone that locks the entire Phase 2 pipelined proving architecture into place. Without it, the newly written srs_manager.rs and pipeline.rs modules would exist as orphan code, never reachable from the running engine. This message represents the moment when an architectural abstraction (the pipeline mode) becomes a runtime-selectable reality.

The Broader Context: Phase 2 of the cuzk Proving Engine

To understand why this message matters, one must understand what cuzk is building. The cuzk project ([msg 430]) is a pipelined SNARK proving daemon for Filecoin proof generation, designed to replace the monolithic proving pipeline used by the supraseal-c2 library. The project is organized into six phases spanning 18 weeks.

Phase 0 established the scaffold: a gRPC API, a priority scheduler, and a working PoRep C2 prover that called filecoin-proofs-api::seal_commit_phase2() monolithically. Phase 1 extended this to support all four Filecoin proof types (WinningPoSt, WindowPoSt, SnapDeals) and added a multi-GPU worker pool with CUDA_VISIBLE_DEVICES isolation.

Phase 2 is the architectural pivot. It replaces the monolithic PoRep C2 prover — which synthesizes all ten partitions at once, consuming ~136 GiB of intermediate memory — with a per-partition pipelined architecture. The pipeline separates the CPU-bound circuit synthesis phase from the GPU-bound proof generation phase, connected by a bounded channel. This reduces peak intermediate memory from ~136 GiB to ~13.6 GiB per partition, making the system viable on 128 GiB machines.

The implementation of Phase 2 required three new modules:

1. srs_manager.rs — Direct SRS (Structured Reference String) parameter loading via SuprasealParameters::new(), bypassing the private GROTH_PARAM_MEMORY_CACHE that the monolithic path relies on.
2. pipeline.rs — The SynthesizedProof type and the split synthesize_porep_c2_partition() / gpu_prove() functions, leveraging the bellperson fork's newly exposed synthesize_circuits_batch() and prove_from_assignments() API.
3. Configuration wiring — The pipeline field in the Config struct, which is the subject of this message.

Why This Message Was Written: The Reasoning and Motivation

The message exists because of a deliberate architectural decision: the pipeline mode must be optional and runtime-configurable. The assistant could have hard-coded the pipeline as the only proving path, replacing the Phase 1 monolithic prover entirely. Instead, the design treats the pipeline as a configurable feature that coexists with the legacy path.

This decision reflects several considerations:

Backward compatibility. The monolithic prover has been validated through Phase 0 and Phase 1 testing, producing correct proofs against golden test data. The pipeline is new code that has only been validated through unit tests (the three new pipeline tests in <msg id=498>). Until end-to-end integration testing with a GPU build confirms correctness, the legacy path must remain available as a fallback.

Graceful degradation. If the pipeline encounters an error or produces incorrect proofs, operators can disable it by toggling a single configuration flag and return to the proven monolithic path. This is critical for a production system that may be handling real Filecoin sector commitments.

Selective applicability. The pipeline is specifically designed for PoRep C2 proofs, which are the most memory-intensive (~200 GiB peak). PoSt and SnapDeals proof types are much smaller (WinningPoSt ~0.45 GiB, WindowPoSt ~16 GiB) and do not benefit from the pipeline's memory partitioning. The configuration toggle allows the engine to route only PoRep C2 jobs through the pipeline while sending other proof types through the monolithic path.

Operational flexibility. Different deployments have different hardware constraints. A machine with 256 GiB RAM might prefer the monolithic path for its simplicity, while a 128 GiB machine requires the pipeline to avoid OOM. The configuration field makes this a deployment-time decision rather than a compile-time or code-change decision.

The Design Decisions Embedded in This Message

Although the message itself is a single edit, it encodes several design decisions:

The pipeline is a first-class configuration section, not a boolean flag. The assistant did not add a single pipeline_enabled: bool field to the Config struct. Instead, <msg id=500> first added a PipelineConfig section (likely containing fields like enabled, synthesis_slots, max_in_flight_partitions, etc.), and then <msg id=501> added the pipeline field to the top-level Config struct. This means the pipeline has its own configuration namespace, which can grow as Phase 2 evolves to include cross-proof overlap scheduling, memory budget limits, and other operational parameters.

The field is likely Option<PipelineConfig> rather than PipelineConfig. This is inferred from the assistant's earlier work on the config system ([msg 499]). Making the pipeline section optional means that if the config file omits the [pipeline] section entirely, the engine defaults to the monolithic path — a sensible default for backward compatibility.

The engine will check this field at job dispatch time. The next message in the sequence ([msg 503]) confirms this: "When pipeline.enabled = true, the engine uses SrsManager to load params directly... When pipeline.enabled = false, the existing Phase 1 monolithic path is used." This means the decision between pipeline and monolithic mode is made per-job, not at startup, allowing the engine to dynamically adapt.

Assumptions Made by the Assistant

The assistant makes several assumptions in this message:

The PipelineConfig type already exists. The assistant assumes that the pipeline configuration section added in <msg id=500> defined a PipelineConfig struct (or similar) that can be referenced as a field type. This is a safe assumption because the assistant wrote that code in the immediately preceding message.

The Config struct already has other fields. The assistant assumes the Config struct exists with fields like engine, server, scheduler, etc., and that adding a pipeline field is a straightforward extension. This is correct — the config system was built in Phase 0 and extended in Phase 1.

The field name pipeline does not conflict with existing fields. The assistant assumes no existing field named pipeline exists in the Config struct. This is a reasonable assumption given the assistant wrote the entire config system and knows its structure.

The edit will compile. The assistant assumes that adding the field, combined with the pipeline configuration section from <msg id=500>, will compile without errors. This is validated in subsequent messages — the next cargo check (implied by the flow) passes without warnings from cuzk code.

The pipeline mode is worth the complexity. The assistant assumes that the additional complexity of a configurable pipeline mode — with its conditional branching in the engine, its separate SRS manager, and its per-partition state management — is justified by the memory reduction and throughput improvements. This assumption is grounded in the detailed memory analysis from earlier segments ([chunk 0.0]), which showed that the monolithic path's ~200 GiB peak memory makes it impractical for many deployment scenarios.

Input Knowledge Required to Understand This Message

To understand what this message is doing, one needs:

Knowledge of the cuzk architecture. The message references Config and config.rs without explanation. One must know that cuzk uses a TOML-based configuration system with typed Rust structs, that the Config struct is the top-level configuration container, and that the engine reads this config at startup to determine its behavior.

Knowledge of Phase 2's pipelined architecture. The message does not explain what the pipeline field enables. One must know that Phase 2 introduces a per-partition synthesis/GPU split, that this requires an SRS manager for direct parameter loading, and that the pipeline mode is specifically for PoRep C2 proofs.

Knowledge of the existing config structure. One must know that the Config struct already has sections for engine, server, scheduler, and possibly others, and that the new pipeline field is one more section in this hierarchy.

Knowledge of the edit sequence. The message is the second of three edits: (1) add pipeline config section ([msg 500]), (2) add pipeline field to Config struct ([msg 501]), (3) refactor engine to use pipeline mode ([msg 503]). Understanding the sequence reveals the dependency chain.

Output Knowledge Created by This Message

This message creates:

A runtime-configurable pipeline toggle. The most important output is the ability to enable or disable the pipeline mode without code changes. This is the mechanism that makes the pipeline a coexisting alternative to the monolithic path, not a replacement.

A foundation for future pipeline configuration. By making the pipeline a full configuration section (not just a boolean), the message creates space for future parameters: synthesis slot counts, memory budgets, preload/evict policies, and cross-proof overlap settings. These are anticipated by the Phase 2 design but not yet implemented.

A clear separation of concerns. The configuration boundary between the pipeline and the engine means that the pipeline module can evolve independently. New pipeline features (like cross-proof overlap in Phase 3) can be added by extending PipelineConfig without modifying the engine's core dispatch logic.

A testable configuration path. The unit tests for config parsing ([msg 498]) include test_default_config, which will now validate that the pipeline field is properly serialized and deserialized. This ensures configuration changes don't break existing deployments.

The Thinking Process Visible in This Message

The assistant's thinking process, visible across the message sequence, reveals a methodical approach to architectural change:

Top-down decomposition. The assistant first identifies the need for pipeline configuration ([msg 499]: "Now let me update the config and add the pipeline configuration"), then implements it in layers: first the configuration section type, then the field on the Config struct, then the engine logic that reads it.

Dependency ordering. The assistant respects dependency order: the pipeline configuration section must exist before it can be referenced as a field type, and the field must exist before the engine can read it. The message sequence follows this chain precisely.

Minimal changes. The assistant does not refactor the entire config system. It adds exactly what is needed: a configuration section and a field. This minimizes the risk of introducing bugs in unrelated parts of the system.

Implicit validation. The assistant does not explicitly run cargo check after this edit (unlike earlier messages where compilation was verified). This suggests confidence that the edit is trivially correct — adding a field to a struct that already has fields of the same type is a low-risk operation. The validation comes implicitly when the engine refactoring in <msg id=503> compiles successfully.

Conclusion

The message <msg id=501> — "Now add the pipeline field to the top-level Config struct" — is a small edit with large architectural significance. It transforms the pipeline from an abstract design into a runtime-selectable mode of operation. It embodies the design philosophy of coexistence: the new pipeline does not replace the old monolithic path but lives alongside it, giving operators the choice. This choice is essential for a production system that must balance innovation with reliability, and it reflects the careful, methodical approach that characterizes the entire cuzk project.