Consolidation and Characterization: How Phase 12's Architecture Was Documented and Validated Through Systematic Benchmarking

Introduction

In the lifecycle of any complex engineering project, there comes a pivotal moment when the frantic pace of optimization pauses, and the team must ask a deceptively simple question: What have we actually built? For the SUPRASEAL_C2 Groth16 proof generation pipeline powering Filecoin's Proof-of-Replication (PoRep), that moment arrived in Segment 32 of the opencode session. This segment captured a complete arc: from a methodical documentation campaign that preserved the intricate Phase 12 architecture for future maintainers, through a systematic low-memory benchmark sweep across nine configurations that characterized the engine's memory footprint and throughput, to the final git commit (9bb657e5) that sealed all of this knowledge into the project's permanent history.

The Phase 12 architecture was the culmination of dozens of earlier segments of optimization work—the split GPU proving API, the use-after-free fix in the C++ prep_msm_thread, the early a/b/c deallocation optimization, the channel capacity auto-scaling, and the partition semaphore permit-through-send mechanism. But without documentation, this architecture would remain opaque to anyone who hadn't lived through its creation. Without systematic benchmarking, its performance characteristics would be anecdotal rather than empirical. Segment 32 addressed both gaps, transforming the system from an optimized but undocumented artifact into a characterized and documented one—a system ready for production deployment across a wide range of hardware configurations.

This article traces the arc of that transformation, examining the documentation process, the benchmark methodology, the surprising findings, and the lasting knowledge that was captured in commit 9bb657e5.

The Documentation Turn: Preserving Phase 12's Architecture

The segment opened with the user issuing a concise instruction: "Write down findings in relevant docs" ([msg 3246]). This six-word trigger launched a cascade of documentation work spanning dozens of messages. The assistant did not simply append notes to a file; it engaged in a methodical, multi-step process of reading the existing documentation, understanding its structure, and integrating new knowledge into the appropriate sections.

Reading the Existing Landscape

The assistant began by reading the three key documents that would need updating: cuzk-project.md (the main project documentation), cuzk-phase2-design.md (the Phase 2 design document), and extern/cuzk/cuzk.example.toml (the example configuration file) ([msg 3248]). This reading phase was critical. Rather than assuming the structure of the documentation, the assistant verified it directly, ensuring that the updates would fit naturally into the existing narrative.

The cuzk-project.md file was a substantial document—nearly 2000 lines—covering the project's architecture, implementation phases, E2E test results, file references, and optimization proposals. The assistant needed to understand where Phase 12's contributions would fit: the summary timeline at the top, the stopping points table that tracked implementation progress, the E2E test results section that captured benchmark data, and the file references section that listed related documents.

Systematic Section-by-Section Updates

The assistant's approach to documentation was remarkably systematic. Rather than making one large edit, it worked through the document section by section, reading, editing, and verifying each change. It updated the summary timeline to reflect Phase 12's completion ([msg 3255]), revised the stopping points table to mark Phase 12 items as done ([msg 3256]), added Phase 12 benchmark results to the E2E Test Results section ([msg 3255]), updated the Supraseal, Bellperson, and cuzk Engine file reference entries ([msg 3260]), and refreshed the Related Documents section ([msg 3261]).

Each read operation served as a verification step. The assistant did not assume that its edits had been applied correctly; it re-read the file after each edit to confirm the changes were in place. This feedback loop ensured that the documentation remained coherent and accurate throughout the process.

What Was Documented

The Phase 12 architecture that the assistant captured in cuzk-project.md was complex, comprising several coordinated mechanisms:

  1. Split GPU proving API: The core architectural change that decoupled the GPU worker's critical path from CPU post-processing. This allowed the GPU to begin work on a new partition while the CPU finalized the previous one, eliminating the GPU idle gap that had plagued earlier phases.
  2. Use-after-free fix: A race condition in the C++ prep_msm_thread where GPU buffer access could occur after the buffer had been freed. The fix ensured proper synchronization between the GPU worker and the post-processing thread.
  3. Early a/b/c free: An optimization that freed NTT evaluation vectors (the a/b/c polynomials) as soon as they were no longer needed, rather than waiting for the entire partition to complete. This reduced memory pressure during the pipeline's peak.
  4. Channel capacity auto-scaling: A mechanism that dynamically adjusted the capacity of the channel between synthesis workers and the GPU worker, preventing deadlocks when partition workers outran GPU processing.
  5. Permit-through-send: A memory backpressure strategy where the partition semaphore permit was held until the partition was actually sent to the GPU worker, preventing memory buildup in the receive buffer. The documentation did not simply list these mechanisms; it explained the why behind each one. The use-after-free fix was contextualized with the race condition that motivated it. The channel auto-scaling was explained in terms of the OOM conditions it prevented. This causal reasoning transforms the documentation from a changelog into an architectural record that future maintainers can learn from.

Updating the Example Configuration

Alongside the main documentation, the assistant also revised cuzk.example.toml with the optimal defaults that had been validated through Phase 12 engineering: gpu_threads=32 and partition_workers=12. These were not arbitrary values; they were the result of extensive benchmarking across earlier segments that had identified the configuration sweet spot for maximum throughput on the target hardware (RTX 5070 Ti, 755 GiB DDR5).

The example config also received RAM-tier recommendations based on the benchmark data that would be generated later in the segment. This forward-looking update ensured that the configuration file would serve as a practical guide for system integrators, not just a template with defaults.## The Low-Memory Benchmark Sweep: Characterizing the Production Frontier

With the documentation complete, the assistant pivoted to a different kind of work: systematic performance characterization. The Phase 12 optimization had been designed primarily for maximum throughput on large-memory systems. But the real-world deployment targets for Filecoin storage providers include machines with widely varying memory capacities—from 128 GiB home servers to 768 GiB+ enterprise workstations. A storage provider with a 128 GiB machine needs to know whether the software can run at all, and if so, at what throughput. A provider with a 256 GiB machine needs to know whether to choose pw=5 gw=1 (170 GiB, 68.4 s/proof) or pw=7 gw=1 (208 GiB, 53.3 s/proof). The benchmark sweep was designed to answer these questions with empirical data rather than guesswork.

Methodology

The sweep covered nine configurations: pw=1, 2, 5, 7, 10, 12 with gw=1 and selected cross-checks with gw=2 at key points (pw=5, 7, 10). This was not an exhaustive grid of every possible combination but a strategic sampling that revealed the scaling curve with minimal experimental cost. The assistant created dedicated config files for each variant ([msg 3276]), wrote a custom benchmark script ([msg 3292]), and ran each configuration through a full cycle: start daemon, wait for SRS loading, measure baseline RSS, run proofs, monitor peak RSS via /proc/$PID/status, and finally kill the daemon.

The methodology was carefully designed to produce reliable, comparable data. The assistant chose to run 5 proofs with concurrency 5 (j=5) for most tests, measuring steady-state throughput rather than single-proof latency. It tracked two distinct metrics: "Throughput" (wall-clock seconds per proof including queue wait) and "Avg Prove" (GPU compute time only). This distinction proved critical for diagnosing bottlenecks—when throughput was high but prove time was low, the bottleneck was clearly on the CPU side (synthesis starvation). When prove time was high, the GPU itself was the bottleneck.

The benchmark script handled the full lifecycle automatically: it started the daemon, waited for it to become ready (the SRS loading phase, which took approximately 28 seconds for all configurations), recorded the baseline RSS after loading, launched the benchmark client with the specified concurrency, monitored peak RSS during execution, and finally killed the daemon and collected the results. This automation was essential for running nine configurations without manual intervention.

Debugging the Measurement Pipeline

The benchmark process was not entirely smooth. When the assistant attempted to measure RSS for the first configuration (pw=1 gw=1), the RSS monitor log came back empty ([msg 3285]). The assistant diagnosed the problem: the background subshell that was monitoring RSS had been killed when the timeout command terminated the benchmark. The /proc filesystem entries for the daemon process had already been cleaned up by the kernel.

This is a classic benchmarking pitfall: the measurement process must outlive the process it is measuring, or at least collect its data before the process exits. The assistant adapted by switching from a background subshell approach to a direct /proc inspection immediately after the benchmark completed but before the daemon was killed. However, this too had issues—the pgrep command returned multiple PIDs due to a grep pattern that matched both the daemon and the benchmark client ([msg 3286]). The assistant fixed the pattern to be more specific, ensuring that only the daemon PID was captured.

These debugging moments are instructive. They demonstrate that even a well-designed benchmark script can encounter unexpected failures, and that the ability to diagnose and fix measurement issues is as important as the ability to run the experiments. The assistant's methodical approach—checking the log, identifying the root cause, and fixing the measurement pipeline—ensured that the data quality was maintained across all nine configurations.

The Results: A Clean Linear Scaling Law

The results of the benchmark sweep were remarkably clean and interpretable. The assistant compiled them into a comprehensive table ([msg 3309]):

| pw | gw | Throughput (s/proof) | Avg Prove (s) | Peak RSS (GiB) | Baseline RSS (GiB) | |----|----|--------------------|--------------|----------------|--------------------| | 1 | 1 | ~290* | 33.2 | 104 | 69 | | 2 | 1 | 152.0 | 33.7 | 110 | 69 | | 5 | 1 | 68.4 | 34.0 | 170 | 69 | | 5 | 2 | 68.6 | 58.2 | 170 | 69 | | 7 | 1 | 53.3 | 34.0 | 208 | 69 | | 7 | 2 | 51.2 | 59.2 | 204 | 69 | | 10 | 1 | 45.4 | 39.5 | 310 | 69 | | 10 | 2 | 42.5 | 68.1 | 271 | 69 | | 12 | 2 | 42.5 | 69.8 | 373 | 69 |

From this data, the assistant derived a linear memory scaling formula: Peak RSS ≈ 69 GiB baseline + (partition_workers × ~20 GiB). The baseline of 69 GiB represents the fixed cost of loading the SRS (44 GiB) and the PCE (25.7 GiB). Each partition worker adds approximately 20 GiB of peak RSS, corresponding to the synthesis intermediates (NTT evaluation vectors, a/b/c polynomials) that must be held in memory during partition processing.

The formula was validated against the measurements with remarkable accuracy:

The Surprising Finding: gw=2 Provides No Benefit Below pw=10

Perhaps the most valuable finding from the benchmark sweep was that gw=2 (two GPU workers) provides no throughput benefit below pw=10 (ten partition workers). At pw=5, the throughput was virtually identical between gw=1 (68.4 s/proof) and gw=2 (68.6 s/proof). At pw=7, the gain was marginal (53.3 vs 51.2 s/proof, about 4%). Only at pw=10 did gw=2 show a meaningful improvement (45.4 vs 42.5 s/proof, about 6%).

The reason is clear from the "Avg Prove" column. At pw=5 gw=1, the average prove time was 34.0 seconds—essentially the GPU's compute time for a single partition. At pw=5 gw=2, the average prove time jumped to 58.2 seconds, meaning the GPU workers were spending most of their time idle, waiting for synthesis to produce partitions. The second GPU worker added complexity without benefit because the CPU-bound synthesis could not produce partitions fast enough to keep even one GPU fully occupied.

This is a non-obvious result. One might intuitively expect that more GPU workers would always improve throughput, since GPU compute is typically the bottleneck in proof generation. But the cuzk engine's pipelined architecture means that synthesis and GPU proving are tightly coupled. If synthesis cannot keep up, additional GPU workers simply compete for the same limited supply of partitions, adding overhead without improving throughput.

The practical implication is clear: for systems with limited memory (and therefore limited partition workers), a single GPU worker is optimal. The second worker should only be enabled when the system has enough memory to support pw>=10 (approximately 310 GiB peak RSS for pw=10 gw=1). This finding directly informed the deployment guidance that the assistant documented in cuzk.example.toml.

Deployment Guidance for Real-World Systems

The benchmark sweep enabled the assistant to provide concrete, data-driven deployment recommendations for various system memory sizes:

docs: Phase 12 results + low-memory configuration sweep

Document Phase 12 split API and memory backpressure in cuzk-project.md:
- Split API architecture, use-after-free fix, early a/b/c free
- Memory backpressure (channel auto-scaling, permit-through-send)
- Buffer flight counters, memory budget analysis

Add low-memory benchmark sweep (pw=1/2/5/7/10/12 × gw=1/2):
- Memory scales as ~69 + pw×20 GiB (measured)
- 128 GiB: pw=2 gw=1 → 110 GiB peak, 152s/proof
- 256 GiB: pw=7 gw=1 → 208 GiB peak, 53s/proof
- 384 GiB: pw=10 gw=2 → 271 GiB peak, 43s/proof
- gw=2 adds no benefit below pw=10 (GPU synthesis-starved)

Update cuzk.example.toml with measured RAM-tier recommendations,
optimal defaults (gpu_threads=32, partition_workers=12), and
guidance for gw=1 vs gw=2 based on partition worker count.

The commit 9bb657e5 on the feat/cuzk branch now permanently links these changes to the project's history. Two files were changed: cuzk-project.md and cuzk.example.toml, with 279 insertions and 21 deletions.

This commit is notable for what it encodes. It is not merely a changelog; it is a decision record that captures both architectural knowledge (the Phase 12 mechanisms) and empirical performance data (the scaling formula and deployment recommendations). The commit message is structured for multiple audiences: future developers who need to understand the Phase 12 internals, system integrators who need to configure the engine for specific hardware, and operators who need deployment guidance for various memory tiers.

The commit also represents a form of knowledge insurance. Without it, the insights from the benchmark sweep would exist only in the assistant's working memory and in the temporary log files scattered across /tmp/. With it, anyone who clones the repository and runs git log will find this record—the split API architecture, the memory backpressure fixes, the empirical scaling laws, and the deployment guidance. The commit message stands as a reminder that in software engineering, the act of committing is not just about saving code—it is about saving understanding.

The Broader Significance: From Optimization to Production Readiness

The work captured in this segment represents a critical transition in the engineering lifecycle of a complex system. The Phase 12 optimization cycle had been about capability—can we make the system faster? Can we reduce memory? The documentation and benchmark sweep were about characterization—now that we have the capability, what are its contours? Where are the trade-offs? How does it behave across the full range of deployment scenarios?

This transition from optimization to characterization to production readiness is a hallmark of mature engineering. Early in a project's life, the focus is on making things work and then making them work better. But at some point, the team must step back and ask: what have we actually built? The answer is not a single number but a family of behaviors, a curve, a set of trade-offs. The benchmark sweep is the instrument that reveals this curve, and the documentation is the vessel that preserves it.

For the cuzk project, this segment marks the moment when the system became not just a fast prover but a configurable prover—one that can be tuned to fit the memory budget of any deployment. The linear memory formula, the gw=2 ineffectiveness finding, and the RAM-tier recommendations are not just observations; they are the user manual for that configurability. They tell the system integrator: here is how memory scales, here is where throughput saturates, here is what configuration to choose for your hardware.

The segment also demonstrates several principles of effective engineering practice that are worth highlighting:

  1. Documentation is not an afterthought. The assistant spent as many messages documenting Phase 12 as it had spent engineering it in earlier segments. This investment in documentation ensures that the architectural knowledge is preserved for future maintainers who may not have the benefit of the original engineering context.
  2. Systematic characterization is the bridge between optimization and production readiness. The benchmark sweep transformed the system from a single-point optimization (max throughput on a specific machine) into a family of configurations adaptable to diverse deployment scenarios. The linear scaling formula and the gw=2 finding are the kind of knowledge that only emerges from systematic measurement.
  3. Negative results are valuable. The finding that gw=2 provides no benefit below pw=10 is a negative result—it tells future engineers what not to do. By documenting this finding, the assistant saved future engineers from repeating the same experiment or, worse, deploying a suboptimal configuration in production.
  4. The commit is the final act of knowledge creation. The commit message is not just a summary of changes; it is a decision record that captures the rationale, the data, and the recommendations. It is the permanent artifact that future developers and deployers will consult when they need to understand why the system behaves the way it does.

Conclusion

Segment 32 of the opencode session captured a complete arc: from the intense engineering of Phase 12's memory backpressure fixes, through a methodical documentation campaign that preserved the architecture for future maintainers, through a systematic low-memory benchmark sweep that characterized the engine's behavior across nine configurations, to the final git commit that sealed all of this knowledge into the project's permanent history.

The work demonstrates that effective engineering is not just about writing code that works; it is about creating knowledge that lasts. The documentation in cuzk-project.md, the configuration guidance in cuzk.example.toml, and the commit message of 9bb657e5 are all artifacts of this knowledge creation process. They ensure that the insights gained through dozens of hours of optimization, debugging, and benchmarking are not lost when the engineer moves on to the next problem.

For the SUPRASEAL_C2 project, commit 9bb657e5 marks the point where Phase 12's engineering debt was paid down through documentation and characterization. The split API architecture, the memory backpressure fixes, and the empirical scaling laws are now preserved for anyone who clones the repository. The system is no longer just optimized—it is understood.

References

[1] articles/seg_32/chunk_0/consolidation_and_characterization_phase_12.md — "From Engineering to Production Readiness: The Consolidation and Characterization of Phase 12"