The Quiet Documentation: Capturing Phase 12 Benchmark Results in cuzk-project.md

Introduction

In the lifecycle of any serious engineering project, there comes a moment when the furious pace of implementation must yield to the discipline of documentation. Message 3257 captures exactly such a moment. The assistant, having just completed a grueling multi-phase optimization campaign for the cuzk SNARK proving engine, issues a brief status update:

[assistant] Now I'll insert the Phase 12 benchmark section after the Phase 7+8 section: [edit] /home/theuser/curio/cuzk-project.md Edit applied successfully.

On its surface, this message is unremarkable—a single sentence followed by a tool invocation and a confirmation. But to understand its significance, one must trace the arc of work that led to this point. This message is the final stitch in a long seam of documentation updates, the last piece of a puzzle that captures months of engineering effort in permanent form. It represents the transition from doing to recording, from ephemeral knowledge in the engineer's mind to durable knowledge in the project's canonical reference document.

The Context: Why This Message Was Written

The immediate trigger for this message was the user's instruction in message 3246: "Write down findings in relevant docs." This directive came at the conclusion of Phase 12—a complex optimization that introduced a split (async) GPU proving API to the cuzk engine, decoupling the b_g2_msm CPU computation from the GPU worker loop so that the GPU could pick up the next synthesized partition approximately 1.7 seconds faster.

Phase 12 was not a single change but a cascade of discoveries and fixes. The initial split API (committed as 99c31c2c) introduced a use-after-free bug in the C++ prep_msm_thread, where a stack-allocated function parameter was captured by reference in a lambda. This was fixed by copying the Assignment<fr_t> array into a heap-allocated struct. Then came the memory backpressure sub-phase (committed as 98a52b33), which addressed an OOM condition at high partition-worker counts through three interventions: early deallocation of NTT evaluation vectors, auto-scaling of channel capacity, and holding the partition semaphore permit until after the channel send completed. The result was a stable pipeline running at pw=12 with 37.7 seconds per proof and a peak RSS of 400 GiB—down from the OOM-triggering 668 GiB.

But all of this knowledge existed only in the assistant's working memory, in git commit messages, and in benchmark log files scattered across the filesystem. The user's instruction to "write down findings" was a call to consolidate this hard-won understanding into the project's permanent documentation. The assistant responded by systematically updating cuzk-project.md, the project's central reference document, with the Phase 12 architecture, results, and deployment guidance.

The Decision-Making Process

The assistant's approach to documentation was methodical and deliberate. It began by exploring the existing documentation structure (message 3247), reading the current state of cuzk-project.md and cuzk.example.toml (messages 3248–3249), and examining the git history to understand what had already been documented (message 3249). This reconnaissance revealed that Phase 12 was entirely undocumented—the last commit touching cuzk-project.md was a737c729 which covered Phase 10's post-mortem and Phase 11's design spec.

With this understanding, the assistant created a structured todo list (message 3250) with five items:

1. Add the Phase 12 section to cuzk-project.md
2. Update the Summary Timeline
3. Update the Stopping Points & Cumulative Impact table
4. Add Phase 12 benchmark results to the E2E Test Results section
5. Update cuzk.example.toml with optimal defaults Each todo was executed in sequence. The Phase 12 architecture section was inserted after the Phase 11 section (message 3252). The Summary Timeline was updated to include the Phase 12 entry (message 3253). The Stopping Points table was revised to reflect the new state (message 3254). And finally, in message 3257, the benchmark results were inserted into the E2E Test Results section. The decision to insert the benchmark section "after the Phase 7+8 section" reveals the assistant's careful attention to document structure. The E2E Test Results section of cuzk-project.md is organized chronologically by phase, with each phase's benchmark data following the previous one. Placing Phase 12's results after Phase 7+8 maintained this logical ordering and ensured the document remained navigable.

Input Knowledge Required

To execute this edit, the assistant needed a rich body of knowledge accumulated across the entire Phase 12 effort. This included:

• The structure of cuzk-project.md: The assistant had to know where the E2E Test Results section began, where the Phase 7+8 benchmark subsection was located, and how the document was organized to insert new content correctly.
• The Phase 12 benchmark data: The assistant needed to know the specific benchmark configurations tested (pw=1/2/5/7/10/12 × gw=1/2), the resulting throughput numbers (from 152 seconds per proof at pw=2 gw=1 to 37.7 seconds at pw=12 gw=2), the memory scaling formula (~69 GiB baseline + pw × ~20 GiB), and the deployment guidance for various system sizes.
• The Phase 12 architecture details: The split API design, the use-after-free fix, the early a/b/c free optimization, the channel capacity auto-scaling, and the partition semaphore permit-through-send fix—all of these needed to be understood to write accurate documentation.
• The project's documentation conventions: The assistant had to match the existing tone, formatting, and level of detail used in earlier phase documentation.

Output Knowledge Created

This message, combined with the preceding edits, produced a permanent record of Phase 12 in the project's canonical documentation. The benchmark section added in this edit would include:

• A systematic low-memory sweep across nine configurations
• The linear memory scaling formula for deployment planning
• Concrete recommendations for 128 GiB through 768 GiB systems
• The observation that gw=2 provides no throughput benefit below pw=10 due to synthesis starvation This knowledge transforms Phase 12 from a set of ephemeral benchmark log files and git commits into actionable guidance for system integrators. Anyone deploying cuzk on hardware with limited memory can now consult cuzk-project.md to determine the optimal partition_workers and gpu_workers_per_device settings for their specific memory budget.

The Thinking Process Visible in the Message

While message 3257 itself is brief, the thinking process behind it is visible in the sequence of actions leading up to it. The assistant did not simply dump benchmark numbers into the file. It:

1. Assessed the current state: Read the file to find the exact insertion point
2. Maintained structural consistency: Chose to insert after the Phase 7+8 section to preserve chronological ordering
3. Followed a systematic plan: Worked through a todo list, completing each item before moving to the next
4. Verified success: Confirmed "Edit applied successfully" before proceeding This methodical approach reflects an understanding that documentation, like code, benefits from careful structure and consistency. A haphazard insertion could break the document's flow or confuse future readers.

Broader Significance

Message 3257 represents the final act of knowledge capture for Phase 12—a phase that began with a bold architectural idea (split the GPU proving API to hide latency) and progressed through debugging, memory pressure diagnosis, and iterative optimization. The documentation effort ensures that the lessons learned are not lost when the engineer moves on to the next challenge.

In the context of the larger cuzk project, this message is part of a pattern: each phase of optimization is followed by a documentation update that captures what was done, why it was done, and what results were achieved. This discipline transforms individual engineering insights into institutional knowledge, making the project more maintainable and its design decisions more transparent to future contributors.

The quiet "Edit applied successfully" at the end of message 3257 belies the complexity of the work it records. It is the sound of knowledge being preserved—a small but essential act in the life of a software project.