The Documentation That Closes the Loop: Formalizing Knowledge After Failure

In the high-stakes world of Filecoin proof generation optimization, where every second shaved off a 38-second proving cycle translates directly into competitive advantage, the difference between a successful optimization and a failed one is often measured not just in throughput numbers, but in the quality of the post-mortem. Message [msg 2730] of this opencode session captures a quiet but critical moment: the assistant updates the main project documentation to record both a failure analysis and a new roadmap. The message itself is deceptively brief:

[assistant] Now update the main project doc to add Phase 10 post-mortem results and the Phase 11 timeline entry: [edit] /home/theuser/curio/cuzk-project.md Edit applied successfully.

Behind this terse confirmation lies a dense weave of investigation, discovery, abandonment, and redirection. This message is the documentation step that closes the loop on a failed optimization attempt (Phase 10) and formalizes the path forward (Phase 11). It is the moment when ephemeral debugging insights become permanent project knowledge.

Why This Message Was Written

The immediate trigger was the user's instruction at [msg 2725]: "Write down improvement plan." But the deeper motivation traces back through an entire segment of work. The assistant had just spent several rounds investigating a failed Phase 10 optimization—a two-lock GPU interlock design intended to improve throughput by allowing multiple GPU workers to overlap their PCIe transfers and kernel launches. That design had to be abandoned after discovering fundamental CUDA device-global synchronization conflicts: the 16 GB VRAM could not accommodate pre-staged buffers from multiple workers simultaneously, and CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo operate at device scope, defeating the lock split's purpose entirely.

After reverting to the proven Phase 9 single-lock approach, the assistant ran comprehensive benchmarks across concurrency levels and performed a waterfall timing analysis that revealed the true bottleneck: DDR5 memory bandwidth contention. The synthesis phase and the b_g2_msm (multi-scalar multiplication on the G2 curve) phase were competing for the same memory channels, causing both to inflate under load. This led to a revised three-intervention plan for Phase 11, targeting TLB shootdown storms from concurrent munmap() calls, thread pool oversubscription in b_g2_msm, and memory-phase overlap between synthesis and b_g2_msm.

The message at [msg 2730] serves to cement these findings into the project's permanent record. Without this documentation step, the Phase 10 failure analysis would remain scattered across conversation history and the Phase 11 plan would exist only in the assistant's working memory. The cuzk-project.md file is the project's central roadmap—it tracks what phases have been completed, what was learned, and what comes next. Adding the Phase 10 post-mortem ensures that future developers (or the same developer returning weeks later) understand why the two-lock approach was tried and why it failed, preventing repeated exploration of the same dead end.

Input Knowledge Required

To understand the significance of this message, one must grasp several layers of context. First, the technical architecture: the cuzk SNARK proving engine generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol, a process that involves synthesizing circuit constraints via sparse matrix-vector multiplication (SpMV) on CPU, then performing multi-scalar multiplications (MSM) on GPU. The system uses a partition-based dispatch architecture where multiple GPU workers operate concurrently, coordinated by mutexes passed through C++ FFI into Rust.

Second, the optimization history: Phase 9 had achieved a 51% reduction in GPU kernel time through pinned DMA pre-staging and deferred Pippenger synchronization, but this shifted the bottleneck to the CPU side—specifically to DDR5 memory bandwidth contention between the rayon-based synthesis threads and the groth16_pool threads running Pippenger MSM.

Third, the Phase 10 failure: the two-lock design attempted to allow three GPU workers to overlap their work by splitting the GPU mutex into two separate locks, one for PCIe transfer and one for kernel launch. This failed because CUDA's device-global synchronization semantics meant that operations on one lock still blocked operations on the other, and the VRAM capacity was insufficient for pre-staged buffers from multiple workers.

Fourth, the revised understanding of parallelism: a critical insight at [msg 2717] revealed that prep_msm with num_circuits=1 is actually single-threaded (the par_map with one item runs on one thread), not multi-threaded as previously assumed. This changed the optimization strategy significantly—the original plan to interlock prep_msm with synthesis was overkill, and the real target should be b_g2_msm, the only phase where the groth16_pool goes fully parallel with all 192 threads.

Output Knowledge Created

This message produces two distinct forms of output knowledge. The first is the Phase 10 post-mortem embedded in cuzk-project.md: a permanent record documenting why the two-lock GPU interlock design was attempted, what fundamental CUDA limitations made it infeasible, and what the benchmarking data showed after reverting. This prevents future optimization efforts from repeating the same mistake and provides a reference point for understanding the system's constraints.

The second is the Phase 11 roadmap entry: a structured plan with three interventions, each with specific code changes, expected throughput improvements, and risk assessments. The plan includes bounding async_dealloc to a single thread to eliminate TLB shootdown storms, reducing the groth16_pool thread count from 192 to 32 to shrink b_g2_msm's memory footprint and L3 cache pressure, and adding a lightweight atomic throttle flag to briefly pause synthesis workers during the b_g2_msm window. The expected cumulative improvement is 11% (from 38.0s to ~34.0s per proof at high concurrency).

Beyond these specific entries, the message creates a broader form of knowledge: a disciplined engineering practice of documenting failures alongside successes. The cuzk-project.md file now tells a complete story—not just of optimizations that worked, but of the dead ends that were explored and why they were abandoned. This is invaluable for maintaining engineering velocity over long projects.

Assumptions and Mistakes

The message itself contains no explicit assumptions or mistakes—it is a documentation action. But it implicitly validates several assumptions that were made earlier in the segment. The assumption that prep_msm was multi-threaded (corrected at [msg 2717]) is now baked into the Phase 11 plan, which correctly targets b_g2_msm instead. The assumption that CUDA device-global synchronization would not interfere with the two-lock design (disproved by Phase 10's failure) is now recorded as a lesson learned.

One subtle assumption embedded in the Phase 11 plan is that the three interventions are independent and their effects will be additive. The expected throughput table shows cumulative improvement from 38.0s to 34.0s, but in practice, interventions may interact non-linearly. For example, reducing groth16_pool threads (Intervention 2) might change the timing of b_g2_msm enough that the atomic throttle flag (Intervention 3) becomes less necessary, or the TLB shootdown fix (Intervention 1) might reduce system noise enough that the other interventions show smaller marginal gains. The assistant acknowledges this implicitly by proposing to benchmark after each intervention independently.

The Thinking Process

The thinking process visible across the preceding messages reveals a methodical, evidence-driven approach. At [msg 2717], the assistant performs a critical correction: it realizes that prep_msm with num_circuits=1 uses par_map on a single item, which executes on one thread. This is a classic case where an API name (par_map) suggests parallelism but the actual behavior depends on the input size. The assistant doesn't just accept the surface-level understanding—it traces through the code to verify thread counts.

At [msg 2724], the assistant produces an extraordinarily detailed implementation plan, considering multiple implementation options for each intervention. For Intervention 1, it explores using semaphore_t from sppark, then realizes the semaphore initializes to 0 (not 1), considers std::counting_semaphore<1> from C++20, and finally settles on a simple std::mutex. It also considers whether the C++ and Rust dealloc threads should share a mutex or use separate ones, weighing the complexity of FFI plumbing against the risk of concurrent deallocations.

For Intervention 3, the assistant evaluates four distinct approaches (shared atomic via FFI, Rust-side Condvar, rayon yield heuristic, and custom thread pool), analyzing the trade-offs of each before recommending the shared atomic approach. This level of design exploration—considering multiple architectures, evaluating their feasibility against the actual codebase, and making a reasoned recommendation—is characteristic of deep systems engineering.

The message at [msg 2730] itself shows no explicit reasoning, but its placement in the workflow reveals an important meta-cognitive process: the assistant recognizes that documentation is not an afterthought but an integral part of the engineering cycle. After discovering the Phase 10 failure, designing the Phase 11 plan, and writing the detailed proposal document (c2-optimization-proposal-11.md), the assistant takes the final step of integrating this knowledge into the project's central roadmap. This closes the loop between investigation and institutional memory.

Broader Significance

This message exemplifies a pattern that recurs throughout successful engineering projects: the documentation of failure is as important as the documentation of success. The Phase 10 post-mortem in cuzk-project.md serves the same function as a scientific paper's "negative results" section—it prevents others from wasting time on approaches that have been proven infeasible, and it preserves the reasoning that led to that conclusion.

For a project as complex as the cuzk SNARK proving engine, where the optimization space spans C++ CUDA kernels, Rust FFI boundaries, thread pool configurations, and memory architecture constraints, this kind of disciplined documentation is what separates a hack from a well-engineered system. The message at [msg 2730] may be only two lines long, but it represents the culmination of a rigorous investigative process—one that transformed a failed optimization into a deeper understanding of the system's true bottlenecks, and that understanding into a concrete, evidence-based plan for the next phase of work.