From Failure to Blueprint: Documenting the Phase 11 Memory-Bandwidth Optimization Plan

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, where every second of proof latency translates directly into operational cost, the difference between a successful optimization and a failed one is measured not just in throughput numbers, but in the quality of the post-mortem. This article examines a pivotal segment of an opencode coding session—spanning messages 2719 through 2740—where an AI assistant and a human user navigated the aftermath of a failed optimization attempt (Phase 10) and produced a detailed, evidence-based plan for the next phase (Phase 11) of the SUPRASEAL_C2 Groth16 proof generation pipeline.

The arc of this chunk is remarkable: it begins with the assistant reading source files to understand the Rust-side deallocation architecture, continues through a systematic documentation effort that produces a comprehensive design specification, and culminates in disciplined git hygiene as the team prepares to commit their knowledge to version control. Along the way, we see a masterclass in systems optimization—diagnosing bottlenecks through careful measurement, validating assumptions through code reading, designing targeted interventions that address root causes rather than symptoms, and documenting everything for posterity.

The Context: A Pipeline Under Pressure

To appreciate the work in this chunk, one must understand the battlefield. The SUPRASEAL_C2 system is a Groth16 proof generator for Filecoin storage proofs, a pipeline that orchestrates CPU-based constraint synthesis (the "PCE MatVec" phase, which evaluates sparse matrix-vector multiplications across partitioned circuits) with GPU-accelerated multi-scalar multiplications (MSM) and number-theoretic transforms (NTT). By the time we reach this chunk, the system has already undergone ten phases of optimization. Phase 9 introduced PCIe transfer optimization, achieving 14.2% throughput improvement. Phase 10 attempted a bold two-lock GPU interlock architecture—and failed spectacularly.

The Phase 10 failure was not a simple bug; it was a fundamental architectural mismatch. The two-lock design attempted to allow multiple GPU workers to overlap their compute and memory operations by splitting the GPU mutex into separate locks for PCIe transfer and kernel launch. But CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations—they synchronize across all threads on a device, defeating the purpose of splitting the lock. Furthermore, the 16 GB VRAM could not accommodate pre-staged buffers from multiple workers simultaneously. The design was abandoned, and the code was reverted to Phase 9's proven single-lock approach.

But the failure was not wasted. The post-mortem analysis of Phase 9 benchmarks revealed a surprising finding: at high concurrency, GPU utilization reached 90.8%, but throughput plateaued at ~38 seconds per proof. The bottleneck was not GPU compute, not PCIe bandwidth, and not lock contention—it was DDR5 memory bandwidth. The CPU-side phases of the pipeline—synthesis (sparse matrix-vector multiplication) and prep_msm/b_g2_msm—were competing for the same memory channels, inflating each other's execution times under load.

This diagnosis led to the design of Phase 11, a three-intervention plan targeting memory-bandwidth-aware pipeline scheduling. The interventions were:

  1. Bound async_dealloc to a single thread, eliminating TLB shootdown storms from concurrent munmap() calls freeing ~37 GiB (C++ side) and ~130 GiB (Rust side) per proof.
  2. Reduce the groth16_pool thread count from 192 to ~32, shrinking b_g2_msm's memory footprint and L3 cache competition with synthesis.
  3. Add a lightweight atomic throttle flag to briefly pause some synthesis workers during the b_g2_msm window, when all 192 groth16_pool threads run Pippenger simultaneously with 192 rayon synthesis threads.

The Research Phase: Reading Before Writing

The chunk opens with the assistant engaged in systematic information gathering. Message [msg 2719] shows the assistant reading lines 388–396 of /home/theuser/curio/extern/bellperson/src/groth16/prover/supraseal.rs—the Rust-side deallocation code. This is the counterpart to the C++ async deallocation that the assistant had already examined. The comment at line 393 is telling: "Move large synthesis data (~130 GB for 10 circuits of 32 GiB PoRep) into a background thread for deallocation, so the caller gets results immediately."

This read operation reveals the dual-deallocation architecture: both C++ and Rust sides of the FFI boundary perform their own background-thread deallocations. The C++ code frees ~37 GiB of GPU-side buffers via std::thread(...).detach(). The Rust code frees an even larger ~130 GiB of ProvingAssignment data using a similar pattern. Both create the same problem: unbounded concurrent deallocation threads that trigger TLB shootdown storms.

The assistant's decision to read this specific file at this specific moment reveals a methodical research process. Having just received user approval for the revised Phase 11 plan, the assistant needs to translate high-level intervention descriptions into concrete code changes. Intervention 1—"bound async_dealloc to a single thread"—requires understanding both deallocation paths. Without reading the Rust-side code, the implementation plan would be incomplete, addressing only half the problem.

This pattern of systematic information gathering continues in subsequent messages. The assistant reads the FFI boundary structure to understand the b_g2_msm/prep_msm timing and call structure ([msg 2720]), verifies that the gpu_threads configuration parameter is properly plumbed through the system ([msg 2721]), examines the SpMV evaluation code to understand where the throttle check should be inserted ([msg 2722]), and reads the synthesis pipeline to trace how the throttle pointer would be threaded through ([msg 2723]).

Each of these read operations serves a specific purpose in grounding the implementation plan in the actual codebase structure. The assistant is not designing from abstract principles; it is designing from concrete code analysis. This is the hallmark of disciplined performance engineering: verify assumptions against code before writing the plan.

The Design: A Comprehensive Implementation Blueprint

Message [msg 2724] is the centerpiece of this chunk—a comprehensive implementation plan for Phase 11, organized around the three targeted interventions. The assistant produces an extraordinarily detailed document covering every file to modify, every risk to assess, and every expected outcome.

Intervention 1: Bound async_dealloc

The first intervention targets TLB shootdown storms. When the pipeline finishes processing a partition, it must deallocate massive data structures. The C++ side frees ~37 GB via a detached thread inside generate_groth16_proofs_c(). The Rust side frees ~130 GB via a background thread spawned after the FFI call returns. Under high concurrency, multiple dealloc threads can run simultaneously, each calling munmap() on large memory regions. This triggers Translation Lookaside Buffer (TLB) shootdown inter-processor interrupts (IPIs) across all CPU cores, forcing them to flush TLB entries and reload page table entries.

The fix is elegant: serialize deallocation to a single thread. The assistant proposes adding a static std::mutex dealloc_mtx to the C++ code, wrapping the dealloc thread body with lock_guard. On the Rust side, a separate static Mutex<()> serializes the Rust dealloc thread. The assistant carefully considers whether to use a shared mutex passed through FFI, but correctly judges that separate mutexes are sufficient since the C++ and Rust deallocs run sequentially for the same proof.

The risk analysis is thorough: if dealloc takes 2.9 seconds and partition completion happens every 2 seconds, the dealloc mutex becomes a bottleneck with one queued dealloc. But this is acceptable because the detached threads simply pile up slightly, with only one running at a time. The expected impact is 2-5% throughput improvement at high concurrency.

Intervention 2: Reduce groth16_pool Size

The second intervention targets the b_g2_msm phase directly. With the default configuration, the groth16_pool contains 192 threads. When b_g2_msm runs its Pippenger MSM on G2 points, all 192 threads participate, each allocating bucket arrays. With ~22 million G2 points and a window size of ~15 bits, each thread allocates roughly 6 MB of bucket RAM. Across 192 threads, that's ~1.1 GiB of bucket RAM—a significant allocation that evicts useful data from the L3 cache.

The fix is a configuration change: set gpu_threads = 32 in the [gpus] section of the cuzk TOML configuration file. This reduces the groth16_pool from 192 threads to 32 threads. The impact on b_g2_msm is a slowdown from ~0.4 seconds to 0.5-0.7 seconds, as fewer threads partition the Pippenger work. But the reduction in L3 cache pollution is dramatic: 32 threads × 6 MB = 192 MB total bucket RAM, compared to 1.1 GiB with 192 threads—a 6x reduction.

The key insight is that b_g2_msm runs after the GPU lock is released, concurrent with the next worker's GPU kernels. A 0.3-second slowdown in b_g2_msm does not affect GPU throughput because the GPU is already busy with the next partition's kernels. The net effect on overall throughput should be positive because synthesis workers retain better L3 cache residency.

Intervention 3: Synthesis Pause During b_g2_msm

The third intervention is the most complex and ambitious. It addresses the direct competition between b_g2_msm's 192 groth16_pool threads and the 192 rayon synthesis threads during the 0.4-0.7 second b_g2_msm window. During this window, 384 threads compete for 12 L3 cache domains on the Zen4 processor.

The design is a cross-language signaling mechanism. When b_g2_msm starts, the C++ code sets a shared atomic flag. The Rust synthesis code periodically checks this flag during its SpMV loop. If the flag is set, a fraction of the synthesis threads yield, briefly reducing memory pressure and giving b_g2_msm uncontested bandwidth.

The assistant explores four implementation options:

The Discovery That Reshaped Everything

One of the most critical moments in this chunk is the assistant's discovery that prep_msm with num_circuits=1 is actually single-threaded. This discovery, made by tracing through the par_map implementation in thread_pool_t.hpp ([msg 2710]), fundamentally changed the optimization strategy.

The par_map function computes num_workers = min(pool_size, num_steps) where num_steps = (num_items + stride - 1) / stride. With num_items=1 and stride=1, num_steps=1, so num_workers=1 regardless of pool size. The entire prep_msm body runs on a single thread.

This had cascading implications:

The Documentation Phase: From Conversation to Canonical Record

After the user's directive at [msg 2725]—"Write down improvement plan"—the assistant shifts from design to documentation. This is a deliberate choice by the user, reflecting a project management philosophy that values documentation as a forcing function for clarity.

The assistant first reads the Phase 10 proposal document ([msg 2728]) to ground the Phase 11 documentation in the lessons learned. This is a crucial step: the Phase 11 proposal is not a fresh start but a direct response to Phase 10's failure. By reading the original proposal, the assistant ensures the post-mortem accurately describes what was attempted and why the approach was fundamentally flawed.

The assistant then writes c2-optimization-proposal-11.md ([msg 2729]), a detailed design specification covering all three interventions with implementation details, risk assessments, and expected outcomes. The document includes a comprehensive summary table mapping each intervention to the specific files that need modification, spanning four programming languages (C++, CUDA, Rust, and TOML configuration) and crossing multiple library boundaries.

Next, the assistant updates cuzk-project.md ([msg 2730]) with the Phase 10 post-mortem and Phase 11 roadmap entry. This is the project's central roadmap file, and adding the post-mortem ensures that future developers understand why the two-lock approach was tried and why it failed, preventing repeated exploration of the same dead end.

The assistant then verifies the documentation by reading the "Related Documents" table in cuzk-project.md ([msg 2736]), confirming that the Phase 11 entry correctly references the new proposal document. This attention to cross-referencing ensures the documentation remains internally consistent.

The Git Hygiene: Disciplined Version Control

The chunk concludes with a series of messages focused on git hygiene. At [msg 2738], the assistant runs git status, git diff --stat, and git log --oneline -5 to understand the repository state before committing. This reveals a nuanced situation: extern/supraseal-c2/cuda/groth16_cuda.cu is already staged (from the Phase 10 reversion), while cuzk-project.md is modified but unstaged (the new documentation).

At [msg 2739], the assistant investigates the staged change more closely, running git diff --cached --stat to confirm that it consists of 2 insertions and 17 deletions—exactly the signature of the Phase 10→Phase 9 revert. This verification step ensures that the working tree is in the correct state before proceeding with Phase 11 implementation.

The user's response at [msg 2740] is an empty message—a silent signal that communicates approval without the need for explicit confirmation. In the context of a well-functioning human-AI partnership where trust has been earned and context is shared, this empty message functions as a continuation signal, closing the loop on the documentation phase and unlocking the next phase of implementation work.

The Broader Significance

This chunk exemplifies several principles of disciplined performance engineering that are worth highlighting:

Diagnose before prescribing. The assistant spent substantial effort diagnosing the DDR5 memory bandwidth contention through waterfall timing analysis, TIMELINE event extraction from daemon logs, and careful analysis of thread pool behavior. Only after understanding the root cause did it propose interventions.

Verify assumptions against code. The discovery that prep_msm is single-threaded—a fact that could only be determined by reading the par_map implementation—fundamentally changed the optimization strategy. Without this verification, the team would have wasted effort on a semaphore interlock with minimal impact.

Research both sides of every boundary. The FFI boundary between Rust and C++ is a common source of optimization blind spots. By reading both the C++ deallocation code and the Rust deallocation code, the assistant ensured its plan addresses the complete picture.

Document failures alongside successes. The Phase 10 post-mortem in cuzk-project.md serves the same function as a scientific paper's "negative results" section—it prevents others from wasting time on approaches that have been proven infeasible, and it preserves the reasoning that led to that conclusion.

Maintain disciplined git hygiene. The assistant's careful verification of staged changes before committing ensures that the commit history accurately reflects the project's evolution, providing a clean baseline for future implementation work.

Conclusion

The chunk spanning messages 2719–2740 represents a complete cycle of systems optimization: diagnose the bottleneck, understand its root causes, design targeted interventions, document the plan with sufficient detail for execution, and preserve the knowledge in version control. The Phase 11 plan, with its three interventions targeting TLB shootdown storms, L3 cache pollution, and memory-phase overlap, is a model of evidence-based engineering.

But perhaps the most impressive aspect of this chunk is its intellectual honesty. The assistant does not present the plan as a guaranteed success. It characterizes the estimates as conservative, acknowledges the risks, and provides fallback options. It documents what was learned from the Phase 10 failure and how that learning informed the Phase 11 design. This is the hallmark of a mature engineering approach: learning from failure, designing with humility, and measuring with rigor.

Whether the three interventions achieve their estimated 11% throughput improvement remains to be seen—that will be determined by the benchmarks that follow. But the plan itself, and the disciplined process that produced it, is a model of how to approach system optimization at scale.