The Memory-Bandwidth-Aware Pipeline: Designing Phase 11 of the cuzk SNARK Proving Engine

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, every second of proof latency translates directly into operational cost. The cuzk SNARK proving engine, a sophisticated pipeline that orchestrates GPU-accelerated Groth16 proof generation for Filecoin storage proofs, had already undergone ten phases of optimization. Each phase had squeezed more throughput from the system: Phase 7 introduced per-partition dispatch, Phase 8 implemented a dual-GPU-worker interlock, and Phase 9 optimized PCIe transfers for a 14.2% throughput improvement. Phase 10 attempted a bold two-lock GPU interlock design—and failed spectacularly, discovering fundamental CUDA device-global synchronization conflicts that made the design untenable.

The message under analysis—message index 2724 in the conversation—represents a pivotal moment in this optimization journey. After the Phase 10 failure, the assistant performed a comprehensive post-mortem, reverted to the Phase 9 single-lock architecture, ran exhaustive benchmarks across concurrency levels, and performed a detailed waterfall timing analysis. That analysis revealed the true bottleneck: DDR5 memory bandwidth contention. With this diagnosis in hand, the assistant designed Phase 11, a three-intervention plan to reduce memory bandwidth contention in the pipeline. The message is the detailed implementation plan for Phase 11, covering every file to modify, every risk to assess, and every expected outcome.

This article examines that message in depth: the reasoning that produced it, the assumptions that shaped it, the knowledge it required, and the knowledge it created. It explores how the assistant navigated the tension between parallelism and contention, how it corrected earlier misunderstandings about thread pool behavior, and how it designed a cross-language signaling mechanism to coordinate work across C++, Rust, and CUDA boundaries.

The Context: A Pipeline Under Pressure

To understand the significance of this message, one must first understand the architecture it targets. The cuzk SNARK proving engine is a complex pipeline that generates Groth16 proofs for Filecoin's proof-of-replication protocol. The pipeline operates at massive scale: each proof requires processing circuits with hundreds of millions of constraints, streaming gigabytes of sparse matrix data through CPU memory, and orchestrating GPU kernels for multi-scalar multiplication (MSM) and number-theoretic transforms (NTT).

The pipeline's throughput is measured in proofs per hour, and the key metric is the time to generate a single proof under full load. At Phase 9, the baseline was approximately 38.0 seconds per proof at maximum concurrency (c=20 workers, j=15 jobs). The theoretical limit, given two GPU workers each processing 10 partitions with an average partition time of 1.5 seconds, is approximately 30.0 seconds per proof—a 21% improvement still on the table.

The Phase 10 two-lock design attempted to improve GPU utilization by allowing two GPU workers to overlap their lock-hold times, effectively doubling GPU throughput. But the design ran into a fundamental constraint: CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations. When two workers each held their own lock and called these APIs, the operations conflicted, causing out-of-memory errors and performance regressions. The design was abandoned, and the code was reverted to Phase 9's proven single-lock approach.

The post-mortem analysis of Phase 9 benchmarks revealed a surprising finding: at high concurrency, GPU utilization reached 90.8%, but throughput plateaued at ~38 seconds per proof. The bottleneck was not GPU compute, not PCIe bandwidth, and not lock contention—it was DDR5 memory bandwidth. The CPU-side phases of the pipeline—synthesis (sparse matrix-vector multiplication) and prep_msm (bitmap classification and tail MSM population)—were competing for the same memory channels, inflating each other's execution times under load.

The Discovery That Changed Everything

The immediate predecessor to the subject message is a chain of investigation that began with the user's concern: "We should be careful to not kill parallelism btw" ([msg 2704]). This concern reflected a fundamental tension in the optimization approach. Every intervention that throttles parallelism to reduce memory contention risks starving the GPU of synthesized partitions, which would increase GPU idle time and reduce overall throughput.

The assistant's response ([msg 2705]) laid out a careful analysis of where parallelism is critical and where it is wasteful. The critical list included the 10 concurrent partition syntheses, rayon parallelism within a single synthesis, prep_msm thread pool parallelism, and NTT/MSM GPU overlaps. The wasteful list included async_dealloc (pure cleanup), the groth16_pool at 192 threads, and simultaneous prep_msm with full synthesis pack.

But then the assistant made a critical discovery. By reading the actual code in groth16_cuda.cu and the thread_pool_t.hpp implementation, it discovered that par_map(num_circuits, ...) with num_circuits=1 (the per-partition pipeline mode) runs on exactly one thread. The par_map implementation computes num_workers = min(pool_size, num_steps) where num_steps = (num_items + stride - 1) / stride. With num_items=1 and stride=1, num_steps=1, so num_workers=1 regardless of pool size.

This was a pivotal insight. The prep_msm phase, previously assumed to benefit from the full 192-thread groth16_pool, was actually single-threaded. The 1.9 seconds of prep_msm execution was a sequential memory scan, not a parallel computation. This meant that reducing the groth16_pool size would not affect prep_msm at all—it only affected b_g2_msm, the Pippenger MSM on G2 points that runs after the GPU lock is released.

The corrected understanding of parallelism is captured in a table in the assistant's analysis ([msg 2717]):

| Operation | With num_circuits=1 | Thread Pool | Threads Used | |---|---|---|---| | prep_msm pass 1 (bitmap classification) | par_map(1, ...) → single thread | groth16_pool | 1 | | prep_msm pass 2 (populate bases/scalars) | par_map(1, ...) → single thread | groth16_pool | 1 | | b_g2_msm (Pippenger) | mult_pippenger(..., &get_groth16_pool()) → full pool | groth16_pool | all (192 default) | | PCE MatVec (synthesis) | par_chunks_mut(8192) + rayon::join | rayon | all (192 default) |

This discovery fundamentally reshaped the optimization strategy. The original plan had targeted a semaphore interlock between prep_msm and synthesis, but since prep_msm was only one thread, the contention it created was minimal. The real contention point was b_g2_msm, where all 192 groth16_pool threads ran Pippenger simultaneously with 192 rayon synthesis threads—384 threads competing for 12 L3 cache domains on a Zen4 processor.

The Subject Message: Phase 11 Detailed Implementation Plan

With the corrected understanding in hand, and after receiving user approval for the revised three-intervention plan ([msg 2717]), the assistant produced the subject message: a comprehensive implementation plan for Phase 11, organized around three targeted interventions to reduce DDR5 memory bandwidth contention.

Intervention 1: Bound async_dealloc to One Concurrent Thread

The first intervention targets a subtle but pernicious source of system-wide slowdown: TLB shootdown storms caused by concurrent munmap() calls.

When the pipeline finishes processing a partition, it must deallocate massive data structures. The C++ side frees approximately 37 GB of split vectors, bit vectors, and tail MSM bases. The Rust side frees approximately 130 GB of proving assignments containing a/b/c scalar vectors. These deallocations are intentionally moved to detached threads to avoid blocking the caller—the C++ dealloc thread is detach()ed inside generate_groth16_proofs_c() before it returns, and the Rust dealloc thread is spawned after the FFI call returns.

Under high concurrency, multiple dealloc threads can run simultaneously. Each thread calls munmap() on large memory regions, which triggers Translation Lookaside Buffer (TLB) shootdown inter-processor interrupts (IPIs) across all CPU cores. These IPIs force all cores to flush their TLB entries and reload page table entries, causing a system-wide stall. With 2-3 dealloc threads running concurrently, the TLB shootdown storms can significantly degrade the performance of all other threads—including the synthesis workers that are actively computing sparse matrix-vector multiplications.

The fix is elegant in its simplicity: serialize deallocation to a single thread. The assistant proposes adding a static std::mutex dealloc_mtx to the C++ code, wrapping the dealloc thread body with lock_guard. On the Rust side, a separate static Mutex<()> serializes the Rust dealloc thread. Since the C++ dealloc and Rust dealloc run sequentially for the same proof (C++ starts first, Rust starts ~1ms later after the FFI returns), and with two GPU workers, the worst case is two C++ dealloc threads and two Rust dealloc threads. Separate mutexes limit this to one C++ plus one Rust dealloc, which is acceptable.

The assistant carefully considers a more sophisticated approach—passing a shared mutex through FFI—but correctly judges that the simpler approach is sufficient. The risk analysis is also thorough: if dealloc takes 2.9 seconds and partition completion happens every 2 seconds, the dealloc mutex becomes a bottleneck with one queued dealloc. But this is acceptable because the detached threads simply pile up slightly, with only one running at a time.

The expected impact is 2-5% throughput improvement at high concurrency, primarily from eliminating TLB shootdown IPIs during steady-state proving.### Intervention 2: Reduce groth16_pool Size

The second intervention targets the b_g2_msm phase directly. With the default configuration, the groth16_pool contains 192 threads. When b_g2_msm runs its Pippenger MSM on G2 points, all 192 threads participate, each allocating bucket arrays for the multi-scalar multiplication algorithm. With approximately 22 million G2 points and a window size of approximately 15 bits, each thread allocates roughly 6 MB of bucket RAM (32K entries × 192 bytes per bucket_fp2_t). Across 192 threads, that's approximately 1.1 GiB of bucket RAM—a significant allocation that evicts useful data from the L3 cache.

The fix is a configuration change: set gpu_threads = 32 in the [gpus] section of the cuzk TOML configuration file. This reduces the groth16_pool from 192 threads to 32 threads. The impact on b_g2_msm is a slowdown from approximately 0.4 seconds to 0.5-0.7 seconds, as fewer threads partition the Pippenger work. But the reduction in L3 cache pollution is dramatic: 32 threads × 6 MB = 192 MB total bucket RAM, compared to 1.1 GiB with 192 threads—a 6x reduction.

The key insight here is that b_g2_msm runs after the GPU lock is released, concurrent with the next worker's GPU kernels. A 0.3-second slowdown in b_g2_msm does not affect GPU throughput because the GPU is already busy with the next partition's batch addition and tail MSM kernels. The net effect on overall throughput should be positive because synthesis workers (running in the rayon thread pool) retain better L3 cache residency, making their SpMV computations faster.

The assistant correctly notes that prep_msm is unaffected by this change because it always runs on a single thread regardless of pool size. The only phase affected is b_g2_msm. This is a clean, targeted intervention with no negative side effects on GPU throughput.

Intervention 3: Synthesis Pause During b_g2_msm

The third intervention is the most complex and ambitious. It addresses the direct competition between b_g2_msm's 192 groth16_pool threads and the 192 rayon synthesis threads during the 0.4-0.7 second b_g2_msm window. During this window, 384 threads compete for 12 L3 cache domains on the Zen4 processor, each thread streaming large working sets through the memory hierarchy.

The design is a cross-language signaling mechanism. When b_g2_msm starts, the C++ code sets a shared atomic flag. The Rust synthesis code periodically checks this flag during its SpMV loop. If the flag is set, a fraction of the synthesis threads yield, briefly reducing memory pressure and giving b_g2_msm uncontested bandwidth.

The assistant explores four implementation options:

Option A: Shared atomic flag via FFI. The C++ code allocates a gpu_resources struct containing both the GPU mutex and a std::atomic<int> membw_throttle. Before b_g2_msm, the flag is set to 1; after, it is cleared to 0. The Rust code receives a pointer to this atomic and checks it periodically in the SpMV loop. This is the most precise and controllable approach.

Option B: Rust-side tokio::sync::Notify or std::sync::Condvar. A global static atomic in Rust code, set before calling generate_groth16_proofs_c(). But this requires the C++ code to call back into Rust mid-function, which is architecturally complex.

Option C: Rayon yield_now() heuristic. Instead of explicit signaling, periodically call rayon::yield_now() in the SpMV loop. This lets rayon steal work from other tasks, naturally reducing active synthesis threads when the system is busy. But it doesn't specifically target the b_g2_msm window.

Option D: Reduce rayon parallelism for MatVec only. Use a custom rayon thread pool with fewer threads for the MatVec phase, reserving threads for the groth16_pool. This is architecturally clean but requires creating a second rayon pool.

The assistant recommends Option A, the shared atomic flag, and provides a detailed implementation sketch:

  1. Extend the gpu_mtx opaque pointer to a gpu_resources struct carrying both the GPU mutex and the atomic flag.
  2. In C++: set resources->membw_throttle.store(1, memory_order_release) before b_g2_msm, clear it after.
  3. In Rust: check throttle_flag.load(Ordering::Acquire) periodically in spmv_parallel(). If set, yield or sleep briefly.
  4. Thread the pointer through the synthesis pipeline: engine.rs → pipeline.rs → evaluate_pce() → spmv_parallel(). The overhead analysis is meticulous. With one check per 64 chunks (each chunk is 8192 rows), that's approximately 250 checks per SpMV call. Each check is a single atomic load with acquire semantics—negligible overhead compared to the millions of row evaluations in each SpMV pass. The expected impact is a 20-30% faster b_g2_msm (from uncontested bandwidth) at the cost of approximately 0.4 seconds of reduced synthesis parallelism. The net effect should be positive because the synthesis workers resume immediately when the flag clears.

The Summary Table: A Blueprint for Implementation

One of the most valuable elements of the subject message is the comprehensive summary table mapping each intervention to the specific files that need modification:

| File | Intervention | Change | |---|---|---| | extern/supraseal-c2/cuda/groth16_cuda.cu | 1 | Add static std::mutex dealloc_mtx, wrap dealloc thread | | extern/supraseal-c2/cuda/groth16_cuda.cu | 3 | Change std::mutex* to gpu_resources* struct with membw_throttle atomic | | extern/supraseal-c2/cuda/groth16_cuda.cu | 3 | Set/clear membw_throttle around b_g2_msm | | extern/supraseal-c2/src/lib.rs | 3 | Update FFI signatures for gpu_resources | | extern/bellperson/src/groth16/prover/supraseal.rs | 1 | Add static Mutex<()> for Rust-side dealloc serialization | | extern/bellperson/src/groth16/prover/supraseal.rs | 3 | Thread membw_throttle pointer to caller | | extern/cuzk/cuzk-pce/src/eval.rs | 3 | Add throttle check in spmv_parallel() loop | | extern/cuzk/cuzk-core/src/pipeline.rs | 3 | Pass throttle pointer through synthesis pipeline | | extern/cuzk/cuzk-core/src/engine.rs | 3 | Create gpu_resources instead of raw mutex, pass throttle to synthesis | | Config TOML | 2 | Add gpu_threads = 32 |

This table serves as both a design document and an implementation checklist. It spans four programming languages (C++, Rust, CUDA, and TOML configuration) and crosses multiple library boundaries (supraseal-c2, bellperson, cuzk-pce, cuzk-core). The assistant demonstrates a thorough understanding of the entire codebase, from the low-level CUDA kernel orchestration to the high-level Rust engine configuration.

The Implementation Order and Expected Outcomes

The assistant proposes a careful incremental implementation order:

  1. Intervention 1 (dealloc mutex) — build, benchmark at c=20 j=15
  2. Intervention 2 (gpu_threads=32) — config change only, benchmark
  3. Intervention 3 (membw_throttle atomic) — build, benchmark
  4. Compare all results vs Phase 9 baseline (38.0s/proof at c=20 j=15) This order is strategic. Intervention 1 is the simplest and safest, requiring only a mutex addition. Intervention 2 is a pure configuration change with zero code risk. Intervention 3 is the most complex and risky, requiring cross-language FFI changes and careful benchmarking to tune the throttle behavior. The expected outcomes are presented as a progression: | State | Expected Throughput (c=20 j=15) | Improvement | |---|---|---| | Phase 9 baseline | 38.0s/proof | — | | After Intervention 1 | ~37.0s | ~3% | | After Intervention 1+2 | ~35.5s | ~7% | | After Intervention 1+2+3 | ~34.0s | ~11% | | Theoretical limit | 30.0s | 21% | The assistant is careful to characterize these as conservative estimates, noting that the actual impact of TLB shootdown elimination could be larger if munmap storms are more frequent than estimated. The theoretical limit of 30.0 seconds (21% improvement) represents the ceiling imposed by the pipeline's fundamental structure: two GPU workers, each processing 10 partitions with an average partition time of 1.5 seconds.

The Thinking Process: Reasoning and Decision-Making

The subject message is the culmination of an extended reasoning chain that spans multiple rounds of investigation. To fully appreciate the message, one must trace the thinking process that produced it.

From Assumption to Discovery

The initial assumption was that prep_msm benefited from the full 192-thread groth16_pool. This assumption was reasonable: the code calls get_groth16_pool().par_map(num_circuits, ...), and the pool has 192 threads. But the assistant did not stop at the surface-level API call—it traced through the implementation to discover that par_map with num_items=1 runs on exactly one thread.

This discovery is a textbook example of the value of reading code rather than inferring behavior from APIs. The par_map function in thread_pool_t.hpp computes num_workers = min(pool_size, num_steps) where num_steps = (num_items + stride - 1) / stride. With num_items=1 and stride=1, num_steps=1, so num_workers=1 regardless of pool size. The entire prep_msm body runs on a single thread.

This discovery had cascading implications:

The Trade-off Analysis

The assistant's thinking is characterized by careful trade-off analysis. Every intervention is evaluated for its impact on both the targeted bottleneck and on other parts of the system:

The Cross-Language Coordination Challenge

The most sophisticated reasoning in the subject message concerns the cross-language signaling mechanism for Intervention 3. The assistant explores four options, each with different trade-offs:

Assumptions and Their Validity

The subject message rests on several assumptions, some explicit and some implicit:

Explicit Assumptions

  1. TLB shootdown storms are a significant source of slowdown. This assumption is based on the waterfall timing analysis showing that DDR5 memory bandwidth contention is the bottleneck. TLB shootdowns contribute to this contention by stalling all cores during page table updates. The assumption is reasonable given the large working sets (37 GB C++ dealloc, 130 GB Rust dealloc) and the frequency of deallocation (every ~2 seconds per worker).
  2. b_g2_msm's 0.4s window is the critical contention period. This is based on the TIMELINE analysis showing that b_g2_msm runs after the GPU lock is released, concurrent with the next worker's GPU kernels. The assumption is that this 0.4s window is when memory bandwidth contention is most harmful because both groth16_pool and rayon threads are at peak activity.
  3. Reducing groth16_pool to 32 threads is safe. The assistant estimates that b_g2_msm slows from 0.4s to 0.5-0.7s with 32 threads. This is based on the Pippenger algorithm's scaling behavior: with fewer threads, each thread processes more points, but the total work is the same. The slowdown is bounded by the ratio of threads (192/32 = 6x), but the actual slowdown is less because Amdahl's law applies—there is serial overhead in the Pippenger algorithm that doesn't scale with thread count.
  4. The atomic flag check overhead is negligible. At one check per 64 chunks (each chunk is 8192 rows), there are approximately 250 checks per SpMV call. Each check is a single atomic load (approximately 10-20 nanoseconds on modern hardware). Total overhead: 2.5-5 microseconds per SpMV call, compared to the call's total duration of several seconds.

Implicit Assumptions

  1. The Zen4 processor's 12 L3 cache domains are the relevant contention point. This assumes that the 192 groth16_pool threads and 192 rayon threads are distributed across all 12 L3 domains, with each domain serving 16 threads. The assumption is reasonable for a Zen4 processor with 12 CCDs (each with 8 cores and shared L3 cache).
  2. The C++ dealloc and Rust dealloc threads do not overlap with each other. The assistant notes that the C++ dealloc thread is detached before the FFI returns, and the Rust dealloc thread is spawned after the FFI returns. But with two GPU workers, the C++ dealloc from worker 1 could overlap with the Rust dealloc from worker 2, or with the C++ dealloc from worker 2. The assistant acknowledges this and handles it with separate mutexes.
  3. The gpu_threads config parameter is already plumbed through the system. The assistant verifies this by reading cuzk-core/src/config.rs, confirming that gpu_threads is a recognized configuration parameter with a default value of 0 (meaning "use system default").

Potential Mistakes or Incorrect Assumptions

  1. The impact estimate for Intervention 1 (2-5%) may be conservative. If TLB shootdown storms are a major contributor to the DDR5 bandwidth contention, eliminating them could have a larger effect. The assistant acknowledges this, noting that the actual impact "could be larger if munmap storms are more frequent than estimated."
  2. The assumption that b_g2_msm slowdown doesn't affect GPU throughput depends on the precise timing of the pipeline. If b_g2_msm slows from 0.4s to 0.7s, and the GPU kernels for the next partition complete in 0.5s, then b_g2_msm becomes the new bottleneck. The assistant's analysis assumes that GPU kernels take longer than b_g2_msm, which is supported by the waterfall timing data showing GPU utilization at 90.8%.
  3. The shared atomic flag approach for Intervention 3 assumes that the Rust synthesis code can periodically check the flag without significant overhead. The analysis of one check per 64 chunks is reasonable, but the actual overhead depends on the memory ordering effects of the atomic load. An acquire-load on a frequently-written atomic can cause cache line invalidation traffic if the flag is being written by a thread on a different CCD. This is a subtle but potentially significant effect that the assistant does not fully explore.

Input Knowledge Required to Understand This Message

To fully comprehend the subject message, a reader needs knowledge spanning multiple domains:

CUDA and GPU Programming

Systems Programming and Memory Hierarchy

The cuzk Pipeline Architecture

Cross-Language FFI

The Groth16 Proof System

Output Knowledge Created by This Message

The subject message creates several forms of knowledge that persist beyond the conversation:

A Detailed Implementation Blueprint

The message serves as a complete implementation plan for Phase 11. Any developer familiar with the codebase could implement the three interventions using the message as a guide. The file-by-file breakdown, the code sketches, and the risk assessments provide everything needed for implementation.

A Decision Record for the Phase 10 Post-Mortem

The message implicitly documents why Phase 10 failed and what was learned from it. The key insight—that CUDA device-global synchronization APIs defeat the purpose of splitting locks—is a valuable piece of systems knowledge that informs future design decisions.

A Benchmarking Protocol

The message establishes a clear benchmarking protocol: run at c=20 j=15 (maximum concurrency), measure throughput in seconds per proof, and compare against the Phase 9 baseline of 38.0 seconds. This protocol ensures that each intervention's impact can be measured independently and that the cumulative effect is quantifiable.

A Risk Registry

Each intervention includes a risk assessment that documents potential failure modes:

A Theoretical Performance Model

The expected outcomes table provides a theoretical performance model that maps each intervention to a predicted throughput improvement. This model can be validated against actual benchmark results, and deviations from the model can inform further investigation.

The Broader Significance: From Optimization to Architecture

The subject message represents a shift in thinking about the cuzk pipeline. Earlier optimization phases focused on individual components: making the GPU faster, reducing PCIe transfer time, improving lock efficiency. Phase 11 represents a shift toward system-level optimization that considers the interactions between components.

The key insight is that the bottleneck is not any single component but the competition for a shared resource: DDR5 memory bandwidth. The three interventions target this competition from different angles:

Conclusion

The subject message—message index 2724 in the conversation—is a masterclass in systems optimization. It demonstrates how to diagnose a bottleneck through careful measurement, how to validate assumptions through code reading, how to design targeted interventions that address root causes rather than symptoms, and how to document a complex implementation plan with sufficient detail for execution.

The message is remarkable for its thoroughness: it covers every file that needs modification, provides code sketches for the key changes, analyzes risks and failure modes, estimates expected improvements, and establishes a benchmarking protocol. It spans four programming languages and multiple library boundaries, demonstrating a comprehensive understanding of the entire codebase.

But perhaps the most impressive aspect of the message is its intellectual honesty. The assistant does not present the plan as a guaranteed success. It characterizes the estimates as conservative, acknowledges the risks, and provides fallback options. It documents what was learned from the Phase 10 failure and how that learning informed the Phase 11 design. This is the hallmark of a mature engineering approach: learning from failure, designing with humility, and measuring with rigor.

The Phase 11 plan, as documented in this message, represents the culmination of ten previous optimization phases and a failed eleventh attempt. It is the product of extended investigation, careful measurement, and disciplined reasoning. Whether the three interventions achieve their estimated 11% throughput improvement remains to be seen—that will be determined by the benchmarks that follow. But the plan itself is a model of how to approach system optimization: diagnose the bottleneck, understand its root causes, design targeted interventions, and implement with careful measurement at every step.