The Thundering Herd and the Cross-Sector Insight: How Per-Partition Dispatch Reshaped the cuzk Proving Engine

Introduction

The optimization of a high-performance SNARK proving engine is rarely a linear progression of improvements. More often, it is a journey through successive layers of misunderstanding, each peeled away by empirical data, user insight, and the willingness to abandon cherished assumptions. This article traces one such journey: the multi-session optimization of the cuzk proving engine for Filecoin's Proof-of-Replication (PoRep) protocol, culminating in the Phase 7 per-partition dispatch architecture.

What begins as a seemingly straightforward exercise in parallel synthesis tuning — adding a concurrency parameter to keep the GPU fed — evolves through six distinct phases of discovery, each more surprising than the last. The assistant implements waterfall timeline instrumentation, discovers CPU contention between synthesis and GPU proving, builds thread pool isolation infrastructure, debugs a cross-language static initialization trap, benchmarks exhaustively, and ultimately confronts a fundamental misunderstanding about how the 10 PoRep C2 partitions actually flow through the pipeline. The user's repeated corrections — patient, precise, and insistent — gradually steer the assistant toward a transformative insight: the "thundering herd" created by batching all 10 partitions together is the root cause of GPU idle time, and the solution is to break the batch abstraction entirely, treating each partition as an independent work unit flowing through the pipeline one-by-one.

This article synthesizes the entire arc, examining the key discoveries, the assumptions that were challenged, the knowledge created at each stage, and the architectural vision that emerged. It is a case study in how complex systems optimization really works: not as a smooth progression of improvements, but as a dialectical process of hypothesis, experiment, correction, and reframing.

Part I: The Discovery That Parallel Synthesis Isn't Enough

The story begins with the assistant at a moment of apparent clarity. A comprehensive status report documented the results of an intensive benchmarking campaign. The assistant had implemented two significant features: waterfall timeline instrumentation (emitting structured TIMELINE_EPOCH events tracking each proof's journey through synthesis, channel submission, GPU pickup, GPU start, and GPU end) and parallel synthesis dispatch (using a tokio::sync::Semaphore to allow multiple CPU synthesis tasks to run concurrently).

The benchmark results told a nuanced story. The baseline configuration (sequential synthesis, concurrency=1) achieved 45.3 seconds per proof with 70.9% GPU utilization. With parallel synthesis enabled (synthesis_concurrency=2, client concurrency -j 2), throughput improved to 42.2 seconds per proof — a modest 7% gain — while GPU utilization climbed to 77.8%. But increasing client concurrency to -j 3 degraded performance to 43.1 seconds, and -j 4 was catastrophic at 60.2 seconds.

The root cause was CPU contention. The b_g2_msm step — a multi-threaded Pippenger multi-scalar multiplication that runs on the CPU during GPU proving — competes for the same rayon thread pool as the synthesis tasks. With two concurrent syntheses, both synthesis time (inflated from 39s to 45s) and GPU time (inflated from 27s to 29s) suffered. With three or more concurrent syntheses, the system collapsed under the contention.

The assistant's status report enumerated four options for proceeding: (A) limit rayon threads for synthesis to leave headroom for b_g2_msm, (B) move b_g2_msm off CPU entirely, (C) reduce synthesis time through further optimization, or (D) accept the 7% improvement and ship. The user's response was a masterclass in delegation: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed."

The assistant chose to ask, presenting the user with a structured question. The user selected "Commit + CPU thread isolation" — Option A. This decision set the technical direction for the next phase of work.

Part II: The Thread Isolation Implementation and Its Hidden Trap

With the user's direction established, the assistant began implementing CPU thread pool isolation. The plan was straightforward: create separate rayon thread pools for synthesis and GPU proving (b_g2_msm), preventing them from competing for the same threads. But what should have been a routine configuration change turned into a multi-session debugging odyssey.

The thread isolation implementation required changes across the Rust/C++ boundary. The CUZK_GPU_THREADS environment variable was added to control the number of threads available to the GPU proving pool. The groth16_pool in the C++ CUDA code needed to be initialized lazily rather than statically, because static initialization in C++ runs before the Rust runtime is ready, causing a silent daemon crash.

The debugging journey traced the crash to a C++ static initialization order problem: the groth16_pool was being constructed as a static global variable, which meant it was initialized during C++ static initialization — before the Rust main() function had a chance to set the CUZK_GPU_THREADS environment variable. The fix was to make the pool lazily initialized, using a std::once_flag and std::call_once pattern to defer construction until the first actual use.

This debugging episode is a microcosm of the challenges inherent in cross-language systems programming. The Rust/C++ FFI boundary is a source of subtle timing-dependent bugs that are difficult to reproduce and diagnose. The assistant's systematic approach — using pgrep to check for daemon startup, adding diagnostic logging, tracing the initialization order — demonstrates the discipline required to work effectively at this boundary.

Part III: The Benchmark That Killed a Hypothesis

With thread isolation implemented and the static initialization bug fixed, the assistant ran extensive benchmarks to measure the impact. A comprehensive benchmarking campaign tested multiple configurations: thread pool isolation with gpu_threads=32, gpu_threads=64, and various synthesis concurrency levels.

The results were sobering. The best configuration achieved 42.8 seconds per proof — a 7.2% improvement over the 46.1-second baseline. Thread isolation itself provided only 2-3% gains. The core problem remained stubbornly in place: two concurrent syntheses sharing the 192-thread rayon pool each got approximately 96 effective threads, inflating synthesis from 39 seconds to 45-47 seconds. Since synthesis (45s) still exceeded GPU time (28s), the GPU continued to idle.

The assistant's summary laid out the findings honestly: "Thread pool isolation didn't help much. The fundamental problem is that synthesis time (~39s with full pool) exceeds GPU time (~27s), so even with perfect overlap, the GPU idles waiting for synthesis." The recommended next steps were: (1) reduce synthesis time below 27s, (2) move b_g2_msm to GPU, or (3) ship the current code and integrate with Curio.

This was a pivotal moment. The assistant had invested significant effort in thread isolation — designing, implementing, debugging, and benchmarking it — only to discover that it addressed a secondary symptom rather than the root cause. The willingness to report this honestly, without spinning the results or advocating for further investment in the approach, is a hallmark of intellectual integrity in engineering.

Part IV: The Strategic Pivot and the Research Agents

The user's response redirected the effort: "Start aggents to asses how feasible and likely 1 / 2 is, consider hpc primitives like BLAS." This 14-word message, despite its typos, was a strategic masterstroke. Instead of diving into implementation of either option, the user wanted a feasibility assessment — a risk analysis before committing engineering resources.

The assistant launched two parallel research agents, each with detailed prompts targeting a specific optimization path. The first agent assessed whether synthesis time could be reduced from ~39s to under ~27s through CPU-level optimizations. The second investigated whether the 25-second b_g2_msm computation could be moved from CPU to GPU.

The synthesis of these agent reports contained two critical findings. First, HPC/BLAS libraries are not applicable to BLS12-381 synthesis because the field arithmetic operates on 256-bit modular integers, not IEEE floating-point types. Second, and more importantly, the partitioned pipeline already solves the b_g2_msm problem: when each partition is proved with num_circuits=1, b_g2_msm uses the full multi-threaded CPU Pippenger and completes in 0.4 seconds — not 25 seconds. The 25-second b_g2_msm was an artifact of the batch-all path where 10 circuits shared a single-threaded per-circuit Pippenger.

This second discovery was transformative. It revealed that the entire thread isolation infrastructure — the CUZK_GPU_THREADS environment variable, the lazy initialization of the groth16_pool, the weeks of debugging and benchmarking — was addressing a problem that the partitioned pipeline already solved. The partitioned pipeline, implemented in Phase 6 for memory reduction, had the side effect of eliminating the b_g2_msm bottleneck entirely.## Part V: The Thundering Herd Correction

The assistant's proposed plan was to combine the partitioned pipeline with parallel synthesis, treating each partition as an independent work unit. But the plan still contained a critical misunderstanding. The assistant believed each partition took approximately 4 seconds to synthesize — a number derived from earlier assumptions about the partitioned pipeline's timing.

The user's correction shattered this assumption: "Synthesis is quite sequential and we already parallelise partitions, no? ... looking at cpu use pattert most of the time is just 10 threads = num partitions. This is your misunderstanding — if we treat each individual snark separately we can feed those proofs one-by-one into the gpu."

The user's observation was devastatingly simple. The 10 partitions do not each take ~4s and run sequentially. They each take ~32-37 seconds (25-27s for sequential witness generation plus 7-10s for SpMV evaluation), and they all run in parallel via rayon, finishing simultaneously in a "thundering herd." This means all 10 partitions submit to the GPU at the same time, forcing the GPU to wait until the last partition finishes before starting any work, and keeping all 10 synthesized partitions in memory (~136 GiB) simultaneously.

The assistant's first attempt to understand this correction was close but still wrong. It drew diagrams showing partitions as independent ~4s work units that could be serialized one after another. The user corrected again: "Noo still wrong, the 10 Px synths run in parallel all for 30s, BUT they START and END at the same time + they all submit to the gpu at the same time."

The user introduced the critical metaphor: "if you think of a waterfall, synth happens as a slab of 10 jobs, with a vertical line to GPU work starting." This "vertical handoff" — a slab of parallel syntheses all finishing simultaneously and all submitting to the GPU at once — is the thundering herd problem. The GPU cannot start on any partition until all 10 are ready, creating structural idle time and massive memory pressure.

Part VI: The Breakthrough and the Phase 7 Architecture

The breakthrough moment came when the assistant finally grasped the full implications of the user's correction. The ASCII diagrams crystallized the architectural insight:

Current behavior (the thundering herd):

Synth: [P0 P1 P2 P3 P4 P5 P6 P7 P8 P9] ← all 10 run in parallel via rayon, ~39s wall
       |________________________________| all finish around the same time
                                         ↓ vertical handoff (all 10 at once)
GPU:                                     [P0][P1][P2]...[P9] ← sequential per partition
                                         ↑ must keep all 10 in memory until GPU starts

Desired behavior (per-partition dispatch):

Synth: [P0~~~~][P1~~~~][P2~~~~][P3~~~~][P4~~~~][P5~~~~][P6~~~~][P7~~~~][P8~~~~][P9~~~~]
GPU:        [P0~~][P1~~][P2~~][P3~~][P4~~][P5~~][P6~~][P7~~][P8~~][P9~~]
Memory:  1   1-2   1-2   1-2   ... (only 1-2 partitions in flight at once)

The assistant enumerated five specific wins from per-partition dispatch:

  1. Memory: Only 1-2 partitions in memory at once instead of 10, reducing peak memory for synthesized partitions from ~136 GiB to ~27 GiB
  2. GPU starts ~35s earlier: GPU begins after the first partition completes (~4s) instead of after all 10 (~39s)
  3. Natural pipelining: No thundering herd — synthesis and GPU interleave smoothly without explicit concurrency controls
  4. No contention: Only 1 synthesis thread active at a time, so GPU b_g2_msm gets the full thread pool (0.4s per partition)
  5. Cross-sector pipelining: The next sector's P0 synthesis can start as soon as the current sector's synth thread is free, eliminating inter-sector GPU idle time The message concluded by launching two research agents: one to verify per-partition synthesis timing and another to analyze engine-level partition dispatch. These agents would return the data needed to design the Phase 7 architecture.

Part VII: Simulation Reveals the True Shape of the Pipeline

The assistant's response to this correction was not blind acceptance but rigorous validation. Python simulations were written to test the implications of the corrected timing model. These simulations proved to be the intellectual core of the entire design effort, transforming verbal reasoning about pipeline behavior into quantitative predictions.

The simulation explored five scenarios. The first confirmed the baseline: with all 10 partitions synthesizing in parallel, synth wall time is ~29s, GPU wall time is ~30s (10 × 3s), total is ~59s, and peak memory holds 10 partitions. This is the "thundering herd" baseline that the current implementation produces.

The second scenario — concurrency=1, fully sequential — revealed a catastrophic outcome. Each partition takes 29s to synthesize, then 3s on GPU. But because only one partition is synthesized at a time, the total time balloons to 293 seconds — nearly 5 minutes per proof. The GPU is active for only 30 of those seconds and idle for 234 seconds. This is worse than the current approach by a factor of 5.

The third scenario — concurrency=3 — showed that even with limited parallelism, the GPU experiences idle gaps of ~20s between batches of three partitions. The total time of ~119s is still far worse than the current 59s.

The fourth scenario — 10 concurrent but feeding GPU as each finishes — is essentially what the existing partitioned pipeline already does. All 10 start at t=0, all finish at ~29–36s, GPU starts at ~29s. Total: 59–66s.

The critical discovery came in the fifth scenario: cross-sector pipelining. If Sector B's partitions can start synthesizing while Sector A's partitions are still being GPU-proved, the GPU idle gap between sectors shrinks to nearly zero. The simulation showed that with 10 concurrent synthesis slots, Sector A finishes synthesis at t=29, GPU runs from t=29 to t=59, and Sector B's synthesis (starting at t=29) finishes at t=58 — just as the GPU becomes free.

This was the moment of profound recalibration. The assistant had been operating under the assumption that per-partition dispatch would improve throughput by enabling finer-grained overlap within a single sector. The simulation revealed that this is false: the 10:1 ratio of synthesis time to GPU time means that you need all 10 concurrent syntheses to keep the GPU fed, regardless of how you dispatch them. Per-partition dispatch doesn't change the fundamental arithmetic of a single sector. The real benefit lies entirely in cross-sector pipelining — and this insight became the foundation of Phase 7.

Part VIII: The Architecture Emerges — A Synth Worker Pool

With the corrected model and the cross-sector insight validated, the assistant began designing the Phase 7 architecture in earnest. The design that emerged was a complete, implementable specification built around a pool of 15–20 concurrent synthesis workers.

Each worker is a tokio::task::spawn_blocking that processes a single partition: it runs the sequential witness generation (~25–27s), then the SpMV evaluation (~4–10s using rayon internally for 3-way parallelism across the A, B, and C matrices), and finally submits the resulting SynthesizedJob to the engine's GPU channel. The GPU channel has a bounded capacity of 2–3 items. When the channel is full, workers block on send, providing natural backpressure that throttles memory usage without requiring explicit memory management.

The GPU worker proves each partition independently using num_circuits=1. This is a critical detail: when the GPU proves 10 circuits as a batch, the b_g2_msm operation takes ~25 seconds. When proving a single circuit, it takes only ~0.4 seconds — a 60x improvement. The per-partition approach reduces this overhead from 25s to a total of 4s across 10 partitions, while also enabling the GPU to start work immediately on the first finished partition rather than waiting for all 10.

A ProofAssembler in the JobTracker accumulates completed partitions for each sector. The assembler accepts out-of-order delivery (since partitions may finish in any order) and assembles the final proof only when all 10 are complete. If one partition fails, the entire sector fails — the assembler is marked as failed, and remaining tasks are allowed to run to completion and discard their results.

The memory budget was carefully calculated and validated through a subagent research task. Each partition's synthesis peaks at ~19.4 GiB (due to temporary witness vector allocations and SpMV workspace), and a settled SynthesizedProof occupies ~13.6 GiB. With 20 concurrent workers:

Part IX: From Design to Document — The Craft of Specification

The user's directive was concise: "Write detailed implementation spec as a phase 7 md." But the assistant's response revealed a methodical approach to technical writing that deserves examination.

Rather than immediately generating the document from scratch, the assistant paused to study the existing documentation conventions. A subagent task was launched to read the Phase 6 design document (c2-optimization-proposal-6.md) and the main project documentation (cuzk-project.md), requesting their full table of contents, heading structure, section conventions, and the final section's format. The task returned a detailed structural breakdown.

But the assistant was not satisfied with a summary. It read the actual file directly to verify the preamble format — the exact phrasing of the Goal line, the Impact bullet list, the line breaks. It read specific sections of Part B and Part C to understand how API definitions and memory budgets were presented. This attention to format fidelity reflects an understanding that design documents in a series are not standalone artifacts; they are entries in an ongoing conversation with the codebase, with other engineers, and with future readers. Consistency across documents reduces cognitive load and signals that the proposals are part of a coherent strategy.

The assistant also demonstrated careful verification after writing. It ran wc -l to confirm the document was 807 lines. It ran grep '^#' to extract all markdown headings, verifying that the heading hierarchy was clean and complete. When the grep output showed extra lines from TOML comments inside code blocks (which also start with #), the assistant correctly diagnosed this as a false alarm — the comments would render correctly in markdown.

The resulting document, c2-optimization-proposal-7.md, was committed to the repository with a detailed commit message that captured the rationale, key design points, and expected impact. The commit message itself is a compressed summary of the entire Phase 7 design:

"Proposal 7 replaces the thundering-herd synthesis pattern (all 10 partitions start/finish simultaneously) with a synth worker pool that processes partitions individually and feeds them to the GPU one at a time. Key design points: 20 synth workers (configurable) each synthesize 1 partition (~29s); Workers submit to engine GPU channel; block if full (backpressure); GPU proves each partition with num_circuits=1 (b_g2_msm: 0.4s vs 25s); ProofAssembler in JobTracker accumulates partitions per job_id; Cross-sector overlap: next sector's synth starts on free workers. Expected impact: 42.8s/proof → ~30s/proof steady-state (GPU-limited), ~100% GPU utilization, zero inter-sector GPU idle time."

Part X: The Knowledge Arc — What Was Learned

This segment of the conversation represents a remarkable knowledge arc. The assistant began with a specific, testable hypothesis (parallel synthesis will improve throughput by keeping the GPU fed), tested it rigorously, discovered a secondary bottleneck (CPU contention), implemented a fix (thread isolation), debugged a cross-language initialization bug, benchmarked the fix, and discovered it was insufficient. Then, through the user's guidance, the assistant launched research agents that revealed a deeper truth: the partitioned pipeline already solved one of the two major bottlenecks. And finally, through the user's repeated corrections, the assistant understood the fundamental architectural problem: the thundering herd created by batching all 10 partitions together.

Each stage of this journey produced valuable knowledge:

The Role of the User: Corrections That Shaped the Architecture

Throughout this segment, the user plays a crucial role that goes beyond simply directing the assistant. The user provides empirical grounding — observations of actual CPU utilization patterns ("10 threads = num partitions") that correct the assistant's theoretical models. The user provides architectural insight — the recognition that the batch abstraction itself is the problem, not the speed of individual components. And the user provides strategic direction — redirecting the assistant from implementation to feasibility assessment, and from incremental optimization to holistic refactoring.

The dynamic between user and assistant is a model of effective human-AI collaboration. The assistant brings systematic analysis, rigorous benchmarking, and deep code-level knowledge. The user brings domain expertise, empirical observation, and architectural intuition. Together, they navigate through layers of misunderstanding to arrive at a genuinely transformative design.

The session concluded with an empty message from the user — nothing but whitespace inside the conversation data tags. This silence is not an absence of communication but a specific form of it. After the correction, the simulation, the design iteration, the format study, the document writing, the verification, and the commit, there was nothing left to say. The deliverable had been received, the design was sound, and the shared understanding was complete.

Conclusion

The Phase 7 design journey documented in this segment is a microcosm of effective engineering practice: hypothesis, correction, simulation, insight, design, validation, documentation, and commitment. It demonstrates that the most valuable optimizations often come not from tuning parameters but from correcting fundamental misunderstandings about how a system actually behaves. The 807-line specification committed to the repository is not just a plan for code changes — it is the permanent record of a collaborative discovery process that transformed a flawed mental model into a coherent, implementable architecture promising ~30% throughput improvement and the elimination of structural GPU idle time.

In the end, the lesson is clear: before optimizing individual components, understand how work actually flows through your system. The thundering herd is not a GPU problem or a CPU problem — it is a pipeline problem. And pipeline problems require architectural solutions, not component-level optimizations. The cross-sector pipelining insight that drives Phase 7 would never have been discovered without first understanding why the intuitive approach (staggering within a sector) doesn't work. That willingness to be wrong, to simulate, to correct, and to redesign is the essence of effective systems engineering.

References

[1] "From Thundering Herd to Continuous Pipeline: The Architectural Revolution of the cuzk Proving Engine" — Chunk article covering the discovery arc from parallel synthesis through thundering herd correction.

[2] "From Thundering Herd to Continuous Pipeline: The Phase 7 Design Journey for PoRep C2 Proving" — Chunk article covering the simulation, architecture design, and document creation for Phase 7.