The Rayon Contention Investigation: How a Systematic Code Reading Reshaped the SUPRASEAL_C2 Optimization Strategy

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. The SUPRASEAL_C2 Groth16 proof generation pipeline—responsible for producing zero-knowledge proofs that storage providers use to prove they are storing data correctly—is a marvel of engineering that pushes the boundaries of memory, parallelism, and GPU utilization. With a peak memory footprint of approximately 200 GiB and a complex call chain spanning Go, Rust, C++, and CUDA, optimizing this pipeline requires a deep understanding of every layer of the system.

This article examines a pivotal sub-session within a larger optimization effort: a focused investigation into whether concurrent partition synthesis would cause excessive rayon thread contention during the sparse matrix-vector multiplication (SpMV) phase. The investigation is documented across 16 message articles (see [1]–[16]), each capturing a specific step in the systematic code-reading process. What began as a narrow technical question about thread pool saturation evolved into a comprehensive architectural analysis that fundamentally reshaped the team's understanding of where the real bottlenecks lie. The investigation, spanning 16 messages and involving systematic code reading across multiple source files, demonstrates the power of evidence-based reasoning in complex systems optimization.

The Question That Started It All

The investigation was triggered by a precise technical question from the user (see [4]), who had been studying the PCE synthesis path. The user had been studying the PCE (Precomputed Constraint Evaluation) synthesis path for a single partition in the SUPRASEAL_C2 pipeline and had identified a potential performance concern. The core worry was straightforward: if 20 partition synthesis workers all reached the SpMV phase simultaneously, they would all contend for the global rayon thread pool.

The user's reasoning was sophisticated. They had traced the code path through synthesize_with_pce and evaluate_pce, discovering that even a single partition's SpMV phase uses rayon heavily: rayon::join across the A, B, and C matrices (3-way parallelism), with each spmv_parallel using par_chunks_mut(8192) across approximately 130 million rows, yielding roughly 15,900 chunks per matrix. The arithmetic was sobering: 20 workers × 3 matrices × 15,900 chunks could theoretically flood the rayon scheduler with nearly a million pending work items.

However, the user also identified a crucial mitigating factor. The witness generation phase—which takes 25–27 seconds per partition—is single-threaded, while the SpMV phase takes only 4–10 seconds. This timing asymmetry means that at any given moment, most workers would be in the single-threaded witness phase, and only a few would have progressed to the SpMV phase. The user wanted confirmation of this intuition.

The user's three specific verification requests were: (1) confirm the rayon usage in evaluate_pce, (2) verify that synthesize_partition calls synthesize_auto(vec![single_circuit]), and (3) determine whether rayon's into_par_iter over a single-element vector runs inline on the calling thread or spawns a separate task. These three questions, seemingly narrow, would unlock a much deeper understanding of the pipeline's concurrency architecture.

The Systematic Investigation Unfolds

The assistant's response to this question was a masterclass in systematic code investigation, as captured in [8] (the first step), [14] (the initial file reads), and [5] (the pivot to targeted searching). Rather than speculating or relying on general knowledge about rayon's behavior, the assistant methodically read the actual source code, tracing the call chain from the outermost dispatch logic down to the innermost parallel primitives.

Phase 1: Locating the Evidence

The investigation began with file discovery. In message 1, the assistant used glob commands to confirm that the relevant source files existed at the expected paths: eval.rs in the cuzk-pce crate and pipeline.rs in the cuzk-core crate. This may seem trivial, but it established the foundation for all subsequent analysis.

Message 2 saw the first substantive reads. The assistant read eval.rs to confirm the SpMV evaluator's structure, and pipeline.rs around lines 470 and 1670 to examine the synthesize_with_pce and synthesize_partition functions. These reads were truncated—the tool returned only the first few lines of each section—but they confirmed the file structure and set the stage for deeper investigation.

Phase 2: Tracing the Call Chain

Messages 3 and 4 represented a critical pivot. The assistant used grep to locate the synthesize_auto function (at line 613 of pipeline.rs) and the rayon thread pool configuration (at lines 106–107 of main.rs). These steps are documented in [5] (tracing the thread pool) and [10] (reading the rayon thread pool configuration). The grep results were revealing: the rayon thread pool is configured at daemon startup using rayon::ThreadPoolBuilder::new().num_threads(rayon_threads).build_global(), confirming that a single global pool serves all partition synthesis work.

The assistant then read the actual code at these locations. The synthesize_auto function signature confirmed that it takes Vec<C> circuits and routes to the PCE fast path when a cached PCE is available. The main.rs read confirmed the global pool configuration and also revealed the CUZK_GPU_THREADS environment variable for the C++ groth16_pool.

Phase 3: Verifying the Concurrency Architecture

Messages 5 through 11 drilled into the actual concurrency model. The assistant read the partition synthesis spawning code in pipeline.rs (around line 1748), discovering a documentation comment that revealed critical timing parameters: "GPU time per partition is ~3s, synthesis per partition is ~29s" and "With max_concurrent=3, the pipeline keeps the GPU always fed."

This was a watershed moment. The doc comment revealed that:

• The pipeline uses a producer-consumer architecture with a bounded channel (sync_channel with capacity max_concurrent)
• The synthesis-to-GPU time ratio is approximately 10:1 (29s vs 3s)
• The recommended max_concurrent value is 3, not the 20 the user had hypothesized In message 6, the assistant verified a critical assumption: there is no custom rayon thread pool for partition synthesis (see [11]). A grep for install|spawn_fifo|ThreadPool|thread_pool|pool.install returned "No files found," confirming that all rayon parallelism goes through the global pool. This negative result was just as important as any positive finding—it meant that contention analysis could proceed with the global pool as the sole target. Message 7 attempted to locate the WitnessCS type definition, returning "No files found" (see [15]). This empty result revealed that WitnessCS comes from the external bellperson crate, not from the local codebase. The assistant would need to reason about its behavior from first principles rather than direct code inspection.

Phase 4: Discovering the Actual Parameters

Messages 8 through 11 focused on discovering the actual concurrency parameters. A dual grep for WitnessCS and max_concurrent (message 8) revealed that max_concurrent is documented in config.rs with a specific memory formula: (max_concurrent + 1) × ~13.6 GiB per partition. The recommended value of 3 keeps the GPU continuously fed while bounding memory to approximately 54 GiB.

Message 9 confirmed the partition count: total_partitions: 10 for PoRep 32G (see [13]). This was a critical discovery—the user's "20 workers" scenario was based on an assumption that turned out to be incorrect. The actual architecture uses 10 partitions, not 20. (The "20" scenario only arises with synthesis_concurrency=2 at the engine level, which would run two proofs simultaneously.)

Message 10 read the configuration file to understand max_concurrent's semantics. The documentation confirmed that it "controls how many synthesized partitions can be queued for the GPU simultaneously," acting as a backpressure mechanism rather than a limit on synthesis threads.

Message 11 was perhaps the most architecturally significant. The assistant read the pipeline architecture comment block at line 1370 of pipeline.rs, which contained an ASCII diagram of the producer-consumer pipeline. This comment described the "True producer-consumer pipeline with parallel synthesis" architecture, revealing how partition synthesis threads feed into a bounded channel that feeds the GPU. This was the Rosetta Stone that connected all the previously discovered facts into a coherent system model.

Phase 5: The Engine-Level Dispatcher

Messages 12 through 14 examined the engine-level dispatcher. A precautionary grep for synthesis_concurrency (message 12) revealed that this parameter controls how many entire proofs (each with their own 10-partition pipeline) can run simultaneously. When synthesis_concurrency > 1, multiple proofs can overlap, potentially creating the 20-thread scenario the user had hypothesized.

Message 13 read the engine dispatcher in engine.rs, discovering that it uses a semaphore to bound concurrent dispatches. The comment "When synthesis_concurrency > 1, multiple proofs can be synthesized in parallel. The dispatcher pulls from the scheduler, acquires a semaphore permit, and spawns each proc..." confirmed the bounded-concurrency pattern.

Message 14 was the final data-gathering step. The assistant read the GpuConfig struct definition to understand how synthesis_concurrency interacts with the partition pipeline. The question "whether it multiplies the 10 partition threads" was the last piece of the puzzle.

The Synthesis: Message 15

Message 15 was the culmination of the entire investigation. The assistant synthesized all the gathered evidence into a comprehensive answer that addressed the user's three questions and went far beyond them.

The Three Verifications

First, the assistant confirmed the rayon usage in evaluate_pce. The nested rayon::join calls at lines 125–133 of eval.rs create three-way parallelism across the A, B, and C matrices, and each spmv_parallel uses par_chunks_mut(8192) across the output vector, yielding approximately 15,900 chunks per matrix for ~130 million rows. All of this runs on the global rayon thread pool.

Second, the assistant confirmed the call chain. Lines 1704–1708 of pipeline.rs show that synthesize_partition calls synthesize_auto(vec![circuit]) with a single-element vector, which routes to synthesize_with_pce when a cached PCE is available.

Third, the assistant explained rayon's behavior for single-element parallel iterators. Rayon's work-stealing scheduler, when presented with a single-element parallel iterator, executes the closure inline on the calling thread. No task is spawned or stolen. This means the witness generation phase (25–27 seconds) consumes zero rayon thread pool resources.

The Contention Analysis

With the three verifications established, the assistant pivoted to the broader contention analysis. The key insight was the natural staggering that emerges from the workload's structure:

• Witness generation: 25–27 seconds per partition, single-threaded, no rayon usage
• SpMV: 4–10 seconds per partition, uses rayon heavily With 10 OS threads (one per partition) starting at roughly the same time, small variations in witness generation duration cause the SpMV phases to be naturally staggered. At any instant, statistically 1–3 partitions will be in the SpMV phase while 7–9 are still in witness generation. The probability of all 10 (or 20) hitting SpMV simultaneously is vanishingly small. The assistant then explained why rayon's work-stealing scheduler handles the remaining contention gracefully. Rayon does not create new threads for each rayon::join or par_chunks_mut call. Instead, it posts work items to a fixed-size thread pool (configured to match the number of CPU cores, e.g., 96 or 192 threads). With 2–3 partitions in SpMV simultaneously, the ~47,700 pending work items are distributed across the pool. Rayon is explicitly designed for this kind of oversubscription pattern.

The Real Bottleneck: Memory Bandwidth

The assistant's most important contribution was reframing the entire problem. The user was worried about rayon thread pool saturation, but the real bottleneck is memory bandwidth. Each SpMV reads the CSR matrix (several GiB) and the witness vector. With 2–3 concurrent SpMV evaluations, they compete for DRAM bandwidth. On a typical 8-channel DDR5 server (~300 GB/s), 3 concurrent SpMV evaluations reading ~10 GiB each will saturate bandwidth before they saturate CPU cores.

This reframing had profound implications for the optimization strategy. Rather than worrying about thread pool configuration or custom pool isolation, the team should focus on memory access patterns, data layout, and reducing the memory footprint of concurrent SpMV evaluations.

The Broader Implications

The investigation's findings had implications far beyond the narrow question of rayon contention. By confirming that the pipeline's concurrency architecture is sound and that the real bottleneck is memory bandwidth, the team could focus their optimization efforts on the right targets.

The investigation also validated the Phase 7 per-partition dispatch architecture that was being designed in parallel. The cross-sector pipelining model—where 15–20 concurrent synthesis workers synthesize individual partitions and submit them to a bounded GPU channel—would not suffer from rayon contention. The natural staggering from witness generation ensures that only a few workers hit SpMV simultaneously, and rayon's work-stealing scheduler handles that gracefully.

Furthermore, the investigation demonstrated the value of systematic code reading in performance analysis. The assistant did not guess or speculate—it read the actual source code, traced the exact call paths, and verified each claim against the implementation. This evidence-based approach is the gold standard for performance engineering in complex systems.

Conclusion

The rayon contention investigation that unfolded across 16 messages in this sub-session is a testament to the power of systematic, evidence-based code analysis. What began as a narrow question about thread pool saturation evolved into a comprehensive architectural understanding that reshaped the team's optimization strategy.

The key discoveries—that the pipeline uses 10 partitions (not 20), that a global rayon pool serves all synthesis work, that natural staggering from witness generation prevents contention, and that memory bandwidth is the true bottleneck—could only have been reached through careful, methodical investigation. Each message built on the previous one, narrowing uncertainty and refining the mental model.

For the broader SUPRASEAL_C2 optimization effort, this investigation provided critical validation. The Phase 7 architecture could proceed with confidence, knowing that concurrent partition synthesis would not cause rayon contention. The team could focus their energy on the real challenge: reducing memory bandwidth pressure through better data layout, streaming partitions sequentially, and optimizing the SpMV kernel for cache efficiency.

In the end, the most valuable insight was not about rayon at all—it was about the importance of questioning assumptions and letting the code speak for itself. The user's "20 workers" scenario was a reasonable hypothesis, but only by reading the actual source could the team discover the true architecture and optimize accordingly.## References

[1] "The Pivotal Read: Tracing Rayon Contention Through the Synthesis Dispatcher" — examines message 13, the engine dispatcher read that connected low-level parallelism to high-level orchestration.

[2] "The Data-Gathering Pivot: Tracing WitnessCS and Concurrency Bounds in a Groth16 Proof Pipeline" — examines message 8, the dual grep for WitnessCS and max_concurrent.

[3] "The Precautionary Grep: How One Assistant Verified Rayon Contention Before Answering" — examines message 12, the final verification step before the synthesis.

[4] "The Rayon Contention Question: A Deep Dive into Concurrent Partition Synthesis in SUPRASEAL_C2" — examines message 0, the user's original question.

[5] "Tracing the Thread Pool: A Systematic Investigation of Rayon Contention in a GPU Proof Pipeline" — examines message 3, the grep for synthesize_auto and thread pool configuration.

[6] "The Pivotal Config Read: Unraveling Concurrency in the SUPRASEAL_C2 Partition Pipeline" — examines message 14, the config read for synthesis_concurrency.

[7] "Diagnosing Rayon Contention in Filecoin's Groth16 Proof Pipeline: A Deep Dive into Concurrent Partition Synthesis" — examines message 15, the final synthesis answer.

[8] "The First Step: Initiating a Deep-Dive into Rayon Contention in the SUPRASEAL_C2 Pipeline" — examines message 1, the initial glob commands.

[9] "The Pivot Point: Tracing the '20 Concurrent Workers' Assumption in a Groth16 Proof Pipeline Investigation" — examines message 5, the read that revealed actual timing parameters.

[10] "Reading the Rayon Thread Pool: A Data-Gathering Pivot in a Concurrency Investigation" — examines message 4, the reads of synthesize_auto and main.rs.

[11] "The Pivot Point: Verifying Rayon Pool Isolation in a Groth16 Proof Pipeline Investigation" — examines message 6, the grep for custom rayon pools.

[12] "The Connecting Thread: How a Single Brief Message Anchored a Deep Rayon Contention Analysis" — examines message 10, the config read for max_concurrent.

[13] "Drilling Down into Partition Count: A Subagent's Methodical Verification of Rayon Contention Boundaries" — examines message 9, the partition count verification.

[14] "Reading the Source: A Data-Gathering Pivot in the SUPRASEAL_C2 Rayon Contention Investigation" — examines message 2, the initial file reads.

[15] "The Empty Grep: A Pivotal Negative Result in the Rayon Contention Investigation" — examines message 7, the empty grep for WitnessCS.

[16] "Reading the Blueprint: How a Single read Tool Call Unlocks Concurrency Architecture in a Groth16 Proving Pipeline" — examines message 11, the architecture comment read.