The 3.4% That Changed Everything: A Pivotal Benchmark in the Phase 11 Optimization Journey

Introduction

In the course of an intensive optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, a single benchmark result at message [msg 2774] marked a critical inflection point. The assistant had just completed the second of three planned "memory-bandwidth interventions" in Phase 11, reducing the groth16_pool thread count from 192 to 32 via the gpu_threads = 32 configuration parameter. The result was a throughput of 36.7 seconds per proof — a 3.4% improvement over the Phase 9 baseline of 38.0 seconds, and a clear improvement over Intervention 1's 37.9 seconds. On its surface, this was a modest but meaningful win. But the message's true significance lies not in the headline number, but in what the assistant did immediately after reporting it: it checked the b_g2_msm timing and discovered a critical side effect that would drive the entire next phase of the project.

This article examines that single message in depth: the reasoning behind the benchmark, the assumptions that were validated and invalidated, the investigative pivot that followed, and how this moment of measurement became the seed for Phase 12's split-API architecture.

The Message: A Benchmark and an Investigation

The subject message is brief but dense. The assistant writes:

36.7s/proof — that's a meaningful improvement! Down from 38.0s (Phase 9 baseline) and 37.9s (Int 1 only). That's a 3.4% improvement from reducing the groth16_pool to 32 threads. The prove time average also dropped from 60.4s to 57.4s.

Let me also check the b_g2_msm timing: [bash] grep "b_g2_msm_ms" /home/theuser/cuzk-p11-int12.log | tail -20 CUZK_TIMING: b_g2_msm_ms=1850 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=2376 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=2079 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1415 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1714 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1885 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1709 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1416 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1526 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=1591 num_circuits=1 CUZK_TIMING: b_g2_msm_ms=2026 num_circuits=1 CUZK_...

The message is structured in two distinct parts. The first part announces and contextualizes the benchmark result. The second part — a shell command and its output — is the beginning of an investigative probe. The assistant does not wait for a subsequent round to act; it immediately issues a grep command to extract timing data from the daemon log, because the benchmark result has raised an implicit question: what happened to b_g2_msm?

The Reasoning Behind the Benchmark

To understand why this message matters, one must understand the chain of reasoning that led to it. The Phase 11 optimization proposal (documented in c2-optimization-proposal-11.md) identified three sources of throughput degradation when moving from single-proof isolation (32.1 s/proof) to high-concurrency operation (38.0 s/proof at c=20 j=15). Waterfall analysis had shown that GPU per-partition time inflated from 4.9s to 7.5s under load, prep_msm from 1.7s to 2.7s+, and synthesis from 35s to 54s. The system had shifted from being GPU-bound (as it was in Phase 8) to being CPU memory-bandwidth-bound.

The three proposed interventions were:

Serialize async_dealloc: Bound TLB shootdown storms by serializing the asynchronous deallocation of pinned host memory, using a static std::mutex in both C++ and Rust.
Reduce groth16_pool to 32 threads: Cut L3 cache thrashing by limiting the thread pool used for the b_g2_msm Pippenger MSM computation from 192 threads (all CPU cores) to 32 threads (approximately two CCDs worth on the AMD architecture).
Memory-bandwidth throttle during b_g2_msm: Use a global atomic flag to signal to Rust's SpMV (Sparse Matrix-Vector multiplication) that it should yield during the b_g2_msm window, reducing contention. The assistant had implemented Intervention 1 in [msg 2750]–[msg 2754], adding a static std::mutex dealloc_mtx in groth16_cuda.cu and a corresponding DEALLOC_MTX static Mutex in supraseal.rs. The benchmark at [msg 2765]–[msg 2766] showed 37.9 s/proof — essentially identical to the 38.0 s baseline. The TLB shootdown reduction from serialization alone was marginal. Intervention 2 was purely a configuration change: setting gpu_threads = 32 in the daemon's TOML config file. The assistant created a new config at [msg 2768], killed the daemon, restarted it at [msg 2770], verified the setting was picked up at [msg 2772], and ran the benchmark at [msg 2773]. The result is the subject message.

Assumptions Validated and Invalidated

The Phase 11 design spec made a specific prediction about Intervention 2: reducing the groth16_pool from 192 to 32 threads would slow the b_g2_msm Pippenger MSM from approximately 0.5 seconds to approximately 0.5–0.7 seconds. This prediction was based on the assumption that the Pippenger algorithm, which is highly parallelizable, would see a roughly linear slowdown proportional to the thread count reduction, but that the reduction in L3 cache contention would more than compensate.

The benchmark validated the overall throughput prediction: 36.7 s/proof was a genuine improvement. But it invalidated the b_g2_msm timing prediction: instead of 0.5–0.7 seconds, the actual b_g2_msm times ranged from 1.4 to 2.4 seconds, with an average around 1.7 seconds. This was a 3–4× slowdown relative to the 192-thread baseline, far more than the model had predicted.

Why did this happen? The assistant's subsequent analysis (in the following messages, [msg 2775]–[msg 2778]) reveals the reasoning: with only 32 threads, the single-circuit Pippenger is fundamentally slower because it cannot saturate the CPU's vector units and memory bandwidth as effectively. The Pippenger algorithm's parallelism scales with the number of points being processed, and for a single circuit's b_g2_msm, the problem size is fixed. Reducing threads from 192 to 32 means each thread handles more work, but the total computation is still bound by the same arithmetic intensity — and with fewer threads, the algorithm loses the ability to overlap memory latency as effectively.

The Investigative Pivot

The most revealing aspect of this message is what it doesn't say explicitly but what the assistant's actions imply. The assistant does not simply report the 36.7 s/proof result and move on to Intervention 3. Instead, it immediately checks b_g2_msm_ms timing. This is the behavior of an engineer who understands that every optimization has a cost, and that the cost must be measured even when the headline number looks good.

The assistant's mental model of the system is sophisticated enough to know that b_g2_msm is a critical-path operation. Even though it runs after the GPU lock is released (thanks to the Phase 9 restructuring), it still blocks the GPU worker thread from picking up the next job. The worker must call prep_msm_thread.join() before it can loop back to acquire the GPU lock for the next partition set. If b_g2_msm takes 1.7 seconds and the GPU kernels for the next partition set are ready earlier, the worker stalls.

This investigative pivot is what separates a superficial optimization exercise from a deep systems-engineering effort. The assistant is not just collecting benchmark numbers; it is building a causal model of the system's behavior under load. The b_g2_msm timing data is a crucial input to that model, and it reveals that the Phase 11 spec's assumptions about thread scaling were incomplete.

Input Knowledge Required

To fully understand this message, one needs knowledge of several layers of the system:

The Groth16 proving pipeline: The proof generation process involves multiple phases: synthesis (building the circuit and computing witness values), MSM (multi-scalar multiplication) for the A, B, C vectors, NTT (number-theoretic transform), and the final b_g2_msm (the G2-group MSM for the B proof element). The b_g2_msm is specifically the MSM that computes the G2 element of the proof, which is CPU-bound and runs after the GPU kernels have completed their work.
The thread pool architecture: The groth16_pool is a C++ thread pool used for CPU-side computations, primarily the Pippenger MSM. In the baseline configuration, it used 192 threads (all hardware threads on the 96-core AMD EPYC system with SMT). Reducing it to 32 threads was intended to reduce L3 cache contention.
The GPU worker interlock: Phase 8 introduced a per-GPU mutex that allows multiple GPU workers to share a single GPU device. Phase 9 restructured the code so that the GPU lock is released before b_g2_msm completes, allowing another worker to start its GPU kernels while the first worker's CPU-side MSM finishes.
The benchmark methodology: The benchmark uses c=20 j=15 (20 proofs total, 15 concurrent jobs) against a daemon configured with gpu_workers_per_device=2 and partition_workers=10. The "prove time" metric is the time from when a job enters the GPU worker to when its proof is complete.
The AMD EPYC memory architecture: The system has multiple CCDs (Core Complex Dies), each with its own L3 cache. With 192 threads, the groth16_pool can span all CCDs, causing L3 thrashing as threads compete for cache lines. Reducing to 32 threads confines the pool to approximately two CCDs, reducing cross-CCD cache traffic and TLB coherence overhead.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A validated benchmark result: 36.7 s/proof at c=20 j=15 with gpu_threads=32, establishing a new performance baseline for the combined Intervention 1+2 configuration.
A quantified improvement: 3.4% over the Phase 9 baseline, confirming that the L3 cache pressure reduction hypothesis was correct, even if the magnitude was modest.
A critical measurement anomaly: b_g2_msm timing of ~1.7s (range 1.4–2.4s), far exceeding the predicted 0.5–0.7s. This becomes the driving problem statement for Phase 12.
A revised mental model: The assistant now understands that the b_g2_msm slowdown is a significant side effect of thread pool reduction, and that the overlap budget between GPU kernel execution and CPU-side MSM is tighter than previously modeled.

The Path to Phase 12

The b_g2_msm timing data from this message directly motivates the next major architectural change. In the following messages ([msg 2775]–[msg 2778]), the assistant analyzes whether b_g2_msm now creates a bottleneck in the GPU worker's critical path. The analysis reveals that with gw=2 (two GPU workers per device), while worker A is blocked on prep_msm_thread.join(), worker B already holds the GPU lock and is running kernels — so the stall doesn't directly hurt GPU utilization. But it does prevent worker A from picking up the next synthesis result, creating a pipeline bubble.

The user then asks whether b_g2_msm could be shipped to a separate thread to unblock the GPU worker entirely. This question, prompted by the data in the subject message, leads to the design of the Phase 12 split API: generate_groth16_proofs_start_c returns an opaque handle after GPU unlock, and a separate finalize_groth16_proof call joins the b_g2_msm thread, runs the epilogue, and writes the final proof. This architectural change decouples the GPU worker's critical path from the CPU post-processing, allowing the worker to loop back immediately and pick up the next job.

Conclusion

Message [msg 2774] is a masterclass in systems-oriented benchmarking. It demonstrates that a good engineer does not simply collect throughput numbers and declare victory; they investigate the side effects of every change, even when the headline number is positive. The 3.4% improvement from Intervention 2 was real, but the discovery that b_g2_msm had slowed by 3–4× was equally important. This measurement anomaly became the seed for Phase 12, transforming a modest throughput gain into a architectural insight that would reshape the proving pipeline.

The message also illustrates the iterative nature of performance optimization: each intervention reveals new bottlenecks, each measurement refines the mental model, and each discovery opens new avenues for improvement. The 36.7 seconds per proof was not the end of Phase 11 — it was the beginning of Phase 12.