The Benchmark That Killed a Throttle: Evaluating Phase 11's Three Memory-Bandwidth Interventions

In the high-stakes world of Filecoin SNARK proving, every millisecond counts. The cuzk proving engine, a custom-built Groth16 prover for Filecoin's Proof-of-Replication (PoRep) circuits, had been pushed through ten optimization phases already — from GPU interlock refinements to PCIe transfer pipelining — when the team hit a wall. The bottleneck had shifted to DDR5 memory bandwidth contention, a subtle system-level problem where CPU threads competing for memory access were slowing each other down through L3 cache thrashing and TLB shootdowns. Phase 11 was designed to address this with three targeted interventions. Message <msg id=2806> captures the moment those interventions were put to the test: a full benchmark sweep running all three together, producing the data that would determine which ones earned their place in the pipeline.

The Context: Three Interventions Against Memory Contention

The subject message is deceptively simple — a single bash command launching a benchmark — but it represents the culmination of a deep investigation into memory subsystem bottlenecks. The assistant had spent the preceding messages implementing three interventions, each targeting a different source of memory bandwidth contention:

Intervention 1 serialized the async_dealloc calls with a static mutex. The GPU worker's asynchronous memory deallocations were creating TLB shootdowns and cache line conflicts when multiple threads freed memory simultaneously. By serializing these deallocations through a C++ std::mutex mirrored in Rust, the assistant aimed to reduce the chaos of concurrent page table updates.

Intervention 2 reduced the groth16_pool thread count from 192 to 32 via gpu_threads = 32. This was the most impactful change. The b_g2_msm computation — a CPU-side multi-scalar multiplication on the G2 curve — was spawning 192 threads that thrashed the L3 cache and saturated memory bandwidth. Reducing to 32 threads (matching the two CCDs on the AMD EPYC processor) cut the thread-induced pressure while still keeping the computation within the GPU kernel overlap window.

Intervention 3 was the most architecturally creative: a global atomic throttle flag. The idea was that when C++ started b_g2_msm (which runs on CPU and competes for memory bandwidth), it would set a flag that Rust's SpMV (Sparse Matrix-Vector multiplication, used during synthesis) would check. If the flag was set, SpMV threads would yield periodically, reducing their memory pressure during the b_g2_msm window. This was implemented as a #[no_mangle] pub extern "C" function in the cuzk-pce crate, called from C++ code in groth16_cuda.cu around the b_g2_msm computation.

The Benchmark: What the Message Actually Shows

The message executes:

FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 20 --concurrency 15 2>&1 | tee /home/theuser/cuzk-p11-int123-c20j15.log

This runs 20 PoRep proofs with concurrency 15 (meaning up to 15 proofs in flight simultaneously), using the pre-computed C1 output from a 32 GiB sector. The --concurrency 15 setting is significant — it's high enough to stress the memory subsystem and reveal contention effects. The output is piped through tee to both display and log to a file for later analysis.

The partial output shows the first five completions:

[1/20] COMPLETED — 59.2s (prove=47795 ms, queue=295 ms)
[2/20] COMPLETED — 95.7s (prove=51390 ms, queue=737 ms)
[3/20] COMPLETED — 129.7s (prove=50156 ms, queue=1167 ms)
[4/20] COMPLETED — 165.5s (prove=51803 ms, queue=1599 ms)
[5/20] COMPLETED — 200.3s (prove=55...

Each line shows the elapsed time from benchmark start, the prove time (actual GPU proving latency in milliseconds), and the queue time (how long the job waited before being picked up). The prove times hover around 48–52 seconds, with the queue time growing linearly as more proofs are submitted.

The Result: A Null Finding That Changed the Direction

The critical insight from this benchmark came in the next message <msg id=2808>, where the assistant analyzed the results:

"36.8s/proof with all 3 interventions, essentially the same as Int1+Int2 alone (36.7s). The throttle (Intervention 3) didn't add measurable benefit."

This was a null result — and a valuable one. The membw throttle, despite being a clever cross-language coordination mechanism, added zero throughput improvement. The assistant's reasoning was sound: with gpu_threads=32, the b_g2_msm computation already used fewer threads, so the L3 contention during its execution window was already reduced to the point where additional throttling of SpMV had no effect.

The benchmark also revealed something else: the per-proof prove times (48–52s) were actually higher than the Phase 9 baseline's ~60s. This was because with all three interventions active and concurrency=15, the system had 15 synthesis jobs running in parallel alongside the GPU workers. The prove time includes queue wait, and at high concurrency, the CPU-side synthesis pipeline becomes the bottleneck — not the GPU.

Assumptions Made and Lessons Learned

The assistant made several assumptions during Intervention 3's design:

1. That b_g2_msm and SpMV meaningfully compete for L3 cache. This was reasonable — both are CPU-intensive memory-bound operations. But with gpu_threads=32, the b_g2_msm thread count was already low enough that its L3 footprint didn't significantly perturb SpMV.
2. That a global atomic flag with yield_now would be a lightweight enough mechanism. The implementation used a single AtomicI32 with std::thread::yield_now() in the SpMV loop. The overhead of checking the flag was negligible, but the benefit was also negligible.
3. That the throttle would need to be active for the full duration of b_g2_msm (~1.7s). In practice, b_g2_msm runs concurrently with GPU kernel execution. The GPU kernels take ~18s total (1.8s × 10 partitions), and b_g2_msm starts after the first partition's prep_msm completes. So b_g2_msm's 1.7s window is well within the GPU execution window, and the throttle's effect was already subsumed by the GPU overlap.

Input Knowledge Required

To understand this message, one needs knowledge of:

• Groth16 proof structure: The proof has multiple partitions, each requiring GPU kernel execution and CPU post-processing (b_g2_msm, epilogue).
• Memory bandwidth contention: How multiple CPU threads competing for DDR5 bandwidth cause L3 cache thrashing, TLB shootdowns, and degraded throughput.
• The cuzk pipeline architecture: How GPU workers, synthesis dispatchers, and post-processing threads interact, and how the GPU interlock mutex controls access to the GPU.
• The Phase 11 interventions: The three specific changes being tested and their hypothesized effects.
• Benchmark methodology: The c=20 j=15 parameters and what they mean for system stress.

Output Knowledge Created

This message produced concrete benchmark data showing that:

1. All three interventions together yield 36.8s/proof at c=20, j=15.
2. Intervention 3 (membw throttle) adds no measurable benefit beyond Intervention 2 alone.
3. The prove times (48–52s) are inflated by CPU-side contention at high concurrency, suggesting the next bottleneck is CPU synthesis throughput, not GPU utilization. This data directly informed the next decision: the user asked to try 3 and 4 GPU workers per device (<msg id=2807>), and the assistant ran those experiments next, finding that gw=3 produced 37.2s/proof (worse) and gw=4 was even worse, confirming that CPU contention, not GPU pipeline depth, was the limiting factor.

The Thinking Process Visible in the Message

While the message itself is just a command execution, the reasoning behind it is visible in the surrounding context. The assistant had just finished implementing Intervention 3 after a careful design process. The initial plan was to alias C++ atomic memory into Rust, but the assistant reconsidered and chose a simpler approach: a global AtomicI32 in the cuzk-pce crate with an FFI setter called from C++. This decision shows a pragmatic engineering mindset — preferring clean, maintainable code over clever but fragile memory aliasing.

The assistant also demonstrated systematic thinking by building and verifying each intervention before combining them. Intervention 3 was built on top of Interventions 1 and 2, and the benchmark tested all three together. When the result came back neutral, the assistant didn't second-guess the data — it accepted the null finding and pivoted to the user's next suggestion (GPU worker scaling).

This message, for all its apparent simplicity, represents a critical decision point in the optimization journey. It validated that the memory-bandwidth throttle was unnecessary, saving future maintenance complexity, and it redirected attention to the next bottleneck: CPU-side synthesis throughput. In the iterative world of performance engineering, knowing what not to do is often as valuable as knowing what to do.