The Quietest Commit: A Rust Mutex in the War Against Memory Bandwidth Contention

Message Summary

The subject message (msg 2754) is deceptively simple:

[assistant] [edit] /home/theuser/curio/extern/bellperson/src/groth16/prover/supraseal.rs Edit applied successfully.

Seven words. A tool result confirmation. On its surface, nothing remarkable — just another file edit in a long optimization session. But this message represents the completion of a critical step in Phase 11 of an intensive performance optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep. The edit added a static DEALLOC_MTX — a std::sync::Mutex<()> — to the Rust side of the FFI boundary, mirroring a C++ static std::mutex dealloc_mtx added moments earlier in groth16_cuda.cu (see msg 2750). Together, these two mutexes formed Intervention 1 of Phase 11: serializing asynchronous deallocations to bound TLB shootdown storms.

The Strategic Context: Why This Edit Exists

To understand why a single mutex deserves an article, we must understand the battlefield. The cuzk proving engine had been optimized through nine previous phases, each targeting a different bottleneck. Phase 9 had achieved an impressive 32.1 seconds per proof in isolation, but under realistic production load — 20 concurrent jobs (c=20) with 15-way concurrency (j=15) — throughput degraded to 38.0 seconds per proof. This 18% degradation was the problem Phase 11 was designed to solve.

The root cause, identified through waterfall timing analysis, was CPU memory bandwidth contention. The system had shifted from being GPU-bound (in Phase 8) to being CPU memory-bandwidth-bound. Three specific mechanisms were identified:

TLB shootdown storms: When multiple CPU cores concurrently called munmap() on large synthesis data buffers (~400 MiB each), the kernel had to flush TLB entries across all cores, causing expensive inter-processor interrupts (IPIs) and serializing memory management.
L3 cache thrashing: The groth16_pool thread pool was sized at 192 threads (matching the total CPU thread count), but the working set of concurrent SpMV (sparse matrix-vector multiply) operations was so large that it evicted useful data from L3 cache, increasing memory pressure.
Uncontrolled memory bandwidth consumption: The b_g2_msm (G2 multi-scalar multiplication) phase, which runs after the GPU lock is released, was competing with synthesis threads for DDR5 bandwidth. Intervention 1 — the edit in our subject message — targeted the first mechanism. The async_dealloc function in the C++ code spawns a detached thread to munmap() large synthesis buffers asynchronously, so the main thread doesn't block. But when multiple GPU workers finish simultaneously, these deallocation threads all run munmap() concurrently, generating a storm of TLB shootdowns that degrade memory performance across the entire system.

The Implementation: What the Edit Actually Did

The edit in msg 2754 added a static Mutex<()> in the Rust file supraseal.rs, specifically in the groth16 prover module. This Rust mutex is the counterpart to the C++ static std::mutex dealloc_mtx added in msg 2750. The two mutexes work together through the FFI boundary: the C++ code acquires dealloc_mtx before spawning the async deallocation thread, and the Rust code acquires DEALLOC_MTX before calling into the C++ deallocation path.

The design is deliberately minimal — a global lock with no data, just synchronization. The mutex serializes all async deallocations across all GPU workers, ensuring that only one munmap() call executes at a time. This bounds the TLB shootdown rate to at most one core's worth of TLB flushes per deallocation, rather than N-cores' worth of flushes for N concurrent deallocations.

The choice of a static Mutex<()> (rather than a more granular lock or a lock-free approach) reflects a pragmatic engineering judgment: the deallocation path is not on the critical path for throughput (it runs after the proof is complete), so serializing it adds negligible latency to any individual proof while reducing system-wide memory subsystem contention. The lock is never held for more than the duration of a single munmap() call, which is typically sub-millisecond for the buffer sizes involved.

Assumptions Embedded in the Design

The intervention rests on several assumptions, some explicit and some implicit:

Assumption 1: TLB shootdowns are a significant contributor. The Phase 11 design spec hypothesized that concurrent munmap() calls cause TLB shootdown storms that degrade memory bandwidth for the entire system. This assumption was based on the waterfall timing analysis showing that GPU per-partition time inflated from 4.9 seconds to 7.5 seconds under load — a symptom consistent with memory bandwidth contention, though not uniquely attributable to TLB shootdowns.

Assumption 2: Serializing deallocations won't hurt throughput. Since deallocation happens after the proof is complete and doesn't block the GPU worker from picking up the next job (the deallocation runs in a detached thread), serializing it should not increase latency on the critical path. The worst case is that a deallocation thread waits briefly for a previous one to finish, but this wait is invisible to the proof pipeline.

Assumption 3: The mutex is uncontested enough to avoid becoming a bottleneck. With only one deallocation per GPU worker per proof cycle, and with the deallocation completing in milliseconds, the lock contention should be negligible even with multiple workers.

Assumption 4: The C++ and Rust mutexes are properly aligned. The design uses separate mutexes on each side of the FFI boundary rather than sharing a single mutex across the boundary. This avoids the complexity of passing a C++ mutex pointer to Rust (which would require careful lifetime management) but assumes that both mutexes together provide the same serialization guarantee. This is correct because the Rust code always acquires DEALLOC_MTX before calling into the C++ deallocation path, and the C++ code always acquires dealloc_mtx before spawning the deallocation thread — so only one deallocation can be in flight at any time across the entire process.

What Actually Happened: The Benchmark Results

The benchmark results (visible in subsequent messages, particularly msg 2766) revealed that Intervention 1 alone produced essentially no measurable improvement: 37.9 seconds per proof versus the Phase 9 baseline of 38.0 seconds. The dealloc serialization did change behavior — individual deallocation times increased (some up to 3.5 seconds) because they now waited for previous deallocations to complete — but total throughput was unaffected.

This negative result is itself informative. It suggests that TLB shootdowns from concurrent munmap() were not the primary driver of memory bandwidth contention at this workload level. The real win came from Intervention 2 (reducing groth16_pool from 192 to 32 threads via gpu_threads = 32), which delivered 36.7 seconds per proof — a 3.4% improvement. Intervention 3 (a global atomic throttle flag) had negligible additional impact.

The failure of Intervention 1 to produce a measurable improvement does not mean the edit was wasted. In performance optimization, disproving a hypothesis is as valuable as confirming one. The serialization of async dealloc remains in the codebase as a safety measure — it prevents a class of pathological behavior that could become significant at different scales or on different hardware — and the benchmarking process validated that the real bottleneck lay elsewhere.

Input Knowledge Required

To understand this message, one needs:

The Phase 11 optimization framework: Knowledge that the system is operating under a specific hypothesis about memory bandwidth contention, with three proposed interventions targeting different mechanisms.
Understanding of TLB shootdowns: Knowledge that munmap() requires TLB invalidation across all cores, and that concurrent munmap() calls from multiple threads can cause performance degradation through inter-processor interrupts.
The async deallocation pattern: Understanding that groth16_cuda.cu spawns detached threads to deallocate large synthesis buffers asynchronously, and that these threads compete for memory management resources.
The C++/Rust FFI boundary: Knowledge that the proving engine spans C++ (CUDA kernels) and Rust (higher-level orchestration), and that synchronization primitives must be coordinated across this boundary.
The benchmarking methodology: Understanding that throughput is measured at c=20 j=15 (20 concurrent jobs, 15-way concurrency) and compared against a Phase 9 baseline of 38.0 seconds per proof.

Output Knowledge Created

This message produced:

A Rust static Mutex<()> in extern/bellperson/src/groth16/prover/supraseal.rs, named DEALLOC_MTX, that serializes all async deallocation calls from the Rust side.
A paired C++ static std::mutex dealloc_mtx in extern/supraseal-c2/cuda/groth16_cuda.cu (from msg 2750), that serializes the actual munmap() calls.
A benchmark result (37.9 s/proof) that disproved the hypothesis that TLB shootdowns from concurrent deallocation were a primary bottleneck at this workload level.
A validated experimental methodology: the intervention was implemented, tested, measured, and the result was used to redirect focus to the next intervention.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, reveals a methodical approach to performance optimization:

Hypothesis-driven design: The assistant doesn't randomly add locks. The Phase 11 design spec (msg 2744) was written first, identifying three specific mechanisms and proposing targeted interventions. Each intervention is a test of a specific hypothesis about where memory bandwidth is being wasted.

Iterative measurement: The assistant commits code, rebuilds, runs benchmarks, and checks results before moving to the next intervention. The benchmark of Intervention 1 (msg 2766) is followed immediately by analysis ("The dealloc serialization alone doesn't produce a measurable improvement") and a pivot to Intervention 2.

Cross-boundary awareness: The assistant understands that a synchronization change in C++ must be mirrored in Rust, and checks the Rust imports (msg 2752) before making the edit. This shows awareness of the FFI contract and the need for coordinated changes.

Pragmatic minimalism: The choice of a static Mutex<()> — the simplest possible synchronization primitive — reflects a design philosophy of making the smallest change that could plausibly work, then measuring. The assistant doesn't over-engineer or add complexity that might not be needed.

Intellectual honesty: When the benchmark shows no improvement, the assistant doesn't try to justify the change or find a different interpretation. The result is accepted ("essentially identical") and the focus shifts to the next hypothesis. This scientific approach to optimization is the hallmark of the entire session.

Conclusion

Message 2754 is a single line in a conversation spanning thousands of messages. It confirms that a file was edited. But that edit was a carefully considered intervention in a complex system, born from detailed waterfall timing analysis, grounded in an understanding of operating system memory management, implemented with cross-language FFI awareness, tested with rigorous benchmarking, and — critically — found to disprove its own hypothesis. In the broader narrative of the cuzk optimization campaign, this message represents a hypothesis tested and found wanting, clearing the path for the interventions that would actually move the needle. It is a quiet testament to the scientific method in systems engineering: propose, implement, measure, learn, and move on.