A Single Data Point: The Systematic Benchmark That Revealed DDR5 Contention

Introduction

In the high-stakes world of Filecoin proof generation optimization, progress is rarely linear. The path from one optimization phase to the next is paved with failed experiments, reverted code, and hard-won empirical data. This article examines a single message ([msg 2673]) from a coding session that captures a pivotal moment: the first data point in a systematic benchmark sweep that would ultimately reveal DDR5 memory bandwidth contention as the dominant bottleneck in the cuzk SNARK proving engine.

The message itself is deceptively simple — a brief report of benchmark results followed by the launch of the next test:

c=5 j=5: 41.4s/proof throughput, 50.3s average prove time. Now c=10 j=10:

But behind this terse summary lies a rich story of architectural backtracking, disciplined experimental methodology, and the kind of evidence-driven debugging that separates effective optimization from guesswork.

Context: The Phase 10 Failure

To understand why this message was written, one must first understand what came immediately before. The assistant had been implementing "Phase 10" — a two-lock GPU interlock design intended to improve throughput by allowing multiple GPU workers to overlap their memory operations with kernel execution. The idea was elegant: split the single gpu_lock into a mem_mtx (for VRAM allocation and data upload) and a compute_mtx (for GPU kernel execution), allowing one worker to pre-stage its data while another worker's kernels were still running.

In practice, the design collapsed under the weight of hardware reality. The system's GPU had only 16 GB of VRAM, and pre-staging for a single proof consumed approximately 12 GB. When the first worker pre-staged its buffers under mem_mtx and then released that lock, its 12 GB of VRAM remained allocated. A second worker, entering compute_mtx first (or attempting its own pre-stage), would find insufficient VRAM and crash with an out-of-memory error. Even the fallback path — which used cudaDeviceSynchronize and pool trimming to reclaim memory — could not prevent OOM failures because CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations. They cannot selectively reclaim one worker's memory without interfering with another worker's running kernels.

The assistant's diagnosis was precise: "This is a fundamental design flaw in the two-lock approach: pre-staged VRAM allocated under mem_mtx persists until consumed under compute_mtx, but another worker may enter compute_mtx first and find VRAM exhausted." The Phase 10 code was reverted to Phase 9's proven single-lock architecture, and the assistant pivoted to a different strategy: systematic characterization of the existing bottleneck.

The Benchmarking Methodology

The sweep that begins with this message ([msg 2673]) was not arbitrary. The assistant chose to run benchmarks at concurrency levels of c=5, c=10, c=15, and c=20, incrementally increasing the load on the system. This阶梯式 approach is a classic experimental technique: start at a moderate load to establish a baseline, then increase systematically to observe how performance degrades or scales.

The parameters c=5 j=5 deserve explanation. In the cuzk-bench framework, --count 5 specifies the total number of proofs to generate, while --concurrency 5 controls how many proofs are in-flight simultaneously. With 5 concurrent proofs and 10 partitions per proof, the system's pipeline — which includes synthesis workers, GPU workers, and the daemon's scheduling logic — is exercised at a moderate level. The Phase 9 configuration used gpu_workers_per_device = 2, meaning two GPU workers share the single physical GPU via the mutex lock.

The first result was revealing: 41.4 seconds per proof throughput, with an average prove time of 50.3 seconds. The gap between throughput (total wall time divided by proof count) and average prove time reflects queueing effects — proofs spend time waiting for earlier proofs to complete before their own partitions can be processed. At c=5, the system was not yet saturated, but the 50.3s average prove time already showed significant inflation over the single-proof isolation benchmark of ~32 seconds.

What the First Data Point Revealed

The 41.4s/proof throughput at c=5 is interesting not for its absolute value but for what it portends. In the subsequent benchmarks (c=10, c=15, c=20), the assistant would discover that throughput actually improved with higher concurrency, plateauing at approximately 38 seconds per proof. This counterintuitive result — higher concurrency yielding better throughput — indicated that at low concurrency, the GPU was not fully utilized. The synthesis pipeline could not keep the GPU fed with partitions, leaving it idle between kernel launches.

As concurrency increased, GPU utilization climbed, reaching 90.8% at the highest levels. But the throughput plateau at ~38s/proof signaled a new bottleneck: DDR5 memory bandwidth contention. At high concurrency, the system's 192 synthesis threads (from partition_workers = 10 and rayon's thread pool) competed with the prep_msm and b_g2_msm computations for the same memory channels. Both synthesis (CPU-bound SpMV evaluation) and prep_msm (CPU-bound MSM preprocessing) inflated under load as they thrashed the DDR5 bus, each slowing the other.

This finding — that memory bandwidth, not GPU compute, was the limiting factor — would become the foundation for Phase 11's three-intervention design: bounding async deallocation to a single thread (to eliminate TLB shootdown storms), reducing the groth16_pool thread count (to shrink b_g2_msm's memory footprint), and adding a lightweight semaphore to pause synthesis workers during b_g2_msm's critical window.

The Thinking Process Visible in the Message

Even in this brief message, the assistant's systematic thinking is evident. The report "c=5 j=5: 41.4s/proof throughput, 50.3s average prove time" is followed immediately by "Now c=10 j=10:" — no pause for analysis, no premature conclusion-drawing. The assistant is executing a pre-planned experimental protocol, collecting data points before attempting interpretation. This discipline is crucial in performance optimization, where a single data point can mislead. Only by observing the full curve — how throughput changes as concurrency increases — can the true bottleneck be identified.

The message also reveals the assistant's comfort with command-line tooling and benchmark orchestration. The bash command chains FIL_PROOFS_PARAMETER_CACHE, the cuzk-bench binary, and tee for logging, all in a single invocation. This fluency with the toolchain allows the assistant to move quickly from hypothesis to data, a key capability when exploring a complex optimization space.

Assumptions and Required Knowledge

This message assumes significant domain knowledge on the part of the reader (or the system recording these logs). One must understand that c=5 j=5 refers to 5 proofs with 5-way concurrency, that 41.4s/proof throughput is calculated as total wall time divided by proof count, and that the Phase 9 architecture uses a single mutex to serialize GPU access across workers. The path /data/32gbench/c1.json points to pre-computed C1 output used as input for the Groth16 proof pipeline. The environment variable FIL_PROOFS_PARAMETER_CACHE points to the directory containing Filecoin's parameter files (the Structured Reference Strings or SRS).

The message also assumes that the Phase 9 codebase is stable and that the benchmark results are reproducible. This assumption proved valid — the subsequent sweep produced clean, interpretable data that led directly to the Phase 11 design.

Broader Significance

This message, for all its brevity, represents a critical inflection point in the optimization journey. The failure of Phase 10 had demonstrated that architectural complexity (two locks, pre-staging, speculative allocation) could not overcome hardware constraints (16 GB VRAM, device-global synchronization). The pivot to systematic benchmarking — starting with this c=5 data point — represented a return to first principles: measure first, then optimize.

The data collected in this sweep would directly inform the Phase 11 design, which the assistant documented in c2-optimization-proposal-11.md. That proposal's three interventions — bounding async deallocation, reducing thread pool oversubscription, and adding a synthesis throttle — were all grounded in the empirical observation that DDR5 bandwidth, not GPU compute, was the limiting factor. Without the systematic sweep that begins with this message, the team might have continued chasing GPU-side optimizations (faster kernels, better MSM algorithms) while the real bottleneck sat in the CPU memory subsystem.

In the end, the most valuable optimization insight came not from clever code but from a simple question: what happens when we increase the load and watch how the system responds? This message captures the moment that question began to be answered.