The Waterfall That Revealed the Truth: How DDR5 Bandwidth Contention Became the Bottleneck in Groth16 Proof Generation

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk SNARK proving engine, a sophisticated piece of infrastructure powering Filecoin's Proof-of-Replication (PoRep) protocol, had been through nine phases of optimization—each targeting a different bottleneck in the pipeline that transforms sector data into Groth16 proofs. Phase 9 had implemented PCIe transfer optimization, achieving a respectable 14.2% throughput improvement. Phase 10, a bold two-lock GPU interlock design intended to further parallelize GPU work, had just crashed and burned, consuming 16 GB of VRAM in a failed experiment that revealed fundamental limitations of CUDA's device-global memory management APIs.

Now, in message 2686 of this sprawling optimization session, the assistant presents a deceptively concise summary: a waterfall timing analysis of the reverted Phase 9 single-lock approach, benchmarked at maximum concurrency (c=20 workers, j=15 jobs). The message is only a few paragraphs, but it represents a critical turning point. It is the moment when the team stops chasing GPU scheduling improvements and recognizes the true adversary: DDR5 memory bandwidth contention.

This article examines that message in depth—its reasoning, its assumptions, its revelations, and the knowledge it creates for the optimization journey ahead.## Context: The Optimization Journey That Led to This Moment

To understand message 2686, one must appreciate the path that led to it. The cuzk SNARK proving engine had been under intensive optimization across multiple phases. Phase 7 introduced a per-partition dispatch architecture. Phase 8 implemented a dual-worker GPU interlock with narrowed C++ mutex scope, achieving 13-17% throughput improvement. Phase 9 optimized PCIe transfers for another 14.2% gain in single-worker mode. Each phase had been methodically designed, implemented, benchmarked, and documented.

Phase 10 was supposed to be the next leap forward: a two-lock GPU interlock that would allow two workers to independently claim GPU resources without contending on a single mutex. The design was elegant on paper—split the lock so that one worker could begin GPU work while the other was still preparing its data. But when implemented and tested, it failed catastrophically. The 16 GB VRAM on the single GPU could not accommodate pre-staged buffers from multiple workers simultaneously. Worse, the CUDA APIs that would have made the split work—cudaDeviceSynchronize and cudaMemPoolTrimTo—are device-global operations. They synchronize with any running kernel on the device, not just the calling thread. This meant the two-lock design was fundamentally impossible on a single GPU: the locks would always serialize at the CUDA driver level, defeating their entire purpose.

The assistant reverted the code to Phase 9's proven single-lock approach and ran a comprehensive benchmark sweep across concurrency levels from c=5 to c=20. Message 2686 is the analysis of the highest-concurrency run (c=20 j=15), and it contains the key insight that reorients the entire optimization strategy.## The Message: A Waterfall Timing Analysis

The subject message is presented as a structured summary under the heading "Waterfall Timing Analysis — Phase 9 gw=2." It is organized into six sections, each addressing a different dimension of the benchmark results. Let us examine it in full before dissecting its implications.

The message opens by documenting the Timeline Event System, noting that events are emitted from cuzk-core/src/engine.rs:39-48 to stderr in CSV format: TIMELINE,<offset_ms>,<event>,<job_id>,<detail>. Six event types are defined: SYNTH_START, SYNTH_END, CHAN_SEND, GPU_PICKUP, GPU_START, GPU_END. The assistant adds a telling observation: "No visualization tools exist — raw grep/sort only." This is a subtle but important piece of context—the analysis was performed by manually extracting and processing raw CSV data from daemon logs, not by running a polished dashboard. The waterfall was constructed through Python scripts that parsed these events, computed gaps, merged intervals, and derived utilization metrics.

The Key Findings table presents the headline numbers: throughput of 38.0 seconds per proof (1.58 proofs per minute), GPU utilization at 90.8% (658.8 seconds busy out of 725.2 seconds of wall time), GPU idle totaling 66.5 seconds across 12 gaps, and average GPU time per partition degrading to 6.0 seconds from a baseline of 4.9 seconds under lower concurrency.

The Waterfall Pattern section describes the behavioral evolution across the benchmark run. For the first five proofs, GPU idle gaps of 5-11 seconds appear between proofs—this is the synthesis lead time, the period during which the CPU is synthesizing the circuit partitions before any GPU work can begin. Starting at proof six, pipelining kicks in: "negative gaps" appear, meaning the GPU starts processing the next proof before finishing the current one. This is the desired steady-state behavior. But the section also flags two concerning trends: GPU time per partition degrades from 4.9 to 7.5 seconds as concurrency grows, and synthesis time inflates from 34 to 54 seconds under full load. Both are attributed to DDR5 bandwidth contention between synthesis workers and the prep_msm CPU-bound phase.

The GPU Idle Gaps section quantifies the wasted time: 12 gaps totaling 66.5 seconds, ranging from 2.6 to 10.6 seconds each, occurring at proof boundaries when synthesis cannot keep pace with GPU consumption.

The Inter-partition Worker Transitions section drills into the per-worker timing. Worker 0 experiences an average gap of 1334 milliseconds between partitions, with a maximum of 12.8 seconds (the gap between proofs). Worker 1 shows similar behavior: average 1151 milliseconds, maximum 10.8 seconds. Within-proof transitions are much tighter: typically 10-450 milliseconds, dominated by lock handoff overhead and cudaDeviceSynchronize calls.

The Why Phase 10 Two-Lock Was Abandoned section serves as both post-mortem and justification for the reversion. It confirms that 16 GB of VRAM cannot accommodate pre-staged buffers (~12 GB) from multiple workers simultaneously, and that CUDA's device-global synchronization APIs defeat any attempt at fine-grained locking. The conclusion is stark: "Phase 9's single-lock approach is already near-optimal for single-GPU."

Finally, the Throughput Bottleneck section delivers the core insight: at 79% theoretical efficiency (30 seconds theoretical versus 38 seconds observed), the gap comes from DDR5 bandwidth saturation. "The primary optimization path is reducing memory bandwidth pressure, not GPU scheduling."## The Reasoning: Why This Message Was Written

Message 2686 was written at a pivotal moment. The assistant had just abandoned Phase 10 after discovering its fundamental flaw. The natural response would be to move on to the next idea. But instead, the assistant chose to perform a deep diagnostic analysis of the reverted Phase 9 baseline. Why?

The answer lies in the optimization methodology that had been followed throughout this project. Each phase had been designed based on a hypothesis about where the bottleneck lay. Phase 7 hypothesized that per-partition dispatch would improve GPU utilization. Phase 8 hypothesized that dual workers would reduce lock contention. Phase 9 hypothesized that PCIe transfers were the bottleneck. Phase 10 hypothesized that a two-lock design would enable better GPU overlap. Each hypothesis was tested, measured, and either validated or refuted.

But after Phase 10's failure, the assistant recognized a deeper problem: the optimization strategy had been chasing GPU-side bottlenecks, but the data was pointing elsewhere. The waterfall analysis was not just a summary of Phase 9's performance—it was a diagnostic tool to answer a specific question: If GPU scheduling is already near-optimal, what is actually limiting throughput?

The message's structure reflects this diagnostic purpose. It begins by documenting the measurement infrastructure (the timeline event system), establishing that the data is trustworthy. It then presents the headline metrics, which immediately reveal something surprising: GPU utilization is already at 90.8%. This is a high number—there is not much GPU idle time to reclaim. The message then decomposes the remaining idle time into its components: synthesis lead time (unavoidable for the first proof), inter-proof gaps (synthesis not keeping up), and within-proof transition overhead (lock handoff). Each component is quantified.

The critical insight comes in the final two sections. By comparing theoretical throughput (30 seconds per proof, derived from pure GPU kernel time) against observed throughput (38 seconds), the assistant computes a 79% efficiency. The 21% gap is attributed to DDR5 bandwidth saturation, which manifests as inflated GPU time per partition (6.0 seconds versus 4.9 seconds) and inflated synthesis time (54 seconds versus 34 seconds). The conclusion is that memory bandwidth, not GPU scheduling, is the bottleneck.

This reasoning is significant because it reorients the entire optimization strategy. Instead of further GPU-side improvements (which would yield diminishing returns given 90.8% utilization), the next phase of optimization must target memory bandwidth contention. This insight directly leads to Phase 11, which proposes three interventions: bounding async deallocation to a single thread to eliminate TLB shootdown storms, reducing the groth16_pool thread count to shrink memory footprint, and adding a lightweight semaphore to briefly pause synthesis workers during the b_g2_msm window.## Assumptions Embedded in the Analysis

Every analysis rests on assumptions, and message 2686 contains several that deserve scrutiny.

The first assumption is that the timeline events are accurate and complete. The assistant notes that events are emitted from cuzk-core/src/engine.rs:39-48 to stderr in CSV format. This instrumentation was presumably added during an earlier phase of optimization, and the assistant assumes that it captures all relevant state transitions. If events are dropped, misordered, or imprecisely timestamped, the waterfall analysis would be compromised. The assistant implicitly trusts the instrumentation, and the Python parsing scripts do not validate event ordering or detect missing events.

The second assumption is that the six event types (SYNTH_START, SYNTH_END, CHAN_SEND, GPU_PICKUP, GPU_START, GPU_END) are sufficient to characterize the pipeline's behavior. Notably, there is no event for prep_msm start or end, even though the analysis identifies prep_msm as a significant contributor to the 6.0-second GPU partition time. The assistant infers prep_msm timing indirectly—it is the difference between the reported gpu_ms (which includes prep_msm join time) and the actual CUDA kernel execution time. This inference is reasonable but introduces uncertainty: the breakdown of "CUDA kernels: ~3.8s, prep_msm join: ~1.7s, Lock handoff: ~0.5s" is presented without direct measurement of each component.

The third assumption is that the theoretical throughput of 30 seconds per proof is a valid baseline. This figure is derived by taking the average GPU time per partition (6.0 seconds) under load, multiplying by 10 partitions per proof, and dividing by 2 workers. But this calculation embeds the degraded 6.0-second figure—it is not a "pure" theoretical maximum. A truly uncontaminated theoretical would use the 3.8-second CUDA kernel time, yielding 19 seconds per proof (3.8 × 10 / 2). The assistant's choice to use 6.0 seconds reflects a pragmatic acknowledgment that some overhead is inherent, but it also means the 79% efficiency metric is conservative.

The fourth assumption is that DDR5 bandwidth contention is the root cause of both GPU time inflation and synthesis time inflation. This is a correlation-based inference: as concurrency increases, both metrics degrade in tandem. The assistant does not directly measure memory bandwidth utilization (e.g., via perf stat or numastat) to confirm saturation. The inference is plausible and consistent with the observed behavior, but it remains an inference rather than a direct measurement.

These assumptions do not invalidate the analysis—they are necessary simplifications. But they frame the conclusions as probabilistic rather than certain, and they point toward the next diagnostic step: direct measurement of memory bandwidth utilization to confirm the hypothesis.## Input Knowledge Required to Understand This Message

Message 2686 is dense with domain-specific concepts. A reader unfamiliar with the cuzk SNARK proving engine or Groth16 proof generation would need substantial background knowledge to interpret its findings.

First, one must understand the partition-based architecture of Groth16 proof generation. A Filecoin PoRep proof is not generated monolithically—it is split into 10 partitions, each of which goes through a synthesis phase (CPU-bound circuit evaluation) followed by a GPU-bound phase (NTT, multi-scalar multiplication, and other elliptic curve operations). The two phases are connected by a channel: synthesized partitions are sent to a GPU worker pool for processing. This architecture means that the CPU and GPU can work in parallel on different partitions, but the pipeline's throughput is limited by the slower of the two.

Second, one must understand the dual-worker model. The cuzk engine uses two GPU worker threads (gw=2) that compete for a single mutex to claim GPU resources. Each worker picks up a synthesized partition, performs GPU computation, and releases the lock. With two workers, the GPU can be kept busy while one worker is preparing its data (the prep_msm phase) and the other is executing CUDA kernels. The lock handoff overhead is the time between one worker releasing the mutex and the next worker acquiring it.

Third, one must understand DDR5 memory bandwidth contention. On a modern CPU, all cores share access to the same DDR5 memory channels. When multiple threads simultaneously access memory—as happens when 10 synthesis workers and 192 groth16_pool threads compete for bandwidth—the memory controller must arbitrate, causing each thread's effective bandwidth to drop. This manifests as increased latency for memory-bound operations, which is exactly what the waterfall analysis reveals: both synthesis time and GPU partition time inflate under load.

Fourth, one must understand CUDA device-global operations. The cudaDeviceSynchronize function does not synchronize only the calling thread's kernels—it synchronizes all pending operations on the entire device. Similarly, cudaMemPoolTrimTo operates on the device's memory pool, not a per-context pool. These semantics mean that fine-grained locking strategies (like Phase 10's two-lock design) are fundamentally incompatible with single-GPU deployments, because any CUDA API call that touches device state will serialize with concurrent operations from other threads.

Fifth, one must understand the timeline event system and how waterfall analysis works. The assistant extracted events from daemon logs, parsed CSV-formatted lines, grouped them by job ID and worker, computed GPU busy intervals, merged overlapping intervals, and derived utilization metrics. The concept of "negative gaps" (where the next proof's GPU start occurs before the current proof's GPU end) is counterintuitive but essential for understanding pipelining behavior.

Without this knowledge, the message's conclusions would appear as opaque numbers. With it, the waterfall analysis becomes a compelling narrative about resource contention and pipeline dynamics.## Output Knowledge Created by This Message

Message 2686 creates several pieces of actionable knowledge that directly shape the subsequent optimization phases.

The most important output is the identification of DDR5 bandwidth contention as the primary bottleneck. Before this message, the optimization strategy had focused on GPU scheduling, lock contention, and PCIe transfer efficiency. The waterfall analysis reframes the problem: at 90.8% GPU utilization, there is little GPU idle time to reclaim. The 21% efficiency gap (79% theoretical efficiency) is caused by memory bandwidth saturation, which inflates both synthesis time and GPU partition time. This insight redirects optimization effort from GPU-side improvements to memory-bandwidth reduction.

The second output is the quantification of the efficiency gap. The assistant computes theoretical throughput of 30 seconds per proof (6.0 seconds per partition × 10 partitions / 2 workers) versus observed throughput of 38 seconds. This 8-second gap represents the optimization opportunity. By decomposing the 6.0-second partition time into CUDA kernels (~3.8s), prep_msm join (~1.7s), and lock handoff (~0.5s), the assistant identifies which components are amenable to optimization. The prep_msm join time, in particular, is a CPU-bound phase that competes for memory bandwidth—reducing it would directly improve throughput.

The third output is the post-mortem of Phase 10's two-lock design. The message documents why the design failed: 16 GB VRAM cannot accommodate pre-staged buffers from multiple workers, and CUDA device-global APIs defeat fine-grained locking. This knowledge prevents future attempts at similar strategies and establishes that Phase 9's single-lock approach is near-optimal for single-GPU deployments. This is a valuable negative result—knowing what doesn't work saves future engineering effort.

The fourth output is the characterization of the waterfall pattern. The message describes how the pipeline transitions from an initial state (proofs 1-5 with GPU idle gaps of 5-11 seconds) to a steady state (proofs 6+ with overlapping proofs and negative gaps). This characterization helps distinguish between unavoidable startup overhead (synthesis lead time for the first proof) and systemic inefficiency (memory bandwidth contention under sustained load). It also reveals that the pipeline does achieve overlap—the GPU is not waiting for synthesis in steady state—but the overlap is incomplete because synthesis slows down under memory pressure.

The fifth output is the per-worker transition timing data. Worker 0 averages 1334ms between partitions, Worker 1 averages 1151ms. Within-proof transitions are 10-450ms. These numbers provide a baseline for evaluating future locking improvements. If a future optimization reduces lock handoff overhead, these transition times should decrease.

The sixth output is the methodology for waterfall analysis. The assistant demonstrates how to extract structured data from unstructured CSV logs, compute GPU utilization from event timestamps, merge overlapping intervals across workers, and derive throughput metrics. This methodology can be reused for future benchmark runs, providing a consistent measurement framework across optimization phases.## Mistakes and Incorrect Assumptions

While message 2686 is analytically rigorous, it contains some limitations that deserve examination.

The most significant potential error is the attribution of causality to correlation. The assistant observes that as concurrency increases, both GPU time per partition and synthesis time increase. The inference that DDR5 bandwidth contention is the cause is reasonable, but it is not proven. Other factors could contribute: increased cache pressure from more active threads, NUMA effects if threads are scheduled across sockets, or thermal throttling under sustained load. The assistant does not present evidence ruling out these alternative explanations. A more rigorous approach would involve direct measurement of memory bandwidth utilization (e.g., using Intel PCM or AMD's perf counters) to confirm saturation.

The second limitation is the reliance on inferred rather than measured timing breakdowns. The assistant decomposes the 6.0-second GPU partition time into 3.8 seconds of CUDA kernels, 1.7 seconds of prep_msm join, and 0.5 seconds of lock handoff. But these numbers are not directly measured—they are estimates based on the assistant's understanding of the code paths. The CUDA kernel time is likely derived from GPU-side timers (e.g., CUDA events), but the prep_msm join time is inferred as the difference between the reported gpu_ms and the kernel execution time. This inference assumes that lock handoff is the only other component, which may not be accurate if there are other unmeasured overheads.

The third issue is the theoretical throughput calculation. The assistant computes 30 seconds per proof as the theoretical maximum (6.0s × 10 / 2), but this uses the degraded 6.0-second partition time. A truly uncontaminated theoretical would use the baseline 4.9-second figure (observed at lower concurrency), yielding 24.5 seconds per proof. Alternatively, using the pure CUDA kernel time of 3.8 seconds would give 19 seconds. The choice of 6.0 seconds makes the theoretical less ambitious and the efficiency gap (79%) smaller than it could be. This is a conservative choice that understates the optimization opportunity.

The fourth limitation is the absence of statistical rigor. The benchmark run (c=20 j=15) produced 15 proofs, but the waterfall analysis treats each proof as an independent data point. There is no discussion of variance, confidence intervals, or outlier detection. The 10.6-second GPU idle gap between proofs 1 and 2, for example, could be a one-time startup artifact rather than a representative measurement. The assistant acknowledges this implicitly by distinguishing between "proofs 1-5" and "proofs 6+" behavior, but does not quantify the uncertainty.

These limitations do not undermine the message's core insight—DDR5 bandwidth contention is clearly a significant factor—but they suggest that the analysis should be treated as diagnostic rather than definitive. The Phase 11 interventions that follow from this analysis are well-motivated, but their impact should be measured against a more rigorous baseline.## The Thinking Process Visible in the Message

Message 2686 is a summary, but it reveals a sophisticated thinking process that unfolded across the preceding messages ([msg 2681] through [msg 2685]). Understanding this process illuminates how the assistant arrived at its conclusions.

The thinking process began with raw data. In [msg 2681], the assistant parsed the timeline CSV file and identified four benchmark runs separated by large gaps in GPU events. It computed GPU utilization for each run, discovering that utilization ranged from 84.9% to 90.8%. It also noticed that GPU idle gaps of 7-10 seconds appeared between proofs, and that average GPU time per partition degraded from 5.0 seconds to 6.0 seconds as concurrency increased. At this point, the assistant was still in data exploration mode—it had not yet formed a hypothesis about the root cause.

In [msg 2682], the assistant shifted to detailed waterfall visualization. It focused on the fourth run (c=20 j=15), normalized timestamps to t=0, and printed the first 60 GPU events. This revealed the alternating pattern of Worker 0 and Worker 1 claiming partitions, and the gaps between GPU_END and the next GPU_START. The assistant then computed per-worker transition gaps, finding that Worker 0 averaged 1334ms between partitions and Worker 1 averaged 1151ms. The within-proof transitions were typically 10-450ms. This was the first concrete evidence that lock handoff overhead was small relative to the total partition time.

In [msg 2683], the assistant built a per-proof waterfall table, showing synthesis start time, first GPU time, last GPU time, synthesis duration, total GPU time, and the gap between proofs. This was the decisive analysis. The table revealed that synthesis time grew from 34 seconds (proof 1) to 54 seconds (proof 15) under load, and that GPU time per partition grew from 4.9 seconds to 7.5 seconds. The "gap" column showed negative values starting at proof 6, confirming that pipelining was working. The assistant then computed overall GPU utilization at 90.8% and identified 12 idle gaps totaling 66.5 seconds.

In [msg 2684], the assistant synthesized these observations into a throughput analysis. It computed theoretical throughput (30 seconds per proof) and compared it to observed throughput (38 seconds), yielding 79% efficiency. It then decomposed the 6.0-second GPU partition time into CUDA kernels (~3.8s), prep_msm join (~1.7s), and lock handoff (~0.5s). The key insight was stated explicitly: "At c=20 j=15, gpu_ms per partition inflates from 4.9s to 6.5s+ due to DDR5 bandwidth contention (synthesis + prep_msm competing for memory BW). This is the primary reason throughput plateaus at ~38s/proof."

In [msg 2685], the assistant updated its todo list, marking Phase 10 as abandoned and Phase 11 as the next step. This set the stage for message 2686, which consolidates all findings into a structured summary.

The thinking process visible across these messages is methodical and hypothesis-driven. The assistant does not jump to conclusions—it explores data, visualizes patterns, quantifies effects, and only then proposes root causes. The waterfall analysis is not a single insight but the culmination of a multi-step diagnostic process that transforms raw CSV logs into actionable knowledge.## Conclusion: The Message That Changed the Optimization Trajectory

Message 2686 is, on its surface, a brief summary of benchmark results. But within the context of the cuzk optimization project, it represents a fundamental reorientation. For nine phases, the optimization strategy had been directed at GPU-side bottlenecks: lock contention, PCIe transfer efficiency, worker scheduling. Each phase had delivered incremental improvements, but the gains were diminishing. Phase 10's spectacular failure—a two-lock design that crashed with OOM errors because CUDA's device-global APIs made fine-grained locking impossible—forced a reassessment.

The waterfall analysis in message 2686 provided the evidence needed for that reassessment. By demonstrating that GPU utilization was already at 90.8%, the assistant showed that there was little GPU idle time left to reclaim. By computing the efficiency gap (79% of theoretical throughput) and attributing it to DDR5 bandwidth contention, the assistant identified the true bottleneck. By decomposing the partition time into its components, the assistant pointed toward specific interventions: reducing memory bandwidth pressure from synthesis workers, from the groth16_pool threads, and from deallocation storms.

This message is a masterclass in diagnostic reasoning. It does not merely present data—it interprets data through the lens of the system's architecture, identifies the limiting factor, and provides a roadmap for the next optimization phase. The Phase 11 interventions that follow—bounding async deallocation to a single thread, reducing the groth16_pool thread count, and adding a lightweight semaphore to pause synthesis during b_g2_msm—are all direct consequences of the analysis presented here.

In the broader narrative of the cuzk optimization project, message 2686 is the turning point where the team stopped optimizing for GPU throughput and started optimizing for memory bandwidth efficiency. It is the moment when the waterfall revealed the truth, and the truth was that the bottleneck was not where anyone had been looking.