The Commit That Captured a Bottleneck Shift: Phase 9's Pivotal Finding

Introduction

In the course of optimizing the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), a single git commit captured a critical turning point. Message [msg 2556] is a deceptively simple bash command — a git add followed by git commit — but its commit message encapsulates the most important diagnostic discovery of the entire Phase 9 investigation. This message marks the moment when the team's understanding of the bottleneck fundamentally shifted: from PCIe transfer latency and GPU kernel execution time to CPU memory bandwidth contention on the host's DDR5 channels.

The message reads:

[assistant] [bash] git add extern/supraseal-c2/cuda/groth16_cuda.cu && git commit -m "feat(cuzk): Phase 9 fine-grained pre-staging timing instrumentation

Add per-stage timing to prestage setup: sync_ms, trim_ms, alloc_ms, upload_ms.

Key findings with c=15 j=15 gw=1:
- Pre-staging overhead: 18ms avg (negligible - PCIe gen5 is fast)
- GPU kernels: 1824ms avg/partition
- CPU critical path (prep_msm + b_g2_msm): 2393ms avg/partition
- CPU is the bottleneck, not GPU — DDR5 bandwidth wall
  with 10 concurrent synthesis workers competing for memory
- Throughput: 41.3s/proof (steady-state)
- c=30 j=20 causes OOM/crash from memory pressure"
[feat/cuzk 599522de] feat(cuzk): Phase 9 fine-grained pre-staging timing instrumentation
 1 file changed, 17 insertions(+), 2 deletions(-)

Context: The Phase 9 Investigation

To understand why this message was written, one must trace back through the preceding investigation. Phase 9 was designed as a PCIe transfer optimization for the cuzk SNARK proving engine. The earlier Phase 8 dual-worker GPU interlock had achieved a 13–17% throughput improvement, and the team was now focused on reducing the overhead of transferring data between host memory and GPU device memory.

The assistant had spent the previous several messages (from [msg 2527] through [msg 2555]) running a systematic series of benchmarks. The daemon was configured with gw=1 (one GPU worker per device) and the team tested concurrency levels of 15, 20, and 30 proofs running simultaneously. The benchmarks used cuzk-bench batch with --type porep --c1 /data/32gbench/c1.json, running 15 to 30 proofs at various concurrency levels.

The initial hypothesis was that PCIe transfer overhead was the primary bottleneck. The Phase 9 optimization had added pre-staging logic to move data to GPU memory ahead of kernel execution. But the benchmarks at c=15 and c=20 showed a puzzling plateau: throughput stabilized at approximately 41 seconds per proof regardless of concurrency. At c=30, the system crashed with an out-of-memory (OOM) error.

The Diagnostic Breakthrough

The critical diagnostic data came from fine-grained timing instrumentation that the assistant had added to the pre-staging path in groth16_cuda.cu. By instrumenting each stage of the pre-staging setup — sync_ms (device synchronization time), trim_ms (memory pool trimming time), alloc_ms (VRAM allocation time), and upload_ms (PCIe upload time) — the assistant was able to decompose where time was actually being spent.

The results were startling. The pre-staging overhead was only 18 milliseconds on average — negligible in the context of per-partition times measured in seconds. The PCIe Gen5 bus was fast enough that transfers were not a bottleneck. The GPU kernel execution time averaged 1,824 milliseconds per partition, which was healthy and well-optimized.

But the CPU-side operations — prep_msm (preparing multi-scalar multiplication point tables) and b_g2_msm (G2-group multi-scalar multiplication) — averaged 1,909ms and 484ms respectively, totaling 2,393ms per partition. This CPU critical path was longer than the GPU kernel time. The GPU was finishing its work in 1.8 seconds but then sitting idle for approximately 600 milliseconds per partition, waiting for the CPU thread to finish its computation before the function could return.

This was the moment of insight captured in the commit message: "CPU is the bottleneck, not GPU — DDR5 bandwidth wall with 10 concurrent synthesis workers competing for memory."

Why This Message Was Written

The commit served multiple purposes simultaneously. First, it was a practical necessity: the code changes to groth16_cuda.cu — adding the per-stage timing instrumentation — needed to be committed to version control to preserve the diagnostic capability for future investigations. The 17 insertions and 2 deletions represented the instrumentation code that made the benchmark analysis possible.

Second, the commit message functioned as a research notebook entry. In a fast-moving optimization campaign where dozens of benchmarks were being run and analyzed, the commit message captured the key findings in a durable, timestamped form. The assistant was not merely checking in code; it was documenting the empirical discovery that the bottleneck had shifted from GPU execution to CPU memory bandwidth contention.

Third, the message served as a communication artifact for the user (the human collaborator). The user had previously suggested that CPU memory bandwidth might be the limiting factor (referenced in [msg 2527] as "CPU memory bandwidth problem, as you suggested"). The commit message confirmed this hypothesis with hard data from the instrumented benchmarks, closing the loop on the diagnostic investigation.

Assumptions and Their Validation

Several assumptions underpinned this message. The first was that the pre-staging instrumentation would not itself distort the measurements — that adding sync_ms, trim_ms, alloc_ms, and upload_ms timing calls would not significantly alter the execution path or introduce measurement artifacts. This assumption proved valid, as the pre-staging overhead was confirmed to be negligible at 18ms.

A second assumption was that the benchmark at c=15 j=15 (15 proofs with 15 concurrent workers) represented a steady-state measurement rather than a transient artifact. The queue times in the benchmark output showed a clear pattern: the first batch of proofs had growing queue times (314ms, 785ms, 1216ms, 1663ms...) as all 15 proofs were submitted simultaneously and processed one at a time, but subsequent waves had small queues (363–388ms). The prove times were consistent at 35–36 seconds, confirming steady-state behavior.

A third assumption was that the c=30 j=20 crash was caused by memory pressure rather than a software bug. The daemon log showed async_dealloc_ms=5799 (5.8 seconds just to deallocate synthesis results), prep_msm_ms=10600 (10.6 seconds, 6× the normal 1.7s), and b_g2_msm_ms=4497 (4.5 seconds, 12× the normal 380ms). These pathological numbers confirmed memory bandwidth saturation as the root cause.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The Groth16 proving system and its role in Filecoin's Proof-of-Replication is foundational context — the pipeline synthesizes circuit witnesses, evaluates constraints, and produces zero-knowledge proofs through a sequence of NTT (Number Theoretic Transform) operations and MSM (Multi-Scalar Multiplication) computations. The prep_msm operation builds split vectors from multi-gigabyte point tables, which requires heavy sequential memory reads. The b_g2_msm operation performs a G2-group MSM that is CPU-bound.

Knowledge of the hardware architecture is also essential. The system has an 8-channel DDR5 memory configuration, and 10 concurrent synthesis workers compete for this bandwidth. Each proof's synthesis uses approximately 7–8 GiB for witness and constraint data, and the SRS (Structured Reference String) occupies 44 GiB in memory. At c=20 concurrency, approximately 150 GiB of synthesis data plus the 44 GiB SRS creates extreme memory bandwidth pressure.

The PCIe Gen5 characteristics matter too: the pre-staging overhead of 18ms confirms that the PCIe bus is not a bottleneck, which was the original motivation for Phase 9's optimization focus.

Output Knowledge Created

This message created several forms of knowledge. Most immediately, it established the definitive bottleneck analysis for the Phase 9 configuration: the GPU kernel time (1,824ms) is not the limiting factor; the CPU critical path (2,393ms) is. This shifted the optimization strategy from GPU-focused improvements to CPU memory bandwidth mitigation.

The message also created a reproducible benchmark baseline: 41.3 seconds per proof at c=15 j=15 with gw=1. This baseline became the reference point for evaluating future optimizations, particularly the Phase 10 two-lock design that would follow.

The commit itself created instrumented code that could be reused for future diagnostic work. The per-stage timing in the pre-staging path (sync_ms, trim_ms, alloc_ms, upload_ms) became a reusable diagnostic tool.

Perhaps most importantly, the message created negative knowledge: the understanding that c=30 concurrency causes OOM crashes. This bounded the feasible operating region of the system and informed the design of subsequent phases, which would need to work within the memory constraints of the hardware.

The Thinking Process Visible

The commit message reveals a clear chain of reasoning. The assistant had started with a hypothesis that PCIe transfers were the bottleneck (the Phase 9 premise). After adding instrumentation and running benchmarks, the data forced a revision of that hypothesis. The commit message presents the findings in order of significance: pre-staging is negligible (18ms), GPU kernels are fast (1,824ms), but the CPU critical path dominates (2,393ms). The conclusion follows logically: the CPU is the bottleneck due to DDR5 bandwidth contention with 10 concurrent synthesis workers.

The inclusion of the c=30 crash observation is particularly telling. It shows that the assistant was thinking about the boundary conditions of the system — not just the steady-state throughput, but the point at which the system breaks. This systems-thinking approach is characteristic of performance engineering at scale.

Mistakes and Limitations

One limitation of the analysis captured in this commit is that it does not fully decompose the prep_msm and b_g2_msm operations into their constituent memory access patterns. The commit identifies these operations as the bottleneck but does not specify which memory access patterns within them are most problematic — whether it's cache misses, TLB misses, or raw bandwidth saturation. This level of detail would require hardware performance counters (e.g., perf stat with MEM_LOAD events) which were not available in the containerized environment.

Another limitation is that the benchmark was run with only one GPU worker (gw=1). The earlier Phase 8 dual-worker configuration had shown different characteristics, and it was not yet clear how the CPU memory bandwidth bottleneck would manifest with two GPU workers competing for the same host memory bus.

The commit also does not address the synthesis workers' memory bandwidth consumption directly. It notes that "10 concurrent synthesis workers competing for memory" is the cause, but does not quantify how much bandwidth each worker consumes or whether throttling synthesis workers could improve overall throughput.

Conclusion

Message [msg 2556] is far more than a routine code commit. It is the documentary record of a pivotal diagnostic discovery in a complex optimization campaign. The 17 lines of instrumentation added to groth16_cuda.cu revealed that the bottleneck in the SUPRASEAL_C2 Groth16 proof pipeline had shifted from GPU kernel execution and PCIe transfers to CPU memory bandwidth contention — a finding that would fundamentally redirect the optimization strategy toward the Phase 10 two-lock design and beyond. The commit message, with its concise bullet points and clear hierarchy of findings, serves as both a technical reference and a narrative milestone in the ongoing effort to architect a continuous, memory-efficient proving pipeline for Filecoin's decentralized storage network.