The Moment the Bottleneck Shifted: Diagnosing CPU Memory Bandwidth Contention in a GPU Proving Pipeline

Introduction

In the high-stakes world of Filecoin proof generation, every millisecond counts. The SUPRASEAL_C2 Groth16 proving pipeline—responsible for generating the cryptographic proofs that underpin Filecoin's storage verification—had already undergone eight phases of optimization, each peeling back another layer of bottleneck. Phase 9 had targeted PCIe transfer optimization, and early results were promising. But something was wrong. Throughput had plateaued at roughly 41 seconds per proof, and the GPU utilization was "jumpy"—alternating between bursts of activity and puzzling idle periods. Message 2554 in this coding session represents the diagnostic breakthrough: the moment the assistant quantified exactly where the bottleneck had shifted, and in doing so, revealed that the optimization battle had moved from the GPU to a completely unexpected front—CPU memory bandwidth contention.

This message is a masterclass in performance analysis under real-world constraints. It combines a concise summary of benchmark results, a sophisticated ad-hoc timing analysis using shell scripting and AWK, and the raw output that would become the foundation for Phase 10's two-lock architecture. To understand this message is to understand how performance engineering works at the frontier of GPU-accelerated cryptography.

The Message: A Diagnostic Snapshot

The message opens with a succinct summary:

41.3s/proof with c=15 j=15. Very consistent with the c=20 j=15 result (41.6s). The queue times grow linearly (all 15 submitted at once, processed one at a time), but prove times are steady 35-36s. Last two proofs faster (29-31s) because fewer synths competing.

This single sentence encapsulates a wealth of information. The assistant has just completed a benchmark run with 15 concurrent proofs (c=15) and 15 total proofs (j=15). The 41.3 seconds per proof is a steady-state measurement—consistent with the earlier c=20 run at 41.6 seconds—indicating that the system has hit a fundamental limit. The observation that "queue times grow linearly" confirms that all 15 proofs were submitted simultaneously and processed sequentially by a single GPU worker. The observation that the last two proofs completed faster (29-31 seconds) because "fewer synths competing" is the first clue that the bottleneck involves CPU-side resources, not GPU compute.

But the real substance of the message lies in what follows: a sophisticated shell one-liner that parses the daemon's timing logs to extract five critical metrics, and the output that reveals the true bottleneck.

The Bash Command: A Portable Performance Analyzer

The assistant constructs a multi-stage AWK script that processes the daemon's TIMELINE log entries. Let's examine what this script does, because it reveals the assistant's deep understanding of the system's instrumentation.

First, it filters for GPU_START and GPU_END events, sorting them by timestamp. It tracks the first GPU_START and last GPU_END to compute the total wall-clock span. It computes the gap between consecutive GPU_END and GPU_START events—the idle time between GPU operations. It extracts gpu_ms from each GPU_END line to compute total GPU kernel time. The output gives:

Why This Message Was Written: The Diagnostic Imperative

This message exists because the assistant had reached an impasse. Phase 9's PCIe optimization had been implemented and committed, but the expected throughput improvement had not materialized. The GPU utilization was erratic—sometimes high, sometimes low—and the root cause was unknown. The assistant needed to answer a specific question: where is the time going?

The earlier benchmarks had shown that GPU kernel time had dropped dramatically to ~1.8 seconds per partition (down from ~2.4 seconds before Phase 9). Yet the per-proof wall time remained stubbornly at ~41 seconds. Something was stealing cycles, and the assistant needed to quantify exactly what.

The message is therefore a diagnostic pivot. Rather than continuing to optimize GPU-side operations, the assistant recognized that the bottleneck had shifted and needed to be precisely characterized before the next optimization phase could be designed. This is a classic performance engineering pattern: when throughput plateaus despite clear improvements to one subsystem, the bottleneck has moved elsewhere, and you must find it before you can fix it.

Input Knowledge Required

To understand this message, one needs knowledge of:

  1. The SUPRASEAL_C2 proving pipeline architecture: The pipeline consists of multiple phases per partition: pre-staging (VRAM allocation and data upload), GPU kernels (NTT, MSM, batch additions, tail MSMs), and CPU-side preprocessing (prep_msm, b_g2_msm). The gpu_prove() C++ function encompasses all of these, and the TIMELINE instrumentation measures the entire call.
  2. The instrumentation system: The daemon emits TIMELINE log entries with GPU_START and GPU_END markers, each containing a timestamp and gpu_ms field. The C++ code also emits gpu_total_ms, prep_msm_ms, b_g2_msm_ms, and total_ps_ms timing lines. Understanding the relationship between these metrics is crucial.
  3. The benchmark configuration: c is concurrency (number of proofs in flight simultaneously), j is total proofs. gw=1 means one GPU worker. The daemon runs persistently, serving proofs submitted by the benchmark client.
  4. The synthesis process: Each proof requires synthesis (witness generation) which is CPU-intensive and memory-intensive, consuming ~7-8 GiB per proof. With 10 synthesis workers running alongside the GPU pipeline, they compete for memory bandwidth.
  5. The hardware: The system has an 8-channel DDR5 memory configuration, which provides a finite memory bandwidth budget shared across all CPU cores.

Output Knowledge Created

This message produced several critical pieces of knowledge:

  1. The exact timing breakdown per partition: GPU kernels take ~1.8s, prep_msm takes ~1.9s, b_g2_msm takes ~0.48s, pre-staging takes ~18ms. The total per-partition wall time inside gpu_prove() is approximately 3.7s (1.8s GPU + 1.9s CPU overlap + gap).
  2. The bottleneck has shifted from GPU to CPU: The GPU kernels complete in 1.8s, but the CPU-side prep_msm takes 1.9s and b_g2_msm takes 0.48s. Since these CPU operations run on a separate thread that must be joined before the function returns, the CPU path is now the critical path.
  3. The GPU is idle ~600ms per partition: The gap between GPU_END and GPU_START averages 305ms, but this includes the b_g2_msm and epilogue time. The GPU finishes its kernels and then waits for the CPU thread to complete prep_msm + b_g2_msm before it can start the next partition.
  4. Memory bandwidth contention is real: The earlier c=30 run showed prep_msm inflating to 10.6s (6× normal) and b_g2_msm to 4.5s (12× normal) when 20 concurrent proofs were synthesizing. Even at c=15, the synthesis workers are competing with the CPU MSM operations for DDR5 bandwidth.
  5. The pre-staging setup is negligible: At only 18ms, the VRAM allocation and data upload is not a significant contributor to the per-partition time.
  6. TIMELINE utilization is 92.2%: This is a misleadingly high number because TIMELINE gpu_ms includes CPU wait time. The actual GPU kernel utilization is much lower—the GPU is actively computing only ~1.8s out of every ~3.7s partition window.

The Thinking Process: What the Message Reveals

The assistant's thinking is visible in several aspects of this message:

Hypothesis-driven analysis: The assistant doesn't just collect data randomly. The AWK script is carefully designed to answer specific hypotheses. The gap calculation (GPU_END→GPU_START) tests whether the GPU is waiting for something between partitions. The separate extraction of prep_msm and b_g2_msm tests whether CPU-side operations are the bottleneck. The pre-staging extraction tests whether VRAM allocation overhead is significant.

System-level understanding: The assistant understands the relationship between the TIMELINE instrumentation (which wraps the entire gpu_prove() call including CPU thread joins) and the C++ kernel timing (which measures only GPU kernel execution). The 92.2% TIMELINE utilization is recognized as misleading—it includes CPU wait time. The assistant knows that the true GPU utilization is the ratio of C++ kernel time to the partition wall time.

Cross-referencing multiple data sources: The assistant doesn't rely on a single metric. It cross-references TIMELINE data, C++ kernel timing, prep_msm timing, b_g2_msm timing, and pre-staging timing to build a complete picture. The consistency between the c=15 and c=20 benchmarks (41.3s vs 41.6s) confirms the result is reproducible.

The "fewer synths competing" insight: The observation that the last two proofs completed faster because fewer synthesis workers were active is a crucial diagnostic clue. It points directly to CPU resource contention as the bottleneck, not GPU compute capacity.

Recognition of the bottleneck shift: The assistant had been optimizing GPU-side operations (PCIe transfers, kernel execution) for multiple phases. This message represents the moment of recognizing that those optimizations have succeeded to the point where the bottleneck has moved to a completely different subsystem—CPU memory bandwidth.

Assumptions and Potential Pitfalls

The analysis makes several assumptions:

  1. The TIMELINE instrumentation is accurate: The assistant assumes that the GPU_START and GPU_END markers correctly bracket the gpu_prove() call and that the gpu_ms field accurately represents the wall time of that call. If there are instrumentation overheads or race conditions in the logging, the numbers could be off.
  2. The C++ timing metrics are independent: The assistant treats gpu_total_ms, prep_msm_ms, and b_g2_msm_ms as separate, non-overlapping measurements. In reality, prep_msm runs on a separate thread that overlaps with GPU kernel execution. The prep_msm time includes time that overlaps with GPU compute, so the "CPU critical path" analysis may overstate the problem slightly.
  3. Linear scalability of queue times: The assistant notes that queue times "grow linearly" because all 15 proofs are submitted at once. This assumes the daemon processes proofs in FIFO order with no scheduling overhead. If the daemon reorders proofs or if there are priority effects, the queue time distribution could be different.
  4. The hardware configuration is stable: The analysis assumes that memory bandwidth contention is the primary cause of prep_msm/b_g2_msm inflation, not other factors like CPU frequency scaling (thermal throttling), NUMA effects, or OS scheduling interference.
  5. The benchmark is representative: The assistant assumes that the c1.json input file (the C1 output from a 32GiB sector) is representative of production workloads. If production sectors have different characteristics, the timing breakdown could shift.

Mistakes or Incorrect Assumptions

While the analysis is sound, there are subtle points worth examining:

The TIMELINE utilization metric is misleading: The assistant reports 92.2% TIMELINE utilization, which sounds high. But as the assistant implicitly understands, this metric includes CPU wait time inside gpu_prove(). The actual GPU compute utilization is closer to 1.8s / 3.7s ≈ 49%. The 92.2% figure could give a false impression of efficiency if taken at face value.

The gap calculation may miss context: The average gap of 305ms between GPU_END and GPU_START includes the time to release the mutex, unregister host memory, serialize the Rust-side proof, and acquire the mutex for the next partition. Some of this is unavoidable serialization that no amount of overlap can eliminate.

The pre-staging time may be underestimated: The 18ms pre-staging time is measured when VRAM is already freed. In a scenario where VRAM is fragmented or the memory pool needs trimming, pre-staging could take longer. The Phase 10 two-lock design would later encounter this exact issue when the first worker's VRAM allocation was still live, causing OOM for subsequent workers.

The Significance: Foundation for Phase 10

This message is the diagnostic foundation upon which Phase 10 was built. The key insight—that CPU memory bandwidth contention is the bottleneck, and that the GPU is idle for ~600ms per partition waiting for CPU operations—led directly to the two-lock design proposed in c2-optimization-proposal-10.md.

The Phase 10 design aimed to increase gpu_workers_per_device from 1 to 3, using a mem_mtx for VRAM allocation and a compute_mtx for kernel execution. The idea was to overlap one worker's CPU-side memory management with another worker's GPU kernel execution, hiding the CPU overhead. The expected per-partition wall time reduction was from ~3.7s to ~1.8–2.0s, with a potential 30–38% throughput improvement.

However, as the subsequent chunk reveals, this design ran into a fundamental CUDA device constraint: cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations that serialize across all workers on the same GPU. The two-lock abstraction broke down because memory management operations on a single CUDA device cannot be fully isolated from compute operations. This led to OOM failures and performance regressions, ultimately requiring a redesign.

Conclusion

Message 2554 represents a pivotal moment in the optimization journey. It is the moment when the assistant stopped optimizing GPU-side operations and started analyzing the CPU-side bottleneck. The message demonstrates the essence of performance engineering: measure precisely, cross-reference multiple data sources, form hypotheses, and let the data guide the next optimization target. The AWK-based timing analysis is a testament to the power of simple, portable tools applied with deep system knowledge. And the recognition that the bottleneck had shifted from PCIe bandwidth to CPU memory bandwidth—a completely different subsystem—is a classic example of how performance optimization is an iterative process of finding and removing the current bottleneck, only to discover the next one waiting underneath.