Benchmarking the Channel Capacity Fix: Validating Memory Backpressure in Phase 12's Split GPU Proving API

Introduction

In the high-stakes world of Filecoin proof generation, every millisecond counts and every gigabyte of memory must be accounted for. Message <msg id=3155> captures a pivotal moment in the optimization journey of the cuzk SNARK proving engine: the first benchmark run after implementing a critical channel capacity auto-scaling fix for Phase 12's split GPU proving API. This message is not merely a routine benchmark execution — it is the empirical validation of a carefully designed memory backpressure mechanism intended to solve an out-of-memory (OOM) failure that had prevented the system from operating at its optimal configuration.

The message shows the assistant executing cuzk-bench with 15 proofs at concurrency 15, using the porep (Proof-of-Replication) circuit type, against a daemon configured with partition_workers=10, gpu_workers_per_device=2, and gpu_threads=32. The output captures the first five completions, with per-proof times ranging from 67.1s to 215.1s (cumulative), revealing a throughput of approximately 38.8 seconds per proof. This single command represents the culmination of a multi-step debugging and optimization process that had consumed the preceding several rounds of the conversation.

The Context: Phase 12's Split API and the Memory Crisis

To understand why this message matters, one must appreciate the architectural shift that Phase 12 introduced. The original proving pipeline was monolithic: a single function call would perform CPU synthesis (building circuits from vanilla proofs), then hand the result to the GPU for the heavy number-theoretic transforms and multi-scalar multiplications. The problem was that the GPU's b_g2_msm operation — a multi-scalar multiplication on the G2 curve — ran asynchronously after the main GPU lock was released, meaning the GPU worker could not immediately loop back for the next job. The Phase 12 split API (implemented in the preceding segment, Segment 30) decoupled this by splitting proving into prove_start (which returns after releasing the GPU lock, with b_g2_msm still running in a background stream) and finish_pending_proof (which joins the background work). This allowed GPU workers to pipeline: worker A could start proving partition N+1 while worker B's b_g2_msm for partition N was still completing.

However, the split API introduced a memory pressure crisis. With partition_workers=10 (or higher), up to 10 CPU threads would synthesize partitions concurrently. Each synthesized partition held approximately 16 GiB of evaluation vectors (a/b/c), plus auxiliary data. When synthesis completed faster than the GPU could consume — which it did, since synthesis took ~29 seconds per partition while GPU proving took only ~5-6 seconds — the completed outputs would pile up. The synthesis→GPU channel had a hardcoded capacity of 1, meaning 9 out of 10 completed syntheses would block on synth_tx.send(), each holding their full 16 GiB allocation while waiting. At partition_workers=12, this caused the process to OOM at 668 GiB peak RSS, far exceeding the available memory.

The Channel Capacity Fix

The assistant's analysis (visible in the reasoning of <msg id=3142> through <msg id=3144>) identified the root cause: the channel capacity was too small. With capacity 1, only one completed synthesis could be buffered; the rest blocked on send() while holding memory. The fix was conceptually simple but required careful reasoning about the interaction between the partition semaphore (which caps concurrent synthesis) and the bounded channel (which provides backpressure against the GPU).

The assistant implemented the fix in <msg id=3144> by auto-scaling the channel capacity to max(synthesis_lookahead, partition_workers) instead of the hardcoded 1. The reasoning was that when partition_workers > 0, up to partition_workers partitions could complete synthesis concurrently and then compete to send on the channel. Sizing the channel to partition_workers would allow all of them to enqueue without blocking, while the semaphore would still bound total in-flight work. The log line confirms this: effective_lookahead=10 (matching partition_workers=10).

The Benchmark Command: What It Measures

The command executed in <msg id=3155> is:

FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 15 --concurrency 15

This runs the cuzk-bench tool in batch mode, generating 15 Proof-of-Replication proofs with a concurrency of 15 (meaning all 15 proofs are submitted to the daemon simultaneously, without staggering). The --c1 flag points to a pre-computed C1 output file — the first stage of the Filecoin proving pipeline, which produces the circuit inputs. The FIL_PROOFS_PARAMETER_CACHE environment variable points to the directory containing the Structured Reference String (SRS) parameters, which are the elliptic curve parameters needed for the Groth16 proving system.

The benchmark measures two key metrics per proof: prove (the actual proving time in milliseconds) and queue (the time the proof spent waiting in the daemon's job queue before processing began). The cumulative completion times (67.1s, 106.5s, 144.1s, etc.) show that proofs complete at roughly 38-second intervals, which is the steady-state throughput of the pipeline.

Interpreting the Output

The output shows the first five completions. The first proof completes in 67.1 seconds (65.4s prove + 0.3s queue). The second completes at 106.5s (39.4s later), the third at 144.1s (37.6s later), the fourth at 179.6s (35.5s later), and the fifth at 215.1s (35.5s later). The inter-completion intervals are stabilizing around 35-39 seconds, suggesting the pipeline has reached steady state by the third or fourth proof.

The per-proof prove times (65.4s, 71.9s, 78.6s, 71.3s, ...) show some variance — the third proof took 78.6s while the fourth took only 71.3s. This variance is expected in a pipelined system where partitions from different proofs interleave. The queue times grow steadily (295ms, 741ms, 1171ms, 1619ms), indicating that proofs submitted later wait longer as the pipeline fills — a normal characteristic of a bounded-throughput system.

Importantly, the benchmark succeeds — it does not OOM. This is the primary validation: the channel capacity fix has eliminated the memory pressure that caused the pw=10 configuration to crash in earlier tests. The RSS monitor (started in <msg id=3154>) would later confirm a peak of approximately 390 GiB, well within the system's 755 GiB budget.

Assumptions and Their Implications

The assistant made several assumptions in this benchmark run. First, it assumed that partition_workers=10 with gpu_workers_per_device=2 and gpu_threads=32 was the correct configuration to validate — this was the known-good configuration from Phase 11, and the fix was expected to make it work reliably without regression. Second, it assumed that running 15 proofs at concurrency 15 would provide sufficient data to assess throughput (roughly 10 minutes of wall-clock time). Third, it assumed that the channel capacity auto-scaling would not introduce a throughput regression — that the larger channel buffer would not cause memory fragmentation or allocator pressure that could slow things down.

A subtle but important assumption was that the channel capacity should match partition_workers exactly. The assistant reasoned: "if up to pw partitions can complete synthesis concurrently, the channel should have room for all of them." This assumption would later be questioned in subsequent messages (starting at <msg id=3156>), when the assistant discovered that the fix produced a ~5% throughput regression (38.8s/proof vs the Phase 12 baseline of 37.1s/proof). The larger channel allowed more synthesis outputs to accumulate in memory, potentially causing allocator fragmentation that slowed allocation and deallocation patterns.

The Thinking Process Visible in the Message

The message itself is a tool call — a bash command execution — and does not contain explicit reasoning. However, the reasoning is visible in the sequence of messages leading up to it. The assistant had just completed the channel capacity edit in <msg id=3144>, verified compilation in <msg id=3146> and <msg id=3147>, started the daemon in <msg id=3150>, confirmed the effective lookahead in <msg id=3151>, waited for SRS preload in <msg id=3152>-<msg id=3153>, and started an RSS monitor in <msg id=3154>. Each step reflects a deliberate methodology: implement, build, deploy, instrument, then benchmark.

The choice to run the benchmark immediately after the daemon became ready (rather than waiting for a separate test run) reflects an assumption that the fix was correct and the primary question was how much memory it would save and whether throughput would be affected. The assistant was not expecting a regression — it was validating that the fix worked without breaking anything.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, the Filecoin proving pipeline: the distinction between C1 (circuit synthesis) and C2 (proof generation), the role of the Groth16 proving system, and the concept of partition-level parallelism where a single proof is split into multiple partitions that are synthesized independently and then aggregated by the GPU. Second, the CUDA programming model: GPU streams, asynchronous operations, and the b_g2_msm operation that motivated the split API. Third, concurrent programming patterns in Rust: bounded channels for backpressure, semaphores for resource limiting, and the spawn_blocking pattern for CPU-intensive work in an async runtime. Fourth, the specific history of this optimization project: Phase 9 (PCIe transfer optimization), Phase 10 (abandoned two-lock design), Phase 11 (memory-bandwidth-aware scheduling), and Phase 12 (split API).

Output Knowledge Created

This message produced concrete empirical data: the benchmark output showing completion times, prove durations, and queue wait times for the first five of fifteen proofs. This data would be used in the immediately following messages to assess whether the fix was successful. The assistant would go on to compute the aggregate throughput (581.7s total / 15 = 38.8s/proof), compare it to the Phase 12 baseline (37.1s/proof), and investigate the ~5% regression through buffer counter analysis and GPU timing logs.

More broadly, this message represents a critical validation point in the optimization cycle. The channel capacity fix was not the final solution — the assistant would later discover that the regression required a more nuanced approach involving queue-depth semaphores and smaller channel buffers — but it was a necessary step in the learning process. The benchmark proved that the fix eliminated the OOM condition, even if it introduced a small throughput penalty. In the iterative optimization methodology employed throughout this project, a fix that trades 5% throughput for OOM stability is a net positive — it opens the door to further tuning.

Conclusion

Message <msg id=3155> is a deceptively simple benchmark command that encapsulates hours of analysis, debugging, and careful engineering. It represents the moment when a theoretical fix meets empirical reality — when the assistant's model of memory backpressure is tested against actual hardware with real memory allocators and real GPU scheduling. The benchmark succeeded in its primary goal (no OOM), but revealed a secondary concern (throughput regression) that would drive the next iteration of optimization. In the broader narrative of the cuzk optimization project, this message is the fulcrum between the memory-pressure crisis of Phase 12 and the more refined queue-depth control that would follow.