A Pivotal Design Decision: Adding a Hard Cap on Parallel Synthesis

The Message

The existing synthesis_concurrency is for the monolithic path. I'll add a new max_parallel_synthesis field for the pipeline path, default 18. Then cap synth_worker_count with it.

This single sentence, followed by a successful file edit, represents a critical design decision in the ongoing development of the CuZK GPU proving pipeline. At first glance, it appears to be a straightforward configuration change — add a field, set a default, apply a cap. But beneath the surface lies a rich tapestry of reasoning about system architecture, resource contention, and the subtle interplay between synthesis throughput and GPU utilization.

Context and Motivation

The message was written in response to a direct user request at [msg 3657]: "Do we have a simple hard cap on parellel synthesis? We should just set it to default 18, and keep configurable, anything more will probably choke on ddr5 systems." The user corrected their typo at [msg 3660] ("parallel running"), but the intent was clear: the system needed a mechanism to prevent too many proof synthesis tasks from executing simultaneously.

This request did not emerge from abstract design concerns. It arose from a concrete operational reality. The CuZK pipeline had been through an intensive tuning cycle spanning multiple segments (segments 21–26 of the conversation). The team had already implemented a PI-controlled dispatch pacer, a zero-copy pinned memory pool, re-bootstrap detection, and asymmetric integral clamping — all in pursuit of eliminating GPU idle time. But as the pipeline was pushed harder, a new bottleneck emerged: CPU and DDR5 memory bandwidth contention from too many concurrent syntheses.

The user's observation — "anything more will probably choke on ddr5 systems" — reflected a practical understanding of the hardware. DDR5 memory bandwidth, while fast, is a shared resource. When too many synthesis tasks run in parallel, each consuming significant memory bandwidth for constraint evaluation, witness generation, and proof construction, they collectively saturate the memory bus. The result is that all syntheses slow down, the GPU starves for work, and the carefully tuned PI controller cannot compensate because the bottleneck has shifted from dispatch timing to memory bandwidth.

The Reasoning Process

The assistant's reasoning, visible in the preceding message at [msg 3661], reveals a careful analysis of the problem. The first step was understanding the current state: synth_worker_count was computed as max_partitions_in_budget.min(64).max(4). With a 400 GiB memory budget and approximately 9 GiB per partition, this yielded 28 concurrent synthesis workers. All 28 could run synthesis simultaneously, and on a DDR5 system, that was too many.

The assistant considered two approaches. One option was introducing a semaphore or similar concurrency-limiting mechanism that would sit between the dispatcher and the synthesis workers, throttling how many could run at once. But the assistant recognized a simpler solution: since each worker pulls from a bounded channel and runs one synthesis at a time, capping synth_worker_count directly would naturally enforce the concurrency limit. Fewer workers means fewer concurrent syntheses, with no additional synchronization overhead.

The critical design insight was distinguishing between the monolithic path and the pipeline path. The existing synthesis_concurrency configuration field (defaulting to 1) was designed for the monolithic proving path — the older, non-pipelined approach. The new pipeline path needed its own configuration field because the two paths have fundamentally different concurrency characteristics. In the monolithic path, concurrency is about how many independent proofs to run in parallel. In the pipeline path, concurrency is about how many synthesis tasks to overlap with GPU work — a much more delicate balance because synthesis and GPU compete for memory bandwidth.

Input Knowledge Required

To understand this message, one needs knowledge of several interconnected domains. First, the CuZK proving engine architecture: the distinction between monolithic and pipelined proving paths, the role of synth_worker_count in determining how many synthesis tasks run concurrently, and the channel-based communication between the dispatcher and workers. Second, the hardware constraints of GPU-accelerated proving: the memory bandwidth competition between CPU-side synthesis and GPU-side computation, the characteristics of DDR5 memory subsystems, and the practical limits of concurrent memory-intensive workloads. Third, the prior tuning work: the PI controller, the pinned memory pool, the re-bootstrap logic — all of which interact with synthesis concurrency because they determine how quickly work flows through the pipeline.

Output Knowledge Created

This message created both a conceptual artifact and a concrete implementation. Conceptually, it established the principle that synthesis concurrency in the pipeline path should be independently configurable from the monolithic path, with a hardware-aware default. Practically, it initiated the implementation of the max_parallel_synthesis field in the pipeline configuration, with a default value of 18. The subsequent messages ([msg 3664] through [msg 3668]) completed the wiring: adding the default function, updating the Default impl, capping synth_worker_count with min(max_parallel, ...), and verifying compilation.

The choice of 18 as the default was not arbitrary. It reflects a heuristic about DDR5 memory bandwidth: on typical server systems, 18 concurrent synthesis tasks approach but do not exceed the memory bandwidth ceiling. This is high enough to keep the GPU fed (especially with the PI pacer managing dispatch timing) but low enough to avoid the catastrophic collapse where memory contention slows every synthesis to a crawl.

Assumptions and Potential Pitfalls

The assistant made several assumptions worth examining. The assumption that capping synth_worker_count is equivalent to capping concurrency relies on the channel-based worker model — each worker runs one synthesis at a time. This is correct for the current architecture, but it means the cap is a static limit rather than a dynamic one. If synthesis times vary significantly, a static worker count may not be optimal: during easy proofs, more workers could run without contention; during hard proofs, even 18 might be too many.

The assistant also assumed that 18 is a reasonable default for DDR5 systems. This is a heuristic that may need adjustment for different hardware configurations — systems with more memory channels, higher bandwidth, or different CPU architectures might benefit from different values. The configurability mitigates this risk, but the default becomes a hidden assumption for anyone who doesn't read the configuration documentation.

Another assumption is that the monolithic path's synthesis_concurrency and the pipeline path's max_parallel_synthesis should remain independent. This creates a maintenance burden: two configuration knobs that control similar behavior in different paths. A future refactoring might unify them, but for now the separation reflects the different operational contexts.

The Broader Significance

This message is a microcosm of the entire tuning effort. The CuZK pipeline had been optimized for GPU utilization — minimizing idle time, balancing dispatch rates, managing memory transfers. But optimization is never complete; it merely shifts the bottleneck. Once the GPU was well-fed, the bottleneck moved to CPU-side memory bandwidth. The hard cap on parallel synthesis is a recognition that the system must be constrained not just by what the GPU can consume, but by what the memory subsystem can sustain.

The decision also reflects a philosophy of simplicity. Rather than building a sophisticated dynamic concurrency controller (which would have been tempting after the PI pacer work), the assistant chose a static cap with a sensible default. This is a "worse is better" approach: a simple limit that works well enough, with configurability for edge cases, rather than a complex adaptive system that might introduce new failure modes.

In the end, this message is about knowing when to stop optimizing and start constraining. The PI pacer, the pinned memory pool, the re-bootstrap logic — all of these made the pipeline faster. But the hard cap on synthesis concurrency is what keeps it from falling over.