The Synthesis Cap: When Control Theory Meets DDR5 Reality

In the middle of a marathon session tuning a PI-controlled dispatch pacer for a GPU-accelerated zero-knowledge proof pipeline, a single user message cuts through the control-theory abstraction and lands squarely on hardware reality:

Do we have a simple hard cap on parellel synthesis? We should just set it to default 18, and keep configurable, anything more will probably choke on ddr5 systems

This message, sent at index 3657 of the conversation, is deceptively simple. On its surface, it is a question about whether a configuration parameter exists. But beneath that lies a significant architectural insight: the recognition that no amount of PI controller tuning can overcome a fundamental hardware bottleneck, and that the correct solution is not a smarter algorithm but a simpler constraint.

The Context: A Deep Dive into GPU Pipeline Tuning

To understand why this message matters, one must appreciate the journey that preceded it. The conversation up to this point had been consumed with an intricate control-theory problem. The system under development is a zero-knowledge proof engine (cuzk) that uses a pipelined architecture: proof partitions are synthesized on CPU cores, then dispatched to GPUs for the proving step. The challenge is keeping the GPU fed without overwhelming the system's memory budget.

The assistant had implemented a PI (proportional-integral) controller that dynamically adjusts the dispatch rate based on GPU queue depth. This controller went through multiple iterations: normalizing error by target, tuning proportional and integral gains, adding asymmetric integral clamping to prevent pipeline-draining backoff, implementing re-bootstrap detection for when the pipeline drains between proof batches, and adding a synthesis throughput cap that ultimately proved counterproductive (it created a vicious cycle where slow dispatch reduced synthesis throughput, which tightened the cap further).

The most recent deployment, pitune3, had adjusted the integral gain from 0.02 down to 0.001 while expanding the integral cap from ±2.0 to +100/-20. This was deployed and running on a remote machine with 755 GB of memory and an NVIDIA GPU. The assistant had just reported the new parameters and was waiting for feedback.

Why This Message Was Written

The user's question emerges from a specific observation about system behavior. The PI controller was managing dispatch timing, but the synthesis step itself — the CPU-side work of constructing proof partitions — was running unchecked. The synth_worker_count was computed dynamically as max_partitions_in_budget.min(64).max(4), which on the current machine with ~400 GiB budget and ~9 GiB per partition yielded 28 concurrent synthesis workers. All 28 could run synthesis simultaneously.

The user recognized that this was problematic. DDR5 memory bandwidth is a shared resource, and when too many CPU cores are simultaneously performing synthesis — which involves heavy memory allocation and data movement — they saturate the memory bus. Each individual synthesis slows down, and total throughput collapses. This is not a problem the PI controller can solve, because the PI controller manages dispatch rate, not synthesis concurrency. The synthesis workers are upstream of the GPU dispatch; they produce the work items that the dispatcher feeds to the GPU. If synthesis itself is bottlenecked by memory bandwidth contention, the GPU starves regardless of how well-tuned the dispatch pacer is.

The message also reflects a pragmatic engineering philosophy. Rather than attempting to build a sophisticated model of memory bandwidth contention and dynamically adjusting synthesis concurrency, the user advocates for a simple hard cap. "We should just set it to default 18, and keep configurable" — the emphasis is on simplicity and configurability. The number 18 is not arbitrary; it reflects the user's understanding that on a typical 64-core DDR5 system, running more than 18 concurrent syntheses causes memory bandwidth saturation. This is experiential knowledge, derived from observing the system's behavior under load.

Assumptions Embedded in the Message

The message makes several assumptions worth examining. First, it assumes that the current system does not already have such a cap — the question "Do we have a simple hard cap on parellel synthesis?" is genuinely exploratory. The user is checking whether this feature exists before requesting its implementation.

Second, it assumes that a hard cap is the correct approach. This is a design decision that prioritizes predictability over optimality. A hard cap is simple, easy to reason about, and guarantees that synthesis concurrency never exceeds the specified limit. But it is also inflexible: it cannot adapt to variations in partition complexity, memory pressure, or system load. An alternative approach might use a feedback loop similar to the PI controller, dynamically adjusting synthesis concurrency based on measured throughput or memory bandwidth utilization. The user implicitly rejects this complexity in favor of a fixed limit.

Third, the message assumes that 18 is a reasonable default. This is a domain-specific judgment that depends on the hardware configuration (number of CPU cores, memory channels, DDR5 speed) and the characteristics of the synthesis workload (memory footprint per partition, allocation patterns, compute intensity). The user's confidence in this number suggests prior experience with similar systems.

Fourth, the message contains a typo — "parellel" instead of "parallel" — which the user corrects in a follow-up message (msg 3660: "* parallel running"). This is a minor detail but worth noting for accuracy.

The Assistant's Response: From Question to Implementation

The assistant's response to this message reveals how a simple question can trigger a cascade of implementation work. The assistant first confirms the current state of affairs by grepping for synth_worker_count and synthesis_concurrency, discovering that the existing synthesis_concurrency field is for the monolithic (non-pipeline) path and defaults to 1. A new field is needed for the pipeline path.

The assistant then reads the config file to understand the existing structure, adds a max_parallel_synthesis field with a default of 18, implements the default function, updates the Default impl, and modifies the pipeline code to cap synth_worker_count at this new limit. The change is minimal: the line let synth_worker_count = max_partitions_in_budget.min(64).max(4) becomes let synth_worker_count = max_partitions_in_budget.min(max_parallel).max(2), where max_parallel comes from the config. The lower bound is also reduced from 4 to 2, giving more flexibility at the low end.

The entire implementation — config changes, engine changes, compilation check, and commit — takes only a handful of messages. The assistant commits the change as 6acd3a27 with the message: "cuzk: add max_parallel_synthesis config (default 18). Hard cap on concurrent synthesis workers in the pipeline path. Too many concurrent syntheses causes CPU contention and DDR5 memory bandwidth saturation, making each synthesis slower and reducing overall throughput."

Input and Output Knowledge

To understand this message, one needs knowledge of: the cuzk proving pipeline architecture (synthesis → GPU dispatch → proving), the concept of memory bandwidth contention on DDR5 systems, the existing synth_worker_count computation logic, the distinction between the monolithic and pipeline code paths, and the configuration system used by the application.

The message creates several outputs: a new configuration field (max_parallel_synthesis) with documentation, a modified worker count computation that respects the cap, a startup log line that reports the capped count, a git commit with a descriptive message, and — most importantly — a behavioral change in the running system. After this change, the pipeline path will never run more than 18 concurrent syntheses by default, preventing the memory bandwidth saturation that was silently degrading performance.

The Broader Significance

This message represents a transition point in the conversation. The preceding messages were consumed with fine-tuning the PI controller — adjusting ki from 0.02 to 0.001, expanding integral caps from ±2.0 to +100/-20, normalizing error terms, and analyzing saturation dynamics. These are sophisticated control-theory adjustments that operate at the level of dispatch timing. But the user's message shifts the focus to a different layer of the system: the synthesis pipeline itself.

The insight is that the PI controller, no matter how well-tuned, can only manage the rate at which completed syntheses are dispatched to the GPU. It cannot prevent the synthesis workers from competing with each other for memory bandwidth. The synthesis concurrency cap addresses a bottleneck upstream of the PI controller's jurisdiction. It is a reminder that in complex systems, performance problems often have multiple causes at different levels of abstraction, and solving them requires interventions at each level.

The message also illustrates a pattern common in engineering conversations: a seemingly simple question ("Do we have X?") that encodes a deeper design insight. The user is not just asking about the existence of a configuration parameter; they are proposing a architectural change based on observed hardware behavior. The assistant recognizes this and implements the change immediately, without debate or alternative proposals. Both participants understand that the cap is necessary and that 18 is a reasonable starting point.

In the broader arc of the session, this message marks the end of the PI tuning phase and the beginning of the production deployment phase. After committing the synthesis cap, the assistant shifts to building the latest cuzk using the main Dockerfile and configuring default memory budgets for vast.ai environments. The synthesis cap is the last algorithmic change before the focus moves to infrastructure and deployment.