The Synthesis Concurrency Cap: A Simple Fix for a Subtle Bottleneck

In the course of an intensive GPU pipeline optimization session, the assistant delivered a brief but consequential message confirming a commit that added a hard cap on parallel synthesis workers. The message, message 3670, reads in full:

Committed as 6acd3a27. The pipeline path now caps at 18 concurrent syntheses by default (was 28 on this machine). Configurable via max_parallel_synthesis in the pipeline config section. Want me to build and deploy this + the ki tuning?

At first glance, this appears to be a routine status update — a commit confirmation followed by a question about next steps. But this short message sits at the intersection of several converging threads of work: days of PI controller tuning, a deep investigation into GPU underutilization, and a growing understanding of the system's memory bandwidth constraints. To appreciate what this message represents, one must trace the reasoning that led to it.

The Context: A Pipeline Under Tuning

The preceding hours had been spent iterating on a PI (proportional-integral) dispatch pacer that controlled how quickly synthesized proofs were fed to the GPU. The pacer had gone through multiple versions: a semaphore-based reactive throttle, a PI-controlled pacer with EMA feed-forward, a re-bootstrap mechanism to handle pipeline drains between batches, and a series of tuning adjustments to the integral gain (ki), proportional gain (kp), and clamp limits. Each deployment — pitune1, pitune2, pitune3, pitune4 — had revealed new failure modes: integral saturation that drained the entire pipeline, re-bootstrap spam that triggered dozens of unnecessary warmup cycles, and a vicious cycle where the synthesis throughput cap slowed dispatch which reduced synthesis concurrency which tightened the cap further.

By message 3657, the user had been observing the pitune4 deployment and noticed a pattern: when the system hit the memory ceiling, the integral went deeply negative and the pipeline fully drained before resuming synthesis. The assistant had been tuning the PI controller to address this — lowering ki from 0.02 to 0.001, raising the integral cap asymmetrically (positive 100, negative -20), and tightening the rate multiplier clamp. But the user's observation in message 3657 shifted the focus from the PI controller to a different concern entirely:

Do we have a simple hard cap on parellel synthesis? We should just set it to default 18, and keep configurable, anything more will probably choke on ddr5 systems

This was a crucial insight. The PI controller was managing the rate at which syntheses were dispatched to the GPU, but it didn't control how many syntheses could run simultaneously. The pipeline architecture worked by spawning a pool of synthesis workers — each worker pulled a partition from a work queue, ran the synthesis computation (a CPU-intensive task involving polynomial arithmetic over large fields), and handed the result to the GPU worker. The number of workers was the concurrency limit: if there were 28 workers, up to 28 syntheses could be in flight at once.

The Problem with Unbounded Concurrency

The assistant's reasoning, visible in message 3661, reveals the analytical process:

Currently synth_worker_count is set to max_partitions_in_budget.min(64).max(4) which with 400 GiB budget and ~9 GiB per partition = 28 workers. All 28 can run synthesis simultaneously.

The budget-based calculation was designed to maximize memory utilization — if you have 400 GiB of memory and each partition needs ~9 GiB, you should be able to run 28 syntheses in parallel. But this ignored a critical resource: DDR5 memory bandwidth. Synthesis is not just memory-capacity-bound; it is memory-bandwidth-bound. Each synthesis worker reads and writes large vectors of field elements, and with DDR5 systems, the memory bus can become saturated long before the memory capacity is exhausted. When too many workers compete for bandwidth, each individual synthesis slows down, and total throughput can actually decrease — a classic concurrency thrashing scenario.

The user's intuition was that 18 concurrent syntheses was a reasonable ceiling for a 64-core DDR5 system. This number likely came from empirical observation: at 28 workers, the system was probably experiencing bandwidth contention where individual syntheses took longer, reducing the effective dispatch rate to the GPU and creating a self-reinforcing slowdown.

The Implementation: Elegant Simplicity

The assistant's approach to implementing the cap was remarkably straightforward. Rather than adding a semaphore or a separate throttling mechanism, the assistant recognized that the simplest fix was to cap synth_worker_count directly. The reasoning, captured in message 3661:

The simplest fix is to just cap synth_worker_count directly at 18 — if there are only 18 workers pulling from the bounded channel, then at most 18 syntheses can run concurrently, which naturally enforces the limit without needing a separate semaphore.

This is a classic systems design insight: when the worker pool size directly controls concurrency, limiting the pool size is the most natural and least invasive way to cap concurrency. No new synchronization primitives, no complex state management — just change the number from max_partitions_in_budget.min(64).max(4) to max_partitions_in_budget.min(max_parallel).max(2).

The assistant then implemented the change across two files:

1. config.rs: Added a new max_parallel_synthesis field to PipelineConfig with a default value of 18, along with a serde deserialization default function. The field was documented as a hard cap on concurrent synthesis workers to prevent CPU contention and DDR5 memory bandwidth saturation.
2. engine.rs: Changed the synth_worker_count calculation from max_partitions_in_budget.min(64).max(4) to max_partitions_in_budget.min(max_parallel).max(2), where max_parallel comes from the new config field. The minimum was lowered from 4 to 2 to allow very constrained configurations. The change was minimal — 37 insertions and 10 deletions across two files — and compiled cleanly with only pre-existing warnings. The assistant verified that the startup log line at line 1843 would now show 18 instead of 28, providing immediate visibility into the new cap.

The Commit Message: Capturing the Rationale

The commit message written by the assistant (message 3669) is worth examining for the clarity of its reasoning:

Hard cap on concurrent synthesis workers in the pipeline path. Too many concurrent syntheses causes CPU contention and DDR5 memory bandwidth saturation, making each synthesis slower and reducing overall throughput. 18 is a good default for 64-core DDR5 systems.

This captures the key insight: the problem is not just about memory capacity (which the budget system already handles) but about memory bandwidth. DDR5 systems have high bandwidth, but it's still a finite shared resource. When 28 workers all try to read and write their working sets simultaneously, the memory bus becomes the bottleneck, and each worker's effective throughput drops. The system as a whole may produce fewer completed syntheses per second than it would with 18 workers, even though the memory budget could theoretically support 28.

The commit also documents the config semantics: max_parallel_synthesis defaults to 18, and setting it to 0 means "use default" (though the implementation actually just uses the default value directly — 0 would cap at 0, which would be nonsensical, so the config parsing likely handles this).

Assumptions and Limitations

The implementation makes several assumptions worth examining:

18 is a good default for 64-core DDR5 systems. This is an empirical claim, not a theoretical one. The user's experience with the specific hardware (likely an AMD EPYC or Intel Xeon system with DDR5 memory) suggested that 18 workers produced better throughput than 28. But this number is highly hardware-dependent: a system with more memory channels, higher clock speeds, or different CPU architecture might benefit from a different cap. The configurable nature of the parameter acknowledges this — the default is a starting point, not a universal optimum.

The worker count directly controls concurrency. This assumption is correct for the current architecture: each worker is a long-lived task that pulls one item from the channel, runs synthesis to completion, and then pulls the next. There is no work-stealing or sub-delegation within a worker. So capping the number of workers is equivalent to capping the number of concurrent syntheses.

The cap should apply to the pipeline path only. The assistant noted that the existing synthesis_concurrency config field was for the monolithic (non-pipeline) path. The new max_parallel_synthesis field applies only to the pipeline path, which is the path being optimized. This is correct separation of concerns but means the monolithic path remains unaffected.

The Deeper Significance

Message 3670, for all its brevity, represents a shift in the optimization strategy. The previous approach had been to control the rate of dispatch through a sophisticated PI controller with EMA smoothing, re-bootstrap detection, and anti-windup. The PI controller was treating the system as a continuous process where the key variable was the GPU queue depth — how many synthesized proofs were waiting for the GPU.

But the user's observation about DDR5 bandwidth pointed to a different kind of constraint: a concurrency limit rather than a rate limit. Even with a perfectly tuned PI controller that kept the GPU queue at the optimal depth, if each synthesis took longer because of bandwidth contention, the overall throughput would suffer. The concurrency cap is a complementary mechanism: it prevents the system from entering a regime where adding more workers reduces total output.

This is a classic insight in queueing theory and systems design: sometimes the right control is not a rate limiter but a concurrency limiter. Rate limiting works well when the bottleneck is a single server with known capacity. Concurrency limiting works better when the bottleneck is a shared resource (like memory bandwidth) where adding more clients reduces each client's effective throughput — the system exhibits a "throughput collapse" beyond a certain concurrency level.

The assistant's question — "Want me to build and deploy this + the ki tuning?" — acknowledges that these two changes are complementary. The PI tuning (lower ki, higher integral cap) prevents the controller from overreacting to transient pipeline drains. The concurrency cap prevents the system from overcommitting the memory bus. Together, they address two different failure modes: the PI controller handles the temporal dimension (how fast to dispatch), while the concurrency cap handles the spatial dimension (how many to run at once).

Conclusion

Message 3670 is a confirmation message that belies the depth of reasoning behind it. It marks the culmination of a multi-day optimization effort where the assistant and user iteratively discovered the system's constraints: first GPU queue depth, then pipeline fill time, then integral saturation, then memory bandwidth contention. Each discovery led to a new control mechanism: the PI pacer, the re-bootstrap logic, the asymmetric integral clamps, and finally the concurrency cap.

The elegance of the concurrency cap lies in its simplicity. After days of tuning PI gains, EMA alphas, and clamp limits — all continuous parameters requiring careful calibration — the cap is a single integer with a clear physical interpretation: "no more than this many syntheses at once." It's a discrete, hard constraint that prevents the system from entering a pathological regime, rather than a continuous controller that tries to steer it away. In complex systems, the best optimization is often not a better controller but a better constraint.