The Quiet Cap: How a Single Edit Wired a Hard Limit on Parallel Synthesis

Message 3666 in this opencode session is, on its surface, almost laughably brief. The assistant writes: "Now cap synth_worker_count in engine.rs:" followed by an edit command that reports success. Two lines. A reader glancing at the conversation might dismiss it as a trivial mechanical step. But this message represents the culmination of a multi-threaded chain of reasoning spanning hardware constraints, control theory, pipeline architecture, and production deployment — and it embodies a fundamental design philosophy shift that had been brewing across dozens of earlier messages.

To understand why this message matters, one must trace the threads that converged upon it.

The Context: A Pipeline Under Tension

The broader session (Segment 26 of the opencode conversation) had been consumed with tuning a PI (proportional-integral) dispatch pacer — a feedback controller that governs how quickly synthesized proofs are dispatched to GPU workers. The assistant had iterated through multiple deployments (pitune1 through pitune4), wrestling with integral saturation, re-bootstrap timing, and the vicious cycle created by a synthesis throughput cap. The core tension was this: the GPU could process proofs faster than the CPU could synthesize them, but synthesizing too many proofs in parallel would exhaust system memory and cause thrashing on DDR5 bandwidth.

The PI controller was the assistant's elegant solution — it used the GPU queue depth as the controlled variable, adjusting dispatch rate to keep the queue optimally filled. But the PI controller only governed dispatch rate; it did not directly limit how many proofs the CPU was synthesizing concurrently. The synthesis worker count was derived from the memory budget: max_partitions_in_budget.min(64).max(4). On the target machine with 400 GiB of memory budget and approximately 9 GiB per partition, this yielded 28 concurrent synthesis workers. All 28 could be running synthesis simultaneously, competing for CPU cores, memory bandwidth, and — crucially — DDR5 memory channels.

The User's Observation

At message 3657, the user interjected with a practical observation grounded in hardware reality:

"Do we have a simple hard cap on parellel synthesis? We should just set it to default 18, and keep configurable, anything more will probably choke on ddr5 systems"

This was not a theoretical concern. DDR5 memory, while faster than DDR4, has specific bandwidth characteristics and channel topologies. Beyond a certain number of concurrent memory-intensive operations — and proof synthesis is deeply memory-bound — the memory controller becomes a bottleneck, and additional parallelism degrades rather than improves throughput. The user's suggested default of 18 reflected an empirical understanding of where that inflection point lay for their hardware.

The user corrected "parellel synthesis" to "parallel running" at message 3660, clarifying that they meant concurrent execution, not the number of worker threads in the pool.

The Assistant's Reasoning Chain

The assistant's response at message 3661 reveals the thinking process. The key insight was recognizing that synth_worker_count — the number of worker tasks pulling from the synthesis channel — directly controls concurrency. Each worker runs one synthesis at a time, so capping the worker count naturally caps parallelism without needing a separate semaphore or admission control mechanism.

The assistant considered and rejected a more complex approach: "We need a simple concurrency limit — a semaphore or similar." But then realized: "The simplest fix is to just cap synth_worker_count directly at 18 — if there are only 18 workers pulling from the bounded channel, then at most 18 syntheses can run concurrently, which naturally enforces the limit without needing a separate semaphore."

This is a textbook example of leveraging existing architecture. The channel-based worker model already provided the right abstraction; the fix was to reduce the number of consumers on that channel. No new synchronization primitives, no new failure modes, no additional runtime overhead.

The Config Wires

Before the subject message could execute its edit, the assistant needed to lay the config infrastructure. Messages 3663-3665 added a new max_parallel_synthesis field to PipelineConfig in config.rs, with a default value of 18. The assistant was careful to distinguish this from the existing synthesis_concurrency field, which served the monolithic (non-pipeline) path. The new field was specific to the pipeline path, with documentation explaining its purpose.

The assistant also needed to add a default function and wire it into the Default impl for PipelineConfig. These are the invisible plumbing steps — unglamorous but essential — that make a config field actually work with serde deserialization.

The Subject Message: Wiring It Together

Message 3666 is where theory meets practice. The edit to engine.rs modifies the line:

let synth_worker_count = max_partitions_in_budget.min(64).max(4);

to something like:

let synth_worker_count = max_partitions_in_budget.min(64).max(4).min(config.max_parallel_synthesis);

Or more precisely, the assistant likely replaced the computation with a direct cap from the config. The exact diff is not shown in the message text, but the intent is unambiguous: the previously unbounded (up to 64) worker count is now clamped to the configurable maximum.

This single change has profound operational implications. Previously, if the memory budget allowed 50 partitions, the system would spawn 50 synthesis workers, all potentially running simultaneously. After this change, even with budget for 50 partitions, only 18 workers would be created, limiting concurrent synthesis to 18 regardless of available memory. The memory budget still gates how many proofs can be in flight (via memory reservations), but the worker cap gates how many can be actively synthesizing at any moment.

Assumptions and Design Decisions

Several assumptions underpin this message:

  1. Worker count equals concurrency: The assistant assumes each worker runs exactly one synthesis at a time. This is correct for the current architecture — workers are tokio::spawned tasks that loop, pulling one item from the channel, synthesizing it, and pushing the result. No worker parallelizes internally.
  2. Channel-based backpressure is sufficient: By reducing the number of consumers, the channel naturally fills up, providing backpressure to the dispatcher. The assistant assumes this backpressure integrates cleanly with the existing PI controller and memory budget system.
  3. 18 is a reasonable default: The user suggested 18, and the assistant accepted it without analysis. The assumption is that this value will be tuned empirically in production, and the config field makes that easy.
  4. The monolithic path is separate: The assistant deliberately kept the existing synthesis_concurrency field for the monolithic path and added a new field for the pipeline path. This assumes the two paths will continue to have different concurrency characteristics.

What This Message Creates

The output knowledge created by this message is a deployed system with a hard upper bound on CPU-side parallelism. Before this change, the system was purely budget-gated — if memory allowed 50 concurrent syntheses, it would attempt 50. After this change, there is an explicit operator-configurable ceiling.

This is significant because the PI controller and the memory budget system could not, by themselves, prevent excessive parallelism. The PI controller governs dispatch rate (how fast completed proofs are sent to GPU), not synthesis rate. The memory budget limits how many proofs can be allocated, but with 400 GiB of budget and ~9 GiB per proof, the budget could accommodate 44 concurrent syntheses — far more than the DDR5 system could handle efficiently. The hard cap closes this gap.

The Broader Significance

Message 3666 represents a shift from purely feedback-driven control (the PI pacer) to a hybrid approach combining feed-forward limits with feedback tuning. The PI controller handles dynamic adjustments based on real-time queue depth; the hard cap provides a static bound that prevents the system from entering regimes where performance collapses due to memory bandwidth contention.

This is a common pattern in production systems: feedback controllers excel at fine-tuning within a stable operating region, but they cannot always prevent excursions into pathological territory. Hard limits — whether on concurrency, memory usage, or request rate — act as safety rails that keep the system within the region where the feedback controller can operate effectively.

The message also illustrates a key principle of systems engineering: the simplest fix is often the best. The assistant could have implemented a semaphore, a rate limiter, or a token bucket. Instead, it recognized that the existing worker-pool architecture already provided the right mechanism — it just needed a smaller pool.