The Floor of Four: A Single Line Fix That Prevented Benchmark Starvation

Introduction

In the middle of a sprawling, multi-day coding session to build a CUDA-based zero-knowledge proving daemon (cuzk) for the Filecoin network, a single-line message appears that is deceptively simple:

Now fix the benchmark concurrency minimum in entrypoint.sh — add a floor of 4 after the MAX_CONC calculation: [edit] /tmp/czk/docker/cuzk/entrypoint.sh Edit applied successfully.

This is message <msg id=3725>, and it represents the third of four critical deployment fixes discovered during live testing on vast.ai cloud instances. To understand why this single line matters—why a simple "floor of 4" deserves deep analysis—we must understand the intricate web of memory budgets, GPU pipeline scheduling, Docker cgroup limits, and the fragile economics of proving Filecoin proofs on rented GPU hardware. This article dissects that message, the reasoning behind it, the assumptions it encodes, and the knowledge it both consumes and produces.

The Context: A System Under Pressure

By the time we reach <msg id=3725>, the assistant and user have been iterating on the cuzk system for dozens of rounds. They have implemented a CUDA pinned memory pool to eliminate GPU underutilization caused by slow host-to-device PCIe transfers, built a PI-controlled dispatch pacer to stabilize GPU queue depth, tuned synthesis concurrency to 18 (the sweet spot for DDR5 systems with 64 cores), and deployed the system to vast.ai cloud instances. But real-world deployment has revealed a set of configuration bugs that only manifest under live conditions.

The system's architecture is complex. The cuzk daemon runs inside Docker containers on vast.ai instances, which are themselves virtual machines with varying amounts of RAM. The benchmark workflow—used to measure proving throughput before putting a node into production—involves starting the cuzk daemon, running a series of proof computations, and recording timing. The entrypoint.sh script orchestrates this lifecycle, including calculating how many concurrent benchmarks the machine can support based on available memory.

The calculation at issue is the MAX_CONC variable. It divides the available system memory (in GB) by the per-proof memory requirement (in GB) times a safety factor of 3, yielding the maximum number of concurrent proof benchmarks that can run without causing an out-of-memory (OOM) kill. On a machine with 251 GB of RAM and a per-proof requirement of roughly 28 GB (for PoRep C2 proofs), this calculation yields MAX_CONC = 251 / (28 * 3) = 251 / 84 ≈ 2.98, which truncates to 2 in integer arithmetic. Even with a more generous per-proof estimate, the result hovers around 3. This is dangerously low.

Why a Floor of 4?

The decision to impose a floor of 4 concurrent benchmarks is not arbitrary. It stems from a deep understanding of the proving pipeline's dynamics. The benchmark script runs proofs sequentially from a single input file, but the cuzk daemon itself is a concurrent system: it can synthesize multiple proof partitions simultaneously, queue them for GPU processing, and pipeline the work. Running only 2 or 3 concurrent benchmarks means the system never reaches its full throughput potential, producing misleadingly low performance numbers that could cause an operator to think the hardware is underperforming.

More critically, the benchmark serves a dual purpose. It is not merely a performance measurement tool—it is also a stress test that validates the configuration before the node enters production. If the benchmark runs with too few concurrent proofs, it may not exercise the memory system enough to trigger latent OOM conditions. A floor of 4 ensures that even on memory-constrained machines, the benchmark generates enough memory pressure to reveal problems before real proving work begins.

The assistant's reasoning, visible in the surrounding conversation, reveals another layer: the benchmark was being restructured into a three-phase model (warmup, timed, cooldown) without restarting the daemon between phases. This change eliminated the long warmup overhead from SRS reloading but made the concurrency calculation even more important—the timed phase needed to run at full throttle to produce accurate measurements.

Assumptions Embedded in the Fix

Every fix carries assumptions, and this one is no exception. The assistant assumes that:

  1. The MAX_CONC calculation is the sole governor of benchmark concurrency. There is no other mechanism that might cap concurrency lower. If another layer (e.g., the cuzk daemon's own synthesis_concurrency config) were set lower than the floor, the floor would be ineffective. However, the assistant had already raised synthesis_concurrency from 4 to 18 in both run.sh and benchmark.sh (messages <msg id=3722> and <msg id=3723>), so this assumption is well-founded.
  2. Running 4 concurrent benchmarks is safe on any machine that passes the memory calculation. The floor of 4 overrides the calculated value only when the calculation yields less than 4. On a machine with only, say, 100 GB of RAM, MAX_CONC would be 100 / 84 ≈ 1, but the floor would raise it to 4. This could cause OOM on truly memory-starved machines. The assistant implicitly assumes that any machine deployed for proving work has at least enough memory for 4 concurrent benchmarks—or that the operator will catch the OOM and adjust. This is a reasonable assumption for the target deployment environment (vast.ai instances with 251+ GB RAM), but it is an assumption nonetheless.
  3. The per-proof memory estimate and safety factor of 3 are correct. The MAX_CONC calculation uses PER_PROOF_GB * 3 as the divisor. This safety factor accounts for the SRS (~44 GiB pinned), PCE (~26 GiB heap), and per-partition working memory (~14 GiB for PoRep). If these estimates are wrong—if a new proof type requires more memory, or if pinned pool allocations grow beyond expectations—the floor of 4 could push the system into OOM territory. The assistant had already observed OOM kills on 251 GB machines when the pinned pool grew to 139 GiB (see the context in <msg id=3717>), so this assumption is grounded in painful experience.

The Thinking Process: A Glimpse Into Debugging Under Pressure

The surrounding messages reveal the assistant's thought process. In <msg id=3724>, the todo list shows the fix status transitioning from "in_progress" to "completed" for the synthesis_concurrency fixes, and "in_progress" for the benchmark concurrency minimum. The assistant is methodically working through a checklist of four deployment issues, applying each fix in sequence.

The choice to add a floor rather than adjust the divisor or change the calculation is telling. The assistant could have reduced the safety factor from 3 to 2, or changed the per-proof memory estimate, or made the calculation more sophisticated (e.g., accounting for the pinned pool separately). Instead, the assistant chose the simplest possible intervention: a post-calculation clamp. This reflects a pragmatic engineering philosophy: when deploying to production, make the smallest change that fixes the observable problem. A floor of 4 is easy to understand, easy to verify, and easy to revert if it causes issues. It is a patch, not a redesign.

Input Knowledge Required

To understand this message, a reader must know:

Output Knowledge Created

This message produces:

Potential Mistakes and Risks

The floor of 4 is not without risk. On a machine with borderline memory, forcing 4 concurrent benchmarks could trigger an OOM kill during the benchmark itself, preventing the node from ever entering production. The assistant does not add any warning or conditional logic—no check that the machine actually has enough memory for 4 concurrent proofs, no fallback to a lower value if memory pressure is detected. The fix is unconditional.

Additionally, the floor interacts with the three-phase benchmark model being developed concurrently. If the warmup phase runs 5 proofs sequentially and the timed phase runs 10 proofs with concurrency 4, the total memory pressure during the timed phase could exceed what the warmup phase tested. A machine that survives warmup could still OOM during timed runs.

Conclusion

Message <msg id=3725> is a masterclass in minimal, targeted production debugging. In one line, the assistant fixes a real problem—benchmark concurrency starvation on memory-constrained machines—without over-engineering the solution. The floor of 4 is a clamp, a safety net, a recognition that the calculated MAX_CONC is too conservative for the actual proving workload. It reflects hours of accumulated knowledge about GPU memory allocation, cgroup limits, pinned pool behavior, and the economics of Filecoin proving. And it reminds us that sometimes the most important fix is the simplest one: a floor that says "never go below this."