The Build That Brought OOM Recovery to Life

"#37 [runtime 9/16] COPY --from=builder /build/memprobe /usr/local/bin/memprobe"

In the sprawling narrative of a production deployment session for the CuZK proving engine, most messages are about writing code, editing scripts, or debugging crashes. But message [msg 4017] is different. It is not a moment of creation or discovery—it is a moment of assembly. After hours of writing a memory-probing utility, patching shell scripts with OOM recovery loops, and editing Dockerfiles, the assistant finally runs the command that packages everything into a deployable artifact:

DOCKER_BUILDKIT=1 docker build -f Dockerfile.cuzk -t theuser/curio-cuzk:latest . 2>&1 | tail -40

This single message, a Docker build invocation with its truncated output, represents the culmination of an entire debugging arc that began with a crashed 342 GiB vast.ai instance. To understand why this message matters, we must trace the chain of reasoning that led to it.

The Context: A Memory Crisis on Constrained Instances

The session leading up to [msg 4017] was driven by a concrete operational failure. A vast.ai instance with 342 GiB of cgroup-limited memory was crashing with OOM kills during GPU proving benchmarks. The root cause, identified through painstaking investigation across earlier messages, was a multi-layered accounting problem.

First, the CUDA pinned memory pool (PinnedPool) was operating outside the MemoryBudget system. Pinned buffers returned to the pool after GPU work were never freed from actual RSS, creating a massive discrepancy between what the budget system thought was allocated and what the kernel actually accounted for. Second, kernel and driver overhead—glibc arena allocations, page tables, GPU driver memory—consumed cgroup budget invisibly, with no tooling to measure it. Third, transient spikes during SRS loading (simultaneous mmap of parameter files and cudaHostAlloc for pinned buffers) could push the system over the edge even when average memory usage looked safe.

The existing 10 GiB safety margin was empirically insufficient. The assistant and user needed a data-driven approach.

The Two-Pronged Strategy

The response to this crisis was a two-pronged strategy implemented across four files, all of which converge in message [msg 4017].

Prong one: memprobe.c — a tiny C program ([msg 4000]) that allocates 1 GiB chunks via mmap and memset until it nears the cgroup limit. By stopping 2 GiB before the limit, it measures the actual usable memory on a machine, accounting for all the invisible kernel overhead that /proc/meminfo and cgroup stats miss. This provides a data-driven safety margin derived from empirical measurement rather than guesswork.

Prong two: OOM recovery in benchmark.sh ([msg 4009]) — the benchmark script was restructured with a retry loop. If the daemon is killed with exit code 137 (the OOM kill signal), the budget is reduced by 10% and the benchmark is retried up to three times, with a 30-second wait between attempts for kernel memory reclaim. This transforms an unrecoverable crash into an adaptive self-healing process.

The entrypoint.sh was updated ([msg 4012]) to run memprobe after memcheck and use the more conservative of the two budget estimates. The Dockerfile.cuzk was edited (<msg id=4003, 4004>) to compile memprobe in the builder stage and copy it to the runtime image.

All four changes were committed together ([msg 4016]) with a descriptive commit message that captures the rationale:

"cuzk: add memprobe utility and benchmark OOM recovery"

Then came the build.

Message 4017: The Build as a Boundary Object

What makes [msg 4017] significant is not the Docker build command itself—it is what the build represents. This is the moment where development artifacts cross the boundary from "code that works on my machine" to "code that works in production."

The truncated output shows the runtime stage of a multi-stage Docker build. Each COPY --from=builder line transfers a compiled binary from the builder stage to the final runtime image:

#34 [runtime 6/16] COPY --from=builder /build/cuzk       /usr/local/bin/cuzk
#35 [runtime 7/16] COPY --from=builder /build/cuzk-bench /usr/local/bin/cuzk-bench
#36 [runtime 8/16] COPY --from=builder /usr/local/bin/portavailc /usr/local/bin/portavailc
#37 [runtime 9/16] COPY --from=builder /build/memprobe  /usr/local/bin/memprobe
#38 [runtime 10/16] COPY docker/cuzk/entrypoint.sh  /usr/local/bin/entrypoint.sh

Line #37 is the critical one: COPY --from=builder /build/memprobe /usr/local/bin/memprobe. This single line represents the successful compilation of the C utility and its inclusion in the production image. The memprobe binary, written from scratch to solve a specific operational problem, is now part of the standard deployment.

The use of DOCKER_BUILDKIT=1 is notable. BuildKit provides better caching, parallel stage execution, and more efficient layer handling. The assistant chose this flag deliberately to speed up what would otherwise be a slow rebuild of the entire image. The tail -40 filter shows only the final stages—the assistant is interested in verifying that the runtime stage completes successfully, not in watching the entire build log.

Assumptions Embedded in the Build

Every build encodes assumptions. Message [msg 4017] is no exception.

The assistant assumes that all four edited files are syntactically correct and will compile without errors. The memprobe.c source, written in C with mmap/memset calls, must compile against the builder stage's libc. The shell scripts (benchmark.sh, entrypoint.sh) must be syntactically valid bash. The Dockerfile edits must produce a valid multi-stage build with correct stage names and copy paths.

The assistant also assumes that the build environment has access to the Docker daemon with BuildKit support, that the Docker cache is warm enough to avoid a full rebuild, and that the network is available for any apt-get or go mod download steps in earlier stages.

There is an implicit assumption that the build will succeed—the command is run without a conditional check or error handling. The tail -40 output is captured for verification, but the assistant does not check the exit code or parse the output for error messages. This is a reasonable assumption given that the Dockerfile edits were minimal (adding a RUN gcc command and a COPY line), but it is an assumption nonetheless.

What the Output Reveals (and Conceals)

The truncated output shows stages #34 through #39 of the runtime stage. Each COPY operation completes in 0.1 seconds, indicating that the builder stage artifacts are already cached and the copies are fast file transfers within the Docker layer system.

The output is cut off at #39 [run... — we never see the final stages, the tagging step, or the build summary. This truncation is intentional: the assistant used tail -40 to capture only the tail end of a potentially verbose build log. The assumption is that if the last visible stages complete without error, the build as a whole succeeded. This is a pragmatic choice—Docker builds can produce thousands of lines of output, and the final stages are the most informative for verifying correctness.

The Thinking Process: Methodical and Goal-Driven

The sequence of messages leading to [msg 4017] reveals a highly methodical thinking process. The assistant maintains a running todo list (visible in <msg id=4005, 4010, 4014>) with four items:

  1. Write memprobe.c — completed
  2. Add memprobe to Dockerfile.cuzk — completed
  3. Update benchmark.sh with OOM recovery — completed
  4. Update entrypoint.sh to use memprobe — completed After each item is checked off, the assistant moves to the next. The commit in [msg 4016] bundles all four changes into a single logical unit. Then the build. Then, presumably, a docker push to deploy the image. This is not exploratory programming or debugging—it is production engineering. Every change is deliberate, every file is version-controlled, and the build is the final verification step before deployment. The assistant is thinking in terms of a CI/CD pipeline: write → commit → build → push → deploy.

Input and Output Knowledge

To understand [msg 4017], a reader needs input knowledge about Docker multi-stage builds (the COPY --from=builder pattern), BuildKit (DOCKER_BUILDKIT=1), the project's binary naming conventions (cuzk, cuzk-bench, portavailc, memprobe), and the overall architecture of the CuZK proving engine deployment.

The message creates output knowledge: the Docker image theuser/curio-cuzk:latest now exists with the new memprobe binary and updated scripts. This image is ready to be pushed to a registry and deployed to vast.ai instances. The build output confirms that the multi-stage Dockerfile works correctly and that all binaries are properly copied to the runtime stage.

Why This Message Matters

In a session full of debugging epiphanies and code edits, message [msg 4017] might seem mundane—just a build command. But it is the moment when theory becomes practice. The memprobe utility, written to solve an empirical measurement problem, is no longer a source file on disk—it is a binary in a Docker image, ready to run on production machines. The OOM recovery loop, written to handle the specific failure mode of memory-constrained instances, is now baked into the entrypoint script that every container will execute.

The build is the bridge between "this should work" and "this does work." Message [msg 4017] is that bridge, captured in a single Docker command.