The Docker Extraction Trick: Deploying a C Binary Without a Compiler

In the trenches of production debugging, the most elegant solutions often look mundane. Message <msg id=4021> is a perfect example: a three-line shell pipeline that extracts a compiled binary from a Docker image and delivers it to a running instance. On its surface, it is a routine deployment step. But beneath that simplicity lies a chain of reasoning about environment constraints, build reproducibility, and the art of working around missing tools without missing a beat.

The Context: An OOM Crisis on Memory-Constrained Instances

To understand why this message exists, we must first understand the crisis it resolves. The team had been battling Out-of-Memory (OOM) kills on vast.ai GPU instances running the CuZK proving engine. These instances operate inside Docker containers with cgroup memory limits, but the system's memory detection logic was reading /proc/meminfo, which reports the host's full RAM rather than the container's cgroup limit. This caused the memory budget to be massively over-allocated, leading to the kernel's OOM killer terminating the cuzk daemon mid-benchmark.

The team had already deployed a cgroup-aware memory detection fix (memcheck.sh), but a deeper problem remained: even with correct cgroup limits, the system was still crashing. Investigation revealed that the CUDA pinned memory pool (PinnedPool) was operating outside the MemoryBudget system — pinned buffers returned to the pool after GPU work were never freed from actual RSS, creating a massive accounting discrepancy. Combined with kernel/driver overhead (glibc arenas, page tables, GPU driver allocations) and transient SRS loading spikes, the 10 GiB safety margin was empirically insufficient for constrained instances.

The solution was two-pronged. First, a memprobe utility — a tiny C program that allocates 1 GiB chunks via mmap/memset until it nears the cgroup limit — would provide a data-driven safety margin that accounts for hidden kernel overhead. Second, the benchmark.sh script would gain an OOM recovery loop: if the daemon is killed (exit code 137), the budget is reduced by 10% and the benchmark is retried up to three times.

The memprobe.c source had been written ([msg 4000]), the Dockerfile.cuzk had been updated to compile it in the builder stage and copy it to the runtime stage (<msg id=4003-4004>), and the Docker image had been built and pushed to Docker Hub (<msg id=4017-4018>). The binary existed inside the image theuser/curio-cuzk:latest at /usr/local/bin/memprobe.

The Problem: No Compiler on the Target Instance

The running instance (ID 32897009) was a 256 GiB vast.ai node that had just finished downloading 159 GiB of Filecoin proving parameters and had started benchmarking. The team needed to get the memprobe binary onto this instance to validate the empirical safety margin and confirm the kernel overhead theory.

The obvious approach — SSH into the instance and compile memprobe.c there — was blocked by a simple environmental constraint: the runtime Docker image did not include gcc. This is standard practice for production containers: compilers are excluded from runtime images to minimize size and attack surface. The builder stage of the Dockerfile had gcc and compiled memprobe.c into a static binary, but the runtime stage only received the compiled artifact.

The assistant could have re-written memprobe in Go (which was available on the instance, since cuzk itself is a Go binary), or could have compiled it statically with a different toolchain. But those approaches would have introduced delays and potential inconsistencies. The binary already existed, already compiled, inside the Docker image they had just pushed. The problem was purely one of extraction and transport.

The Solution: docker create + docker cp

The assistant's response is a masterclass in leveraging Docker's tooling for file extraction:

docker create --name memprobe-extract theuser/curio-cuzk:latest true 2>/dev/null \
  && docker cp memprobe-extract:/usr/local/bin/memprobe /tmp/memprobe \
  && docker rm memprobe-extract 2>/dev/null \
  && ls -la /tmp/memprobe

Let us unpack each step.

docker create instantiates a container from the specified image without starting it. The --name memprobe-extract gives the container a label for later reference. The true command is the container's entry point — it does nothing and exits immediately, but it is never executed because the container is never started. The 2&gt;/dev/null suppresses any warnings about the container name already existing from a previous run (the docker rm at the end ensures cleanup).

docker cp copies files from the container's filesystem to the host. This works on stopped containers — Docker maintains the container's filesystem layers as an overlay mount, and docker cp reads directly from that mount without requiring the container to be running. The source path /usr/local/bin/memprobe is the exact location where the Dockerfile.cuzk placed the compiled binary in the runtime stage.

docker rm removes the ephemeral container, cleaning up the named container so the script can be re-run without a name collision.

ls -la verifies that the extracted binary exists, is the expected size (16400 bytes), and has executable permissions.

The output confirms success: a 16,400-byte executable at /tmp/memprobe, ready to be scp'd to the target instance.

Assumptions and Reasoning

This approach makes several implicit assumptions, all of which are sound in this context:

  1. The binary exists at the expected path in the image. This assumption is justified because the Docker build succeeded ([msg 4017]) and the build log explicitly showed COPY --from=builder /build/memprobe /usr/local/bin/memprobe completing successfully.
  2. docker cp works on a non-running container. This is a well-documented feature of Docker. The container's filesystem is materialized from its image layers and any writable layer changes, regardless of whether the container is running. A docker create without docker start is sufficient.
  3. The binary is statically linked and portable. A C program using only mmap, memset, and standard library calls (as memprobe.c does) compiles to a statically linkable binary that does not depend on runtime libraries that might differ between the build environment and the target. The 16 KB binary size is consistent with a statically linked executable.
  4. The binary will work on the target instance's architecture. Both the build machine and the vast.ai instance run x86_64 Linux, so architecture compatibility is guaranteed.
  5. The docker create name collision is handled. The 2&gt;/dev/null suppresses the error if memprobe-extract already exists, and the subsequent docker rm cleans it up. This makes the command idempotent for repeated runs.

Input Knowledge Required

To understand this message, a reader needs:

Output Knowledge Created

This message produces:

  1. A concrete binary artifact: /tmp/memprobe on the build host, ready for deployment.
  2. A reusable extraction pattern: the docker create + docker cp technique, which can be applied to any Docker image to extract tools, configuration files, or debugging utilities without running the container.
  3. Validation of the build pipeline: the successful extraction confirms that the Docker build correctly compiled memprobe.c and placed the binary at the expected path in the runtime image.

The Deeper Pattern: Working Within Constraints

What makes this message noteworthy is not the technical complexity — the commands are straightforward — but the reasoning process it reveals. The assistant faced a constraint (no compiler on the target) and a resource (a Docker image with the pre-compiled binary). Rather than fighting the constraint by installing a compiler or rewriting the tool in another language, the assistant worked with the constraint by finding a path from the resource to the target.

This is the hallmark of mature systems thinking: recognizing that the Docker image is not just a deployment artifact but also a distribution mechanism for compiled tools. The builder stage of a multi-stage Docker build serves double duty — it compiles the application, but it also compiles auxiliary utilities that can be extracted for use outside the container. The docker cp command bridges the build environment and the runtime environment, carrying the compiled artifact across without requiring either environment to change.

The pattern generalizes beyond this specific case. Any time a team maintains a Docker-based build pipeline, the resulting image contains compiled binaries that can be extracted for deployment to environments without compilers. This is particularly valuable for embedded systems, minimal base images (like scratch or alpine), or air-gapped environments where installing development tools is impractical or prohibited.

Conclusion

Message &lt;msg id=4021&gt; is a small but revealing moment in a larger debugging saga. It demonstrates that production engineering is not always about writing complex algorithms or designing elegant architectures. Sometimes it is about knowing that docker cp works on stopped containers, recognizing that the binary you need already exists inside an image you just built, and executing a three-line pipeline that saves hours of workarounds. The 16 KB memprobe binary extracted in this message would go on to provide the empirical validation the team needed — confirming that the 256 GiB instance was operating at 99% of its cgroup limit with only 6 GiB of kernel overhead, and that the new adaptive safety margin and OOM recovery logic were essential for reliable operation on memory-constrained vast.ai nodes.