The Cat That Wasn't There: Extracting a Binary from a Scratch-Based Docker Image

In the middle of deploying a complex memory management system for a GPU-based zero-knowledge proving engine, a seemingly trivial Docker command failed. The message at <msg id=2308> captures a moment where an experienced engineer's assumption about container environments collided with the stripped-down reality of a FROM scratch Docker image. The result is a brief but instructive failure that illuminates the gap between everyday container usage and the lean, production-oriented patterns used in high-performance computing deployments.

The Message

The assistant executed the following command:

docker run --rm cuzk-rebuild:membudget cat /cuzk > /tmp/cuzk-daemon-new && chmod +x /tmp/cuzk-daemon-new && ls -lh /tmp/cuzk-daemon-new

The response was an error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "cat": executable file not found in $PATH

The Broader Context

This message is part of a much larger effort: deploying a unified, budget-based memory manager for the cuzk GPU proving engine. The memory manager replaces a fragile static concurrency limit with a memory-aware admission control system that tracks all major memory consumers—SRS pinned memory, PCE heap allocations, and synthesis working sets—under a single byte-level budget. The assistant had just committed this change (a 2,629-line addition across nine files), built a Docker image containing the new binary, and was now attempting to extract that binary for upload to a remote production machine running an RTX 5090 with 755 GiB of RAM.

The Docker image cuzk-rebuild:membudget was built using Dockerfile.cuzk-rebuild, a minimal rebuild Dockerfile designed for rapid iteration. Unlike the full Dockerfile.cuzk which produces a complete runtime environment, the rebuild variant uses a multi-stage build where the final stage is FROM scratch—a completely empty base image containing nothing but the compiled binary itself. This is a common pattern for producing minimal deployment artifacts, but it comes with a critical constraint: there is no shell, no utilities, and no standard library.

Why This Message Was Written

The assistant needed to get the compiled binary out of the Docker image and onto the remote machine at 141.0.85.211. The previous attempt, in <msg id=2307>, had used docker create followed by docker cp to extract the binary, but that attempt failed with "No such container" because the container creation itself had errored out. The assistant pivoted to a different approach: using docker run --rm with a command that would read the binary and pipe it to a local file.

The choice of docker run --rm cat /cuzk is a natural one. It's concise, it cleans up after itself (the --rm flag removes the container on exit), and it follows a pattern that works reliably with any Docker image that has a shell. The cat command reads the file and writes it to stdout, which is then redirected to /tmp/cuzk-daemon-new. The && chmod +x ensures the extracted file is executable, and ls -lh confirms the result.

The Assumption and the Mistake

The critical assumption was that the Docker image contained a usable cat command. In the vast majority of Docker workflows, this is a safe assumption. Most production images are based on ubuntu, debian, alpine, or distroless variants that include at least a minimal set of POSIX utilities. Even distroless images typically include sh and basic tools. But FROM scratch is different: it is truly empty. The only file in the image is the binary that was copied in during the build—in this case, /cuzk. There is no /bin, no /usr, no cat, no sh, no ls. The container's filesystem is a blank slate containing exactly one file.

The error message from runc makes this explicit: exec: "cat": executable file not found in $PATH. The container runtime attempted to locate cat in the image's filesystem, found nothing, and refused to start.

This mistake is easy to make because the Dockerfile's final stage is often invisible to someone who only interacts with the built image through docker run. The FROM scratch directive appears in the Dockerfile source (which the assistant had read in <msg id=2304>), but it's easy to overlook when focused on the deployment task. The assistant's reasoning in <msg id=2306> shows they were thinking about which binaries to build and how to test the memory manager, not about the intricacies of the Dockerfile's final stage.

Input Knowledge Required

To understand this message, one needs to know:

Output Knowledge Created

The error message itself is the primary output. It reveals:

  1. The Docker image uses a scratch base, which means it has no shell utilities
  2. The docker run approach with inline commands will not work for this image
  3. The binary extraction must use docker cp instead (as the assistant does in <msg id=2309>) More subtly, the error confirms that the Docker build succeeded—the image exists and is runnable, but only with the binary itself as the entrypoint, not with arbitrary commands. This is actually good news: it means the FROM scratch final stage is correctly structured for its intended purpose of producing a minimal deployment artifact.

The Thinking Process Visible in the Message

The assistant's reasoning is not explicitly shown in this message (it's a pure tool call without a reasoning block), but the choice of command reveals a thought process. The assistant had just seen docker create fail in the previous message and was trying a different approach. The docker run --rm pattern is a common alternative to docker cp—it works with most images and is simpler because it doesn't require managing container lifecycle. The assistant likely thought: "The previous docker create failed, let me try docker run with cat to pipe the binary out directly."

The command structure also shows attention to detail: the && chaining ensures that chmod +x and ls -lh only run if the extraction succeeds, and --rm prevents container accumulation. This is disciplined shell scripting, even if the underlying assumption about the image's contents was wrong.

Resolution and Lessons

In the very next message (<msg id=2309>), the assistant recognizes the issue: "The final stage is FROM scratch so there's no shell or cat." They then successfully extract the binary using docker create followed by docker cp—the approach that had failed earlier, now executed correctly. The binary is 27 MB and is subsequently uploaded to the remote machine via scp in <msg id=2310>.

This episode illustrates a recurring tension in infrastructure engineering: the gap between the mental model of a system and its actual configuration. The FROM scratch pattern is powerful for producing minimal, secure deployment artifacts, but it violates the implicit assumption that a Docker image contains a usable operating environment. The error is minor and quickly corrected, but it serves as a reminder that even straightforward deployment steps can fail in surprising ways when the underlying assumptions are wrong.

For the broader memory manager deployment, this was a two-minute detour. The binary was extracted, uploaded, and deployed, and the assistant proceeded to test the memory manager on the remote machine—only to encounter more significant challenges with async blocking panics, concurrency bottlenecks, and OOM conditions that would occupy the remainder of the segment.