The Scratch Image Problem: Extracting a Binary When There's No Shell

In the middle of deploying a critical memory management overhaul for a GPU proving engine, a seemingly trivial task—extracting a compiled binary from a Docker image—turned into a small but instructive puzzle. Message [msg 2309] captures the moment an AI assistant diagnosed and solved this puzzle, bridging the gap between a successful Docker build and a deployable artifact. The message is brief, but it encapsulates a pattern of reasoning that is far more general than the specific Docker command it produces: understanding the constraints of a build artifact, recognizing why a straightforward approach failed, and adapting to the actual affordances of the system.

The Deployment Context

To understand why this message matters, one must appreciate the stakes of the broader session. The assistant had just implemented a unified budget-based memory manager for the cuzk GPU proving engine—a system responsible for generating zero-knowledge proofs for Filecoin storage proofs. The old system used a static concurrency limit (partition_workers) that was fragile and could lead to out-of-memory (OOM) crashes. The new system introduced a MemoryBudget that tracked all major memory consumers—SRS pinned memory, PCE heap allocations, and synthesis working sets—under a single byte-level budget, auto-detected from system RAM. This was a substantial architectural change touching nine files and inserting over 2,600 lines of code.

The user's instruction at [msg 2293] was clear: commit the changes, build the new binary in Docker (since the target machine required a compatible runtime), upload it to a remote production host (141.0.85.211, a 755 GiB machine with an RTX 5090), and test that the memory constraints worked correctly. The assistant had committed the changes at [msg 2296] and successfully built a new Docker image tagged cuzk-rebuild:membudget using a minimal rebuild Dockerfile (Dockerfile.cuzk-rebuild). The build completed, and the binary was inside the image. Now it just needed to come out.

Two Failed Attempts

The assistant's first attempt at extraction, at [msg 2307], used a straightforward script:

docker create --name cuzk-extract cuzk-rebuild:membudget 2>/dev/null; 
docker cp cuzk-extract:/cuzk /tmp/cuzk-daemon-new; 
docker rm cuzk-extract >/dev/null 2>&1; 
ls -lh /tmp/cuzk-daemon-new

This failed with Error response from daemon: No such container: cuzk-extract. The problem was a race condition in the shell script: the docker create command had its stderr redirected to /dev/null, so if it failed silently, the subsequent docker cp would try to copy from a container that was never created. The || chaining was missing proper error handling.

The second attempt, at [msg 2308], tried a different approach:

docker run --rm cuzk-rebuild:membudget cat /cuzk > /tmp/cuzk-daemon-new

This failed with a more informative error: exec: "cat": executable file not found in $PATH. The docker run command tried to execute cat inside the container, but cat didn't exist. This was the critical clue.

The Reasoning: Understanding Scratch Images

Message [msg 2309] opens with the assistant's reasoning, which shows the moment of insight:

Agent Reasoning: The final stage is FROM scratch so there's no shell or cat. I need to use docker cp or use a multi-stage approach. Let me try creating the container first then copying.

This is the key diagnostic step. The assistant had previously read the Dockerfile at [msg 2304], which revealed a multi-stage build. The Dockerfile.cuzk-rebuild used nvidia/cuda:13.0.2-devel-ubuntu24.04 as a builder stage, but the final stage was FROM scratch. In Docker, a scratch image is the minimal possible base—it contains absolutely nothing. No shell, no cat, no ls, no /bin, no /usr. It is an empty filesystem layer, typically used only to host a single statically linked binary. This explained why docker run ... cat /cuzk failed: there was no cat binary to execute.

The assistant correctly reasoned that docker cp was the right tool for this job. Unlike docker run, which requires an executable entrypoint, docker cp operates at the filesystem layer. It can copy files from a container's filesystem regardless of whether the container has a shell or can execute commands. The container doesn't even need to be running—it just needs to exist (in the "created" state).

The Solution: Create, Copy, Clean Up

The command the assistant executed is a textbook example of extracting a binary from a scratch-based Docker image:

docker create --name cuzk-tmp cuzk-rebuild:membudget /cuzk 2>&1 && \
docker cp cuzk-tmp:/cuzk /tmp/cuzk-daemon-new && \
docker rm cuzk-tmp && \
chmod +x /tmp/cuzk-daemon-new && \
ls -lh /tmp/cuzk-daemon-new

The pipeline works in four stages:

  1. docker create instantiates a container from the image without starting it. The /cuzk argument is technically the command to run, but since the container is never started, it doesn't matter. The 2>&1 redirect ensures error output is visible (fixing the mistake from the first attempt where stderr was hidden).
  2. docker cp copies the file from the container's filesystem to the host. This works because docker cp reads the filesystem layers directly, without needing a running process.
  3. docker rm removes the temporary container, cleaning up.
  4. chmod +x ensures the binary has execute permissions (Docker preserves file permissions from the build, but it's good practice to ensure this).
  5. ls -lh confirms the result: a 27 MB binary, dated and ready. The output confirms success: 050a9e2b681d47eb815796c3b137d3b46371e37b656e65f0f721c3bf68dd88a6 (the container ID), cuzk-tmp (the container name), and -rwxr-xr-x 1 theuser theuser 27M Mar 13 15:14 /tmp/cuzk-daemon-new (the file listing).

Assumptions, Mistakes, and Lessons Learned

This sequence of three attempts (msg 2307, 2308, 2309) reveals several assumptions and corrections:

Assumption 1: Docker images have a shell. The assistant initially assumed that docker run ... cat /cuzk would work, which presupposes a shell or at least a cat binary in the image. This is a reasonable default assumption—most Docker images are based on Alpine, Ubuntu, or Debian, which include basic utilities. But scratch images are the exception.

Assumption 2: Error output is noise. In the first attempt, stderr was redirected to /dev/null (2>/dev/null). This hid the error from docker create, causing a cascading failure where subsequent commands silently failed. The corrected version uses 2>&1 to capture errors.

Mistake 1: Silent failure masking. The first attempt used semicolons (;) instead of && chaining, meaning each command ran regardless of whether the previous one succeeded. This is a classic shell scripting pitfall.

Mistake 2: Assuming a running container is needed. The docker run approach assumes the container must be running to access its files. In fact, docker cp works on stopped containers too.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with Docker multi-stage builds and the scratch base image; understanding of docker create vs docker run semantics; knowledge that docker cp operates at the filesystem layer; and awareness of shell scripting idioms like && chaining and stderr redirection.

Output knowledge created by this message is both practical and conceptual. Practically, the binary /tmp/cuzk-daemon-new is now available on the host filesystem, ready to be uploaded to the remote machine (which happens in subsequent messages). Conceptually, the message documents a reliable pattern for extracting binaries from scratch-based Docker images—a pattern that applies to any minimal container image where the goal is to retrieve a build artifact without adding a shell layer.

Broader Significance

While this message is small in scope—a single Docker command—it illustrates a recurring theme in infrastructure engineering: the gap between "build succeeds" and "artifact is deployable." The Docker build produced a correct binary, but the binary was trapped inside an image designed for minimal size. Extracting it required understanding the image's internal structure and choosing the right tool for the job.

This pattern appears in many contexts: extracting binaries from embedded system images, retrieving logs from minimal containers, accessing files in unikernel images. The general principle is: when a system is designed for minimalism (no shell, no utilities), you must interact with it at the layer of its actual capabilities—filesystem operations rather than process execution.

The message also demonstrates a productive debugging cycle: try a straightforward approach, learn from its failure, refine your understanding of the system's constraints, and apply a more targeted solution. The assistant's reasoning—"The final stage is FROM scratch so there's no shell or cat"—is the kind of diagnostic insight that separates a correct fix from a lucky guess. It shows that the assistant understood why the previous approach failed, not just that it failed.

In the larger narrative of the session, this message is a brief pause before the real testing begins. The binary is extracted. The next steps—uploading to the remote host, hot-swapping the daemon, and testing the memory manager under real-world conditions—will reveal deeper issues: an async-blocking panic in the evictor callback, concurrency bottlenecks under tight budgets, and an OOM crash that requires careful safety margin tuning. But those challenges belong to later messages. Here, in message [msg 2309], the assistant simply needed to get the binary out of the box. And it did.