The Bridge Between Build and Deploy: A Docker Extraction Command in the CUDA ZK Proving Pipeline

The Message

docker create --name pinned1-extract cuzk-rebuild:pinned1 /cuzk && \
  docker cp pinned1-extract:/cuzk /tmp/cuzk-pinned1 && \
  docker rm pinned1-extract

Output:

d8d8f710cb7259b9f1db4dadefe5b347000608268f7ddf1b63b1eb3ddd1ba7c1
pinned1-extract

This three-line shell pipeline, executed at message index 3205 in a lengthy optimization session, appears deceptively simple. It creates a temporary Docker container from a pre-built image, copies a single binary out of it, and cleans up the container. Yet this command represents a critical inflection point in a months-long engineering effort to eliminate GPU underutilization in a zero-knowledge proof proving system. It is the moment when carefully crafted code, compiled and packaged, crosses the threshold from "it compiles" to "it can be tested in production."

The Context: A GPU Utilization Crisis

To understand why this message matters, one must understand the problem it was written to solve. The team behind the cuzk (CUDA ZK) proving daemon had been battling a persistent performance issue: the system's RTX 5090 GPU was showing only ~50% utilization, with multi-second idle gaps between partition proofs. Through extensive instrumentation — adding timing probes to C++ GPU code, Rust worker loops, and the finalizer — the root cause was identified. The bottleneck was not GPU compute capability but rather the host-to-device (H2D) memory transfer path. The execute_ntts_single function in the GPU code was performing cudaMemcpyAsync from unpinned heap memory (Rust Vec<Fr>), forcing CUDA to stage data through a tiny internal pinned bounce buffer at only 1–4 GB/s instead of the PCIe Gen5 line rate of ~50 GB/s. With each partition requiring 7.8 GiB (SnapDeals) to 12.5 GiB (PoRep) of a/b/c vector data, these slow transfers dominated the per-partition time, ballooning from a theoretical ~80 ms to as much as 5,617 ms.

The solution was a zero-copy pinned memory pool — a system that allocates CUDA-pinned host memory via cudaHostAlloc and reuses it across synthesis operations, so that the a/b/c vectors are written directly into pinned buffers during circuit synthesis, eliminating both the reallocation copy and the slow staged H2D transfer. This involved creating a new PinnedPool struct in Rust, modifying the bellperson library's ProvingAssignment to support pinned backing, and wiring the pool through the entire synthesis pipeline — nine call sites in pipeline.rs, five dispatch sites in engine.rs, and the evictor callback for memory pressure handling. The code compiled cleanly, and a Docker image tagged cuzk-rebuild:pinned1 was built (message 3198). But compilation is only half the battle.

Why This Message Was Written

The message at index 3205 exists because of a fundamental architectural constraint: the target deployment environment is a Docker container running on a remote machine with an overlay filesystem. The team had learned from painful prior experience that deploying binaries directly to /usr/local/bin/ on this system caused overlay filesystem issues. The safe path was /data/. But the binary was inside a Docker image on the build machine, not on the remote. The extraction command bridges these two worlds.

The choice of the docker create + docker cp + docker rm pattern rather than docker export or docker save reveals deliberate decision-making. docker create instantiates a container without running it, which is perfect for extracting artifacts — it avoids any risk of the container executing unintended code or modifying files. The --name pinned1-extract flag gives the container a human-readable name for cleanup. The command /cuzk passed to docker create is the container's entry point command, but since the container is never started, it's merely a placeholder. The docker cp command then copies the binary from the container's filesystem to the host's /tmp/ directory. Finally, docker rm pinned1-extract removes the stopped container, leaving no trace.

The output confirms success: the container ID (d8d8f710cb7259b9f1db4dadefe5b347000608268f7ddf1b63b1eb3ddd1ba7c1) and the container name (pinned1-extract) are printed, indicating the binary now resides at /tmp/cuzk-pinned1 on the build host, ready for the next step — scp to the remote test machine.

Assumptions and Implicit Knowledge

This message makes several assumptions that are invisible without context. First, it assumes the Docker image cuzk-rebuild:pinned1 exists and contains a binary at the path /cuzk. This assumption was validated in the preceding messages: message 3199 confirmed the image was built (27.7MB 2026-03-13 20:24:30 +0100 CET), and the Dockerfile used to build it was known to produce a static binary at that path. Second, it assumes the build host has Docker installed and the docker command is available — a reasonable assumption given the development environment. Third, it assumes the binary extraction is safe to perform without stopping any running services, which is true because docker create does not execute the container.

A subtle but important assumption is that the binary extracted this way is functionally identical to what would run inside the container. Since the cuzk binary is compiled as a statically linked executable (a common pattern for CUDA applications that link against the CUDA runtime), the filesystem context of the container is irrelevant — the binary carries its own dependencies. This assumption proved correct in subsequent testing.

The message also implicitly assumes that the extraction step is worth doing before deployment. An alternative approach would have been to push the entire Docker image to the remote machine and run it there. But the remote machine's overlay filesystem constraints and the desire to run the binary directly (not inside a container) made extraction the right choice. The team had already established this pattern in earlier deployments — the binary at /data/cuzk-timing2 was extracted the same way.

What This Message Creates

The output knowledge created by this message is tangible: a binary file at /tmp/cuzk-pinned1 on the build host. But more importantly, this message creates the possibility of testing. Before this command, the pinned memory pool existed only as source code and a Docker image. After this command, it exists as an executable artifact that can be deployed, run, and measured. The subsequent steps — scp to the remote machine, stopping the old daemon, waiting for pinned memory to free, and starting the new binary — all depend on this extraction.

The message also creates operational confidence. The clean output (no errors, no warnings) confirms that the Docker image is well-formed and the binary is accessible. In a debugging session where every hypothesis must be tested against real hardware, this confidence is invaluable. The team can now proceed to the critical question: does the pinned memory pool actually fix the GPU utilization problem?

The Thinking Process

The reasoning behind this message is visible in the sequence of commands that precede it. In message 3203, the assistant checked git status to confirm which files were modified, then listed Docker images to verify the build succeeded. In message 3204, the assistant created a todo list with "Extract pinned1 binary from Docker image" as the first high-priority item. The extraction command is the execution of that todo.

The choice to chain three Docker commands with && rather than running them separately reveals an understanding of atomicity. If any command fails, the chain stops — docker create must succeed before docker cp runs, and docker cp must succeed before docker rm cleans up. This prevents orphaned containers if the copy fails. The container name pinned1-extract is chosen to be descriptive and unique, avoiding conflicts with any other extraction containers.

The assistant also chose to run this as a single bash invocation rather than splitting it into multiple tool calls. This suggests a desire for speed and simplicity — the extraction is a well-understood mechanical step that doesn't require intermediate inspection. The output is minimal: just the container ID and name. No verbose flags, no progress bars. This is a professional who knows exactly what they need and trusts the tools.

Broader Significance

In the arc of the optimization effort, this message is the pivot point between theory and experiment. The preceding hours were spent designing, implementing, and compiling the pinned memory pool — intellectual work that could be done entirely in the development environment. The following hours would be spent deploying, monitoring, and debugging — empirical work that requires the real system. The extraction command is the handoff between these two modes.

The subsequent testing would reveal that the pinned pool had critical flaws: budget integration caused silent fallback to heap allocations, PCE caching was blocked by budget exhaustion, and a dispatch burst problem caused thundering herd cudaHostAlloc calls. These issues would be diagnosed and fixed in the following chunks (pinned2, pinned3, pinned4), ultimately achieving near-zero H2D transfer times and full GPU utilization. But none of that debugging could begin without this extraction step.

This message is a reminder that in complex engineering work, the unglamorous operational steps — extracting a binary from a Docker image, copying it to a remote machine, restarting a daemon — are as essential as the elegant algorithmic solutions. The pinned memory pool was a clever design, but it was useless until it ran on actual hardware. Message 3205 is the moment it got its chance.