The Docker Volume Mount Gambit: Extracting a Rebuilt Curio Binary from a Transient Container

Introduction

In the course of debugging a distributed Filecoin proving system, a developer working on the ProofShare subsystem had just fixed two critical production bugs—a deadlock caused by HTTP 429 retries in TaskRequestProofs and a cuzk job ID collision that mixed partition proofs across concurrent challenges. With the code changes committed and verified locally, the next step was straightforward in principle: build the updated Curio binary and deploy it to the remote GPU worker host. In practice, this simple deployment step would reveal a cascade of assumptions about build environments, Docker caching, and container lifecycle management, culminating in the message we examine here—message index 1950—where the assistant recovers from a self-inflicted extraction problem by pivoting to a volume-mount strategy.

This article examines that single message in depth, unpacking the reasoning, the mistakes, the context required to understand it, and the knowledge it produced. The message itself is short—a few lines of agent reasoning and a single bash command—but it sits at the intersection of several complex systems: multi-stage Docker builds, Go cross-compilation with CUDA dependencies, and the operational workflow of patching production GPU workers without full image rebuilds.

Context: The Road to This Message

To understand message 1950, we must first understand what led to it. The assistant had just completed a series of edits to three Go source files in the Curio project (task_request.go, task_prove.go, and provictl.go) to fix two production bugs in the ProofShare system. The first bug was a deadlock: CreateWorkAsk would retry HTTP 429 responses indefinitely, blocking the poll loop from discovering matched work and inserting it into the proofshare_queue. The fix introduced an ErrTooManyRequests sentinel error and made CreateWorkAsk return it immediately on 429, with the caller (the Do() method in task_request.go) handling progress-based exponential backoff. The second bug was a job ID collision: because ProofShare challenges all targeted the same miner and sector (miner=1000, sector=1), concurrent tasks sent identical job_id values to the cuzk GPU proving engine, whose partition assembler keyed on job_id, causing partition results from different proofs to be mixed together. The fix added the harmony task ID to the RequestId format string, making it unique per invocation.

With the code changes passing go build and go vet cleanly, the user issued the instruction: "build curio and send updated binary to the vast host" ([msg 1934]). The assistant's initial attempt was straightforward: build the binary locally using make curio, then scp it to the remote host at ssh -p 40362 root@141.195.21.72. This worked—the binary was built and uploaded—but when the assistant tried to swap it in place on the remote host, the binary failed to run:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

This error revealed a critical assumption: the locally built binary was dynamically linked against libraries (specifically libconfig++.so.15 from the supraseal dependencies) that were not present on the remote host. The remote host was a GPU worker running inside a Docker container built from Dockerfile.cuzk, which included all the CUDA and supraseal runtime libraries. The binary needed to be built inside that same Docker environment to link against the correct shared libraries.

The user clarified: "it is also a curio node, maybe you need to docker-build (Dockerfile.cuzk) and send the binary from docker build" ([msg 1944]).

The Docker Build Attempt and Cache Problem

The assistant then pivoted to building inside Docker. It read the Dockerfile.cuzk to understand the multi-stage build structure ([msg 1945]), then ran docker build -f Dockerfile.cuzk --target builder -t curio-builder:latest . to build the builder stage image ([msg 1946]). However, this produced a cached binary from before the code changes—the Docker build cache had not invalidated because the COPY . . layer was still considered fresh. The assistant recognized this: "The builder stage used a cached curio binary from before my changes" ([msg 1947]).

Rather than rebuilding the entire Docker image from scratch (which would be time-consuming), the assistant devised a clever workaround: create a container from the builder image, copy the modified source files into it using docker cp, then run a build command inside a container that mounts the builder's volumes via --volumes-from. This approach would reuse the cached dependencies and compilation artifacts while incorporating only the changed source files.

The assistant executed this plan in message 1948, copying the three modified files into the container at /build/tasks/proofshare/task_request.go, /build/tasks/proofshare/task_prove.go, and /build/lib/proofsvc/provictl.go. Then in message 1949, it ran:

docker run --rm \
  --volumes-from curio-rebuild \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /tmp/curio-new -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix" \
      ./cmd/curio && ls -lh /tmp/curio-new
  '

This command built the binary successfully—the output showed it downloading dependencies and compiling. But there was a fatal flaw: the --rm flag meant the container would be automatically removed after the command finished, and the binary was written to /tmp/curio-new inside the container. The assistant realized this only after the fact.

The Subject Message: Recognizing the Extraction Problem

Message 1950 opens with the assistant's reasoning:

Hmm, the binary was built but I'm not sure I can get it out of the container since I used --rm. Let me try a different approach - build it and then copy it out. Actually --rm removes the container after it exits, so I need a different approach.

>

Let me use a non-rm container or mount a volume.

>

The binary was built but the container was removed. Let me redo with a volume mount to extract it:

This reasoning is the heart of the message. It reveals a moment of realization: the assistant had built the binary to a path inside a container that no longer existed. The --rm flag, which is typically used to keep the Docker environment clean by automatically removing containers after they exit, had become a liability. The binary was compiled, linked, and sitting at /tmp/curio-new—but that filesystem was ephemeral, tied to the container's lifecycle, and gone with the docker run --rm exit.

The assistant's thinking shows a clear diagnostic process:

  1. Observation: The binary was built, but it's trapped inside a removed container.
  2. Root cause identification: The --rm flag caused automatic container removal.
  3. Solution brainstorming: The assistant considers two options—using a non---rm container (which would persist after the build command exits, allowing a subsequent docker cp to extract the binary) or mounting a volume (-v) to write the binary directly to the host filesystem.
  4. Decision: The assistant chooses the volume mount approach, which is more elegant because it avoids a two-step process (build, then copy) and writes the binary directly to the host in one shot. The command that follows implements this decision:
docker run --rm \
  --volumes-from curio-rebuild \
  -v /tmp:/output \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /output/curio-psfix -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix" \
      ./cmd/curio && ls -lh /output/curio-psfix
  '

Note the key differences from the previous attempt:

Analysis of Assumptions and Mistakes

This message reveals several assumptions and one clear mistake:

Mistake: Using --rm without a volume mount in the first build attempt. The assistant's initial Docker run command (message 1949) combined --rm with an output path inside the container's own filesystem (/tmp/curio-new). This was a classic container lifecycle error: assuming the binary would be accessible after the container was removed. The --rm flag is convenient for ephemeral compute, but it destroys the filesystem along with the container. The assistant recognized this only after the fact, as the reasoning in message 1950 makes explicit.

Assumption: The --volumes-from approach would allow binary extraction. The assistant initially thought that using --volumes-from curio-rebuild (which mounts volumes from the previously created container) would somehow make the binary accessible. But --volumes-from mounts volumes from another container—it doesn't create a persistent volume that survives container removal. The binary written to /tmp/curio-new was on the container's own writable layer, not on any mounted volume.

Assumption: Docker build cache invalidation would handle source changes. Earlier, the assistant assumed that running docker build with the modified source files would automatically detect the changes and rebuild the curio binary. It didn't, because the Docker layer caching system operates at the granularity of Dockerfile instructions, not individual file changes within a COPY instruction. The assistant correctly identified this and pivoted to the --volumes-from approach instead of a full --no-cache rebuild.

Assumption: The remote host had the required shared libraries. The initial local build assumed that a statically linked or self-contained binary could be produced. In reality, the Curio binary with CUDA/supraseal support dynamically links against libraries like libconfig++.so.15 that are only available inside the Docker build environment. This assumption was corrected after the user pointed out the need for Docker-based building.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

  1. Docker container lifecycle: Understanding that --rm removes the container and its filesystem on exit, and that files written to a container's internal paths (like /tmp/curio-new) are lost unless written to a mounted volume.
  2. Docker volume mounts: The -v /host/path:/container/path syntax and how it creates a persistent bridge between host and container filesystems.
  3. --volumes-from: How this flag shares volumes from another container, and its role in the assistant's strategy to reuse the builder image's compilation cache.
  4. Multi-stage Docker builds: The concept of a "builder" stage that contains the full development toolchain (Go, Rust, CUDA, gcc) and produces binaries that are later copied into a minimal runtime stage.
  5. Go build with CGO and FFI: The environment variables FFI_BUILD_FROM_SOURCE, FFI_USE_CUDA, FFI_USE_CUDA_SUPRASEAL, CGO_LDFLAGS_ALLOW, and the -tags "cunative" flag, which control how the Go compiler links against native CUDA and Filecoin FFI libraries.
  6. The ProofShare system context: The two bugs being fixed (HTTP 429 deadlock and job ID collision) and why the Curio binary needs to be updated on the remote host.

Output Knowledge Created

This message produces several forms of knowledge:

  1. A deployable binary: The command produces /tmp/curio-psfix on the host, a 163MB Curio binary compiled with CUDA/supraseal support and containing the ProofShare bug fixes. The version string includes _psfix to distinguish it from previous builds.
  2. A reusable build technique: The combination of --volumes-from (to reuse cached compilation artifacts) with -v (to extract the output) is a pattern for incrementally rebuilding Go binaries inside Docker without full image rebuilds. This is particularly valuable in environments where the full Docker build takes a long time due to CUDA and Rust compilation.
  3. Validation of the build environment: The successful compilation confirms that the three modified Go source files are syntactically correct and compatible with the CUDA/supraseal build environment. Any type errors, missing imports, or API mismatches would have surfaced during this build.
  4. A corrected mental model: The assistant's reasoning demonstrates the importance of understanding Docker container ephemerality. The mistake and its correction serve as a concrete example of why output files must be written to mounted volumes, not container-internal paths, when using --rm.

The Thinking Process

The reasoning in message 1950 is particularly interesting because it shows a real-time debugging process. The assistant starts with uncertainty ("Hmm, the binary was built but I'm not sure I can get it out"), then walks through the logic:

  1. "I used --rm" — identifying the root cause.
  2. "Let me try a different approach" — deciding not to dwell on the mistake but to find a solution.
  3. "Build it and then copy it out" — considering a two-step approach (non---rm container, then docker cp).
  4. "Actually --rm removes the container after it exits, so I need a different approach" — confirming the understanding.
  5. "Let me use a non-rm container or mount a volume" — identifying two viable alternatives.
  6. "The binary was built but the container was removed. Let me redo with a volume mount to extract it" — settling on the volume mount approach as the cleaner solution. This thinking reveals a developer who is comfortable with Docker's semantics but made a simple oversight under the pressure of a multi-step deployment workflow. The correction is swift and precise, and the resulting command is an improvement over the original: by writing directly to /output (mapped to the host's /tmp), the binary is immediately available for the next step (uploading to the remote host) without an intermediate docker cp command.

Conclusion

Message 1950 is a small but instructive moment in a larger debugging and deployment saga. It captures the friction between the idealized workflow of "build and deploy" and the messy reality of heterogeneous build environments, Docker caching, and container lifecycle management. The assistant's mistake—using --rm without a volume mount—is a common one, and the correction demonstrates a pragmatic understanding of Docker's volume semantics. More broadly, this message illustrates that in complex distributed systems, even the deployment step is a minefield of assumptions: about library availability, build cache behavior, and the persistence of container filesystems. Each assumption that proves false is a learning opportunity, and the assistant's ability to recognize, articulate, and correct its mistake in a single reasoning block is a model of effective debugging under pressure.