The Final Cleanup: Why `docker rm curio-rebuild` Tells a Deeper Story

At first glance, message <msg id=1954> appears to be the most mundane of operations — a simple Docker container removal:

[assistant] [bash] docker rm curio-rebuild 2>&1 curio-rebuild

A single line. A container deleted. But this message is not the beginning of a story; it is the end of one. It is the final, quiet exhale after a long and winding journey through production debugging, multi-stage Docker builds, cache invalidation battles, and a remote deployment to a GPU-proving worker on the Filecoin network. To understand why this message was written, one must understand the entire saga that preceded it — a saga that reveals deep truths about the friction of patching distributed systems in production.

The Context: Production Bugs in the ProofShare System

The assistant had been deep in the trenches of the ProofShare system, a component of the Curio Filecoin proving infrastructure. Two critical production bugs had been identified and fixed in the preceding messages. First, a deadlock in TaskRequestProofs where CreateWorkAsk retried HTTP 429 responses indefinitely, blocking the poll loop from discovering matched work and inserting it into proofshare_queue. Second, a cuzk job ID collision where concurrent proofshare challenges all targeted the same miner and sector, causing partition results from different proofs to mix — a bug confirmed by a "partition 0 already inserted" panic in the Rust engine.

These fixes were made to Go source files: task_request.go, task_prove.go, and provictl.go. But fixes on disk are not fixes in production. The code needed to be compiled into a Curio binary and deployed to the remote vast.ai GPU worker host at 141.195.21.72.

The First Attempt: A Lesson in Shared Libraries

The assistant's first instinct was straightforward: build locally and upload via scp. The local build succeeded, producing a 163 MB binary. But when the assistant attempted to verify the binary on the remote host, it encountered a stark failure:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

This error is a classic systems integration failure. The Curio binary links against the supraseal library, which in turn depends on libconfig++. These libraries are available inside the Docker build environment (defined in Dockerfile.cuzk) but not on the host system where the assistant compiled the binary. The binary was built in the wrong environment — it was linked against shared objects that did not exist on the target machine.

The Docker Build: Cache Invalidation Strikes

The user then guided the assistant to build inside Docker using Dockerfile.cuzk. The assistant ran docker build -f Dockerfile.cuzk --target builder -t curio-builder:latest ., expecting the source changes to be picked up. But Docker's layer caching intervened. The COPY . . instruction in the Dockerfile had already been executed in a previous build, and since the Docker build context hadn't changed at the file level (the source modifications were made to files already present in the context), Docker reused the cached layer — including the cached curio binary from an earlier build.

This is a subtle but important point: Docker's layer caching operates on the content of the build context at the time docker build is invoked. If the source files were modified after the previous build, a new docker build would normally detect the changes and invalidate the cache. But the assistant's workflow involved modifying files inside a running container (via docker cp), not in the build context on the host. The Docker build system had no way of knowing the files had changed.

The Creative Workaround: Injecting Source into a Container

Rather than rebuilding the entire Docker image from scratch with --no-cache (which would have been time-consuming and wasteful), the assistant devised a clever workaround. First, it created a container from the builder image without running anything:

docker create --name curio-rebuild curio-builder:latest /bin/true

This created a stopped container named curio-rebuild with all the builder image's filesystem contents. Then, the assistant used docker cp to overwrite the three modified source files inside the container:

docker cp /tmp/czk/tasks/proofshare/task_request.go curio-rebuild:/build/tasks/proofshare/task_request.go
docker cp /tmp/czk/tasks/proofshare/task_prove.go curio-rebuild:/build/tasks/proofshare/task_prove.go
docker cp /tmp/czk/lib/proofsvc/provictl.go curio-rebuild:/build/lib/proofsvc/provictl.go

This is a powerful technique: using a container as a mutable filesystem layer that can be modified independently of the image it was created from. The curio-rebuild container served as a temporary workspace — a sandbox where the assistant could inject the latest code without rebuilding the multi-gigabyte Docker image.

The Build and the Mistake

With the modified files in place, the assistant ran the Go build using --volumes-from curio-rebuild, which mounted the container's filesystem into a new ephemeral container:

docker run --rm \
  --volumes-from curio-rebuild \
  -w /build \
  curio-builder:latest \
  bash -c '... go build ... -o /tmp/curio-new ...'

The build succeeded. But the assistant had made a critical error: the --rm flag on docker run meant the container would be automatically removed after exit. The binary was built to /tmp/curio-new inside the ephemeral container — and when that container was removed, the binary was gone.

This is the kind of mistake that happens to everyone working with Docker. The --volumes-from flag mounts volumes from another container into the new container, but writes to those mounted paths go to the ephemeral container's own filesystem, not to the source container. The assistant realized this immediately in the reasoning block:

"Hmm, the binary was built but I'm not sure I can get it out of the container since I used --rm."

The Recovery

The assistant recovered by re-running the build with a host volume mount:

docker run --rm \
  --volumes-from curio-rebuild \
  -v /tmp:/output \
  -w /build \
  curio-builder:latest \
  bash -c '... go build ... -o /output/curio-psfix ...'

By mounting /tmp from the host into the container at /output, the assistant ensured the binary would survive container removal. The build completed, producing /tmp/curio-psfix on the host. The binary was then uploaded to the remote host via scp, swapped in place, and verified with the custom version string _psfix confirming the new code was compiled in.

The Subject Message: Cleanup

And then came message <msg id=1954>:

docker rm curio-rebuild 2>&1
curio-rebuild

This is the cleanup. The curio-rebuild container — the temporary workspace that held the modified source files — is no longer needed. It was created solely as a vehicle for --volumes-from to inject source changes into the build environment. Now that the binary has been built, deployed, and verified, the container can be removed.

The output curio-rebuild is simply Docker echoing the container name back, confirming successful removal. It is the digital equivalent of putting a tool back in the drawer after finishing a repair.

Why This Message Matters

This message is deceptively simple. On its surface, it is a throwaway cleanup command. But it represents the culmination of a complex, multi-step workflow that involved:

Root cause analysis of two production bugs (deadlock and job ID collision)
Code fixes across three Go source files
Build environment discovery (local build failed due to missing shared libraries)
Docker cache invalidation workaround (using docker create + docker cp + --volumes-from)
Build artifact extraction (recovering from the --rm mistake with volume mounts)
Remote deployment (SCP upload, binary swap, version verification)
Resource cleanup (removing the temporary container) The message also reveals the assistant's thinking about resource hygiene. In a production deployment pipeline, temporary artifacts should be cleaned up to avoid accumulating stale containers, consuming disk space, and causing confusion. The assistant did not leave the curio-rebuild container lying around — it removed it as soon as it was no longer needed.

Assumptions and Knowledge

To understand this message, one must understand Docker concepts: container lifecycle (docker create vs docker run), filesystem isolation, volume mounting (--volumes-from, -v), and layer caching in Docker builds. One must also understand the Go build toolchain, the concept of shared library linking, and the specific constraints of the Curio/cuzk project where FFI libraries (CUDA, supraseal) are required.

The assistant assumed that docker cp would successfully overwrite files in a stopped container (correct), that --volumes-from would make those files visible to a new container (correct), and that the Go build would pick up the modified source files (correct). The mistaken assumption was that --rm would not interfere with artifact extraction — a lesson relearned by every Docker user at some point.

Conclusion

Message <msg id=1954> is a single line of cleanup, but it is also a testament to the iterative, sometimes messy reality of deploying fixes to production systems. The path from "fix the bug" to "fix is running in production" is rarely straight. It involves build environments, cache invalidation, shared library compatibility, and the quiet satisfaction of cleaning up after a job well done. The docker rm curio-rebuild command is not the hero of this story — but it is the closing credits, and every good story needs those.

The Final Cleanup: Why docker rm curio-rebuild Tells a Deeper Story