The Third Attempt: When Docker Build Caches Defeat a Production Fix

In the high-stakes world of distributed GPU proving for Filecoin, a single character can determine whether ten concurrent proof partitions produce valid results or silently corrupt each other's data. This article examines a single message from an opencode coding session — message index 2003 — where an AI assistant copies three Go source files into a Docker container for the third time, attempting to deploy a critical fix for a proof-collision bug that was causing all ProofShare partitions to produce invalid proofs.

The Message

[assistant] [bash] docker cp /tmp/czk/tasks/proofshare/task_request.go curio-rebuild3:/build/tasks/proofshare/task_request.go && docker cp /tmp/czk/tasks/proofshare/task_prove.go curio-rebuild3:/build/tasks/proofshare/task_prove.go && docker cp /tmp/czk/lib/proofsvc/provictl.go curio-rebuild3:/build/lib/proofsvc/provictl.go && echo "copied"
copied

On its surface, this is a mundane command: copy three modified files into a Docker container named curio-rebuild3. But this message sits at a critical inflection point in a debugging saga that had already consumed hours of investigation, multiple rebuilds, and two failed deployments. Understanding why this seemingly redundant operation was necessary reveals deep truths about the friction of patching production GPU-proving systems, the subtle treachery of Docker build caches, and the persistence required to ship fixes to remote hardware.

The Context: A Bug That Corrupts Proofs

The story begins with the ProofShare system, a component of the Curio Filecoin proving infrastructure that allows multiple parties to share GPU resources for generating proofs-of-replication (PoReps). The system uses a CUZK proving engine — a Rust-based GPU acceleration layer — to handle the computationally intensive work of generating zero-knowledge proofs for Filecoin storage verification.

The bug was devastating: when multiple ProofShare challenges ran concurrently (which is the normal operating mode), every single PoRep partition produced an invalid proof. The assistant had traced this to a job ID collision. The RequestId sent to the CUZK engine was formatted as ps-porep-%d-%d using the miner ID and sector number. But in the ProofShare system, all challenges target the same synthetic sector (miner=1000, sector=1), meaning every concurrent task sent the identical job_id of ps-porep-1000-1 to the CUZK engine. The engine's partition assembler keyed on this job_id in a HashMap, so partition results from different proofs would overwrite each other — a fact confirmed by the telltale panic message "partition 0 already inserted".

The fix was straightforward: include the harmony task ID in the RequestId, changing the format string to ps-porep-%d-%d-%d. This would make every concurrent invocation unique. The assistant had already made this change in the Go source code, verified it compiled locally, and deployed a rebuilt binary (curio-psfix2) to the remote host.

The Discovery That Everything Was Wrong

After deploying psfix2, the user reported that the CUZK logs still showed ps-porep-1000-1 — the old format without the task ID. The assistant checked the running binary using grep -ao to search for the format string, and found ps-porep-%d-%d with only two %d specifiers. The fix had not been compiled into the binary at all.

This was the moment captured in message 2003. The assistant had just discovered that the Docker build process had silently ignored the source file changes. The Go build cache, preserved inside the Docker image layers, had served up the old compiled object files, producing a binary identical to the first build. The docker cp command had placed the modified files into the container, but the build process never actually recompiled them.

The Reasoning Behind the Third Attempt

Message 2003 represents the assistant's first response to this discovery: create a new container, copy the files again, and try the same approach with renewed hope. The reasoning, visible in the surrounding context, follows a natural but ultimately flawed logic:

  1. The files were modified locally — the assistant had edited task_prove.go, task_request.go, and provictl.go with the fix.
  2. The local build succeeded — go build ./tasks/proofshare/ compiled cleanly, confirming the code was syntactically correct.
  3. The previous Docker build didn't pick up the changes — perhaps the container was stale, or the docker cp didn't work correctly the first time.
  4. A fresh container will fix it — create curio-rebuild3, copy the files again, and rebuild. The assistant creates a new container from the curio-builder:latest image, copies the same three files, and prepares to rebuild. The container name curio-rebuild3 (the third rebuild attempt) tells its own story of iterative deployment.

The Critical Assumption

The fundamental assumption underlying this message is that docker cp into a container, followed by docker run --volumes-from to mount that container's filesystem, would make the modified source files available to the build process. This assumption had already failed once (producing psfix2 without the fix), but the root cause wasn't yet understood.

The assistant assumed the problem was a stale container or a copy error, rather than a fundamental architectural incompatibility between docker cp and --volumes-from. The --volumes-from flag only mounts volumes that were explicitly declared with the VOLUME directive in the Dockerfile, not the entire container filesystem. The docker cp command places files into the container's writable layer, but --volumes-from doesn't expose that layer — it only shares named volumes. Since the curio-builder image likely doesn't have a VOLUME directive for /build, the copied files were invisible to the build container.

This is a subtle but critical distinction in Docker's architecture. The assistant would discover this in the next message (2004), when even adding touch to bust the Go build cache failed to produce the correct binary. The breakthrough came in message 2005, when the assistant switched to direct bind mounts (-v /tmp/czk/tasks/proofshare/task_prove.go:/build/tasks/proofshare/task_prove.go), which forced the Go compiler to see the modified source files and triggered a full recompile.

Input Knowledge Required

To understand this message, a reader needs familiarity with:

Output Knowledge Created

This message produces a Docker container (curio-rebuild3) with three modified source files at the correct paths. This container is the input to the next build step. However, because the --volumes-from approach doesn't actually expose these files, the output is effectively useless — a fact that becomes apparent only in the subsequent message.

The real output knowledge is negative: the assistant learns that docker cp + --volumes-from is insufficient for modifying Go source files in this build pipeline. This negative result drives the pivot to bind mounts, which ultimately succeeds.

The Thinking Process

The assistant's reasoning, visible in the surrounding messages, follows a systematic debugging process:

  1. Observation: The deployed binary still shows the old format string.
  2. Hypothesis 1: The binary wasn't replaced during deployment. Rejected — the version string showed _psfix2.
  3. Hypothesis 2: The Go build cache prevented recompilation. Accepted — leads to the third rebuild attempt.
  4. Hypothesis 3 (emerging in msg 2004): The --volumes-from approach doesn't share the modified files. Confirmed — leads to the bind mount solution. The assistant's thinking is methodical but initially constrained by an incorrect mental model of Docker's --volumes-from behavior. This is a classic debugging pitfall: when a fix doesn't work, the natural instinct is to try harder with the same approach rather than questioning the fundamental mechanism.

The Broader Significance

This message, for all its apparent simplicity, encapsulates a universal challenge in production systems engineering: the gap between a correct code change and a correctly deployed binary. The fix itself was trivial — adding %d to a format string — but getting that fix into the running binary required navigating Docker's layered filesystem architecture, Go's compilation caching, and the operational complexity of remote GPU-proving infrastructure.

The assistant's persistence through three rebuild attempts, each revealing a new layer of the problem, demonstrates the tenacity required for production debugging. The eventual solution — using direct bind mounts to bypass the volume-sharing issue — was conceptually simple but required the assistant to question and revise its understanding of how Docker containers share files.

In the end, the fix was deployed as curio-psfix3, confirmed by both the version string (_psfix3) and the correct format string (ps-porep-%d-%d-%d) found via grep -ao in the binary. The concurrent ProofShare challenges would finally receive unique job IDs, and the partition assembler would no longer mix results from different proofs. But the journey to that deployment ran through message 2003 — the third attempt, the moment of renewed effort before the breakthrough.