The Final Deployment: How a Three-Character Fix Survived Docker's Build Cache

Introduction

In the life of a distributed proving system, few moments carry as much weight as the one captured in message 2012 of this coding session. After hours of debugging, multiple failed build attempts, and a growing sense of frustration with Docker's opaque caching behavior, the assistant finally types:

Hashes match, ps-porep-%d-%d-%d confirmed, version _psfix3. Now start curio:

This is not a dramatic message. It contains no triumphant exclamation, no sigh of relief, no reflection on the journey that led here. It is clinical and matter-of-fact: a hash verification, a format string confirmation, a service start command. But beneath its brevity lies the resolution of a production outage that had been silently corrupting ProofShare proofs across all ten partitions of every challenge.

The message is the final link in a chain of reasoning that began with a "partition 0 already inserted" panic in the cuzk GPU proving engine and ended with a single character added to a format string—a %d that became %d-%d-%d instead of %d-%d. This article examines that message in depth: why it was written, the decisions it embodies, the assumptions it rests on, and the hard-won knowledge it represents.

The Context: A Job ID Collision in the Proving Pipeline

To understand why message 2012 matters, one must first understand the bug it fixes. The ProofShare system in the Filecoin Curio implementation allows storage providers to prove ownership of sectors they do not actually store, using a challenge-response protocol. When a challenge arrives, the system spawns concurrent tasks—one per partition—each of which invokes the cuzk GPU proving engine with a RequestId that serves as a job identifier.

The original code constructed this identifier as fmt.Sprintf("ps-porep-%d-%d", miner, sector). In production, however, all ProofShare challenges target a single hardcoded bench sector: miner=1000, sector=1. Every concurrent task therefore sent the identical job ID "ps-porep-1000-1" to cuzk. The engine's partition assembler, which keys its internal state on job_id, could not distinguish between tasks. Partition results from different proofs were mixed together, producing invalid proofs that passed cuzk's own internal checks but would fail verification on-chain. The symptom was a "partition 0 already inserted" panic—a clear signal that the HashMap keyed on job_id was being overwritten by concurrent writes.

The fix was deceptively simple: add the harmony task ID to the format string, producing "ps-porep-%d-%d-%d". This single change, made in task_prove.go:338, ensured that each concurrent task had a unique job identifier, eliminating the collision at its root.

The Deployment Ordeal: Docker's Go Build Cache

What should have been a routine rebuild and deployment turned into a multi-round ordeal that consumed the better part of the session. The assistant's first attempt used Docker's --volumes-from mechanism to share source files from a data container into the builder container, combined with touch to bust the Go build cache. Neither worked.

The root cause was subtle. The --volumes-from flag only mounts volumes declared in the Dockerfile with the VOLUME directive. The curio-builder image had no such directive for /build, so the files copied via docker cp into the data container's writable layer were invisible to the build container. The touch command, applied to files that were never actually mounted, had no effect. The Go compiler, seeing no changes to the package hashes, served the cached object files from a previous build. The resulting binary still contained the old format string ps-porep-%d-%d.

The assistant discovered this failure mode by grepping the binary's strings on the remote host. The command grep -ao "ps-porep-%d-%d[^\"]*" /usr/local/bin/curio returned the old two-%d format, confirming that the build had not incorporated the change. This diagnostic step—searching for the exact format string in the compiled binary—became the assistant's primary verification technique, repeated after every build attempt.

The breakthrough came when the assistant abandoned --volumes-from in favor of direct bind mounts with the -v flag:

docker run --rm \
  -v /tmp:/output \
  -v /tmp/czk/tasks/proofshare/task_prove.go:/build/tasks/proofshare/task_prove.go \
  -v /tmp/czk/tasks/proofshare/task_request.go:/build/tasks/proofshare/task_request.go \
  -v /tmp/czk/lib/proofsvc/provictl.go:/build/lib/proofsvc/provictl.go \
  ...

Bind mounts replace the file in the container's filesystem directly, forcing the Go compiler to see the modified timestamps and content hashes. This time, grep -ao "ps-porep-%d-%d[^\"]*" on the output binary returned ps-porep-%d-%d-%d—three format specifiers, confirming the fix was compiled in.## The Deployment That Almost Wasn't

Even after the correct binary was built, deployment nearly failed a second time. The assistant's first attempt to deploy used a chained command: kill $(pgrep -f "curio run") 2>/dev/null; sleep 2; mv /tmp/curio-psfix3 /usr/local/bin/curio. This command produced no output, and when the assistant checked the running binary, it still showed version _psfix2 with the old two-%d format string. The mv had failed silently.

The most likely cause was that the running curio process still held a file descriptor on /usr/local/bin/curio, preventing the mv from replacing it. The chained command killed the process but immediately attempted the move before the kernel had released the file's inode. On Linux, replacing a running executable is possible (the old inode remains live until all file handles are closed), but the timing is delicate. The sleep 2 should have been sufficient, but the assistant's subsequent investigation revealed that a new curio process had already started (PID 210085), suggesting the process had respawned or the kill had not taken effect before the move.

The assistant learned from this failure. In message 2012, the deployment is done step by step:

  1. Kill first, verify separately: The kill command is issued in one SSH session, followed by a sleep 3 and a pgrep check to confirm the process is truly stopped.
  2. Copy, don't move: The replacement uses cp /tmp/curio-psfix3 /usr/local/bin/curio instead of mv, which is more reliable when the destination is a live filesystem path.
  3. Verify the binary on disk: Before starting the service, the assistant checks md5sum of both the source and destination to confirm they match, then greps the format string to confirm the fix is present, and checks curio --version to see the _psfix3 tag.
  4. Start with nohup: The service is launched with nohup and redirected to a log file, ensuring it survives the SSH session's termination. This step-by-step approach reflects a deeper understanding: in distributed systems debugging, each step must be independently verifiable. A chained command that "should work" can fail silently, and the only way to catch that failure is to check the state at each stage.

The Verification: What Message 2012 Actually Confirms

When the assistant writes "Hashes match, ps-porep-%d-%d-%d confirmed, version _psfix3", it is reporting three independent verification checks:

  1. Hash match: The MD5 checksum of /usr/local/bin/curio matches that of /tmp/curio-psfix3, confirming the copy was complete and uncorrupted.
  2. Format string grep: The command grep -ao "ps-porep-%d-%d[^ ]*" /usr/local/bin/curio returns ps-porep-%d-%d-%dinvalid. The grep pattern was designed to match the old two-%d format; the fact that it returns the three-%d version confirms the fix is compiled into the binary. (The trailing invalid is a false positive from the grep's binary search picking up adjacent text—a common artifact of grepping compiled binaries.)
  3. Version tag: The curio --version output shows _psfix3, confirming this is the latest build, not a stale binary. These three checks together provide high confidence that the deployed binary contains the fix. The assistant has learned from earlier failures that a version tag alone is insufficient—the _psfix2 binary was confirmed running but still contained the old format string because the Go build cache had not been busted. Only by examining the actual compiled strings can one be certain the change was compiled in.

The Thinking Process: What the Assistant Assumed

The assistant's reasoning throughout this deployment reveals several assumptions, some validated and some corrected:

Assumption 1: --volumes-from shares the filesystem. This was incorrect. The assistant assumed that docker cp into a data container followed by --volumes-from would make those files visible to the build container. In reality, --volumes-from only mounts named volumes, not the container's writable layer. The correction came from reasoning about why touch had no effect: if the files weren't actually mounted, touching them was meaningless.

Assumption 2: touch busts the Go build cache. This is true in principle but irrelevant if the source files aren't visible to the compiler. The assistant correctly identified that the build cache issue was a symptom, not the root cause.

Assumption 3: A chained kill; mv will reliably replace a running binary. This failed because the process may not have released the file handle in time, or the process respawned before the move. The correction was to separate the steps and verify each one.

Assumption 4: The format string grep is a reliable indicator of correctness. This is validated. The assistant used this technique repeatedly and it correctly distinguished between the old and new binaries. The only caveat is that grep on binary files can produce false positives from adjacent strings, but the core pattern match is trustworthy.

Input and Output Knowledge

To understand message 2012, the reader needs several pieces of input knowledge:

Conclusion

Message 2012 is a study in operational discipline. It represents the culmination of a debugging session that spanned multiple subsystems—Go Curio, Rust cuzk, Docker build infrastructure, and remote SSH deployment—and required the assistant to develop a theory of the bug, implement a fix, navigate Docker's caching behavior, and deploy with rigorous verification. The message itself is only 45 words, but it encodes hours of debugging, three build attempts, two failed deployments, and a growing understanding of how distributed proving systems can fail in unexpected ways.

The most important lesson is that in production systems, verification is not optional. The assistant could have deployed the _psfix2 binary and assumed it was correct based on the version tag alone. That would have left the bug unfixed and invalid proofs flowing through the pipeline. Instead, the assistant grepped the binary, found the old format string, and persisted through multiple rebuilds until the fix was genuinely compiled in. That persistence—and the methodological rigor that supported it—is what makes message 2012 more than just a deployment log entry. It is a testament to the discipline required to maintain distributed proving systems in production.