The Build Cache Betrayal: When the Fix Isn't Really Fixed

"There it is — ps-porep-%d-%d with only TWO %d format specifiers. The Docker Go build cache didn't recompile task_prove.go."

This single sentence, uttered by an AI assistant in the middle of a high-stakes production debugging session, represents one of the most frustrating moments in any developer's life: the moment you realize that your carefully crafted fix never actually made it into the running binary. Message [msg 2002] captures the precise instant of discovery—a grep against a deployed binary revealing that the format string ps-porep-%d-%d remained unchanged, despite two rounds of Docker builds, file copies, and deployments. The message is a masterclass in operational debugging, build system archaeology, and the quiet horror of realizing that your deployment pipeline has been lying to you.

The Context: A Production Firefight

To understand why this message was written, we must first understand the fire it was trying to extinguish. The ProofShare system—a distributed proving pipeline for Filecoin's proof-of-replication (PoRep) protocol—was producing invalid proofs. Every partition of every proof was failing. The root cause, identified in the preceding messages ([msg 1978] through [msg 1996]), was a job ID collision: concurrent proofshare challenges all targeted the same miner (1000) and sector (1), generating identical job_id values of ps-porep-1000-1. The cuzk proving engine's partition assembler keyed its internal HashMap on job_id, so when multiple concurrent tasks submitted proofs for the same sector, their partition results collided and overwrote each other. The smoking gun was a panic message: "partition 0 already inserted".

The fix seemed straightforward: add the harmony task ID to the RequestId format string, changing fmt.Sprintf("ps-porep-%d-%d", miner, sector) to fmt.Sprintf("ps-porep-%d-%d-%d", miner, sector, taskID). This pattern was already used by the Snap computation path, so it was a known-good approach. The assistant edited the source file, built a new binary (psfix2) inside a Docker container using the project's curio-builder:latest image, copied the binary to the remote host, and deployed it. The version string confirmed _psfix2 was running.

And yet, the logs still showed ps-porep-1000-1.

The Moment of Discovery

Message [msg 2002] opens with the assistant's realization. The user had reported in [msg 1997] that the cuzk logs still showed the old job_id format, even after deployment. The assistant initially speculated that old jobs might still be queued in the cuzk pipeline ([msg 1998]), but the timestamps suggested these were new jobs submitted by the updated curio. A check of the binary's version string confirmed psfix2 was running ([msg 2000]), but when the assistant grepped the binary for the format string, the truth emerged: ps-porep-%d-%d with only two format specifiers. The fix had never been compiled in.

The message then documents the assistant's response: creating a new Docker container (curio-rebuild3) to attempt a proper rebuild. The tone is matter-of-fact, but the subtext is palpable—this is a developer who has just wasted hours deploying a non-fix, who must now go back to square one with a new understanding of the problem.

The Assumptions That Failed

This message exposes several critical assumptions that proved incorrect:

Assumption 1: docker cp into a --volumes-from container would trigger a recompile. The assistant had used docker cp to copy the modified source files into a container created with --volumes-from curio-rebuild2, then ran the build inside that container. The assumption was that Go's build system would detect the changed source file and recompile it. In reality, Go's build cache is sophisticated—it uses content hashes of source files, dependency analysis, and compiler output caching. But the Docker build cache layer added another level of indirection. The --volumes-from container shares volumes with the builder image, but the Go build cache (typically in /root/.cache/go-build or the module cache) persisted from previous builds. Go's compiler may have determined that the object file for task_prove.go was already up-to-date based on its cached hash, or the build system may have used a cached dependency graph that didn't reflect the file change.

Assumption 2: The version string _psfix2 guaranteed the fix was included. The version string was set via -X ldflags at build time, which injects a string into a Go variable. This is a linker operation that happens after compilation. The version string would be updated even if no Go source files were recompiled, because the ldflags are processed independently of the source code. This created a false positive: the binary reported _psfix2, but only the metadata had changed, not the actual proving logic.

Assumption 3: The build pipeline was deterministic and reproducible. The assistant assumed that running the same build commands with updated source files would produce a binary incorporating those changes. The Docker build system, however, introduced non-determinism through cached layers, volume mounts, and the Go build cache's incremental compilation logic.

The Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

Go build system internals: Understanding that Go uses content-addressable caching for compiled packages, that go build checks modification times and content hashes, and that the build cache can produce stale artifacts if not properly invalidated. The -tags "cunative" flag and CGO_LDFLAGS_ALLOW=".*" environment variable indicate this is a CGo project with native CUDA bindings, adding another layer of compilation complexity.

Docker build mechanics: The --volumes-from flag shares volumes from an existing container, but the Go build cache resides in the container's filesystem, not necessarily in the shared volumes. The docker cp command copies files into the container's filesystem at a specific path, but if the Go build cache has a pre-compiled object for the old version of the file, it may not recompile.

The cuzk proving engine architecture: Understanding that job_id is used as a key in the engine's JobTracker.assemblers HashMap, that concurrent jobs with the same key cause partition result collisions, and that the fix requires a unique identifier per invocation.

The ProofShare protocol: Knowing that proofshare challenges all target the same bench sector (miner=1000, sector=1) for testing purposes, making them vulnerable to ID collisions that wouldn't occur in production with real sectors.

The Output Knowledge Created

This message creates several important pieces of knowledge:

A confirmed bug in the deployment process: The Docker Go build cache does not reliably recompile files copied via docker cp into --volumes-from containers. This is a concrete, reproducible finding that changes the deployment strategy going forward.

A diagnostic technique: Using grep -ao against a binary to search for format strings is a quick way to verify that source-level changes were actually compiled in. This is a valuable operational debugging technique for any Go deployment.

A refined understanding of the problem space: The issue is not just the code fix itself, but the entire pipeline from source edit to running binary. The build system is part of the critical path and must be treated with the same rigor as the application code.

The Thinking Process Revealed

The reasoning visible in this message and its surrounding context shows a systematic debugging approach:

1. Observation: Logs show old format string despite deployment of new binary
2. Hypothesis generation: Old jobs still in queue? Binary not actually updated? Build cache issue?
3. Evidence gathering: Check version string (confirms psfix2), grep binary for format string (reveals old format)
4. Root cause identification: Build cache didn't recompile the changed file
5. Corrective action: Create new container for rebuild with fresh cache The assistant's thinking is methodical and evidence-driven. Rather than assuming the fix worked (which would be the easy, comforting conclusion), the assistant verified the binary's actual contents. This is the hallmark of a mature debugging practice: trust nothing, verify everything.

The Broader Lessons

Message [msg 2002] illustrates a universal truth about software engineering: the gap between "fixing the code" and "fixing the system" is often wider than we think. The code change was correct. The logic was sound. But the deployment pipeline—the Docker build cache, the Go incremental compilation, the volume mount semantics—introduced a failure mode that nullified the fix.

This is why continuous deployment pipelines include cache-busting mechanisms, why build systems have --no-cache flags, and why experienced developers know to verify not just that the build succeeded, but that the output actually contains the expected changes. The assistant's grep -ao against the binary is a form of integration testing at the binary level—a practice more teams should adopt.

The message also demonstrates the importance of persistence in debugging. When the user reported that the logs still showed the old format, the assistant didn't dismiss it as a transient issue or blame the reporting tool. Instead, the assistant dug deeper, checking the binary directly, and found the real problem. This willingness to question one's own work—to assume the fix might not have taken effect—is a crucial trait for any engineer working on production systems.

Conclusion

Message [msg 2002] is a small moment in a larger debugging saga, but it encapsulates a profound lesson: the build system is not a black box. Every layer of the deployment pipeline—Docker, Go's compiler cache, volume mounts, linker flags—can introduce failures. The only defense is systematic verification at each step. The assistant's discovery that ps-porep-%d-%d still lurked in the binary, despite two deployment attempts, is a reminder that in distributed systems debugging, the most dangerous assumption is that your fix actually made it into production.