The Moment the Build Cache Betrayed Us: A Post-Mortem of a Docker Go Build Failure in Production

Introduction

In the high-stakes world of distributed GPU proving for the Filecoin network, few moments are as deflating as deploying what you believe is a perfect fix, only to watch the logs prove you wrong. Message <msg id=2001> captures exactly such a moment. The assistant, having diagnosed a critical job-ID collision bug in the ProofShare system and deployed a rebuilt binary (psfix2) to a production GPU worker, receives confirmation from the user that the cuzk logs still show the old format string ps-porep-1000-1 — missing the task ID that was supposed to make concurrent proof challenges unique. What follows is a masterclass in debugging deployment pipelines: the assistant systematically rules out hypotheses, reaches for low-level binary inspection tools, and uncovers the uncomfortable truth that the Go build cache inside Docker had silently defeated the fix.

This message is the pivot point of the entire segment. It is not a triumphant "fix deployed" message, nor a confused "I don't know what's wrong" message. It is a precise diagnostic turn: the assistant confirms the binary identity (psfix2), confirms it has been running for 150 seconds, and then, with surgical clarity, deduces that the format string must not have been recompiled. The subsequent grep -ao command on the remote binary delivers the verdict: ps-porep-%d-%d with only two format specifiers. The build cache had won this round.

The Message in Full

The subject message reads:

Agent Reasoning OK so the binary is confirmed as psfix2 and has been running for 150 seconds. But the cuzk log still shows ps-porep-1000-1 without the taskID. This means either:

>

1. The Go build cache in Docker didn't actually recompile task_prove.go with our change 2. The docker cp into the volumes-from container didn't work as expected

>

Let me check the binary more carefully. Since strings isn't available, I can use grep -a to search for the format string in the binary. OK so the binary IS psfix2 but the format string might not have been recompiled due to Docker Go build cache. Let me check:

>

[bash] ssh -p 40362 root@141.195.21.72 'grep -ao "ps-porep-%d-%d[^\"]*" /usr/local/bin/curio | head -5' ps-porep-%d-%dRSealProvFinalCommR mismatchstorage-marketAggregateDealslabel_equalityDecodeSnap: %westimating gasinvalid metricdon't have keyApplyFunction...

The output is a firehose of binary noise — every string in the 163MB binary that happens to follow the pattern ps-porep-%d-%d — but the critical signal is clear: the format string has only two %d placeholders. The fix that added a third %d for the task ID is absent. The build cache had served the old object file.

Context: The Bug That Wasn't Fixed

To understand the weight of this discovery, we must trace the chain of events that led here. The ProofShare system is a distributed proving pipeline where GPU workers prove sector commitments on behalf of miners. Multiple concurrent "PSProve" tasks can be dispatched in parallel. The assistant had previously identified that all proofshare challenges target the same hardcoded sector (miner=1000, sector=1), meaning concurrent tasks generated identical job_id values (ps-porep-1000-1). The cuzk engine's partition assembler used this job_id as a HashMap key, causing partition results from different proofs to collide — confirmed by a "partition 0 already inserted" panic in the Rust engine.

The fix seemed straightforward: change the format string from fmt.Sprintf("ps-porep-%d-%d", miner, sector) to fmt.Sprintf("ps-porep-%d-%d-%d", miner, sector, taskID), matching the pattern already used by the Snap computation path. The assistant edited the Go source, built a new binary (psfix2) inside a Docker container using the --volumes-from approach, deployed it via SSH, and confirmed the version string showed _psfix2. The user then reported that the cuzk logs still showed ps-porep-1000-1. This is where <msg id=2001> begins.

The Reasoning Process: A Window Into Debugging Under Pressure

The assistant's reasoning in this message is exemplary in its structure. It begins by stating the confirmed facts: the binary is psfix2, it has been running for 150 seconds (confirmed by the user via the harmony task uptime metric), yet the cuzk log output still shows the old format. From these facts, the assistant generates exactly two hypotheses, both of which point to the same root cause category: the build process did not incorporate the source change.

Hypothesis 1 — "The Go build cache in Docker didn't actually recompile task_prove.go with our change" — is the more likely culprit. Go's build cache is aggressive: if the compiler determines that a package's source hasn't changed (based on content hash, not just mtime), it will reuse previously compiled object files. The docker cp command copies files into the container's writable layer, but the Go build cache from the original image build still contains compiled artifacts keyed to the original source content hash. Unless the copied file's content actually differs from what the cache knows, the build will skip recompilation.

Hypothesis 2 — "The docker cp into the volumes-from container didn't work as expected" — is a deployment plumbing concern. The --volumes-from flag only shares volumes declared in the Dockerfile via VOLUME directives, not the entire container filesystem. If the curio-builder image does not declare /build as a volume, then files copied into that path in the source container are invisible to the build container using --volumes-from. This is a subtle Docker semantics issue that has tripped up many engineers.

The assistant does not stop at hypothesis generation. It immediately moves to verification, noting that strings (the standard Unix binary string extraction tool) is not available on the remote machine, and adapts by using grep -ao (binary grep) to search for the format string pattern directly in the ELF binary. This is a pragmatic, resourceful choice — grep -a treats the binary as text, and -o outputs only the matching text. The regex pattern ps-porep-%d-%d[^\"]* is clever: it matches the known prefix and then captures everything up to the next double-quote character, which in Go's string representation would terminate the format string.

The Verdict: A Single Line of Evidence

The output of the grep command is devastatingly clear. The binary contains ps-porep-%d-%d followed by a jumble of unrelated strings from other parts of the binary — RSealProvFinalCommR, mismatch, storage-market, AggregateDeals, and so on. These are not part of the format string; they are simply the next bytes in the binary that happen to follow the matched pattern before a double-quote character appears. But the critical detail is the absence of a third %d. The format string has exactly two format specifiers. The fix did not make it into the binary.

This single line of evidence collapses both hypotheses into a single conclusion: the build process failed to incorporate the source change. The assistant now knows it must find a different way to force a recompile.

Assumptions Made and Broken

Several assumptions underlay the assistant's initial deployment strategy, and this message reveals where they failed.

Assumption 1: docker cp followed by --volumes-from provides the modified source to the build container. This assumption was incorrect. The --volumes-from flag in Docker does not share the entire filesystem of the source container; it only shares volumes explicitly declared with the VOLUME instruction in the Dockerfile. The curio-builder image likely does not declare /build as a volume, so the copied files were invisible to the build container. The build container saw the original, unmodified source files from the image layer.

Assumption 2: The Go build cache would be invalidated by the modified source file. Even if the source file had been visible, Go's build cache uses content hashing, not file modification time. If the content of the file as seen by the compiler is identical to the cached version, no recompilation occurs. The assistant later discovers (in subsequent messages <msg id=2005>) that switching to direct bind mounts (-v) forces a full recompile because the mount overlay changes the file identity from the compiler's perspective.

Assumption 3: The version string embedded in the binary (_psfix2) confirms the fix is present. The version string is set via a linker flag (-X github.com/filecoin-project/curio/build.CurrentCommit=+git_..._psfix2) and is independent of whether any source files were actually recompiled. The binary can have the new version string while still containing old compiled code for packages whose source did not change. This is a subtle but critical point: version strings reflect the build invocation, not the code content.

Assumption 4: The cuzk logs reflect requests from the new binary. The assistant initially considered that old jobs might still be in the cuzk pipeline from before the restart, but the 150-second uptime of the new binary, combined with the fact that PSProve tasks are short-lived, makes this unlikely. The logs are indeed from the new binary — which is why the discovery that the binary lacks the fix is so significant.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

Go build system internals: Understanding that Go's compiler uses content-based caching for package object files, and that go build will skip recompilation if the source content hash matches a cached entry. This is distinct from Makefile-style timestamp-based caching.

Docker filesystem semantics: The distinction between a container's writable layer (where docker cp writes), named volumes (declared via VOLUME in the Dockerfile), and bind mounts (specified with -v at runtime). The --volumes-from flag only shares named volumes, not the writable layer or the image layers.

ELF binary structure: The ability to search for format strings in a compiled binary using grep -ao, and understanding that the output will include arbitrary subsequent bytes until the pattern's termination condition (a double-quote) is met.

The ProofShare architecture: Understanding that multiple concurrent PSProve tasks target the same hardcoded sector (miner=1000, sector=1), that job_id is used as a HashMap key in the cuzk engine's partition assembler, and that uniqueness of this key is essential for correct proof isolation.

The deployment workflow: The sequence of building inside a Docker CUDA environment (required for GPU code), copying the binary out, SCP-ing to the remote host, and hot-swapping the running process.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The format string in the deployed binary is ps-porep-%d-%d (two specifiers), not ps-porep-%d-%d-%d (three specifiers). This is definitive evidence that the source change was not compiled in.
  2. The --volumes-from approach for Docker builds is unreliable for incremental source changes. The assistant's subsequent attempts (visible in <msg id=2002> through <msg id=2006>) show that even touch on the modified files inside the container does not help, and that switching to direct bind mounts (-v) is required.
  3. Version strings in Go binaries built with linker flags are not reliable indicators of code content. The binary reported _psfix2 but contained old code. This is a valuable lesson for CI/CD pipelines: version tags should be derived from source content, not build invocation metadata.
  4. grep -ao on a remote binary is a viable substitute for strings when the latter is unavailable. This is a practical debugging technique that can be applied in constrained environments.
  5. The build cache defeat pattern is reproducible and diagnosable. The assistant's method — check binary identity, check runtime behavior, inspect binary content for expected strings — forms a template for debugging similar deployment failures.

The Broader Significance

This message is a microcosm of the challenges inherent in patching production GPU-proving systems. Unlike a typical web service where you can git pull && make && systemctl restart, these systems require:

Conclusion

Message <msg id=2001> is the turning point in a debugging odyssey that spans Docker filesystem semantics, Go build cache internals, and the gritty reality of production GPU proving. It is the moment when the assistant realizes that a fix can be "deployed" without being "compiled" — that the build pipeline itself is a source of failure. The subsequent messages show the assistant iterating through touch, --volumes-from, and finally direct bind mounts to force a genuine recompile, culminating in a binary with the correct ps-porep-%d-%d-%d format string.

For any engineer who has ever pushed a hotfix, watched it fail, and discovered that the build server cached its way around the change, this message will feel painfully familiar. It is a reminder that in complex distributed systems, the deployment pipeline is not a neutral conduit — it is an active participant in the correctness of the system, and it can betray you in ways that no amount of version-string checking will catch.