The Touch That Fixed It: Beating Go's Build Cache in a Docker GPU-Proving Pipeline

Introduction

In distributed systems debugging, the most frustrating class of bug is the one that should have been fixed but stubbornly persists. The developer applies the correct patch, deploys the new binary, verifies the version string, restarts the service—and the logs still show the old behavior. This is the story of exactly such a moment in the opencode session: a production fix that was logically correct but was silently neutralized by Go's build cache inside a Docker container. The subject message, <msg id=2004>, represents the third attempt to deploy a critical uniqueness fix for GPU proof generation, and the key insight that finally broke the stalemate was a humble touch command.

The Message

The assistant issued a single bash command:

docker run --rm \
  --volumes-from curio-rebuild3 \
  -v /tmp:/output \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"

    # Touch the changed files to bust Go build cache
    touch tasks/proofshare/task_prove.go tasks/proofshare/task_request.go lib/proofsvc/provictl.go

    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /output/curio-psfix3 -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix3" \
      ./cmd/curio && \
    grep -ao "ps-porep-%d-%d[^\"]*" /output/curio-psfix3 | head -3
  ' 2>&1 | tail -10

The output shows only the tail of the build log—Go module downloads scrolling past—but the critical detail is the grep command appended after the build, designed to verify that the format string ps-porep-%d-%d (the old, broken format with only two %d placeholders) does not appear in the output binary. This is a self-verifying build: compile, then immediately check that the fix actually took effect.

The Reasoning and Context

To understand why this message exists, we must trace the chain of events that led to it. The production system was a Filecoin Curio node running GPU-accelerated proof generation via the CuZK engine. A feature called ProofShare allowed multiple concurrent proof challenges against the same sector (miner=1000, sector=1). Each challenge was a different random seed, but they all shared the same job_id because the request ID was formatted as ps-porep-%d-%d using only the miner and sector numbers.

The CuZK engine's partition assembler used this job_id as a key in a HashMap. When multiple concurrent proofs submitted their partition results with the same key, the partitions from different proofs overwrote each other, producing corrupted proofs. The symptom was a panic: "partition 0 already inserted". The fix was straightforward: include the harmony task ID in the request ID, making it ps-porep-%d-%d-%d (miner, sector, taskID).

The assistant implemented this fix in <msg id=1980>–<msg id=1983>, changed the format string in task_prove.go, and built a new binary (psfix2) inside a Docker container using --volumes-from to access the source files. The binary was deployed, the version string confirmed as _psfix2, and the service restarted.

Yet the cuzk logs still showed ps-porep-1000-1 — the old format without the task ID. The user reported this in <msg id=1997>. The assistant's first instinct was that old jobs might still be in the pipeline, but the timestamps showed they were from after the restart. Something was wrong.

The breakthrough came in <msg id=2002>: the assistant grepped the binary for the format string and found ps-porep-%d-%d — only two %d specifiers. The Go build cache had not recompiled the changed files. The docker cp command had copied the modified source files into the container's volume, but Go's compiler cache saw that the object files were already up-to-date and skipped recompilation. The binary was built from stale object code, and the fix was never compiled in.## The Touch: A Minimal Intervention with Maximum Impact

The subject message represents the third build attempt (psfix3), and the critical difference is the touch command. The assistant explicitly comments in the bash script: # Touch the changed files to bust Go build cache. This is the key insight.

Go's build cache works by comparing file modification times and content hashes. When a source file's modification timestamp is older than the corresponding compiled object file, the compiler assumes nothing has changed and skips recompilation. The docker cp command, while it replaces the file content, does not necessarily update the modification timestamp to a value that Go's cache logic recognizes as "newer." The touch command explicitly sets the file's modification time to the current moment, forcing Go to see the files as recently modified and invalidating any cached object files.

This is a subtle but well-known pitfall in Docker-based Go builds. When you copy files into a container using docker cp or --volumes-from, the files inherit their original timestamps from the host filesystem. If those timestamps are older than the cached build artifacts (which were created during the initial docker build of the curio-builder:latest image), Go's compiler will skip recompilation. The result is a binary that appears to be freshly built—new version string, new build commit—but contains stale compiled code for the specific changed files.

The assistant's decision to add touch was not a guess; it was a deduction based on evidence. The previous build (psfix2) had produced a binary with the correct version string (_psfix2) but the wrong format string. The version string is injected via -ldflags -X at link time, not at compile time, so it would reflect the new value regardless of whether the Go source files were recompiled. The format string, however, lives in the compiled code of task_prove.go. If that file wasn't recompiled, the old format string would persist. The mismatch between the version string (new) and the format string (old) was the telltale sign of a build cache issue.

The Self-Verifying Build Pattern

A notable design choice in this message is the grep command appended to the build step:

grep -ao "ps-porep-%d-%d[^\"]*" /output/curio-psfix3 | head -3

This is a self-verifying build: after compilation completes, the script immediately searches the output binary for the old format string. If the fix took effect, the old format string should not appear (or should appear only as part of the new three-placeholder string ps-porep-%d-%d-%d). If the old string still appears, the build failed to incorporate the change.

This pattern is powerful because it catches build cache issues at build time, not after deployment. Without this check, the assistant would have deployed psfix3, restarted the service, waited for new proofs to flow through, and only then discovered the fix wasn't working—repeating the exact same cycle that wasted the psfix2 deployment.

The grep uses -a (treat binary as text) and -o (only matching) to search the raw binary file. The regex ps-porep-%d-%d[^\"]* matches the old two-placeholder format, stopping at any quote character. If the binary contains ps-porep-%d-%d (old), the grep will find it. If it only contains ps-porep-%d-%d-%d (new), the regex won't match because the third -%d follows the second %d before any quote. This is a cleverly crafted test that distinguishes the two formats.

Assumptions and Knowledge

The message makes several implicit assumptions:

  1. Go build cache behavior: The assistant assumes that touch will invalidate the cache. This is correct for Go's default cache strategy, which uses modification times as a heuristic (though Go 1.20+ also uses content hashes, making touch less reliable in some cases). In practice, touch worked because the cached object files had earlier timestamps than the touched sources.
  2. Docker volume semantics: The assistant assumes that --volumes-from curio-rebuild3 provides access to the modified source files. This is correct—the docker cp commands in <msg id=2003> copied the files into the named container's volumes, and --volumes-from mounts those same volumes into the build container.
  3. The grep will catch the issue: The assistant assumes that if the old format string is absent from the binary, the fix is correct. This is a reasonable heuristic but not a complete verification—it doesn't check that the new format string is present, nor does it verify the surrounding code logic. The knowledge required to understand this message includes: Go build cache mechanics, Docker volume mounts and --volumes-from, the structure of Go -ldflags -X version injection, the CuZK engine's job ID architecture, and the ProofShare concurrent proving pipeline. Without this context, the touch command would appear to be a superstitious gesture rather than a targeted fix for a specific cache invalidation failure.

Mistakes and Incorrect Assumptions

The primary mistake in the broader sequence was the initial assumption that docker cp into a --volumes-from container would produce a correct rebuild. This assumption failed because:

  1. File timestamps are preserved: docker cp copies files with their original modification times. If the source files were last modified hours or days ago, their timestamps are older than the cached build artifacts from the Docker image build.
  2. The version string is misleading: The -ldflags -X mechanism updates the version string at link time, which happens even if no source files are recompiled. This creates a false signal: the version string says "psfix2" or "psfix3," implying the fix is included, but the actual compiled logic is unchanged.
  3. The initial deployment sequence was rushed: In <msg id=1989>, the assistant chained kill and mv commands, which failed silently because the running process held the file lock. This was a separate operational error, but it compounded the confusion by making it unclear whether the binary was actually replaced. A secondary mistake was the initial failure to verify the binary content before deployment. In <msg id=1996>, the assistant declared the fix deployed and summarized the root cause, but had not confirmed that the binary actually contained the new format string. The verification only happened after the user reported the logs still showed the old format. The psfix3 build rectifies this by embedding verification directly in the build script.

Output Knowledge Created

This message creates several pieces of output knowledge:

  1. A verified binary: The curio-psfix3 binary, built with the correct format string ps-porep-%d-%d-%d, ready for deployment.
  2. A build procedure: The combination of touch + grep verification establishes a repeatable pattern for Docker-based Go builds where source files are injected post-hoc. This pattern can be reused for future fixes.
  3. Diagnostic evidence: The grep output (or lack thereof) confirms whether the build cache issue is resolved. If the old format string is absent, the fix is compiled in. If present, the build needs further adjustment.
  4. A lesson about Go build caches in Docker: The session now contains a documented instance of this failure mode, which can inform future debugging.

The Thinking Process

The reasoning visible in this message reflects a methodical progression from symptom to root cause to countermeasure:

  1. Symptom: Cuzk logs show ps-porep-1000-1 despite deploying psfix2.
  2. Hypothesis A: Old jobs still in pipeline. Rejected — timestamps are post-restart.
  3. Hypothesis B: Binary wasn't actually replaced. Partially true — the first deployment had a kill/mv issue, but the second deployment (after manual cleanup in <msg id=1992>) confirmed the binary was in place with the correct version string.
  4. Hypothesis C: Build cache prevented recompilation. Confirmed — grep -ao on the binary shows the old format string.
  5. Countermeasure: Add touch to bust cache, add grep to self-verify. The assistant's thinking is visible in the progression from <msg id=1997> (user reports the issue) through <msg id=1998> (assistant puzzles over the discrepancy) to <msg id=2002> (the grep discovery) and finally <msg id=2004> (the fixed build). Each step narrows the hypothesis space until the root cause is identified.

Conclusion

The touch command in <msg id=2004> is a small but pivotal moment in a complex debugging saga. It represents the intersection of deep system knowledge—Go build internals, Docker volume semantics, GPU proving pipeline architecture—with practical operational experience. The fix itself (adding a task ID to a format string) was trivial; the challenge was getting that fix into a running production system without rebuilding the entire Docker image. The touch + self-verifying grep pattern is a reusable technique for anyone building Go binaries in Docker with post-hoc source injection. More broadly, this message illustrates a fundamental truth about distributed systems debugging: the hardest bugs are often not the logical errors in your code, but the silent failures in your build and deployment pipeline that prevent your fixes from ever reaching production.