The Docker Build That Didn't Take: A Lesson in Go Cache Invalidation

docker run --rm \
  --volumes-from curio-rebuild2 \
  -v /tmp:/output \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /output/curio-psfix2 -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix2" \
      ./cmd/curio && ls -lh /output/curio-psfix2
  ' 2>&1 | tail -5
go: downloading github.com/zondax/hid v0.9.2
go: downloading github.com/zondax/golem v0.27.0
go: downloading github.com/libp2p/go-cidranger v1.1.0
go: downloading gitlab.com/yawning/tuplehash v0.0.0-20230713102510-df83abbf9a02
-rwxr-xr-x 1 root root 163M Mar 13 10:06 /output/curio-psfix2

At first glance, message [msg 1987] looks like a routine CI build step: a Docker invocation that compiles a Go binary, succeeds, and produces a 163 MB executable. But this message sits at a critical inflection point in a production debugging session that had already consumed hours of root-cause analysis. The build command is the bridge between a discovered bug and its deployed fix — and, as the subsequent messages reveal, this bridge had a hidden structural flaw that would only be discovered after the binary was already running in production.

The Context: A Job ID Collision in the ProofShare System

To understand why this message matters, we must understand the crisis that precipitated it. The user was operating a production Filecoin proving system built around the Curio daemon and the cuzk GPU proving engine. The ProofShare subsystem allows multiple parties to collaboratively prove storage commitments by issuing "challenges" — cryptographic tasks that require generating zk-SNARK proofs for randomly selected sectors of stored data.

The user had deployed a new cuzk binary built inside a Docker container, which passed benchmarks successfully. Yet when the system ran real ProofShare workloads, every single one of the ten PoRep (Proof of Replication) partitions produced invalid proofs. The log screamed "PER-PARTITION VERIFICATION: 0/10 valid" — a complete failure.

The assistant's investigation in the preceding messages ([msg 1977] through [msg 1984]) traced the root cause to a job ID collision. The ProofShare challenges all targeted the same artificial sector (miner=1000, sector=1), so concurrent tasks sent identical job_id values — "ps-porep-1000-1" — to the cuzk engine. Inside the Rust-based proving pipeline, a JobTracker maintained a HashMap of partition assemblers keyed on job_id. When two concurrent proofs shared the same key, their partition results overwrote each other. The symptom was a Rust panic: "partition 0 already inserted".

The fix was conceptually simple: add the harmony task ID to the RequestId format string, changing "ps-porep-%d-%d" to "ps-porep-%d-%d-%d". The assistant had already edited the source file (task_prove.go) and verified it compiled locally ([msg 1983]). The non-proofshare cuzk path was audited and found safe because it used real (unique) sector identities. Now the fix needed to be compiled into a Curio binary and deployed to the production GPU worker.

The Build Command: A Carefully Constructed Deployment Artifact

Message [msg 1987] is the assistant's attempt to produce that binary. Every element of the Docker command reflects deliberate decisions shaped by the production environment.

The use of --volumes-from curio-rebuild2 is the first notable choice. In message [msg 1985], the assistant had created a throwaway container from the curio-builder:latest image using docker create. Then in message [msg 1986], it copied the three modified source files into that container using docker cp. Now, instead of copying files again, the assistant mounts that container's filesystem into a new ephemeral build container. This is a clever Docker pattern for incremental builds: the "rebuild2" container acts as a writable overlay where only the changed files are placed, while the rest of the build context (dependencies, vendored libraries) comes from the base image.

The environment variables reveal the complexity of GPU-accelerated Go builds. FFI_BUILD_FROM_SOURCE=1 forces compilation of the Filecoin FFI (Foreign Function Interface) bindings rather than using prebuilt binaries — necessary because the build must link against CUDA. FFI_USE_CUDA=1, FFI_USE_CUDA_SUPRASEAL=1, and FFI_USE_GPU=1 enable the GPU proving pipeline. The LIBRARY_PATH and LD_LIBRARY_PATH variables point to the CUDA toolkit libraries, including the stubs directory needed for linking. The GOAMD64=v3 flag enables modern x86-64 instruction set extensions (AVX2, etc.) for better performance.

The ldflags inject version metadata into the binary: IsOpencl=0 disables OpenCL (using CUDA instead), and CurrentCommit embeds the git commit hash and a _psfix2 suffix for traceability. This version string is what appears in logs and --version output, allowing operators to confirm which binary is running.

The output is written to /output/curio-psfix2, a path that maps to /tmp/ on the host via the -v /tmp:/output bind mount. The psfix2 suffix distinguishes this build from earlier attempts (psfix1 presumably being the deadlock fix from earlier in the session).

What the Output Reveals — and What It Conceals

The build output shows four Go module downloads (zondax/hid, zondax/golem, libp2p/go-cidranger, yawning/tuplehash) followed by the final binary listing: 163 MB, timestamped March 13 at 10:06. The downloads suggest these modules weren't in the container's module cache, possibly because the --volumes-from approach exposed a different GOPATH or module cache than the original build. The binary size — 163 MB — is typical for a Go binary that statically links CUDA and FFI libraries.

The build succeeded. The binary was produced. The assistant moved on to deployment.

But here's the hidden flaw: the Go build cache was not busted. The --volumes-from approach exposed the modified source files through the container's filesystem, but Docker's build cache and Go's own build cache may have determined that the compiled package was already up-to-date. In the very next chunk ([chunk 13.1]), the assistant discovers that the deployed binary still contains the old "ps-porep-%d-%d" format string. The fix was never actually compiled in.

This is the central irony of message [msg 1987]. It is a technically sophisticated build command that produces a binary with all the right characteristics — correct size, correct version string, correct CUDA linkage — but it fails at its primary objective: incorporating the source-level fix. The Go toolchain's incremental compilation, combined with Docker's layered filesystem semantics, conspired to produce a binary that was indistinguishable from the previous build.

The Deeper Lesson: Build Reproducibility in GPU-Proving Systems

This message illuminates a broader challenge in operating distributed proving infrastructure. The proving pipeline spans three languages: Go (Curio daemon, orchestration), Rust (cuzk engine, GPU kernel management), and C++/CUDA (supraseal, the actual GPU proving library). The build environment must reproduce the exact same compiler versions, CUDA toolkit, and FFI bindings that the original developers used. Any discrepancy — a cached object file, a different module version, a stale build artifact — can silently produce a binary that behaves differently from the one that passed testing.

The assistant's choice to use --volumes-from rather than a full docker build with a Dockerfile reflects an operational tradeoff. A full image rebuild would take longer and might introduce unrelated changes from the base image. The incremental approach is faster and more targeted. But it introduces the risk of cache staleness — a risk that materialized here.

In the subsequent messages, the assistant pivots to direct bind mounts (-v /tmp/czk/tasks/proofshare/task_prove.go:/build/tasks/proofshare/task_prove.go) which forces Go to detect the file modification and recompile. The lesson is hard-won: in Docker-based Go builds, --volumes-from does not guarantee that the Go compiler sees file modifications the same way a native filesystem would. The build cache, stored in the container's layers, may have a different timestamp or content hash than the overlaid file.

Conclusion

Message [msg 1987] is a moment of apparent success that masks a subtle failure. The Docker command is a carefully engineered piece of deployment infrastructure, reflecting deep knowledge of CUDA builds, Go toolchain quirks, and production proving requirements. Yet it falls victim to one of the most insidious classes of bugs in software engineering: the build that succeeds but doesn't incorporate the intended change. The 163 MB binary it produces is, in a sense, a phantom — it looks right, it has the right version tag, but it doesn't contain the fix that the entire debugging session was working toward. The message stands as a testament to the gap between "the build succeeded" and "the fix is deployed," and the vigilance required to bridge that gap in complex, multi-language, GPU-accelerated systems.