The Cache Trap: A Docker Build Detective Story in Production Proving Systems

Introduction

In the high-stakes world of Filecoin proof generation, where GPU-powered workers churn through cryptographic computations to sustain the decentralized storage network, even a seemingly routine deployment task can unravel into a subtle debugging puzzle. This article examines a single message from an opencode coding session — message index 1947 — in which an AI assistant confronts the realization that a Docker build has silently ignored its source code changes, and devises a clever surgical workaround to avoid a full rebuild. The message is a masterclass in understanding build system caching behavior, the friction between development and deployment environments, and the pragmatic tradeoffs that define production engineering.

Context: The ProofShare Production Bugs

To understand message 1947, we must first understand the crisis that precipitated it. The session's broader context involves two critical production bugs in a system called "ProofShare" — a protocol for generating and sharing Filecoin proofs across a distributed network of GPU workers.

The first bug was a deadlock in TaskRequestProofs. The CreateWorkAsk function, which negotiates proof work with a remote service, would retry HTTP 429 (Too Many Requests) responses indefinitely using exponential backoff. Because the retry loop was blocking, the main poll loop could never return to check whether any previously submitted asks had been matched with work. The system was stuck: it kept asking for more work while never discovering the work it already had. The fix introduced a sentinel ErrTooManyRequests error that causes CreateWorkAsk to return immediately on 429, allowing the poll loop to continue, with progress-based exponential backoff to avoid hammering the service.

The second bug was a job ID collision in the cuzk proving engine. ProofShare challenges all targeted the same miner and sector (miner=1000, sector=1), so concurrent tasks sent identical job_id values to the cuzk daemon. The engine's partition assembler keyed on job_id, causing partition results from different proofs to get mixed together — confirmed by a "partition 0 already inserted" panic. The fix added the harmony task ID to the RequestId, making it unique per invocation.

Both fixes were implemented in Go source files within the Curio project. The assistant had edited three files — task_request.go, task_prove.go, and provictl.go — and verified that the code compiled cleanly with go build and go vet. The next step was to deploy the updated binary to the production GPU worker.

The Deployment That Went Wrong

Message 1947 is the culmination of a deployment attempt that went sideways. Let's trace the chain of events.

The assistant initially built the Curio binary locally using make curio (message 1937), producing a 163 MB executable. This was uploaded to the vast host at 141.195.21.72 via scp (message 1940). But when the assistant tried to swap the binary and verify it, the remote execution failed:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

The locally-built binary was linked against shared libraries present in the development environment but absent from the production GPU worker. The user then clarified (message 1944):

it is also a curio node, maybe you need to docker-build (Dockerfile.cuzk) and send the binary from docker build

This revealed a crucial architectural detail: the vast host was not just a GPU worker running the cuzk daemon — it was also a Curio node. And the Curio binary needed to be built inside the Docker environment defined by Dockerfile.cuzk, which includes the CUDA 13 development toolchain, the supraseal library, and all the FFI dependencies that the locally-built binary was missing.

Message 1947: The Cache Realization

The assistant followed the user's guidance and ran a Docker build using the existing Dockerfile.cuzk (message 1946):

cd /tmp/czk && docker build -f Dockerfile.cuzk --target builder -t curio-builder:latest .

But the output revealed a problem. The builder stage completed, but the curio binary inside the image was dated Mar 13 08:45 — before the source code changes had been made. The Docker build system, in its efficiency, had used cached layers. Since the COPY . . instruction's layer hash hadn't changed (the source files had been modified, but Docker's layer caching depends on more than just file timestamps), the build skipped the compilation step entirely.

This is the moment captured in message 1947. The assistant's reasoning text reveals the thought process:

The builder stage used a cached curio binary from before my changes. The Docker build copies the source at COPY . . but it's using cached layers. I need to make sure the curio binary is rebuilt with the new code changes. Let me check if it picked up the changes.

>

Actually, looking at the output - the curio binary is 163M Mar 13 08:45 which is from 08:45, but I built locally at 09:38. This means the Docker build used cached layers and didn't rebuild curio with my changes.

>

I need to bust the cache. Let me use --no-cache on the relevant layers, or better yet, just use --no-cache-filter or rebuild without cache from the COPY step.

>

Rather than rebuilding everything from scratch with --no-cache, I could spin up a container from the builder image, copy my modified source files into it, rebuild curio there, and extract the updated binary.

The assistant considers two approaches. The first is to use Docker's --no-cache or --no-cache-filter flags to force a rebuild from the COPY step onward. This would work, but it would trigger a full recompilation of the entire Curio project — including all Go dependencies, the Rust cuzk-core library, and the FFI bindings — which could take many minutes.

The second approach is more surgical: create a container from the existing builder image (which already has all dependencies compiled and cached), copy only the three modified source files into it, and rebuild just the Curio binary inside that container. This reuses the cached compilation artifacts for everything except the changed files.

The assistant chooses the second approach and executes:

docker create --name curio-rebuild curio-builder:latest /bin/true && echo "created"

This creates a stopped container named curio-rebuild from the builder image, which can then be used as a volume source for copying files in and out.

The Thinking Process: Tradeoffs Under Time Pressure

What makes message 1947 interesting is the explicit reasoning about tradeoffs. The assistant is operating under implicit time pressure — production bugs are causing invalid proofs, and every minute of delay means more wasted GPU cycles. A full --no-cache rebuild would be the "correct" approach in a CI/CD sense, guaranteeing a clean build from the exact source state. But it would be slow.

The surgical approach — creating a container from the cached image and injecting modified files — is faster but riskier. It assumes that:

  1. The builder image's compilation cache (Go's build cache, Rust's cargo cache) is intact and compatible
  2. The modified source files are the only ones that changed (true, since the assistant only edited three files)
  3. No build configuration changes are needed (also true)
  4. The resulting binary will be functionally identical to a full rebuild (likely true for Go, where incremental compilation is reliable) The assistant also implicitly assumes that Docker's layer caching is the culprit, rather than some other mechanism. This is a correct diagnosis: Docker's build cache uses content hashes of each layer's inputs, and if the COPY layer was cached from a previous build, the subsequent compilation steps won't re-run even if the source files on disk have changed. This is a well-known Docker gotcha, especially when working with multi-stage builds where intermediate layers are aggressively cached.

Input Knowledge Required

To fully understand message 1947, the reader needs knowledge spanning several domains:

Docker build caching mechanics: Understanding that Docker caches each layer by hashing its inputs, and that a cached COPY layer means the compilation step won't re-run even if source files have changed. This is non-obvious to developers accustomed to incremental builds in native toolchains.

Multi-stage Docker builds: The Dockerfile.cuzk uses a multi-stage build with a builder target. The --target builder flag tells Docker to stop after building the first stage, producing an image with all build dependencies but no runtime layer.

The docker create / docker cp workflow: The assistant uses docker create to instantiate a container from the builder image without running it, then plans to use docker cp to copy modified files into it. This is an advanced Docker technique that most practitioners don't encounter daily.

Go build caching: The assistant's confidence that copying modified source files into the container will trigger a partial rebuild relies on understanding Go's build cache, which tracks source file hashes and only recompiles packages whose sources have changed.

The project's architecture: The reader needs to know that Curio is a Go project that depends on CUDA, supraseal, and Filecoin FFI libraries, and that these dependencies are only available in the Docker build environment — hence the need to build inside Docker rather than locally.

Output Knowledge Created

Message 1947 creates several pieces of knowledge:

  1. The Docker cache diagnosis: The assistant has established that the builder image contains a stale binary and that the Docker build system is responsible for the staleness. This is a concrete finding about the build pipeline.
  2. The surgical rebuild strategy: The assistant has formulated a plan to bypass the cache by creating a container from the builder image and injecting modified source files. This strategy is documented in the reasoning text.
  3. The container creation: The docker create command has been executed, producing a container ID (c316f8170cc512cd48065105fa5d070a05ac95164dd1b30888f97eb2c47619f8). This container is now available for file operations.
  4. A decision point: The assistant has committed to the surgical approach over the --no-cache approach, establishing a preference for speed over thoroughness in this context.

Assumptions and Potential Pitfalls

The assistant makes several assumptions that could be wrong:

That the builder image's Go build cache is compatible: If the builder image was created with a different Go version or different build flags than what the surgical rebuild uses, the cache might be invalidated anyway, negating the speed advantage.

That only three files need updating: The assistant assumes that the only changes between the cached binary and the desired binary are in the three edited files. This is true for the proofshare fixes, but if any other files had been modified (e.g., by git operations or other edits), those changes would be missed.

That docker cp will work on a stopped container: The docker cp command works on stopped containers, so this is safe. But the subsequent docker run --volumes-from command (which appears in message 1949) assumes that the container's filesystem is accessible through volumes, which is correct.

That the binary can be extracted: The assistant's plan involves rebuilding inside the container and then extracting the binary. This requires either a volume mount or a docker cp from a running container. The assistant initially forgets this and uses --rm, losing the binary (message 1950), requiring a second attempt with a volume mount.

The Broader Significance

Message 1947 is a microcosm of a pattern that repeats throughout the coding session: the tension between the idealized development workflow and the messy reality of production deployment. The assistant writes clean code, verifies it compiles, and deploys it — only to discover that the deployment environment has its own rules. The Docker build cache, designed to save time, instead becomes a trap that silently ignores changes.

The message also illustrates the value of explicit reasoning in AI-assisted coding. The assistant doesn't just execute commands; it thinks through the problem, considers alternatives, and documents its decision process. This transparency allows the human collaborator to catch mistakes, suggest better approaches, or simply understand what's happening.

In the messages that follow (1948-1951), the assistant executes the surgical rebuild strategy, first copying the modified files into the container, then running the build command. The first build attempt succeeds but the binary is lost due to --rm. The second attempt uses a volume mount and succeeds, producing a 163 MB binary at /tmp/curio-psfix. The fix is then deployed to the production host.

Conclusion

Message 1947 captures a moment of debugging that every production engineer will recognize: the moment when a trusted tool (Docker's layer cache) silently betrays you, and you must decide whether to fight it head-on (with --no-cache) or work around it (with surgical precision). The assistant's choice of the surgical approach reflects a pragmatic understanding of the situation: the build environment is known, the changes are minimal, and time is of the essence. It's not the most elegant solution, but it's the one that gets the fix into production fastest — and in the world of distributed proving systems, that speed translates directly into saved GPU cycles and reliable proof generation.