The Docker Build Trap: When Layer Caching Defeats Production Patching

In the high-stakes world of Filecoin proof generation, deploying a critical bug fix to a production GPU worker is never as simple as recompiling a binary and copying it over. Message 1946 of this opencode session captures the exact moment when an engineer—in this case, an AI assistant—makes a reasonable but flawed decision about how to build a patched binary, setting off a chain of debugging that reveals the subtle ways containerized build environments can betray expectations.

The Context: Two Production Bugs, One Binary

To understand message 1946, we must first appreciate the pressure behind it. The assistant had just spent several rounds diagnosing and fixing two severe production bugs in the ProofShare system—a critical component of the Filecoin proving pipeline. The first bug was a deadlock: the TaskRequestProofs loop would retry HTTP 429 responses from the proof service indefinitely inside CreateWorkAsk, never returning to the poll loop to discover matched work. The second was a job ID collision: because all concurrent proofshare challenges targeted the same hardcoded miner and sector (miner=1000, sector=1), the cuzk engine's partition assembler keyed on the identical job_id, causing partition results from different proofs to mix—confirmed by a "partition 0 already inserted" panic ([msg 1919]-[msg 1933]).

The assistant had implemented fixes across three Go source files (task_request.go, task_prove.go, provictl.go) and verified they compiled cleanly. The user then gave a straightforward instruction: "build curio and send updated binary to the vast host" ([msg 1934]).

The First Attempt: A Locally Built Binary

The assistant's initial approach was natural. It found the make curio target, ran it, and produced a 163 MB binary ([msg 1937]-[msg 1939]). It uploaded this binary to the remote vast host at 141.195.21.72 and attempted to swap it in place. But when it ran curio --version on the remote host, the binary failed with a shared library error:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

This was the first clue that the build environment mattered. The Curio binary depends on libconfig++.so.15, a shared library from the supraseal GPU proving stack, which is only available inside the Docker environment defined by Dockerfile.cuzk. The assistant's local machine lacked these libraries, so the dynamically-linked binary was non-functional on the target host.

The user's response in [msg 1944] clarified the situation: "it is also a curio node, maybe you need to docker-build (Dockerfile.cuzk) and send the binary from docker build." This revealed an unstated assumption: the vast host was not just a cuzk GPU worker—it also ran the Curio node itself, and the binary needed to be built inside the same Docker environment that provides the CUDA/supraseal dependencies.

Message 1946: The Docker Build Decision

This brings us to the subject message. The assistant, having read the Dockerfile.cuzk in the previous message ([msg 1945]), now commits to a Docker-based build strategy:

I'll build just the curio binary inside Docker and extract it. Let me create a minimal build Dockerfile that reuses the cached layers:

>

[bash] cd /tmp/czk && docker build -f Dockerfile.cuzk --target builder -t curio-builder:latest . 2>&1 | tail -40

The reasoning here is straightforward and, on the surface, sensible. The Dockerfile.cuzk is a multi-stage build: stage 1 (builder) compiles both Curio and cuzk-daemon inside a CUDA 13 devel environment with Go 1.24, Rust 1.86, and gcc-13; stage 2 (runtime) packages only the runtime libraries and binaries. By targeting --target builder, the assistant can build just the first stage and extract the Curio binary from it, avoiding the full runtime image build.

The phrase "reuses the cached layers" is particularly telling. The assistant explicitly intends to leverage Docker's layer cache for speed, assuming that the source code changes will be picked up by the build. This is a critical assumption—and it turns out to be wrong.

The Hidden Assumption: Docker Layer Caching and COPY . .

The Dockerfile.cuzk contains a COPY . . instruction that copies the entire source tree into the builder image. Docker caches the result of this instruction based on the content of the source directory. However, the assistant had been editing files inside /tmp/czk/ using the edit tool, and these changes were present in the working directory. The question is whether Docker's cache invalidation would detect these changes.

In theory, Docker's COPY instruction computes a checksum of the source directory contents, so any file modification should bust the cache. But there are edge cases: if the Docker build context includes .dockerignore rules, or if the COPY targets a subdirectory, or if the build was run with --cache-from pointing to a remote image, the cache might not be properly invalidated.

The assistant's assumption that the build would "reuse cached layers" while also picking up the new changes is internally contradictory. You cannot have both: either the cache is reused (old binary) or the source is re-copied and rebuilt (new binary). The assistant wants the speed of caching with the freshness of a rebuild, and Docker's layer caching is not designed to provide both simultaneously when source files have changed.

What the Build Output Reveals

The output shown in message 1946 is truncated but revealing. We see a Rust compiler warning from the cuzk-core crate:

warning: type `JobTracker` is more private than the item `process_monolithic_result`
   --> cuzk-core/src/engine.rs:379:1
    |
379 | / pub(crate) fn process_monolithic_result(
380 | |     t: &mut JobTracker,
381 | |     result: Result<Result<(Vec<u8>, Duration)>, tokio::task::JoinError>,
382 | |     job_id: &JobId,
...   |
391 | |     single_request: Option<&ProofRequest>,
392 | | ) {

This warning is about Rust visibility rules—a pub(crate) function referencing a private type in its signature—and is unrelated to the Go source changes. But its presence tells us something important: the Docker build did recompile the Rust components (cuzk-core). However, the Curio binary (a Go project) may or may not have been rebuilt depending on whether the Go build step detected changes.

The truncated output with tail -40 means we only see the last 40 lines of the build log. The actual go build invocation for Curio may have happened earlier in the output and been cut off. This truncation is itself a minor mistake—the assistant cannot verify that Curio was actually recompiled.

The Mistake Revealed

The assistant's mistake becomes clear in the very next message ([msg 1947]):

The builder stage used a cached curio binary from before my changes. The Docker build copies the source at COPY . . but it's using cached layers. I need to make sure the curio binary is rebuilt with the new code changes.

The Docker build did not pick up the source changes. The curio binary in the builder image was the old one, built from the cached layer. The assistant's plan to "reuse cached layers" backfired: the cache was too effective, and the source changes were not incorporated.

Why did this happen? Several possibilities exist:

  1. The COPY . . layer was cached: If Docker determined that the source directory contents hadn't changed (perhaps because the edit tool modified files in a way that preserved timestamps or because the build was run from a different context), the cached layer would be used.
  2. The Go build layer was cached separately: Even if the source was re-copied, the RUN go build step might have been cached if the Dockerfile uses a separate RUN command for the Go build and the cache key (which includes the command string and input files) matched.
  3. The build used a remote cache: If --cache-from was specified or the Docker daemon was configured to use a registry cache, the builder image might have been pulled from a remote cache rather than rebuilt locally.

The Pivot: A More Surgical Approach

Recognizing the failure, the assistant pivots in [msg 1947] to a more targeted strategy. Instead of relying on Docker's build cache, it creates a container from the existing builder image and copies the modified source files directly into it:

docker create --name curio-rebuild curio-builder:latest /bin/true
docker cp /tmp/czk/tasks/proofshare/task_request.go curio-rebuild:/build/tasks/proofshare/task_request.go
docker cp /tmp/czk/tasks/proofshare/task_prove.go curio-rebuild:/build/tasks/proofshare/task_prove.go
docker cp /tmp/czk/lib/proofsvc/provictl.go curio-rebuild:/build/lib/proofsvc/provictl.go

This approach bypasses Docker's layer caching entirely. By using docker cp to inject the modified files into an existing container filesystem, and then running go build inside that container, the assistant ensures the binary is compiled from the exact source files that were edited. This is a more reliable pattern for surgical patches: instead of rebuilding the entire Docker image, extract the build environment and rebuild only the target binary.

Input Knowledge Required

To understand message 1946, a reader needs knowledge of:

  1. Docker multi-stage builds: The --target builder flag selects a specific build stage, and the resulting image contains the build artifacts from that stage.
  2. Docker layer caching: Each instruction in a Dockerfile produces a cached layer. The COPY instruction caches based on source content. Reusing cached layers is generally desirable for speed but can prevent picking up changes.
  3. The supraseal/CUDA dependency chain: Curio's FFI bindings to Filecoin's GPU proving libraries require libconfig++.so.15 and other shared libraries that are only available in the Docker build environment.
  4. The ProofShare architecture: The fixes being deployed are in the Go Curio binary, not the Rust cuzk daemon. The Curio binary handles task orchestration (requesting proofs, managing the queue), while cuzk handles GPU proving.
  5. The vast.ai deployment model: The remote host is a rented GPU instance accessed via SSH on a non-standard port (40362). Binaries must be compatible with the host's environment.

Output Knowledge Created

Message 1946 creates several important pieces of knowledge:

  1. The Docker builder image exists: curio-builder:latest is now available locally, containing the full CUDA 13 devel environment with Go and Rust toolchains.
  2. The build environment is functional: The Docker build completed successfully (exit code 0), confirming that the Dockerfile.cuzk is valid and all dependencies are resolvable.
  3. The Rust cuzk-core crate has a visibility warning: The process_monolithic_result function's signature uses JobTracker which is more private than the function itself. This is a code quality issue that may need attention but is not blocking.
  4. The approach needs refinement: The truncated output doesn't confirm whether Curio was rebuilt, and the cached layer problem will need to be addressed in the next message.

The Broader Lesson

Message 1946 is a microcosm of a common pattern in production DevOps: the tension between speed and correctness in build pipelines. The assistant wanted to "reuse cached layers" for speed, but this optimization directly conflicted with the need to pick up source changes. The Docker layer cache, designed to accelerate builds by skipping unchanged steps, became the enemy of deployment when changes were expected but not detected.

This is not a failure of the Docker tool itself, but of the mental model applied to it. The assistant assumed that "building inside Docker" meant running docker build with the existing Dockerfile. But the Dockerfile is designed for full image builds, not for surgical binary patches. The correct approach—which the assistant discovers in the next message—is to use the Docker image as a build environment (a container with the right toolchain) rather than as a build recipe (a Dockerfile that produces an image).

The message also highlights the importance of verifying that a build actually incorporated changes. The assistant did not check whether the curio binary inside the builder image had the new version string or the old one. The truncated tail -40 output obscured the critical go build step. In production deployments, this kind of verification—checking binary hashes, grepping for version strings, or comparing build timestamps—is essential to avoid deploying stale binaries.

Conclusion

Message 1946 is a turning point in the deployment saga. It represents the moment when the assistant correctly identifies the need for Docker-based building but applies the wrong Docker workflow. The assumption that docker build with layer caching would produce an updated binary proves incorrect, forcing a more sophisticated approach in the following messages. The lesson is universal: when patching production systems, never trust the cache—always verify that your changes made it into the binary. The Docker build cache is a performance optimization, not a correctness guarantee, and treating it as such can lead to deploying unchanged binaries to production.