Building a Patching Pipeline: Docker Layer Cache Tricks for GPU-Proving Binaries

The Message

docker run --rm \
  --volumes-from curio-rebuild \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /tmp/curio-new -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix" \
      ./cmd/curio && ls -lh /tmp/curio-new
  ' 2>&1 | tail -20

The output shows Go downloading dependencies—secp256k1-voi, ledger-go, uint256, go-eth-kzg, gopsutil, gonum, and others—as the compiler assembles a 163 MB binary with CUDA and supraseal support. This single command, issued at message 1949 in the conversation, represents a critical inflection point in a production debugging session: the moment when the assistant pivots from a failed local build to a successful Docker-based rebuild strategy, leveraging Docker's layer cache to avoid a full image rebuild while still injecting patched source files.

Why This Message Was Written: The Context and Motivation

To understand why this message exists, one must trace the chain of events that led to it. The session had just fixed two severe production bugs in the ProofShare system—a deadlock caused by HTTP 429 retries blocking the poll loop, and a job ID collision in the cuzk proving engine that caused partition proofs from different challenges to mix together. Both fixes were in Go source files: lib/proofsvc/provictl.go, tasks/proofshare/task_request.go, and tasks/proofshare/task_prove.go. These are Curio source files, not cuzk source files, meaning the fixes had to be compiled into the Curio binary.

The user's instruction was straightforward: "build curio and send updated binary to the vast host" ([msg 1934]). The assistant's first attempt was a native build on the development machine using make curio ([msg 1937]), which succeeded and produced a 163 MB binary. That binary was uploaded via SCP to the remote vast host at 141.195.21.72 ([msg 1940]). But when the assistant tried to verify the binary by running curio --version on the remote host, it failed with: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory ([msg 1942]).

This failure exposed a critical assumption: the Curio binary, when built with CUDA and supraseal support, links against shared libraries that exist only inside the Docker build environment defined in Dockerfile.cuzk. The native Go build on the development machine produced a binary that was dynamically linked against libraries not present on the target system. The user then clarified: "it is also a curio node, maybe you need to docker-build (Dockerfile.cuzk) and send the binary from docker build" ([msg 1944]).

The Decision: Layer Cache Optimization

The assistant now faced a choice. The straightforward approach would be to rebuild the entire Docker image from scratch using docker build --no-cache, which would force a recompilation of Curio with the updated source files. But this is expensive: the Dockerfile.cuzk build involves installing CUDA 13 devel tools, Go 1.24, Rust 1.86, gcc-13, compiling FFI bindings with supraseal, and building both Curio and the cuzk daemon. A full rebuild could take 30 minutes or more.

Instead, the assistant chose a more surgical approach. In message 1948, it had already created a container from the cached builder image (curio-builder:latest) and copied the three modified source files into it using docker cp. Now, in message 1949, it runs a build command inside that container using --volumes-from to mount the container's filesystem, with the working directory set to /build (where the source tree lives in the builder image).

This approach exploits Docker's layer cache: the builder image already contains all compiled dependencies, the Go module cache, and the previous build artifacts. Only the three modified Go files are new. The Go compiler will recompile only the packages that changed (and their dependents), which is far faster than a full rebuild. The command also sets a custom CurrentCommit build variable (+git_$(git log -1 --format=%h_%cI)_psfix) to distinguish this build from previous ones, which is crucial for verifying that the correct binary is deployed.

Assumptions Made

Several assumptions underpin this message. First, the assistant assumes that the curio-builder:latest image tag exists and was successfully built in the previous step ([msg 1946]). The Docker build output showed warnings but no fatal errors, and the final lines indicated success, so this is a reasonable assumption—but it's worth noting that the builder image was built with cached layers, meaning it contained the old Curio binary, not the patched one.

Second, the assistant assumes that copying the three modified Go source files into the container's filesystem via docker cp is sufficient to trigger a rebuild. This depends on Go's build cache invalidation: if the Go compiler sees that the source files have changed (different mtimes or content hashes), it will recompile the affected packages. The --volumes-from flag mounts the container's filesystem into the run container, so the modified files are visible at their original paths.

Third, the assistant assumes that the build environment inside the container is fully configured for CUDA and supraseal builds. The environment variables set in the command (FFI_BUILD_FROM_SOURCE=1, FFI_USE_CUDA=1, FFI_USE_CUDA_SUPRASEAL=1, FFI_USE_GPU=1, LIBRARY_PATH, LD_LIBRARY_PATH) mirror the ones set in the Dockerfile's build stage, so this is consistent.

Fourth, the assistant assumes that git log will work inside the container to produce the commit hash for the version string. The builder image copies the entire Git repository (via COPY . . in the Dockerfile), so the .git directory should be present. This is a reasonable assumption, but if the .git directory were excluded (e.g., via .dockerignore), the version string would be empty or the command would fail.

Mistakes and Incorrect Assumptions

The most notable mistake was the initial local build. The assistant assumed that a native make curio build would produce a binary that could run on the remote vast host. This assumption failed because the Curio binary, when built with the cunative tag, links against CUDA runtime libraries and supraseal's libconfig++ that exist only in the Docker environment. The error message libconfig++.so.15: cannot open shared object file was the tell.

This mistake is understandable: the development machine likely has Go and basic tooling installed, but not the full CUDA 13 + supraseal environment. The Dockerfile.cuzk exists precisely to provide this hermetic build environment. The assistant learned this the hard way, and the user's correction was necessary.

A subtler issue is that the docker run command uses --volumes-from curio-rebuild, but the curio-rebuild container was created with /bin/true as its command ([msg 1947]), meaning it immediately exited. A stopped container's volumes can still be referenced with --volumes-from, so this works in Docker—but it's an unusual pattern that might confuse someone unfamiliar with Docker's volume lifecycle. The container's filesystem is ephemeral and exists only as long as the container object itself exists (which it does, since it wasn't removed with --rm).

Input Knowledge Required

To understand this message, one needs knowledge of several domains:

Docker internals: The --volumes-from flag mounts volumes from another container, including the container's own root filesystem if it was created from an image. The --rm flag removes the run container after exit. The builder image was built in a previous step and tagged as curio-builder:latest.

Go build system: The -tags "cunative" flag enables CUDA-specific code paths. The -ldflags with -X sets build-time variables for version information. The GOAMD64=v3 environment variable enables AMD64 microarchitecture level 3 optimizations (AVX2, etc.).

CUDA and supraseal: The environment variables FFI_USE_CUDA, FFI_USE_CUDA_SUPRASEAL, FFI_USE_GPU control whether the Filecoin FFI library is built with GPU proving support. The LIBRARY_PATH and LD_LIBRARY_PATH variables point to CUDA library directories.

Filecoin/Curio architecture: Curio is the storage mining node software that handles proof generation. The cunative build tag enables native CUDA GPU proving. The CurrentCommit build variable embeds a version string into the binary for identification.

The production context: The three modified source files contain fixes for a deadlock and a job ID collision in the ProofShare system. These are critical production bugs that need to be deployed urgently.

Output Knowledge Created

This message produces a rebuilt Curio binary at /tmp/curio-new inside the container, with the three proofshare fixes compiled in and a distinctive version string (+git_<hash>_<date>_psfix). The binary is dynamically linked against the CUDA and supraseal libraries present in the builder image, making it compatible with the runtime environment on the vast host (which runs the same Docker image's runtime stage).

The message also establishes a repeatable pattern for patching production binaries without full Docker image rebuilds: create a container from the cached builder image, copy modified source files into it, and rebuild inside the container. This pattern is valuable because full Docker rebuilds of the CUDA environment can take 30+ minutes, while this incremental approach takes only the time needed to recompile the changed Go packages (typically 2-5 minutes).

The Thinking Process

The reasoning visible in this message reveals a developer who understands the tradeoffs between build speed and correctness. The assistant initially tried the fastest path (native build + SCP), which failed. Then it tried the most robust path (full Docker rebuild), but that used cached layers and didn't pick up the changes. The docker build output in message 1946 showed timestamps from 08:45, confirming the cache hit.

The assistant's pivot to the docker cp + --volumes-from approach is a creative middle ground. It avoids the cost of a full --no-cache rebuild while ensuring the binary is built in the correct environment. The command explicitly sets all the CUDA and FFI environment variables that the Dockerfile would set, ensuring the build is hermetic and reproducible.

The inclusion of && ls -lh /tmp/curio-new at the end of the build command shows attention to verification: the assistant wants to confirm the binary was produced and see its size. The tail -20 on the output shows the dependency download activity, which is normal for a Go build that needs to fetch missing dependencies (likely because the Go module cache in the builder image didn't have all transitive dependencies pre-downloaded).

The custom version string +git_$(git log -1 --format=%h_%cI)_psfix is a deliberate choice. The _psfix suffix distinguishes this build from the unpatched version, making it easy to verify which binary is running on the remote host. This is a common operational pattern: embed a unique identifier in the binary so that deployed instances can be audited.

Conclusion

Message 1949 is a deceptively simple Docker command that encapsulates a sophisticated understanding of build systems, Docker internals, and production deployment workflows. It represents the third attempt to produce a deployable binary after two earlier failures (native build missing shared libraries, cached Docker build missing source changes). The assistant's ability to diagnose each failure and adapt—first recognizing the library dependency issue, then recognizing the Docker cache issue, and finally crafting a surgical rebuild strategy—demonstrates the kind of systematic troubleshooting that production engineering demands. The command itself, with its careful environment variable setup, version string customization, and volume mounting, is a small masterpiece of incremental build optimization.