The Docker Build Pivot: Why a Native Go Compilation Failed for Curio's GPU Proving Stack

Introduction

In distributed systems engineering, few moments are as instructive as the precise instant when a seemingly straightforward build-and-deploy operation fails, revealing hidden assumptions about the environment. Message 1945 in this opencode session captures exactly such a moment. The assistant had just built a Curio binary using make curio, uploaded it to a remote vast.ai GPU host, and attempted to verify it—only to be greeted with a shared library error. The user's response pointed toward Docker-based building, and the assistant's message 1945 represents the pivot: the recognition that the native build approach was insufficient, the acceptance of the Docker build path, and the first concrete step of reading the Dockerfile.cuzk to understand the true build environment.

This message, though brief in its visible output, encapsulates a deep lesson about the nature of modern GPU-accelerated proving systems: they are not simple Go binaries but complex artifacts that depend on a carefully orchestrated chain of native libraries, CUDA toolkits, and FFI bindings. Understanding why the native build failed, what the assistant reasoned about the failure, and how the Docker build approach addresses the root cause reveals the intricate dependency landscape of production Filecoin proving infrastructure.

The Context: A Production Deadlock Fix That Needed Deployment

The broader session context is essential for understanding the stakes of message 1945. The assistant had just implemented critical fixes for two production bugs in the ProofShare system: a deadlock caused by HTTP 429 retries blocking the poll loop, and a cuzk job ID collision that caused partition proof mixing across concurrent challenges ([msg 1933]). These fixes touched three Go source files—provictl.go, task_request.go, and task_prove.go—and required building a new Curio binary to deploy to the production GPU worker.

The assistant's initial approach was straightforward: build Curio natively on the development machine using make curio, then scp the binary to the remote vast.ai host at 141.195.21.72. This worked in the sense that the build succeeded and the 163MB binary was transferred ([msg 1939], [msg 1940]). But when the assistant attempted to verify the binary on the remote host by running curio --version, the error appeared:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

This error is the catalyst for message 1945. It reveals that the native-built binary was dynamically linked against a library (libconfig++.so.15) that exists in the development environment's supraseal toolchain but is absent from the target host's runtime. The binary is not self-contained—it carries runtime dependencies that the target environment cannot satisfy.

The Reasoning: Tracing the Dependency Chain

The assistant's reasoning in message 1945 is explicit and methodical:

The curio binary needs to be built inside the Docker environment because it depends on shared libraries that are available in the Docker build context (like libconfig++.so.15 which comes from the supraseal dependencies).

This analysis identifies the root cause with precision. The libconfig++.so.15 library is not a standard system library—it is part of the supraseal toolchain, a C++ library suite used for GPU-based proving operations in the Filecoin ecosystem. The make curio build command includes the -tags "cunative" flag ([msg 1937]), which enables CUDA-native proving paths. These paths link against supraseal's shared libraries at build time, producing a binary that requires those libraries at runtime.

The assistant correctly infers that the Docker build environment—defined in Dockerfile.cuzk—is the canonical build environment precisely because it includes all these dependencies: CUDA 13 devel tools, Go 1.24, Rust 1.86, gcc-13, and the supraseal libraries. The Docker build is not an alternative build method; it is the correct build method, and the native build was a shortcut that happened to work on the development machine only because the supraseal libraries were installed there too.

The Assumption and the Mistake

The assistant's initial assumption was that make curio produces a standalone binary that can be copied to any Linux host. This assumption is reasonable for many Go projects, where static linking or Go's self-contained runtime produces portable binaries. However, Curio is not a typical Go project. It uses cgo (Go's C interoperability mechanism) to call into the Filecoin FFI library (filecoin-ffi), which in turn links against supraseal's C++ libraries. When Go builds with CGO_ENABLED=1 (the default), and the code imports packages that use cgo, the resulting binary is dynamically linked against any shared libraries those C/C++ dependencies require.

The mistake was not in the build command itself but in the assumption about portability. The assistant had successfully built and deployed a cuzk binary earlier in the session using a similar Docker build approach ([chunk 12.0]), but that experience did not immediately transfer to the Curio build. The error message was the necessary signal that the native build was insufficient.

It is worth noting that this is not a careless mistake—it is a natural consequence of the complexity of the system. The Curio codebase spans Go, Rust (via cgo FFI to filecoin-ffi), C++ (supraseal), and CUDA (GPU kernels). Each layer adds its own linking requirements, and the full dependency graph is not visible from any single vantage point. The Dockerfile exists precisely to encode this complexity into a reproducible build.

Input Knowledge Required

To fully understand message 1945, several pieces of context are necessary:

  1. The supraseal toolchain: A C++ library suite developed for Filecoin GPU proving. It provides the libconfig++ library and other shared objects that Curio's CUDA-native proving paths link against. Without this toolchain, Curio cannot perform GPU-based proof generation.
  2. The cunative build tag: The -tags "cunative" flag in the Makefile enables CUDA-accelerated proving code paths. This is what introduces the dependency on supraseal's shared libraries.
  3. The Dockerfile.cuzk: A multi-stage Docker build file that defines the canonical build environment for Curio's CUDA-enabled binary. Stage 1 uses a CUDA 13 devel image with Go, Rust, and gcc-13 to compile Curio and cuzk-daemon. Stage 2 creates a runtime image with just the CUDA runtime and shared libraries.
  4. The remote host's role: The vast.ai instance at 141.195.21.72 is a GPU worker that runs both cuzk (the GPU proving daemon) and Curio (the task orchestration layer). It needs the full supraseal runtime, which is provided by the Docker runtime image.
  5. The ProofShare system: A distributed proof marketplace where miners request proofs from remote proving services. The fixes being deployed address deadlocks and job ID collisions in this system.

Output Knowledge Created

Message 1945 produces several concrete outputs:

  1. The Dockerfile path: The assistant reads /tmp/czk/Dockerfile.cuzk, confirming its existence and location. This is the blueprint for the correct build process.
  2. The build environment specification: The Dockerfile's multi-stage structure reveals that the build requires CUDA 13 devel tools, Go 1.24, Rust 1.86, and gcc-13. The runtime stage confirms that libconfig++.so.15 and other supraseal libraries are included in the runtime image.
  3. The corrected deployment strategy: Instead of building natively and copying the binary, the correct approach is to build inside the Docker environment (either by running docker build or by using a container to compile) and then extract the binary from the resulting image.
  4. The root cause documentation: The reasoning explicitly documents that libconfig++.so.15 comes from the supraseal dependencies, providing a clear explanation for future debugging.

The Thinking Process: From Error to Action

The assistant's thinking process in message 1945 follows a clear diagnostic pattern:

  1. Observe the symptom: The binary fails with a missing shared library error.
  2. Identify the missing dependency: libconfig++.so.15 is not found on the remote host.
  3. Trace the dependency origin: This library comes from the supraseal toolchain, which is part of the CUDA proving stack.
  4. Recognize the environment mismatch: The development machine has supraseal installed (so the build succeeds), but the remote host does not have the library in its library path (so execution fails).
  5. Recall the canonical build method: The Dockerfile.cuzk exists precisely to provide a consistent build environment that includes all dependencies.
  6. Take the first corrective action: Read the Dockerfile to understand the build process before proceeding. This is textbook root-cause analysis. The assistant does not attempt to fix the symptom (e.g., by copying the missing library to the remote host) but instead addresses the systemic issue: the build must be performed in an environment that matches the target runtime. The Docker build guarantees this match because the runtime stage of the Docker image includes exactly the libraries that the builder stage compiled against.

The Broader Significance

Message 1945 is a microcosm of a recurring challenge in modern infrastructure engineering: the tension between developer convenience and production reproducibility. Native builds are fast and familiar, but they encode implicit assumptions about the environment. Docker builds are slower and more complex, but they guarantee that the binary will run correctly in the target environment.

For GPU-accelerated systems, this tension is amplified. CUDA toolkits, GPU drivers, and specialized C++ libraries like supraseal create a dependency graph that is difficult to replicate outside the canonical build environment. The Dockerfile is not just a build script—it is a specification of the entire dependency chain, from the operating system packages to the CUDA version to the Go toolchain.

The assistant's pivot from native build to Docker build is not a retreat but an advancement in understanding. It transforms the deployment process from a fragile sequence of manual steps into a reproducible, auditable pipeline. The Docker build, once executed, will produce a binary that is guaranteed to work on any host running the Docker runtime image—including the production vast.ai instance.

Conclusion

Message 1945 captures the moment of insight when a failed deployment reveals the true complexity of a system. The missing libconfig++.so.15 library was not an obstacle to be worked around but a signal that the build process needed to be aligned with the runtime environment. The assistant's reasoning—tracing the library to the supraseal toolchain, recognizing the Dockerfile as the canonical build specification, and reading it to understand the correct approach—demonstrates systematic debugging at its best.

The message is brief, but it represents a critical juncture in the session. Before this message, the assistant was pursuing a native build strategy that would have produced a non-functional binary. After this message, the assistant has the information needed to execute the correct Docker-based build, producing a binary that will run correctly in the production environment. The pivot is not just about a single library—it is about understanding that in complex GPU proving systems, the build environment is part of the artifact, not just a tool for producing it.