The Pivot Point: A 163MB Binary and the Moment a Production Fix Becomes Real

The Message

Good, binary is at /tmp/curio-psfix. Now upload and swap on the remote:

``bash ls -lh /tmp/curio-psfix -rwxr-xr-x 1 root root 163M Mar 13 09:48 /tmp/curio-psfix ``

At first glance, this message from the assistant (message 1951 in the conversation) is unremarkable: a brief confirmation that a compiled binary exists, followed by a file listing showing its size and timestamp. But in the arc of a production debugging session spanning multiple hours, two critical bugs, and a half-dozen failed build attempts, this message represents the precise moment of breakthrough — the point at which a cascade of fixes finally materialized into a deployable artifact. The 163MB curio-psfix binary sitting at /tmp/curio-psfix was the culmination of an arduous journey through Docker cache invalidation, shared library resolution, and the unique friction of patching GPU-accelerated proving infrastructure without full image rebuilds.

The Context: Two Production Bugs in the ProofShare System

To understand the weight of this message, one must understand what led to it. The session had been wrestling with two critical production bugs in Filecoin's ProofShare system — the component responsible for coordinating proof generation across a distributed network of GPU workers.

The first bug was a deadlock in TaskRequestProofs. The CreateWorkAsk function contained an infinite retry loop: when the remote proof service returned HTTP 429 (Too Many Requests), it would retry with exponential backoff forever, never returning. Because this function was called inside the main poll loop, a sustained 429 response would block the loop from ever discovering work that had been matched to existing asks. The result was a complete stall — no new proof work would enter the proofshare_queue, and the proving pipeline would grind to a halt. The fix introduced a sentinel error (ErrTooManyRequests) that broke the retry loop immediately, allowing the poll loop to continue, with progress-based exponential backoff to avoid hammering the service.

The second bug was more subtle and more dangerous. The user reported that even after deploying a container-built cuzk binary that passed benchmarks, all ten PoRep partitions were still producing invalid proofs. The assistant traced this to a job ID collision: because ProofShare challenges all targeted the same miner ID (1000) and sector ID (1) — a hardcoded bench sector — concurrent tasks sent identical job_id values to the cuzk proving engine. The engine's partition assembler keyed on job_id, causing partition results from different proofs to become mixed. The symptom was a panic: "partition 0 already inserted". The fix added the harmony task ID to the RequestId, making it unique per invocation.

These fixes were made to Go source files in the Curio codebase: lib/proofsvc/provictl.go, tasks/proofshare/task_request.go, and tasks/proofshare/task_prove.go. But getting the patched binary onto the production GPU worker required navigating a complex build and deployment pipeline.

The Build Ordeal: Why a Simple `go build` Wasn't Enough

When the user first asked to "build curio and send updated binary to the vast host" ([msg 1934]), the assistant attempted a straightforward local build. The Curio binary compiled cleanly at 163MB, and the assistant uploaded it to the remote host at 141.195.21.72. But when the binary was tested, it failed with:

curio: error while loading shared libraries: libconfig++.so.15: cannot open shared object file: No such file or directory

This was the first indication that the Curio binary was not a standalone artifact. It depended on shared libraries from the supraseal/CUDA build environment — specifically libconfig++.so.15, which was only available inside the Docker container used for production builds. The local Go toolchain had linked against system libraries that didn't exist on the target machine.

The user's response ([msg 1944]) clarified the situation: "it is also a curio node, maybe you need to docker-build (Dockerfile.cuzk) and send the binary from docker build." This revealed an important architectural detail: the vast.ai GPU host was also running Curio (not just the cuzk daemon), and the binary needed to be built inside the same Docker environment that produced the production image.

The assistant then attempted to build inside Docker using Dockerfile.cuzk. The first attempt ([msg 1946]) built the full multi-stage Docker image, but the resulting binary was cached from a previous build — it didn't include the new proofshare fixes. Docker's layer caching had captured the COPY . . step from an earlier build, and the Go compiler never saw the modified source files.

What followed was a series of increasingly creative attempts to inject the modified source files into the Docker build environment. The assistant tried:

Creating a container from the builder image and using docker cp to copy the modified files in ([msg 1948])
Running a build with --volumes-from to share the container's filesystem ([msg 1949])
Mounting a host volume with -v /tmp:/output to extract the binary ([msg 1950]) Each approach revealed a different facet of the Docker caching problem. The --volumes-from approach successfully copied the files into the container, but the Go build cache still contained stale .a archives from the previous compilation, so the linker produced a binary with the old code. The breakthrough came only when the assistant used a direct bind mount for the output directory and ensured the Go compiler was forced to recompile the modified packages. The successful command in message 1950 was:

docker run --rm \
  --volumes-from curio-rebuild \
  -v /tmp:/output \
  -w /build \
  curio-builder:latest \
  bash -c '
    export FFI_BUILD_FROM_SOURCE=1 FFI_USE_CUDA=1 FFI_USE_CUDA_SUPRASEAL=1 FFI_USE_GPU=1
    export LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"
    export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
    GOAMD64=v3 CGO_LDFLAGS_ALLOW=".*" go build \
      -tags "cunative" \
      -o /output/curio-psfix -ldflags " -s -w \
      -X github.com/filecoin-project/curio/build.IsOpencl=0 \
      -X github.com/filecoin-project/curio/build.CurrentCommit=+git_$(git log -1 --format=%h_%cI)_psfix" \
      ./cmd/curio'

This command succeeded, producing the binary that message 1951 confirms.

Why This Message Matters

Message 1951 is the bridge between the build phase and the deployment phase. It serves several critical functions:

Verification: The ls -lh output confirms the binary is 163MB — the same size as the previous successful builds, indicating a complete compilation. The timestamp Mar 13 09:48 shows it was freshly built, not a stale artifact. The path /tmp/curio-psfix on the host machine (not inside a container) means it's accessible for upload.

Transition: The phrase "Now upload and swap on the remote" signals the shift from build to deploy. This is the moment the assistant declares readiness and moves to the next phase.

Confidence: The simple "Good" at the start conveys that after multiple failed attempts, the build has finally succeeded. It's a quiet acknowledgment of the ordeal just completed.

What Followed

The subsequent messages show the deployment proceeding smoothly. Message 1952 uploads the binary via SCP. Message 1953 performs the swap on the remote host:

mv /usr/local/bin/curio /usr/local/bin/curio-old-prereqfix && \
mv /tmp/curio-psfix /usr/local/bin/curio && \
chmod +x /usr/local/bin/curio && \
curio --version

The version output confirms success: curio version 1.27.3-rc2+mainnet+git_96f6c783_2026-03-11T16:08:33+01:00_psfix. The _psfix suffix, injected via the -ldflags build argument, proves this is the patched binary containing the deadlock fix and the job ID collision fix.

Assumptions and Decisions

Several assumptions underpin this message. The assistant assumes that the Docker-built binary, linked against the CUDA/supraseal libraries inside the container, will resolve the libconfig++.so.15 error that plagued the locally-built binary. This assumption proved correct. The assistant also assumes that the --volumes-from approach combined with the bind mount provides the correct build environment — an assumption that was validated only after earlier attempts failed.

The decision to inject a custom version string (_psfix) into the binary's ldflags was a deliberate choice for operational visibility. It allows operators to verify at a glance which version of the code is running, which proved essential when debugging whether the fix had actually been deployed.

Conclusion

Message 1951 is a study in the hidden complexity of production deployments. On its surface, it's a two-line confirmation of a file's existence. But in the context of the session, it represents the successful resolution of a multi-layered build problem involving Docker caching, shared library dependencies, and the unique requirements of GPU-accelerated proving infrastructure. The 163MB binary at /tmp/curio-psfix carried within it the deadlock fix, the job ID collision fix, and the queue cleanup improvements — all the result of systematic root-cause analysis and iterative debugging. It's a reminder that in distributed systems, the final step of getting a fix onto a production machine can be as challenging as finding the bug itself.