The Three Percent Difference: How a Format String Bug Nearly Broke a Distributed Proving System

"Now deployed for real. _psfix3 with ps-porep-%d-%d-%d (includes taskID). Concurrent proofshare jobs will have unique cuzk job IDs."

At first glance, message [msg 2013] reads like a mundane status update—a developer confirming that a fix has been deployed. The assistant's tone is matter-of-fact, almost anticlimactic after the storm of debugging that preceded it. But this short message represents the culmination of one of the most frustrating classes of software bugs: a deployment that looks successful but isn't. The journey from psfix2 to psfix3 reveals how the gap between "code changed" and "code running" can swallow hours of engineering effort, and how the humble format string ps-porep-%d-%d versus ps-porep-%d-%d-%d became the difference between a working distributed proving system and one that silently corrupted proofs.

The Context: A Production System Under Stress

To understand why this message exists, we must first understand the system it operates within. The assistant is working on a Filecoin Curio node—a GPU-accelerated proving system that generates cryptographic proofs for the Filecoin network. Within this system, the ProofShare protocol allows multiple parties to collaboratively generate proofs, splitting the work across partitions. The CuZK engine handles the GPU-accelerated proving pipeline, and each job submitted to it is identified by a RequestId (which becomes a job_id inside the engine).

The production bug was devastating: all ten PoRep (Proof of Replication) partitions were producing invalid proofs. The root cause, traced across multiple debugging sessions, was a job ID collision. The RequestId was formatted as ps-porep-%d-%d using only the miner ID and sector number. But ProofShare challenges all target the same artificial benchmark sector (miner=1000, sector=1), so every concurrent PSProve task sent the identical job_id to CuZK. The engine's partition assembler keyed its internal HashMap on job_id, causing partition results from different proofs to mix together—confirmed by a "partition 0 already inserted" panic. The fix was trivial in concept: add the harmony task ID to the format string, producing ps-porep-%d-%d-%d.

The Deployment Ordeal: When Code Changes Don't Survive the Build

What makes [msg 2013] significant is not the fix itself—that was already implemented in psfix2—but the fact that it took three separate build-and-deploy cycles to get the change to actually stick. The assistant's earlier message [msg 1996] confidently declared the fix deployed with _psfix2, yet [msg 1997] showed the CuZK logs still emitting ps-porep-1000-1 without the task ID. Something was wrong.

The assistant's reasoning in [msg 2001] and [msg 2002] reveals the detective work. The binary was confirmed as psfix2 via the version string, but grep -ao "ps-porep-%d-%d[^\"]*" /usr/local/bin/curio returned only the two-%d format. The Docker Go build cache had not recompiled the modified source files. The docker cp approach used in [msg 1986] copied files into the container's writable layer, but the --volumes-from mount in [msg 1987] only shared declared Docker volumes—not the entire filesystem. The Go build cache, stored in /root/.cache/go-build inside the image layers, still had compiled object files from the original build. The touch command in [msg 2004] was a reasonable attempt to bust the cache by updating mtimes, but it failed because the files being touched weren't the ones actually being compiled—the --volumes-from approach wasn't sharing the modified source files at all.

This is a subtle Docker behavior that catches many developers. docker cp writes to the container's ephemeral writable layer, but --volumes-from only exposes volumes declared with the VOLUME directive in the Dockerfile. Since the curio-builder image likely didn't declare /build as a volume, the copied files were invisible to the build container. The assistant's breakthrough came in [msg 2005] by switching to direct bind mounts with -v flags:

-v /tmp/czk/tasks/proofshare/task_prove.go:/build/tasks/proofshare/task_prove.go

This forced the container to see the actual modified files from the host, and the Go compiler, detecting the changed source, performed a full recompile. The resulting binary in [msg 2006] finally contained ps-porep-%d-%d-%d.

The Second Surprise: Deployment Failure After Successful Build

Even with the correct binary built, deployment nearly failed again. In [msg 2007], the assistant chained kill + mv in a single SSH command, but the output was empty—a red flag. [msg 2009] reveals the problem: the binary still showed _psfix2 and the old format string. The mv had failed silently because the running Curio process locked the file at /usr/local/bin/curio. The kill command sent SIGTERM, but the process hadn't fully terminated by the time mv executed.

The assistant's response in [msg 2010] and [msg 2011] demonstrates proper operational discipline: separate the kill, verify the process stopped, then copy the file. This step-by-step approach—kill 210085; sleep 3; pgrep -a curio || echo "curio stopped"—ensured the file was unlocked before replacement. The final verification in [msg 2011] shows matching MD5 hashes between /tmp/curio-psfix3 and /usr/local/bin/curio, the correct format string ps-porep-%d-%d-%d, and the _psfix3 version tag.

The Meaning of "Now Deployed for Real"

Message [msg 2013] is the exhausted exhale after this ordeal. The word "real" carries the weight of three build attempts, two failed deployments, and hours of debugging not the proof system itself but the deployment pipeline. The assistant is making several implicit claims:

1. The binary is correct: The format string ps-porep-%d-%d-%d was confirmed via grep -a on the binary itself, not just inferred from the build process.
2. The binary is running: Curio was started with nohup and confirmed via pgrep.
3. The fix will work: By including the task ID, concurrent ProofShare jobs targeting the same miner/sector will produce unique CuZK job IDs, preventing partition result mixing. The message also reveals an assumption: that the fix is complete. The assistant states "Concurrent proofshare jobs will have unique cuzk job IDs" with certainty. This is a reasonable conclusion given the root cause analysis—the partition assembler's HashMap keyed on job_id was the sole mechanism for proof isolation—but it assumes no other code path uses the same RequestId for different purposes. The assistant later performs a thorough audit of all CuZK RequestId callers in the subsequent chunk, confirming that only the ProofShare PoRep path was vulnerable.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

• Filecoin proving architecture: The distinction between PoRep, WindowPoSt, WinningPoSt, and SnapDeals proof types, and how they flow through the CuZK GPU engine.
• The ProofShare protocol: How collaborative proof generation works, with multiple parties submitting challenges against the same benchmark sector.
• Docker build caching: How Go build artifacts are cached in Docker image layers, and why docker cp + --volumes-from doesn't always work as expected.
• CuZK job routing: How the engine's partition assembler uses job_id as a HashMap key to collect partition results before final proof assembly.
• Harmony task scheduling: How Curio's harmony system manages distributed work items and assigns task IDs.

Output Knowledge Created

This message creates several forms of knowledge:

• A verified deployment procedure: The sequence of build with bind mounts, kill-verify-copy, and binary string verification becomes a reusable pattern for future hot-patches.
• Confirmation of the root cause hypothesis: The fix working (assuming subsequent logs confirm unique job IDs) validates the theory that job ID collision caused partition mixing.
• A documented Docker build cache pitfall: The lesson that --volumes-from doesn't share docker cp-ed files unless the image declares VOLUME directives is now captured in the session history.

The Thinking Process

The assistant's reasoning across messages [msg 1998] through [msg 2012] shows a systematic narrowing of possibilities. When the CuZK logs still showed the old format after psfix2 deployment, the assistant considered three hypotheses: (1) old jobs still in the CuZK pipeline, (2) the binary wasn't actually replaced, or (3) the Go build cache prevented recompilation. Each was tested in order, with the binary string search in [msg 2002] providing the definitive answer. The pivot from --volumes-from to bind mounts in [msg 2005] shows adaptive problem-solving—when one approach fails, try a fundamentally different mechanism rather than tweaking parameters.

The most striking aspect is the assistant's insistence on verifying the binary's contents, not just its metadata. Checking the version string (_psfix2) was insufficient—the format string had to be confirmed directly. This is a lesson in deployment verification: trust the binary, not the build log.

Conclusion

Message [msg 2013] is a single line of text, but it encapsulates hours of debugging, three build cycles, two failed deployments, and the hard-won understanding that in distributed systems, the difference between a working fix and a broken one can be as small as a single %d in a format string—and as large as the gap between "deployed" and "deployed for real."