The Hot-Swap: Deploying a Critical Proof Verification Fix to a Production GPU Proving Cluster

Introduction

In the high-stakes world of Filecoin proof generation, an intermittent failure is more dangerous than a consistent one. A consistent failure stops the pipeline and demands attention; an intermittent one silently corrupts output, allowing invalid proofs to propagate through the system until they are rejected downstream—potentially wasting hours of computation and damaging the node's reputation in the ProofShare challenge protocol. Message [msg 1896] captures the culmination of a deep investigative and engineering effort: the hot-swap deployment of a patched cuzk binary to a production GPU worker, replacing a running daemon that had been silently returning invalid proofs to its callers.

This message is the final act in a multi-round debugging saga that spanned several sub-sessions, crossing Go, Rust, C++, and CUDA codebases. It is a study in surgical production engineering: building a minimal binary locally, transferring it to a remote machine with no Rust toolchain, and swapping it into a live process with minimal downtime—all while ensuring the fix is comprehensive across all code paths.

The Bug: A Diagnostic That Wasn't a Gate

The root cause, uncovered through painstaking analysis in earlier messages, was deceptively simple. The cuzk GPU proving engine, written in Rust and backed by CUDA-accelerated supraseal C++ libraries, had a self-check mechanism for verifying proofs after generation. However, this self-check was diagnostic only: when verify_porep_proof() returned Ok(false)—indicating an invalid proof—the engine logged a warning but still returned JobStatus::Completed with the bad proof bytes to the caller. The Go-side VerifySeal call would then correctly reject the proof, but by then the damage was done: the invalid proof had already been transmitted, the round had been wasted, and the intermittent nature of the GPU proving instability made the failure pattern unpredictable and hard to debug.

The fix, applied in earlier messages to /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs, changed the control flow so that a failed self-check returns JobStatus::Failed instead of JobStatus::Completed. This turned a diagnostic whisper into a hard error, preventing invalid proofs from ever reaching the caller.

Proactive Defense: Fixing All Pipeline Paths

What distinguishes this engineering effort is the systematic approach. After confirming the fix for the Phase 7 pipeline path (the one running on the production machine, configured with partition_workers = 16), the assistant did not stop there. A thorough audit of engine.rs using rg searches revealed that the same diagnostic-only self-check pattern existed in two additional pipeline assembly paths: the batched multi-sector path and the single-sector pipeline path. Both were fixed in rapid succession ([msg 1874] and [msg 1875]), ensuring that all four pipeline modes (Phase 6 slot-based, Phase 7 partition-worker, batched multi-sector, and single-sector) now enforce the self-check as a mandatory gate.

This proactive hardening is a hallmark of mature engineering: recognizing that a bug is not an isolated incident but a class of bugs, and systematically eliminating every instance rather than patching only the symptomatic path.

The Deployment Challenge: No Rust Toolchain on the Remote

With the code fixes in place, the next challenge was deployment. The production machine (host c23b146a4796, an RTX 3090-based GPU worker on the vast.ai network) had no Rust toolchain installed. The cuzk binary at /usr/local/bin/cuzk was a pre-built 27MB executable. Rebuilding the entire Docker image and restarting the container would be slow and disruptive—the user explicitly stated in [msg 1867]: "We don't want to restart the whole node - very slow, so just build the binaries here and update on the remote."

The assistant devised a clever strategy: create a minimal Dockerfile (Dockerfile.cuzk-rebuild) that uses the existing CUDA 13 devel environment (which was already cached locally from previous builds), mounts the source tree, and builds only the cuzk-daemon binary using cargo build --release. The resulting binary would be extracted from the Docker scratch image and transferred via SCP.

This approach leveraged Docker layer caching: the CUDA devel image was already present (nvidia/cuda:13.0.2-devel-ubuntu24.04), and the Rust dependency compilation was likely cached from the previous full Docker build. The build completed quickly, producing a 27MB binary—identical in size to the production binary, confirming that the build configuration was consistent.

The Hot-Swap: Message [msg 1896] in Detail

The subject message executes the hot-swap via a single SSH command that performs six operations in sequence:

ssh -p 40362 root@141.195.21.72 '
# Backup old binary
cp /usr/local/bin/cuzk /usr/local/bin/cuzk-old

# Get the cuzk PID  
CUZK_PID=$(pgrep -f "cuzk.*--config" | head -1)
echo "Current cuzk PID: $CUZK_PID"

# Swap binary
mv /usr/local/bin/cuzk-new /usr/local/bin/cuzk
chmod +x /usr/local/bin/cuzk

# Get the full command line so we can restart with same args
CUZK_CMD=$(cat /proc/$CUZK_PID/cmdline | tr "\0" " ")
echo "Command: $CUZK_CMD"

# Kill the old process
kill $CUZK_PID
sleep 2

# Verify it died
if kill -0 $CUZK_PID 2>/dev/null; then
    echo "Process still alive, sending SIGKILL"
    kill -9 $CUZK_PID
    sleep 1
fi

# Restart with same args, log to same file
nohup /usr/local/bin/cuzk --config /tmp/cuzk-run-config.toml --listen 0.0.0.0:9820 --log-level info > /tmp/cuzk-daemon.log 2>&1 &
NEW_PID=$!
echo "New cuzk PID: $NEW_PID"
sleep 2
# Check if it started OK
if kill -0 $NEW_PID 2>/dev/null; then
    echo "cuzk restarted successfully"
else
    echo "FAILED to start cuzk!"
    tail -20 /tmp/cuzk-daemon.log
fi
'

The output confirms success: the old PID was 9371 (a process that had been running since March 12, with over 4344 minutes of CPU time), and the new PID is 170402. The process required a SIGKILL because the graceful kill did not terminate it within the 2-second window—not unusual for a GPU-accelerated process that may be in the middle of a CUDA kernel call.

Assumptions and Risks in the Deployment

Several assumptions underpin this hot-swap, each carrying risk:

  1. Binary compatibility: The locally built binary assumes the same CUDA runtime library versions are present on the remote machine. The production machine runs the CUDA 13 runtime (matching the devel image), but any mismatch in libcuda.so or libcudart.so versions could cause crashes at load time.
  2. Configuration parity: The command-line arguments were reconstructed from /proc/$PID/cmdline. This captures the exact arguments, but any environment variables or runtime state that was set during the original process launch (e.g., via the entrypoint.sh script) would need to be replicated. The new process starts with the same explicit arguments but inherits the SSH session's environment, which may differ from the original Docker environment.
  3. No data loss: The old process was killed without any graceful shutdown signal (SIGTERM was attempted, but SIGKILL was required). If the process was in the middle of writing proof results to a socket or database, there is a risk of incomplete I/O. However, given that the process is a GPU proving daemon that communicates via network sockets, the clients should handle connection drops gracefully.
  4. SRS parameter loading: The cuzk daemon preloads SRS parameters (porep-32g). The new process must reload these from disk, which takes time and memory. The 2-second sleep before the health check may not be sufficient if parameter loading is slow, but the startup check only verifies that the process is alive, not that it is fully initialized.

Knowledge Input and Output

To understand this message, the reader needs knowledge of:

The Thinking Process: Why This Approach Was Chosen

The assistant's reasoning is visible in the progression of messages leading to [msg 1896]. The initial plan ([msg 1860]) included "Build cuzk on the remote machine" as a pending task. However, after discovering that the remote had no Rust toolchain ([msg 1862]: rustc --version and cargo --version were not checked but the absence was inferred from the minimal OS setup), the assistant pivoted. The user's guidance in [msg 1867] confirmed the strategy: build locally, deploy binary-only.

The Docker-based build was chosen over cross-compilation because:

  1. The CUDA devel image was already cached locally
  2. The build dependencies (Rust 1.86, gcc-13, CUDA headers) were already configured in the Dockerfile
  3. Docker layer caching would reuse the compiled dependency tree from the previous full build The assistant encountered a minor obstacle extracting the binary from a scratch-based Docker image (no cat, no shell), and solved it by using docker create with an explicit --entrypoint /bin/true flag to create a container that could be used with docker cp. This is a non-obvious Docker trick that demonstrates deep familiarity with container internals.

Significance and Impact

This deployment represents the hardening of a critical production component against a class of bugs that could silently corrupt the proof generation pipeline. Before the fix, an intermittent GPU proving error would produce an invalid proof that would be returned to the caller, only to be rejected later by VerifySeal. The proof would be wasted, the sector would need to be re-proved, and the intermittent nature made debugging extremely difficult—the failure might not reproduce under controlled conditions.

After the fix, any invalid proof is caught at the source and rejected immediately, with a clear error message. The caller receives JobStatus::Failed and can retry or escalate appropriately. The diagnostic logging added to task_prove.go (in earlier messages) ensures that when failures do occur, all verification inputs are captured for analysis.

The hot-swap itself is a model of surgical production engineering: minimal downtime, no container restart, no Docker image rebuild, no disruption to other services on the machine. The entire operation, from binary upload to process restart, completed in seconds. The new cuzk daemon (PID 170402) began serving requests immediately, with the self-check logic fully enforced across all four pipeline modes.

Conclusion

Message [msg 1896] is the culmination of a deep debugging journey that spanned multiple programming languages, GPU architectures, and distributed systems concepts. It demonstrates that fixing a bug is not enough—the fix must be deployed, validated, and extended to all related code paths. The hot-swap technique, born from the constraint of no Rust toolchain on the remote machine, shows that production engineering often requires creative solutions that go beyond the standard deployment playbook. The result is a more robust proof generation pipeline that catches invalid proofs at the source, protecting the node's participation in the Filecoin network from intermittent GPU proving instability.