The Verification That Exposed a Deployment Gap: Binary Version Checking in Distributed Systems

Introduction

In the lifecycle of any production software deployment, the moment between "code committed" and "code running" is fraught with peril. A developer can write perfect code, pass every test, and craft an impeccable commit message, but if the binary sitting on the production server doesn't reflect those changes, none of it matters. This article examines a single message from an opencode coding session—message index 2235—where an assistant, having just committed a major refactor to remove a legacy dependency (Lassie) and enable HTTP-only repair workers for a distributed Filecoin storage gateway, pauses to verify that the deployed binaries match the freshly built code. What seems like a mundane status check reveals deep truths about deployment hygiene, the gap between development and operations, and the quiet vigilance required to run distributed systems.

The Message in Full

The subject message reads:

All services running. Let me check the current binary versions vs what we just built:

``` # Check deployed binary hash vs local echo "=== Local kuri binary ===" md5sum /home/theuser/gw/kuri

echo "" echo "=== Deployed kuri on kuri1 ===" ssh 10.1.232.83 "md5sum /opt/fgw/bin/kuri"

echo "" echo "=== Deployed kuri on kuri2 ===" ssh 10.1.232.84 "md5sum /opt/fgw/bin/kuri" ```

``` === Local kuri binary === 73fa47fa22ecd6c4b4219916fc0de814 /home/theuser/gw/kuri

=== Deployed kuri on kuri1 === e58d6ef864a7ac771863c6a3574d66cf /opt/fgw/bin/kuri

=== Deployed kuri on kuri2 === e58d6ef864a7ac771863c6a3574d66cf /opt/fgw/bin/kuri ```

On its surface, this is a straightforward diagnostic. But the implications ripple outward in several directions.

Why This Message Was Written: Reasoning, Motivation, and Context

To understand why this message exists, one must understand what preceded it. The assistant had just completed a significant refactoring effort spanning multiple files across the codebase. The changes included:

Removing the Lassie dependency from go.mod and go.sum, a library for Graphsync-based retrieval that had become legacy code.
Rewriting deal_repair.go to implement HTTP-only group retrieval from storage providers, replacing the old Lassie-based approach with a simpler, more reliable HTTP mechanism.
Enabling repair workers via startRepairWorkers() in the startup path of ribs.go.
Updating Ansible roles to include RIBS_REPAIR_WORKERS, RIBS_REPAIR_STAGING_PATH, and a dedicated directory creation task.
Committing all changes with a detailed commit message summarizing the work. After the commit, the assistant checked service status on all three nodes (S3 proxy, kuri1, kuri2) and confirmed they were all running. But "running" is not the same as "running the right code." The assistant recognized this gap and initiated a binary version comparison. The motivation is rooted in a fundamental operational principle: trust but verify. The services were active, but were they active with the new code? The only way to know was to compare cryptographic hashes of the binaries. This is especially critical in a distributed system where multiple nodes must be coordinated. If kuri1 gets the new binary but kuri2 doesn't, the system enters a mixed-version state that can produce subtle, hard-to-debug failures. The context also matters: this is a QA cluster (the IP addresses 10.1.232.x suggest a private network), and the assistant is iterating rapidly through production deployment debugging. In segment 12 of the session, the themes include "iterative production debugging" and "production deployment iteration." The assistant is operating in a mode where each change must be validated before moving to the next issue. The CIDgravity API timeout problem that triggered this whole line of work is still unresolved, and the assistant needs to ensure that the repair worker changes don't introduce new problems while fixing old ones.## How Decisions Were Made in This Message The decision to check binary hashes rather than simply restarting services or checking version strings reveals a sophisticated understanding of deployment verification. The assistant could have taken several other approaches: - Check a version endpoint: Query the running service for a version string or build timestamp. But version strings can be misleading—they're easy to forget to update, and they don't change when only a subset of files are modified. - Restart the services and assume the new binary is picked up: This assumes the deployment pipeline correctly copies the binary to the right location, which is exactly what needs to be verified. - Check process start time: See if the process was restarted recently. But this doesn't confirm the correct binary is running. Instead, the assistant chose cryptographic hash comparison—the most reliable method available without direct binary inspection on the target machines. MD5 (while not cryptographically secure for security purposes) is perfectly adequate for integrity verification: if two files have different MD5 hashes, they are guaranteed to be different. The choice of md5sum over sha256sum or sha1sum is pragmatic. MD5 is faster, universally available on Linux systems, and sufficient for detecting accidental binary mismatches. The assistant isn't worried about malicious tampering; they're worried about whether the deployment script actually copied the file. The three-way comparison—local binary vs. kuri1 vs. kuri2—is also a deliberate design. By checking both remote nodes, the assistant can detect:
Whether either node has the new binary (if either hash matches local)
Whether the two remote nodes are consistent with each other (if both hashes match)
Whether there's a partial deployment (one node updated, the other not) This is the kind of diagnostic that reveals systemic issues in the deployment pipeline.

Assumptions Made by the User or Agent

This message rests on several assumptions, most of which are reasonable but worth examining:

Assumption 1: The binary is deployed to a known path. The assistant assumes /opt/fgw/bin/kuri is the correct path on both remote nodes. This is based on the Ansible role configuration, which presumably defines this path. If the deployment script had a bug and placed the binary elsewhere, this check would show a hash mismatch even if the correct binary was running from a different location.

Assumption 2: The services are running the binary at that path. Systemd services can be configured with any executable path. The assistant assumes that kuri.service points to /opt/fgw/bin/kuri. If the service file was manually edited or uses a symlink, the hash check could be misleading.

Assumption 3: MD5 hash mismatch implies the binary hasn't been updated. This is logically sound—different hashes mean different files—but it doesn't account for the possibility that the new binary was deployed but the old process is still running (i.e., the service wasn't restarted). The assistant checked service status earlier, so this is partially addressed.

Assumption 4: The local binary represents the desired state. The assistant assumes that the binary built locally (/home/theuser/gw/kuri) is the one that should be deployed. If the build process produced an incorrect binary (e.g., due to a dirty build environment or missing flags), the hash comparison would confirm deployment of a bad binary.

Assumption 5: Both remote nodes should have identical binaries. In this architecture, kuri1 and kuri2 are storage nodes of the same type. If they had different configurations or were running different versions (e.g., during a rolling upgrade), identical hashes would not be expected. The assistant's implicit expectation that both hashes match is correct for this deployment.

Mistakes or Incorrect Assumptions

The most notable finding from this check is that the local binary hash does not match either remote binary hash. The local binary is 73fa47fa22ecd6c4b4219916fc0de814, while both remote nodes show e58d6ef864a7ac771863c6a3574d66cf. This means:

The newly built binary (with the Lassie removal and repair worker changes) has not yet been deployed to either kuri node.
Both kuri nodes are running the same older version, which is good for consistency but means the new code is not in effect.
The deployment pipeline—whether Ansible-based or manual—has not been executed since the commit. This is not necessarily a mistake by the assistant; it's a deliberate pause to assess the situation before proceeding. The assistant could have blindly run ansible-playbook to deploy, but instead chose to first understand the current state. The mistake would be if the assistant assumed the new binary was already deployed and moved on without checking. However, there is a subtle incorrect assumption visible in the message's framing: "All services running. Let me check the current binary versions vs what we just built." The phrase "vs what we just built" implies an expectation that the deployed binaries might match the local one. The assistant may have assumed that the deployment was automatic or that a previous step had already copied the binary. The hash mismatch reveals this assumption was wrong. Another potential issue: the assistant checked service status on the S3 proxy node but only checked binary hashes on the kuri nodes. The S3 proxy binary (s3-proxy) was also rebuilt during make, but its deployed hash was not verified. This is a minor oversight—if the S3 proxy code had also changed, it would need updating too.## Input Knowledge Required to Understand This Message To fully grasp what this message communicates, a reader needs familiarity with several domains: Distributed systems deployment. The concept of multiple nodes (kuri1, kuri2, S3 proxy) that must be kept in sync is fundamental. The reader must understand that in a distributed storage system, all nodes should run the same version of the software to avoid protocol incompatibilities, data corruption, or routing errors. Binary integrity verification. The use of md5sum as a verification tool assumes knowledge of cryptographic hashing. The reader needs to know that an MD5 hash is a fixed-length fingerprint of a file's contents, and that two identical files produce identical hashes. This is a standard DevOps practice, but it's not obvious to someone without deployment experience. SSH and remote command execution. The syntax ssh 10.1.232.83 "md5sum /opt/fgw/bin/kuri" executes a command on a remote host and returns the output. The reader must understand SSH key-based authentication and the concept of running commands on remote machines. The project architecture. The Filecoin Gateway (FGW) system consists of S3 frontend proxies that route requests to Kuri storage nodes, which in turn store data in YugabyteDB. The "kuri" binary is the storage node executable. Understanding that kuri1 and kuri2 are the two storage nodes in a QA cluster is essential context. The deployment pipeline. The binary at /opt/fgw/bin/kuri is placed there by Ansible roles. The reader needs to know that Ansible is the deployment automation tool used for this project, and that the binary path is defined in the Ansible role defaults. The development workflow. The local binary at /home/theuser/gw/kuri is built by running make or go build. The reader must understand that building a Go project produces a binary in the project root, and that this binary must be copied to remote servers to deploy changes.

Output Knowledge Created by This Message

This message produces several concrete pieces of knowledge that drive subsequent actions:

1. A clear deployment gap is identified. The hash mismatch between local (73fa47fa...) and remote (e58d6ef8...) binaries is definitive proof that the new code is not deployed. This is actionable intelligence: the assistant must now run the deployment pipeline.

2. Both remote nodes are consistent. The fact that kuri1 and kuri2 have the same hash (e58d6ef8...) means they are running the same version. This rules out a partial deployment scenario where one node was updated but the other wasn't. If the assistant proceeds with deployment, both nodes can be updated simultaneously.

3. The deployment pipeline has not been triggered. Since the commit was made but the binary hasn't changed on the remote nodes, the assistant knows that the deployment step is still pending. This informs the next action: run the Ansible playbook or manually copy the binary.

4. A baseline for future verification. After deployment, the assistant can re-run the same md5sum commands to confirm that all three hashes match. This creates a closed-loop verification cycle: build → deploy → verify.

5. Documentation of the current state. This message becomes part of the conversation history, serving as a record that at this point in time, the QA cluster was running an older version of the kuri binary. If issues arise later, this timestamped record helps determine what code was running when.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, while not explicitly stated in reasoning tags within this message, can be inferred from the sequence of actions and the structure of the diagnostic:

Step 1: Verify service health. Before checking binary versions, the assistant confirmed that all services were running. This is a logical ordering: if a service is down, binary version is irrelevant—the first priority is restoring service. Only after confirming all services are "active (running)" does the assistant proceed to deeper verification.

Step 2: Choose the right verification method. The assistant could have checked version strings, build timestamps, or file sizes. Instead, it chose MD5 hashes, which are the gold standard for file identity verification. This shows an understanding that version strings can be stale or misleading, while cryptographic hashes are definitive.

Step 3: Design a three-way comparison. The assistant doesn't just check one remote node; it checks both. This reveals a mental model of the system as a distributed entity where all nodes must be synchronized. The comparison structure—local vs. kuri1 vs. kuri2—is essentially a matrix that can detect multiple failure modes.

Step 4: Interpret the results implicitly. The assistant doesn't explicitly state "the new binary is not deployed" in this message. The raw output is presented without commentary. This is a deliberate choice: the data speaks for itself. The assistant trusts the reader (or itself, in a future review) to interpret the mismatch. This is a hallmark of good diagnostic practice—present evidence neutrally rather than jumping to conclusions.

Step 5: Prepare for the next action. The message ends with the hash output and no further commands. This is a pause point. The assistant is gathering information before deciding the next step. The implicit next action is to deploy the new binary, but the assistant hasn't committed to that yet—it's still in assessment mode.

Broader Implications for Deployment Hygiene

This message, while brief, illustrates a principle that separates professional operations from amateur deployments: verification is not optional. Many developers, especially when working on personal projects or small teams, assume that if make succeeds and systemctl restart returns without error, the new code is running. This message demonstrates a more rigorous approach.

The cost of skipping this verification is subtle but real. Consider a scenario where the Ansible playbook fails silently—perhaps the binary copy task errors out due to a permissions issue, but the playbook continues and reports success. The service restarts using the old binary. The developer sees "service active (running)" and assumes the fix is in place. Hours later, when the bug isn't fixed, the developer must retrace steps and eventually discover the deployment failure. The MD5 check catches this immediately.

In distributed systems, the stakes are even higher. If kuri1 gets the new binary but kuri2 doesn't, the two nodes may disagree on protocol versions, data formats, or repair strategies. This can lead to data corruption, unreachable objects, or cascading failures that are extremely difficult to diagnose. The three-way hash comparison catches partial deployments before they cause harm.

The message also highlights the importance of idempotent deployment verification. The same md5sum commands can be run before and after deployment to confirm the change took effect. This creates a repeatable verification script that can be automated into the CI/CD pipeline.

Conclusion

Message 2235 is a masterclass in operational discipline disguised as a routine status check. On the surface, it's three md5sum commands and their output. But beneath that surface lies a sophisticated understanding of distributed system deployment, a commitment to verification over assumption, and a mental model that treats the production environment as something to be measured rather than guessed at.

The hash mismatch it reveals—local 73fa47fa vs. remote e58d6ef8—is not a failure. It's a discovery. It tells the assistant that the deployment step is still pending, that both nodes are consistent, and that the next action is clear. Without this check, the assistant might have moved on to debugging the CIDgravity timeout issue, wasting time investigating a problem that might already be fixed in the undeployed code.

In the high-stakes world of distributed storage systems, where data integrity and availability are paramount, this kind of verification is not optional. It is the difference between hoping the system works and knowing it does. Message 2235 captures that ethos in three lines of shell commands and a table of hexadecimal fingerprints.