Deploying a Data Integrity Fix: The Remote Binary Update That Preserved Benchmark History

The Message

In message 1529 of a sprawling infrastructure development session, the assistant executed a single, dense command to deploy a rebuilt binary to a remote host and restart the management service:

[bash] cat /tmp/czk/vast-manager | ssh 10.1.2.104 'cat > /tmp/vast-manager.new' && ssh 10.1.2.104 'sudo mv /tmp/vast-manager.new /usr/local/bin/vast-manager && sudo chmod +x /usr/local/bin/vast-manager && sudo systemctl restart vast-manager && sleep 1 && systemctl status vast-manager --no-pager | head -10'

● vast-manager.service - Vast Worker Manager Loaded: loaded (/etc/systemd/system/vast-manager.service; enabled; preset: enabled) Active: active (running) since Thu 2026-03-12 12:28:12 UTC; 1s ago Main PID: 121651 (vast-manager) Tasks: 6 (limit: 35761) Memory: 2.0M (peak: 2.8M) CPU: 15ms CGroup: /system.slice/vast-manager.service └─121651 /usr/local/bin/vast-manager --listen :1235 --ui-listen 0.0.0.0:1236 --db /var/lib/vast-manager/state.db

At first glance, this is a routine deployment: pipe a binary over SSH, install it, restart the service, verify it is running. But this message is the culmination of a critical data-integrity bug hunt that exposed a subtle flaw in how the system recorded benchmark performance metrics. Understanding why this single command was necessary requires tracing the chain of reasoning that led to it.

The Bug: When Good Data Overwrites Better Data

The story begins in the preceding messages. The assistant was monitoring a batch of benchmark runs on rented GPU instances managed through a custom vast-manager service. This service orchestrates instances on vast.ai, runs benchmarks to measure proofs-per-hour throughput, and records results in a SQLite database table called host_perf. The table schema used a composite primary key of (machine_id, gpu_name, num_gpus), and the insertion logic used INSERT ... ON CONFLICT DO UPDATE SET — meaning if a record already existed for a given machine, GPU model, and GPU count, the new values would simply overwrite the old ones.

This seemed reasonable until a specific scenario unfolded. Two RTX 4090 instances were deployed simultaneously on the same physical machine (machine ID 15136) in Norway. Both ran benchmarks at the same time, competing for PCIe bandwidth, memory bandwidth, and CPU resources. The first instance completed with an impressive 61.4 proofs/hour — well above the minimum threshold of 59. It passed and was promoted to "running" status. The second instance, contending with the first, managed only 46.2 proofs/hour — below the threshold, so it was destroyed.

But when the second instance reported its result, the ON CONFLICT clause kicked in and overwrote the 61.4 record with 46.2. The higher benchmark score was permanently lost from the database. The assistant spotted this immediately ([msg 1525]):

Wait — the RTX 4090 on machine 15136 shows 46.2 (the second run). But the first run on the same machine got 61.4. The host_perf table uses PRIMARY KEY (machine_id, gpu_name, num_gpus), so the second result overwrote the first! That's a problem — we lost the higher benchmark.

This was a data integrity bug with real consequences. The host_perf table is used to make deployment decisions — which machines to prefer, which to avoid, and how to price proofs. Losing the highest benchmark score meant the system would underestimate a machine's true capability, potentially causing it to be flagged as underperforming or deprioritized for future workloads.

The Fix: MAX Instead of Overwrite

The fix was conceptually simple but required surgical precision. In message 1527, the assistant edited the SQL statement in main.go to use MAX() on the bench_rate column during the conflict update. Instead of DO UPDATE SET bench_rate = ?, the new logic became DO UPDATE SET bench_rate = MAX(bench_rate, ?). This ensured that only a higher benchmark score could overwrite an existing record — lower scores from contended runs would be harmlessly ignored.

The assistant then built the binary in message 1528, noting a pre-existing LSP error about a missing ui.html file that was harmless (the embed pattern lookup path is relative and works correctly during the actual build). The build completed, producing the vast-manager binary at /tmp/czk/vast-manager.

The Deployment: Anatomy of a Remote Update

Message 1529 performs the actual deployment. The command is structured as a shell pipeline with two SSH invocations chained by &&:

Transfer the binary: cat /tmp/czk/vast-manager | ssh 10.1.2.104 'cat > /tmp/vast-manager.new' — This pipes the local binary over SSH to a temporary file on the remote host. Using cat > rather than scp avoids needing a separate file transfer tool and works within the existing SSH trust relationship.
Install and restart: The second SSH command performs four operations atomically: - sudo mv /tmp/vast-manager.new /usr/local/bin/vast-manager — Replaces the old binary atomically (mv is atomic on the same filesystem) - sudo chmod +x /usr/local/bin/vast-manager — Ensures execute permissions - sudo systemctl restart vast-manager — Restarts the systemd service to pick up the new binary - sleep 1 && systemctl status vast-manager --no-pager | head -10 — Waits briefly and verifies the service started correctly The output confirms success: the service is active (running), started at 12:28:12 UTC (just 1 second before the status check), with PID 121651. The command-line arguments show it listens on port 1235 for API traffic and port 1236 for the web UI, with its SQLite database at /var/lib/vast-manager/state.db.

Assumptions and Risks

The assistant made several assumptions during this deployment:

Binary compatibility: The Go binary was cross-compiled with GOOS=linux GOARCH=amd64 (visible in msg 1528). The assistant assumed the remote host runs Linux on amd64 architecture, which is reasonable for a vast.ai instance but not verified in this message.
SSH trust: The command assumes passwordless SSH access to 10.1.2.104 with sufficient privileges to write to /tmp and execute sudo. This trust relationship was established earlier in the session.
Atomic replacement safety: Using mv to replace a running binary is safe on Linux because the kernel keeps the old binary's inode open for running processes. However, if the service restarts before the mv completes, there's a brief window where the binary could be missing. The assistant mitigated this by writing to a temporary file first, then using a single mv.
LSP error dismissal: The assistant assumed the ui.html embed error was harmless. This was correct in this case — the Go embed directive uses a pattern that's resolved at compile time relative to the source file, and the LSP was running from a different working directory. But dismissing LSP errors without full verification carries a risk of shipping broken code.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with Go cross-compilation and the GOOS/GOARCH environment variables; understanding of SQLite ON CONFLICT clauses and the MAX() aggregate function; knowledge of systemd service management (systemctl restart, systemctl status); SSH piping techniques for file transfer; and the broader context of the vast-manager system as a GPU instance orchestrator that benchmarks proof generation throughput.

Output knowledge created by this message includes: a verified deployment of the fixed binary on the remote host; confirmation that the service restarted successfully with the correct command-line arguments; and, implicitly, the guarantee that future benchmark results will no longer overwrite higher scores with lower ones. The host_perf table will now accumulate the best observed performance for each machine, providing a more accurate basis for deployment decisions.

The Thinking Process

The assistant's reasoning chain across messages 1525–1529 reveals a disciplined debugging methodology. First came observation: noticing the discrepancy between the 61.4 and 46.2 scores for the same machine. Then diagnosis: tracing the bug to the ON CONFLICT SQL logic. Then fix design: choosing MAX() as the minimal, correct change. Then implementation: editing the Go source, rebuilding, and deploying. Finally verification: checking the service status output.

The choice of MAX() rather than a more complex approach (like storing all measurements and computing percentiles, or adding a separate "best_score" column) reflects a pragmatic trade-off. The assistant prioritized a minimal, correct fix that could be deployed immediately over a more sophisticated solution. This is characteristic of operations engineering: fix the data loss first, then consider richer analytics later.

Broader Significance

This message, while brief, captures a fundamental tension in distributed systems: the gap between what a benchmark measures and what it means. The RTX 4090 on machine 15136 was capable of 61.4 proofs/hour, but resource contention made it look like 46.2. Without the MAX() fix, the system would have permanently remembered the lower number, making incorrect decisions about that machine's suitability for future work. The deployment in message 1529 closes that gap, ensuring that the system's memory of its hardware's capability reflects its true potential rather than a transient artifact of parallel execution.

In the larger narrative of the session, this fix represents a shift from platform building to platform hardening — the kind of subtle data integrity work that distinguishes a prototype from a production system.