The Third Time's the Charm: Deploying a Critical Binary Through SSH Piping

In the middle of a sprawling development session spanning platform hardening, UI enhancements, and deep protocol-level debugging, there lies a seemingly mundane but richly instructive message: the deployment of a rebuilt Go binary to a remote server. Message [msg 1502] captures the moment when an AI assistant, after two failed attempts to deploy an updated vast-manager binary, pivots to a third strategy and succeeds. The message is a single bash command—a pipeline that cats a local binary through SSH, writes it on the remote host, moves it into place, restarts the service, and verifies the result. But beneath this terse command lies a cascade of reasoning about failure modes, SSH protocol quirks, privilege boundaries, and the operational realities of managing remote GPU compute infrastructure.

The Context: Why This Binary Needed to Reach the Remote Host

The assistant had been working on the vast-manager—a custom management service for orchestrating GPU instances on vast.ai, a cloud marketplace for rented compute. The manager handles instance lifecycle, benchmark orchestration, and log aggregation across dozens of rented machines running Filecoin proof-of-replication (PoRep) workloads using the CuZK proving engine. In the preceding messages ([msg 1486] through [msg 1499]), the assistant had identified a critical gap: when benchmark scripts failed on remote instances, the error output was invisible to the manager because the cuzk-daemon logs were never shipped to the manager's log-push API.

The fix involved two coordinated changes. First, the entrypoint.sh script was modified to ship two new log sources—benchdaemon (the cuzk-daemon log) and benchout (the benchmark output log)—to the manager's /api/log-push endpoint ([msg 1490], [msg 1491]). Second, the UI's log filter tabs needed to include these new sources so operators could filter logs by them ([msg 1497], [msg 1498]). The UI change was a single line edit: adding 'benchdaemon' and 'benchout' to the array of log filter tabs on line 480 of ui.html.

With both code changes complete, the assistant rebuilt the vast-manager Go binary ([msg 1499]). The build succeeded (with only cosmetic sqlite3 warnings), producing a fresh binary at /tmp/czk/vast-manager. Now came the operational step: getting this binary onto the remote controller host at 10.1.2.104 where the vast-manager service ran, and restarting the service to pick up the changes.

Two Failures, Two Lessons

The first deployment attempt ([msg 1500]) used scp:

scp /tmp/czk/vast-manager 10.1.2.104:/usr/local/bin/vast-manager.new

This failed with Permission denied. The assistant had connected as a non-root user, and /usr/local/bin/ is root-owned. The error was straightforward: the user lacked write permission to the target directory. The assistant then tried to work around this by copying to a temp location and using mv with sudo, but the scp command itself—running as the non-root user—could not open the destination file for writing.

The second attempt ([msg 1501]) switched to piping through SSH:

scp /tmp/czk/vast-manager root@10.1.2.104:/usr/local/bin/vast-manager.new

This time the assistant tried root@ to get the necessary privileges. But a different, more cryptic error appeared: Received message too long 1349281121. This is a classic SSH failure mode. The error occurs when the remote shell produces output on stdout during the SSH session setup—typically from a .bashrc, .profile, or system message—which corrupts the SCP protocol handshake. SCP relies on a clean binary protocol over the SSH channel; any stray text output from the remote shell before the SCP subprocess starts is interpreted as protocol data, causing a length mismatch. The number 1349281121 is the corrupted length field that SCP tried to parse.

This failure is particularly frustrating because it's environment-dependent: the remote server's shell configuration (likely a motd banner, a misbehaving .bashrc, or a system-wide profile script) was producing output that broke SCP but not regular interactive SSH. The assistant had no direct way to fix the remote shell configuration from this session—it needed a different transport mechanism.

The Third Approach: Piping Through Cat

Message [msg 1502] represents the assistant's third strategy, one that sidesteps both permission and protocol issues:

cat /tmp/czk/vast-manager | ssh 10.1.2.104 'cat > /tmp/vast-manager.new' && ssh 10.1.2.104 'sudo mv /tmp/vast-manager.new /usr/local/bin/vast-manager && sudo chmod +x /usr/local/bin/vast-manager && sudo systemctl restart vast-manager && sleep 1 && systemctl status vast-manager --no-pager | head -15'

This approach is elegant in its simplicity. Instead of using SCP's binary protocol, it uses a plain SSH channel with stdin forwarding. The local cat reads the binary file and pipes its contents through the SSH connection. On the remote side, cat > /tmp/vast-manager.new reads from stdin and writes to a temporary file in /tmp/—a world-writable directory that doesn't require root privileges. This avoids the Permission denied problem entirely.

The second segment of the command, after the &&, runs a new SSH session that executes the privileged operations: sudo mv to relocate the binary from /tmp/ to /usr/local/bin/, sudo chmod +x to set executable permissions, sudo systemctl restart vast-manager to restart the service, and finally systemctl status to verify the restart succeeded. By splitting the operation into two SSH invocations, the assistant keeps the unprivileged file transfer separate from the privileged installation steps.

The output confirms success. The systemctl status output shows:

● vast-manager.service - Vast Worker Manager
     Loaded: loaded (/etc/systemd/system/vast-manager.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-03-12 12:09:46 UTC; 1s ago
   Main PID: 120490 (vast-manager)
      Tasks: 6 (limit: 35761)
     Memory: 2.1M (peak: 2.8M)
        CPU: 15ms
     CGroup: /system.slice/vast-manager.service
             └─120490 /usr/local/bin/vast-manager --listen :1235 --ui-listen 0.0.0.0:1236 --db /var/lib/vast-manager/state.db

The service is running, the PID is fresh (120490, just 1 second old), and the memory usage is at a clean baseline (2.1M, peak 2.8M), confirming a clean restart. The command line shows the binary is now at /usr/local/bin/vast-manager with the expected flags.

Assumptions and Their Validity

The assistant made several assumptions in this message, most of which proved correct:

That piping binary data through SSH stdin would work reliably. This is a well-known technique, but it depends on the remote shell not interfering with the binary stream. If the remote .bashrc produced output, it would corrupt the binary just as it corrupted SCP. The assistant implicitly assumed the remote shell was quiet for non-interactive SSH commands (as opposed to SCP's subprocess invocation). This assumption held.
That /tmp/ was writable by the SSH user. On virtually all Unix systems, /tmp/ is world-writable, so this was a safe bet.
That sudo was available and the user had sudo privileges. The assistant had already used ssh root@ in the previous attempt, confirming root access was available. Using sudo via a non-root user was a reasonable fallback.
That the binary transfer completed without corruption. The && chaining means the second SSH command only runs if the first (the pipe) succeeds. If the pipe had been interrupted or truncated, the mv would still execute on whatever partial data was written, potentially leaving a broken binary. The assistant implicitly trusted that SSH's stdin forwarding would deliver the complete file. The successful systemctl restart later confirmed the binary was intact.
That systemctl restart vast-manager would succeed without dependency issues. The vast-manager service had been running for 55 minutes before the restart ([msg 1487]). Restarting it would terminate the existing process and start the new one. The assistant assumed no port conflicts, no lingering socket issues, and no database corruption from an abrupt restart. The clean status output validated this assumption.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

SSH protocol mechanics: Understanding why SCP failed with "Received message too long" requires knowing that SCP uses a binary handshake over the SSH channel and that any stray shell output corrupts it. The alternative of piping through cat works because it uses a raw stdin-forwarded SSH session without the binary handshake.
Unix file permission model: The distinction between /usr/local/bin/ (root-owned, requires sudo) and /tmp/ (world-writable) is essential to understanding why the first approach failed and the third succeeded.
Systemd service management: The systemctl restart and systemctl status commands, and interpreting the output (PID, active time, memory, command line) to confirm a clean restart.
Go binary deployment: The assistant had just compiled a Go binary with GOOS=linux GOARCH=amd64 go build. Understanding that Go produces statically linked binaries (or mostly static, with possible cgo dependencies) is relevant—the binary was self-contained and could be deployed without additional runtime dependencies.
The vast-manager architecture: Knowing that this binary is the central orchestrator for GPU instance management, that it listens on ports 1235 (API) and 1236 (UI), and that it uses a SQLite database at /var/lib/vast-manager/state.db.

Output Knowledge Created

This message produced several concrete outputs:

A successfully deployed binary at /usr/local/bin/vast-manager on the controller host, incorporating the UI log filter improvements.
A restarted service with a clean baseline (2.1M memory, PID 120490), confirming the new binary loaded without crashes or initialization errors.
Operational validation that the deployment pipeline works: build on the development machine, transfer via SSH pipe, install with sudo, restart with systemd, and verify with status.
A reusable deployment pattern for future updates: the cat | ssh 'cat >' technique is now established as a reliable fallback when SCP fails due to shell configuration issues.

The Thinking Process Visible in the Message

The structure of the command reveals the assistant's reasoning process. The command is split into two chained SSH invocations separated by &&. This is not accidental—it reflects a deliberate separation of concerns:

Phase 1 (file transfer): Uses a non-root SSH session to pipe the binary into /tmp/. This is the operation most likely to fail due to network issues or shell interference, so it's isolated. If it fails, the && prevents the second phase from executing on a potentially corrupted binary.
Phase 2 (installation and verification): Uses sudo for privileged operations. The assistant could have combined both phases into a single SSH command (e.g., cat binary | ssh user@host 'sudo tee /usr/local/bin/vast-manager > /dev/null && ...'), but chose to split them. This suggests the assistant wanted to minimize the complexity of the privileged command and keep the file transfer step simple and auditable. The inclusion of sleep 1 between systemctl restart and systemctl status shows an understanding of systemd's asynchronous nature: restart returns immediately, but the service may take a moment to reach a steady state. The 1-second sleep is a pragmatic heuristic—long enough for the service to crash if something is fundamentally wrong, short enough to not delay the feedback loop. The use of head -15 to truncate the status output shows awareness of the conversation's context limits. The assistant is not just running commands for its own benefit; it's producing output that will be read by the user and analyzed in subsequent reasoning steps. Keeping the output concise is a courtesy and a practical necessity in a long-running session.

Broader Significance

This message, while appearing to be a routine deployment step, sits at an inflection point in the development session. The vast-manager UI improvements—the new log filter tabs for benchdaemon and benchout—are the foundation for better observability of benchmark failures on remote GPU instances. Without this deployment, the previous 15 messages of code changes would have remained local and unapplied. The successful restart means that the next benchmark failure on a remote instance will produce visible logs in the manager UI, closing a critical feedback loop in the platform.

Moreover, the SSH piping technique demonstrated here is a general-purpose pattern for deploying binaries to remote hosts when SCP is unreliable. It's a tool that any developer working with remote infrastructure should have in their repertoire. The message serves as a case study in diagnosing and adapting to SSH failure modes—a skill that becomes increasingly important as infrastructure grows more distributed.

The message also illustrates a broader principle of operational reasoning: when a straightforward approach fails, the correct response is not to brute-force the same approach with different parameters, but to understand why it failed and choose a fundamentally different mechanism. The assistant could have tried scp with different flags, or debugged the remote shell configuration, but instead recognized that the protocol mismatch was a fundamental incompatibility and switched to a protocol that doesn't have that vulnerability. This is the difference between debugging symptoms and addressing root causes.