The Art of the Hot Deployment: A Single SSH Command as a Window into Production Operations

The Message

[assistant] [bash] ssh 10.1.232.83 "sudo systemctl stop kuri-kuri_01 && sudo cp /tmp/kuri /opt/fgw/bin/kuri && sudo chmod +x /opt/fgw/bin/kuri && sudo systemctl start kuri-kuri_01" 2>&1

At first glance, this message appears to be nothing more than a routine deployment command — a developer SSHing into a server to stop a service, replace a binary, and restart it. In isolation, it is a single line of shell script, chaining four commands with && operators, wrapped in an SSH invocation. But within the context of the Filecoin Gateway (FGW) project's development session, this message represents a critical moment of operational delivery: the culmination of a feature implementation cycle, the execution of a deployment workflow, and the tangible act of pushing new code into a live environment. Understanding this message requires unpacking the layers of context, reasoning, and operational practice that surround it.

The Context: What Led to This Command

To understand why this particular SSH command was executed, one must trace back through the preceding messages in the conversation. The assistant had just completed implementing a significant new feature: L1/L2 cache metrics visibility in the WebUI dashboard. This involved adding a CacheStats struct to the interface layer, implementing a CacheStats() method on the retrieval provider that collects statistics from both the L1 ARC (Adaptive Replacement Cache) in memory and the L2 SSD cache on disk, exposing this data through a new RPC endpoint, and building a React component (CacheStatsTile) to render the metrics with color-coded hit rates and per-tier breakdowns.

The implementation spanned multiple files across the codebase — from the iface package where the data structures were defined, through the rbdeal package where the retrieval provider collected the statistics, to the integrations/web directory where the RPC endpoint and UI component lived. After committing the changes with a descriptive commit message ("feat: add L1/L2 cache metrics to WebUI"), the assistant built a new static binary for the Kuri node — the core storage node in the FGW architecture. The build command (CGO_ENABLED=0 go build -o kuri ./integrations/kuri/cmd/kuri) produced a 126-megabyte statically-linked executable.

The assistant then copied this binary to two remote nodes using scp: first to 10.1.232.83:/tmp/kuri, then to 10.1.232.84:/tmp/kuri. These are the two nodes in the QA (Quality Assurance) test cluster. The binary was placed in /tmp/ — a staging location, not the final deployment path. This is a deliberate pattern: copy to a temporary location first, then use the deployment command to atomically swap it into place. This two-phase approach minimizes the risk of a partial or interrupted copy leaving the production binary in an inconsistent state.

The Command Itself: A Study in Operational Patterns

The command chains four operations with &&, meaning each must succeed before the next proceeds:

sudo systemctl stop kuri-kuri_01 — Gracefully stops the running Kuri node service. The systemctl stop command sends a SIGTERM to the main process, allowing it to clean up resources, flush buffers, and close connections. This is the controlled shutdown phase.
sudo cp /tmp/kuri /opt/fgw/bin/kuri — Copies the new binary from the staging location to the final deployment path. The cp command overwrites the existing binary in place. This is the atomic swap moment.
sudo chmod +x /opt/fgw/bin/kuri — Ensures the binary has execute permissions. While the original binary likely already had these permissions, explicitly setting them after the copy is a defensive measure — it guarantees that the file is executable regardless of the source file's permissions or the cp command's behavior regarding permission preservation.
sudo systemctl start kuri-kuri_01 — Starts the service again with the new binary. systemd picks up the updated binary from the filesystem and launches it as a new process. The use of sudo for each command individually, rather than wrapping the entire chain in a single sudo sh -c '...', is noteworthy. This pattern means that if any command fails (for example, if the stop command times out or the cp fails due to disk space), the chain halts immediately and the service remains in its last known state — stopped, but not restarted with a potentially corrupted binary. Each sudo invocation also creates a separate audit trail in the system's authentication logs, which can be valuable for operational forensics.## The Reasoning: Why This Message Was Written The assistant wrote this command as the final step in a deployment workflow that had been carefully orchestrated across multiple prior messages. The user had requested "Update qa deployment" (message 2780), and the assistant responded by committing the cache metrics changes, building the binary, copying it to both nodes, and then executing the hot-swap commands. Each step built upon the previous one with a clear dependency chain: the code changes had to compile, the binary had to be built, the binary had to be copied to the remote hosts, and finally the service had to be restarted to pick up the new code. The reasoning behind the command is straightforward but critical: the only way for the newly implemented cache metrics feature to become live in the QA environment is to replace the running binary with the newly compiled one and restart the service. The Kuri nodes are statically compiled Go binaries — there is no hot-reload mechanism, no dynamic code loading, no plugin system. A restart is the only path to deployment. But there is a deeper layer of reasoning visible in the choice of commands. The assistant could have used a simpler approach: sudo systemctl restart kuri-kuri_01 after copying the binary. Instead, the explicit stop && cp && chmod && start sequence reveals an understanding of operational best practices. By stopping the service first, the assistant ensures that the binary is not being read by a running process when it is overwritten. On Linux, a running process keeps the inode of the executable file open even if the file is deleted or overwritten, but replacing the file while the process is running can lead to subtle issues — the old binary remains in memory, and the new binary is only picked up on restart. The stop-first approach is cleaner and more predictable.

Assumptions Embedded in the Command

Every deployment command carries assumptions about the target environment, and this one is no exception. The assistant assumes:

SSH access is available to 10.1.232.83 with the current user's key. The absence of a -i flag or user specification (user@host) implies that either the SSH configuration has a default user, or the key has been pre-configured for the theuser user.
sudo privileges are available without a password prompt. The command is non-interactive — there is no -t flag for pseudo-terminal allocation, no expect script, and no password pipe. This means the remote user has passwordless sudo configured, which is standard for automated deployment scenarios.
The systemd service unit kuri-kuri_01 exists and is in a state where it can be stopped and started. If the service were already stopped, the stop command would still succeed (it would be a no-op), but the subsequent start would work correctly.
The target directory /opt/fgw/bin/ exists and is writable by the sudo user. The cp command does not create intermediate directories.
The binary at /tmp/kuri is valid and functional. The assistant built it moments earlier, but there is no checksum verification, no signature check, and no smoke test before the swap. This is a trust-but-verify approach that relies on the build process having succeeded without errors.
The node's IP address 10.1.232.83 is stable and reachable. This is a QA environment on a private network, so IP stability is a reasonable assumption.

Potential Mistakes and Risks

While the command is correct for its purpose, several risks are worth examining. First, there is no pre-flight health check before the stop command. If the node was in the middle of processing a critical write operation, the systemctl stop would send SIGTERM, and the process would need to handle graceful shutdown. The Kuri node's shutdown handler is expected to drain in-flight operations, but there is no guarantee of completion — the systemd unit file may specify a TimeoutStopSec that could force-kill the process if it takes too long.

Second, there is no post-deployment validation within this command. After the start command completes, the assistant does not immediately verify that the service is healthy. (In subsequent messages — 2790 and 2791 — the assistant does check systemctl status and the /healthz endpoint, but these are separate commands, not part of the deployment chain.) This separation is actually good practice: the deployment command should do one thing (swap the binary and restart), and verification should be a distinct step. However, it does mean that if the new binary crashes on startup, the operator would not know until the next command is run.

Third, the command targets only one node (kuri_01). A parallel command targets kuri_02 on the other node. This sequential approach means there is a brief period where the two nodes are running different versions of the software. In a distributed system, version skew during rolling updates can cause protocol mismatches, RPC failures, or inconsistent behavior. The assistant mitigates this by keeping the downtime per node very short (seconds), but the skew window exists.

Knowledge Required and Knowledge Created

To understand this message, a reader needs knowledge of Linux system administration (systemd service management, SSH, sudo, file permissions), the FGW project architecture (Kuri nodes as statically compiled Go binaries, the two-node QA cluster, the deployment paths), and the deployment workflow conventions used in this project (staging in /tmp/, final deployment to /opt/fgw/bin/, systemd unit naming).

The message creates operational knowledge: it records the exact sequence of commands used to deploy a new binary to the QA cluster. This is valuable for reproducibility — if the deployment needs to be rolled back, the operator can reverse the process (copy the old binary back and restart). It also serves as an audit trail, showing who deployed what, when, and to which nodes.

The Thinking Process

The assistant's reasoning, visible through the sequence of actions leading up to this command, reveals a methodical deployment process. The assistant first commits the code changes to create a clean checkpoint. Then builds the binary with CGO_ENABLED=0 to produce a statically-linked executable that has no external library dependencies — a critical choice for deployment to systems where the runtime environment may differ from the build environment. The binary is copied to both nodes' /tmp/ directories before any service is stopped, minimizing the window where the new binary exists on only one node. Finally, the deployment commands are executed sequentially per node, followed by health checks.

This pattern — commit, build, stage, deploy, verify — is a miniaturized version of a production deployment pipeline, executed manually but with the same logical structure. The assistant is not just running a command; it is following an operational discipline that prioritizes safety, auditability, and rapid feedback.