When SSH Goes Silent: Diagnosing Exit Code 255 Through Stderr Capture and Stale Socket Recovery

In distributed systems, the most frustrating bugs are often the ones that produce a single, opaque error message—one that tells you something went wrong but offers no clue as to why. Message [msg 3780] in this coding session is a compact but revealing example of how a seemingly simple fix (capturing SSH stderr and cleaning up stale control sockets) emerges from a deep diagnostic journey through SSH internals, Go's os/exec package, and the architecture of a management proxy.

The Problem: A Silent SSH Failure

The session began with a user report: the vast-manager UI was showing "cuzk: ssh exec failed: exit status 255" for all nodes. Exit code 255 in SSH is a catch-all error code meaning the SSH client itself failed—it could indicate a missing key, a closed port, a DNS resolution failure, a stale control socket, or any number of other connectivity problems. The user noted that it "used to work on one of the running nodes," which meant the SSH setup had been functional at some point, making the sudden universal failure all the more puzzling.

The vast-manager is a Go-based management service that proxies HTTP requests to remote cuzk proving instances via SSH. When the user's browser requests the cuzk status endpoint, the vast-manager SSHes into the remote instance and runs a curl command against the local cuzk status API. This design avoids exposing the cuzk status port to the public internet, but it introduces SSH as a critical dependency.

The Diagnostic Trail

The assistant's reasoning in the preceding messages ([msg 3765] through [msg 3779]) reveals a systematic investigation. The first hypothesis was stale ControlMaster sockets. SSH's ControlMaster=auto option allows connection reuse: the first SSH connection to a host creates a Unix domain socket at a path like /tmp/vast-ssh-root@host:port, and subsequent connections reuse that socket to skip the expensive TCP and key exchange handshake. But if the master SSH process dies unexpectedly—say, because the vast-manager was restarted—the socket file remains on disk as a dead artifact. Any new SSH connection that tries to use that stale socket will fail with exit code 255.

The assistant checked for stale sockets on the local machine (ls -la /tmp/vast-ssh-*) and found none. It then tried to locate the vast-manager process itself, but pgrep and ps searches came up empty—the vast-manager was running on a different machine entirely. This was an important realization: the assistant was working in a development environment, not on the production server where the vast-manager actually ran. The fix would need to be deployed separately.

The second, more fundamental observation was about error reporting. The assistant read the handleCuzkStatus function in main.go and found this critical line:

out, err := cmd.Output()

Go's cmd.Output() captures only the command's stdout. Any error messages SSH writes to stderr—messages like "Permission denied (publickey)", "Connection refused", or "No route to host"—are silently discarded. The only thing the code could report was the exit code: "exit status 255." This is like a car's dashboard showing a generic "Check Engine" light with no diagnostic code: you know something is wrong, but you have no way to know what.

The Fix: Three Changes in One Message

Message [msg 3780] is the assistant's summary after implementing the fix. The message itself is brief—just a few bullet points—but it represents the culmination of a multi-step code modification that touched three distinct areas of the SSH proxy logic.

First, stderr capture. The assistant replaced cmd.Output() with a setup that attaches a bytes.Buffer to cmd.Stderr. This required adding the bytes package to the import list and removing an unused path/filepath import that had been left over from an earlier edit. After the command runs, the stderr buffer is read and included in the error message returned to the UI. This single change transforms the error from the opaque "exit status 255" into something actionable like "Permission denied (publickey)" or "Connection refused."

Second, stale socket cleanup on retry. The assistant added logic that, on exit code 255, removes the control socket file at the computed path (/tmp/vast-ssh-root@host:port) and retries the SSH command once. This is a pragmatic recovery mechanism: if the failure was caused by a stale socket, removing it forces SSH to establish a fresh connection, which will create a new socket. If the failure was caused by something else (like a missing key), the retry will fail again, but now with the stderr message visible.

Third, the build. The assistant compiled the modified code with go build -o /tmp/czk/vast-manager ., producing a new binary. The go vet check passed cleanly (the only warnings were from the vendored C code in the sqlite3 binding, which is expected).

Assumptions and Their Risks

The fix embodies several assumptions, some explicit and some implicit. The most significant assumption is that stale ControlMaster sockets are a likely cause of the universal failure. The assistant's reasoning notes that "ALL nodes fail, including ones that previously worked, which points to something systemic rather than per-host issues." A restart of the vast-manager process would leave behind all the control sockets from previous connections, and if those sockets are now stale, every new SSH attempt would fail. This is a plausible scenario, but it is not the only possible one.

Another assumption is that the vast-manager's SSH configuration is correct aside from the socket issue. The code does not add an -i flag to specify an identity file, relying instead on the default SSH key discovery (typically ~/.ssh/id_ed25519 or ~/.ssh/id_rsa). If the vast-manager runs as a systemd service without a proper home directory or SSH agent, SSH might not find any keys at all. The stderr capture would reveal this ("Permission denied (publickey)"), but the fix does not address the underlying key provisioning.

There is also an assumption about the socket path format. The control path is constructed as /tmp/vast-ssh-%r@%h:%p, which SSH expands to something like /tmp/vast-ssh-root@141.0.85.211:40612. Unix domain socket paths are limited to around 104-108 characters on most systems, and this path is well under that limit. But if the hostname were a very long IPv6 address, the path could potentially exceed the limit. The assistant considered this and judged it safe.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains:

Output Knowledge Created

The message produces several tangible and intangible outputs:

The Thinking Process

What makes this message interesting is not the code changes themselves—they are straightforward—but the reasoning that led to them. The assistant's thinking, visible in the earlier messages, shows a methodical narrowing of possibilities:

  1. Observation: All nodes fail, including one that previously worked. This rules out per-node issues (like a specific instance being down) and points to a systemic cause on the vast-manager side.
  2. Hypothesis generation: The assistant enumerates possible causes—stale sockets, missing SSH agent, key permission changes, too many connections—and evaluates each against the evidence.
  3. Evidence gathering: The assistant checks for stale sockets (ls -la /tmp/vast-ssh-*), checks for the SSH agent (ssh-add -l), and tries to locate the vast-manager process (pgrep -af vast-manager). Each check eliminates a hypothesis or refines the search.
  4. Code inspection: The assistant reads the handleCuzkStatus function and identifies the cmd.Output() call as a critical information bottleneck. The stderr is being thrown away.
  5. Fix design: The assistant decides on a two-pronged approach: capture stderr for better diagnostics, and add retry-with-cleanup for the stale socket scenario. These are complementary: the stderr capture helps diagnose the current failure, while the retry logic provides automatic recovery.
  6. Implementation: The assistant edits the Go file in three passes (import fix, logic replacement, import cleanup), verifies with go vet, and builds the binary. Each edit is checked for LSP errors before proceeding.

Conclusion

Message [msg 3780] is a masterclass in turning an opaque error into actionable information. The fix is small—a few lines of Go code—but the reasoning behind it is substantial. By capturing SSH's stderr, the assistant gives the user a window into the SSH negotiation process. By adding stale socket cleanup, it provides automatic recovery for a common failure mode. And by building the binary and telling the user exactly what to do next, it closes the loop between diagnosis and action.

The message also illustrates a broader principle in systems debugging: when a tool gives you a generic error, the first thing to fix is not the tool but the error reporting. You cannot debug what you cannot see. By making SSH's failures visible, the assistant transforms "exit status 255" from a dead end into a starting point.