The Quiet Verification: What Two ss Commands Reveal About Distributed Systems Deployment

The Message

[assistant] [bash] ssh 10.1.232.83 "ss -tlnp | grep -E '8079|7010|9010|2112'" 2>&1
LISTEN 0      4096               *:9010             *:*          
LISTEN 0      4096               *:7010             *:*          
LISTEN 0      4096               *:8079             *:*          
LISTEN 0      4096               *:2112             *:*          

[bash] ssh 10.1.232.84 "ss -tlnp | grep -E '8079|7011|9010|2112'" 2>&1
LISTEN 0      4096               *:8079             *:*          
LISTEN 0      4096               *:7011             *:*          
LISTEN 0      4096               *:2112             *:*          
LISTEN 0      4096               *:9010             *:*

On its surface, this message from an AI assistant during a distributed systems deployment session is unremarkable: two SSH commands, eight lines of output, four listening ports confirmed on each of two physical nodes. It reads like a mundane checklist item, the kind of verification step that experienced operators perform almost reflexively before moving on to the next task. But in the context of the broader deployment narrative—a multi-node QA cluster for the Filecoin Gateway (FGW) distributed storage system—this message represents a quiet but critical inflection point. It is the moment when theory becomes practice, when configuration becomes operation, and when the assistant transitions from deploying to validating.

The Weight of Context

To understand why this message was written, one must understand what preceded it. The assistant had spent the better part of a session building a QA test cluster across three physical nodes: a head node at 10.1.232.82 running YugabyteDB, and two storage (kuri) nodes at 10.1.232.83 and 10.1.232.84. The deployment had been anything but smooth. Earlier, the kuri daemons had failed to start because of "dirty migration" states left behind in the YugabyteDB CQL keyspaces—a classic artifact of test suites that abort mid-migration, leaving the schema_migrations table with dirty = true flags that prevent subsequent initializations. The assistant had to manually connect to the Cassandra Query Language (CQL) interface on the YugabyteDB node and issue UPDATE schema_migrations SET dirty = false for three separate keyspaces (filecoingw_kuri_01, filecoingw_kuri_02, and filecoingw_s3) before the kuri daemons would even start.

Even then, the first service start attempts failed. The assistant had to dig into journalctl logs, identify the dirty migration errors, fix them, restart, and verify again. This iterative debugging—fix, restart, check logs, fix again—is the unglamorous reality of distributed systems deployment. By the time we reach message 2010, the assistant has successfully started both kuri services and confirmed they are active (running) via systemctl status. But "running" in the systemd sense only means the process hasn't crashed yet. It does not mean the service is actually serving. This is the gap that the ss commands in message 2010 are designed to close.## Why ss and Not curl or systemctl?

The choice of verification tool is itself revealing. The assistant had already used curl in the immediately preceding messages to probe the S3 API endpoint on port 8079 and the Prometheus metrics endpoint on port 2112, both returning data. So why switch to ss? The answer lies in what each tool confirms. curl proves that an HTTP server is responding on a given port, but it is an application-level check—it requires the service to be fully initialized, its routing tables loaded, its handlers registered. A curl success is strong evidence, but it can mask partial failures. systemctl status proves that systemd considers the process alive, but systemd's definition of "alive" is simply that the main PID has not exited. A process could be hung, deadlocked, or listening on the wrong interface and systemd would still report it as active (running).

The ss command (socket statistics) operates at the transport layer. It shows what ports are actually open and listening on the kernel's network stack. This is the lowest-level confirmation that a service has successfully bound to its intended address and is ready to accept TCP connections. By filtering for the specific ports—8079 (S3 API), 7010/7011 (LocalWeb for CAR data uploads), 9010 (internal API), and 2112 (Prometheus metrics)—the assistant is performing a precise, protocol-agnostic inventory of what network services are actually available. It is checking that each kuri node is listening on all the ports it should be, not just the one that happens to respond to HTTP.

The Asymmetry in the Output

A careful reader will notice a subtle but important difference between the two nodes' outputs. On kuri1 (10.1.232.83), the LocalWeb server is listening on port 7010. On kuri2 (10.1.232.84), it is listening on port 7011. This is not a coincidence or a configuration error—it is deliberate. The settings files generated earlier in the session explicitly assigned different LocalWeb ports to each node (EXTERNAL_LOCALWEB_SERVER_PORT="7010" for kuri1, EXTERNAL_LOCALWEB_SERVER_PORT="7011" for kuri2). This per-node differentiation is essential because LocalWeb serves CAR files for deal-making, and each node must have a unique endpoint that the Filecoin network can reach. If both nodes used the same port, there would be port conflicts on shared infrastructure, and the CIDGravity provider selection system would be unable to distinguish which node to route deals to.

This asymmetry also validates a deeper architectural assumption: that the configuration files deployed to each node are truly independent and correct. One of the most common failure modes in distributed deployments is configuration drift—where a template or shared config file is deployed uniformly across nodes without accounting for per-node differences. The fact that port 7010 appears on kuri1 and port 7011 appears on kuri2 confirms that the assistant's earlier work of generating separate settings-kuri1.env and settings-kuri2.env files, and deploying them to the correct nodes, was executed faithfully. The ss output is, in this sense, a cryptographic proof of correct configuration distribution.

What the Message Assumes

The message makes several assumptions, most of them justified by the preceding work. It assumes that the ss command is available on both target nodes (it is a standard Linux utility, part of the iproute2 package). It assumes that SSH key-based authentication is working and that the assistant's SSH agent has the necessary credentials cached—a non-trivial assumption in a multi-hop deployment scenario, but one that had been validated repeatedly during the session. It assumes that the port numbers being filtered correspond to the services configured in the environment files, which in turn assumes that the kuri daemon's internal port binding logic respects the RIBS_S3API_BINDADDR, EXTERNAL_LOCALWEB_SERVER_PORT, and RIBS_PROMETHEUS_PORT environment variables. This is a chain of assumptions that stretches from the application's source code through its configuration parser, through the systemd environment file loader, through the kernel's TCP stack, all the way to the ss output. Any break in that chain would produce a different result.

There is also an implicit assumption about time. The ss command is instantaneous—it shows the state of the network stack at the exact moment of execution. It does not guarantee that the ports were listening five seconds ago or will be listening five seconds from now. The assistant mitigates this by running the check shortly after confirming the services are running, but there is always a window of vulnerability. A service that crashes immediately after binding would still show as listening in the ss output if the check races ahead of the crash. This is a fundamental limitation of point-in-time verification, and the assistant does not address it in this message—though the broader session includes repeated checks and load testing that would eventually surface any such instability.## Input Knowledge Required

To fully appreciate what this message accomplishes, a reader needs a fairly specific set of background knowledge. First, one must understand the FGW architecture: that kuri nodes are storage nodes that serve S3-compatible object storage on port 8079, expose a LocalWeb interface for CAR file staging on a configurable port (7010 or 7011), provide an internal API on port 9010 for cluster topology and management, and emit Prometheus metrics on port 2112. Without this port-to-service mapping, the ss output is just a list of numbers. Second, one needs to know that the deployment spans multiple physical nodes with distinct IP addresses, and that the assistant has SSH access to both. Third, one must understand the deployment history—specifically the dirty migration crisis that preceded this moment—to grasp why a simple port check carries so much emotional weight. It is the "it works" moment after an extended debugging session, and that relief is only legible against the backdrop of failure.

The message also assumes familiarity with the ss tool itself, or at least with the concept of listening sockets. The -tlnp flags filter for TCP (-t), listening sockets (-l), numeric output (-n), and show the process associated with each socket (-p). The grep filters for the specific port numbers. The output format—LISTEN 0 4096 *:9010 *:*—shows the socket state (LISTEN), the send queue size (0), the receive queue size (4096, which is the default net.core.somaxconn backlog), the local address (*:9010 meaning bound to all interfaces on port 9010), and the peer address (*:* meaning accepting connections from any remote address). A reader who can parse this output can independently verify that each service is bound to the wildcard address (not just localhost) and is therefore accessible from other nodes on the network.

Output Knowledge Created

This message creates several distinct pieces of knowledge. The most immediate is a verified inventory of running services on each kuri node. Before this message, the assistant knew that the systemd units were active (running). After this message, the assistant knows that:

The Thinking Process

The reasoning behind this message is visible in its structure. The assistant does not simply check one port on one node; it checks four ports on two nodes, symmetrically. This symmetry is deliberate. By running the same command on both nodes and filtering for the same set of ports (adjusted for the LocalWeb port difference), the assistant creates a comparison baseline. If kuri1 showed five ports and kuri2 showed three, that asymmetry would immediately signal a problem—perhaps a configuration file that failed to deploy, a service that crashed silently, or a firewall rule that blocked binding. The parallel structure of the two commands reveals a systematic, checklist-oriented mindset: verify everything, verify it everywhere, and let the structure of the output reveal anomalies.

The choice of grep -E with a pipe-delimited pattern ('8079|7010|9010|2112') is also telling. The assistant could have run four separate ss commands per node, or used curl to probe each port individually. Instead, it chose a single command that returns all matching ports in one shot. This is efficient but also information-dense: the output shows not just that each port is listening, but how it is listening (the backlog size, the bind address). The assistant is not just checking for existence; it is checking for correctness of the bind address. The *:8079 output confirms that the S3 API is bound to all interfaces, which is essential for the S3 proxy (which will run on the head node) to reach it. If the output had shown 127.0.0.1:8079, the service would be inaccessible from other nodes, and the entire cross-node architecture would fail.

There is also a subtle temporal reasoning at play. The assistant runs these checks after confirming the services are running via systemctl status. This ordering is not accidental. Systemd's active (running) state can lag behind the actual process state, or can report success before the process has fully initialized its network listeners. By waiting a few seconds (the sleep 8 in the restart command) and then checking with ss, the assistant gives the services time to complete their binding. This is a heuristic, not a guarantee, but it is a reasonable one informed by experience with service initialization latencies.

Mistakes and Limitations

The message is not without limitations. The most significant is that ss confirms only that the socket is open, not that the service behind it is functioning correctly. A service could be listening on port 8079 but returning HTTP 500 errors for every request, or could have a corrupted internal state that only manifests under load. The ss check is a necessary but insufficient condition for a healthy deployment. The assistant implicitly acknowledges this by following up with application-level checks (curl probes, load tests) later in the session, but the message itself does not address this gap.

Another limitation is the reliance on exact port numbers. The grep pattern hard-codes 7010 for kuri1, but the command is run against both nodes. On kuri2, the grep for 7010 would return nothing (since kuri2 uses port 7011), but the grep for 7011 is not included in the pattern. Wait—re-reading the message carefully, the assistant actually uses different grep patterns for each node. For kuri1: '8079|7010|9010|2112'. For kuri2: '8079|7011|9010|2112'. This is correct and deliberate. The assistant customized the pattern per node to match the expected LocalWeb port. This is a small but important detail: it means the assistant was aware of the per-node configuration differences and accounted for them in the verification logic. A less careful operator might have used the same pattern for both nodes and missed the asymmetry, or worse, concluded that kuri2 was missing a service.

The Broader Significance

In the grand narrative of the FGW deployment session, message 2010 is the calm before the next storm. The assistant has just resolved the dirty migration crisis, started both kuri services, and confirmed they are listening on all expected ports. But the architecture is not yet complete. The kuri nodes are running as standalone S3 endpoints, each serving only their local blockstore. Cross-node reads—where an object written to kuri1 can be read from kuri2—are not yet possible. The assistant will soon discover this limitation and respond by deploying the s3-proxy frontend on the head node, creating the three-layer architecture (S3 proxy → kuri nodes → YugabyteDB) that the roadmap specifies.

This message, then, is a snapshot of a system in transition. It captures the moment when the foundational layer—the storage nodes themselves—is verified to be operational, setting the stage for the routing and proxy layer to be added on top. It is a testament to the importance of incremental verification in distributed systems: fix one problem, verify, move to the next. The ss commands are the punctuation marks in that process, confirming that the current chapter is complete before the next one begins.

For the reader who has followed the session from the beginning, message 2010 carries an emotional resonance that its terse output belies. The four listening ports on each node are not just sockets; they are the resolution of a debugging arc that involved database schema surgery, systemd service file engineering, secure credential vaulting, and multi-node configuration management. They represent the first moment in the session where both kuri nodes are simultaneously healthy and reachable. It is a small victory, but in distributed systems, small victories are the only kind that exist.