Debugging Distributed Systems: The Art of Checking What's Actually Running

The Message

[assistant] [bash] # Check what's running on the head node
ssh 10.1.232.82 "ss -tlnp | grep -E '5433|9042'"
LISTEN 0      624     10.1.232.82:5433       0.0.0.0:*    users:(("postgres",pid=17270,fd=3))    
LISTEN 0      128     10.1.232.82:9042       0.0.0.0:*    users:(("yb-tserver",pid=17104,fd=148))
LISTEN 0      4096    10.1.232.82:15433      0.0.0.0:*    users:(("yugabyted-ui",pid=17259,fd=5))

This message, at first glance, appears to be a simple diagnostic command: SSH into a remote machine, check which processes are listening on two specific ports, and display the results. But in the context of a complex debugging session spanning a distributed storage system, this single command represents a critical pivot point in the investigation. It is the moment when the engineer steps back from high-level application logic to verify the most fundamental assumption of all: that the infrastructure is actually running the way it is supposed to.

The Context: A Stalled Deal Pipeline

To understand why this message was written, we must understand the debugging journey that led to it. The session was focused on diagnosing why the Filecoin Gateway (FGW) distributed storage system was failing to make deals—that is, why storage deals with Filecoin miners were not being initiated despite the system having data ready to store. The deal pipeline had been stalled, and the team had already resolved several blocking issues.

The first major blocker was a Lotus gateway (the Filecoin blockchain API endpoint) that was unreachable. The endpoint pac-l-gw.devtty.eu had been returning "Connection refused" on port 443. The user confirmed they had fixed this, and the assistant verified the gateway was operational by successfully calling Filecoin.ChainHead, which returned chain height 5,729,846. So the blockchain connection was working.

Yet deals were still not flowing. The deal tracker's cleanup loop was completing successfully (taking 35–43 seconds instead of timing out at 150+ seconds), but no GBAP (Get Best Available Providers) calls were being made to CIDgravity, the service that selects Filecoin miners for storage deals. Group 1, which had approximately 30 GB of data in state 3 (GroupStateLocalReadyForDeals), should have been triggering deal proposals, but nothing was happening.

The assistant had been tracing through the code, checking group states via RPC calls, examining the GetGroupDealStats database query, and trying to understand why makeMoreDeals wasn't being invoked. The investigation had moved from the application layer (checking RPC methods, examining log output) to the data layer (trying to query the YugabyteDB database directly).

The Pivot: From Application Logic to Infrastructure Reality

The immediate trigger for this message was a failed attempt to query the database. The assistant had tried two approaches to inspect the database state directly:

  1. Using psql on the head node—but the command failed with "psql: command not found"
  2. Using /opt/yugabyte/bin/ysqlsh—but this failed with "could not connect to server: Connection refused" on localhost These failures created a puzzle. The YugabyteDB database was supposed to be running on the head node (10.1.232.82). The application was clearly connecting to it, since the deal tracker was completing its cleanup loop and the RPC endpoints were returning group data. But direct database access from the head node itself was failing. This is where the message under analysis becomes the critical diagnostic step. Instead of continuing to chase application-level theories (wrong state transitions, missing code paths, configuration errors), the assistant made a deliberate decision to verify the infrastructure layer. The command ss -tlnp | grep -E '5433|9042' asks the operating system: "What processes are actually listening on the ports that YugabyteDB should be using?"

Why This Specific Command

The choice of ss -tlnp is revealing. ss is a socket statistics utility that shows detailed information about network sockets. The flags break down as follows:

Assumptions Embedded in the Command

This message carries several implicit assumptions that are worth examining:

Assumption 1: The database is supposed to be running on this node. The entire debugging session had been operating under the topology that 10.1.232.82 was the "head node" running YugabyteDB plus the S3 proxy. This assumption was validated by the deployment scripts and Ansible inventory, but it was being tested here.

Assumption 2: The database listens on ports 5433 and/or 9042. These are the standard YugabyteDB ports, but a custom deployment might use different ports. The assistant implicitly trusts that the standard configuration applies.

Assumption 3: SSH access to the head node is sufficient for infrastructure debugging. The assistant had already SSH'd into the kuri nodes (10.1.232.83 and 10.1.232.84) multiple times during this session. The head node was the next logical target.

Assumption 4: The ss command will work and return meaningful output. This assumes the tool is installed, the user has sufficient privileges (the -p flag requires root or CAP_NET_ADMIN), and the output format is parseable.

The Results: A Moment of Validation

The output confirmed that YugabyteDB was indeed running and listening on both expected ports:

The Thinking Process: What This Reveals About Debugging Methodology

This message exemplifies a crucial debugging technique: when higher-level abstractions fail, drop down to lower-level verification. The assistant had been working at the application level (RPC calls, log analysis, code reading) and then at the database client level (trying psql and ysqlsh). When both of these failed to provide answers, the assistant dropped to the operating system level to verify the most basic fact: is the database process actually running and listening for connections?

This is a form of the "trust but verify" principle applied to distributed systems debugging. The deployment scripts and configuration management tools assert that YugabyteDB is running on the head node, but the direct evidence was missing. The ss command provides that evidence directly from the kernel's socket table, which is as close to ground truth as you can get without inspecting the process memory itself.

The message also reveals a disciplined approach to problem-solving. Rather than guessing at why ysqlsh failed (which could have led down a rabbit hole of PostgreSQL configuration debugging), the assistant first confirmed that the database was actually running. Once that was confirmed, the failure of ysqlsh to connect to localhost became a secondary issue—the database was accessible from other nodes (as evidenced by the application working), just not from localhost via the loopback interface.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the system topology: That 10.1.232.82 is the "head node" running YugabyteDB, and that the database uses ports 5433 (YSQL) and 9042 (YCQL).
  2. Knowledge of the debugging context: That the assistant was trying to query the database directly after failing to find why deals weren't flowing, and that previous attempts with psql and ysqlsh had failed.
  3. Knowledge of Linux networking tools: Understanding what ss -tlnp does and how to interpret its output, particularly the significance of the bind address (10.1.232.82 vs 0.0.0.0).
  4. Knowledge of YugabyteDB architecture: That it exposes both a PostgreSQL-compatible interface (YSQL on port 5433) and a Cassandra-compatible interface (YCQL on port 9042), and that the yb-tserver process is the core tablet server.
  5. Knowledge of the session's history: The earlier attempts to use database clients, the failed systemctl command, and the overall goal of diagnosing the stalled deal pipeline.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

  1. Confirmed database availability: YugabyteDB was running and listening on both expected ports. The database layer was not the cause of the deal pipeline stall.
  2. Identified the bind address issue: The database was bound to the external IP (10.1.232.82) rather than 0.0.0.0 or localhost, explaining why localhost connections failed. This is useful operational knowledge for future debugging—any direct database queries must specify the external IP.
  3. Provided process-level detail: The specific PIDs (17270 for postgres, 17104 for yb-tserver, 17259 for yugabyted-ui) could be used for further investigation, such as checking resource usage or tracing system calls.
  4. Eliminated a hypothesis: The database being down was ruled out as a cause of the deal pipeline stall, allowing the investigation to focus elsewhere (which, as the session summary indicates, eventually led to discovering the missing removeUnsealedCopy field in the CIDgravity API request).

Broader Significance

This message, while seemingly mundane, represents a fundamental pattern in distributed systems debugging: the systematic narrowing of hypotheses through layered verification. When the application layer is silent, check the data layer. When the data layer is unreachable through client tools, check the infrastructure layer. When the infrastructure layer is unclear, check the operating system.

The ss -tlnp command is the distributed systems engineer's equivalent of a doctor checking a patient's pulse—it's the most basic vital sign, and if it's absent, nothing else matters. In this case, the pulse was strong, and the investigation could move on to the next hypothesis. The message is a testament to the importance of having a mental model of the entire system stack and knowing which tool to use at each layer when something goes wrong.