The Diagnostic Pivot: How `ps aux` Revealed the Truth About a Stuck Database

Introduction

In the midst of a grueling debugging session for a horizontally scalable S3 storage system, a single diagnostic command cut through layers of abstraction to reveal the true state of a database that appeared to be stuck. This article examines message 1265 from an opencode coding session—a brief but pivotal moment where the assistant, after hours of wrestling with port configurations, stale data directories, and misleading health checks, used a fundamental Unix debugging technique to determine whether a YugabyteDB instance was actually running or fundamentally broken.

The Message

The message reads:

[assistant] YB processes are running! The ysqlsh is also hanging. Let me check if postgres process started: [bash] docker exec test-cluster-yugabyte-1 ps aux 2>&1 | grep postgres | head -5 root 254 0.0 0.0 52472 7532 ? S 15:44 0:00 /home/yugabyte/bin/ysqlsh postgresql://yugabyte:[REDACTED]@127.0.0.1:15433 -c \conninfo root 6412 0.0 0.0 2231244 38788 ? R 15:49 0:00 /home/yugabyte/postgres/bin/postgres -D /root/var/data/pg_data -p 15433 -h 127.0.0.1 -k /tmp/.yb.127.0.0.1:15433 -c unix_socket_permissions=0700 -c logging_collector=on -c log_directory=/root/var/data/yb-data/tserver/logs -c yb_pg_metrics.node_name=biryani:9000 -c yb_pg_metrics.p...

At first glance, this appears to be a routine diagnostic check. But in the context of the preceding two hours of debugging, it represents a critical turning point—the moment when the assistant confirmed that the database server was actually operational, and the real problem lay elsewhere in the connection layer.

The Context: A Debugging Odyssey

To understand why this message matters, we need to trace the debugging journey that led to it. The assistant was building a test cluster for a horizontally scalable S3 architecture consisting of three layers: S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. The cluster was designed to run in Docker with host network mode, meaning all services would bind directly to host ports—a configuration that provides better performance but risks port conflicts with existing services.

The problem began when the assistant attempted to reconfigure YugabyteDB to use non-default ports to avoid conflicts with other running services on the host machine. The default YugabyteDB ports (5433 for YSQL, 9042 for YCQL, 7000 for yb-master HTTP, 7100 for yb-master RPC, 9000 for yb-tserver HTTP, 9100 for yb-tserver RPC, and 15433 for the web UI) all needed to be shifted. The assistant assigned offset ports (15433 for YSQL, 19042 for YCQL, 17000 for yb-master HTTP, 17100 for yb-master RPC, 19000 for yb-tserver HTTP, 19100 for yb-tserver RPC).

This seemingly simple configuration change triggered a cascade of failures. The yugabyted CLI tool, which is the recommended way to start YugabyteDB, didn't support port options directly through its command-line flags. The assistant had to research the correct syntax, discovering that ports must be passed through --master_flags and --tserver_flags parameters that are forwarded to the underlying yb-master and yb-tserver processes. After updating the Docker Compose configuration with the correct syntax, the assistant cleaned the data directory to remove stale state from previous runs and restarted the container.## The Port Binding Success That Wasn't

When the assistant ran ss -tlnp after restarting the container with a clean data directory, the output was encouraging:

LISTEN 0      128                      127.0.0.1:19100      0.0.0.0:*                                                 
LISTEN 0      128                      127.0.0.1:19000      0.0.0.0:*                                                 
LISTEN 0      128                      127.0.0.1:19042      0.0.0.0:*                                                 
LISTEN 0      4096                     127.0.0.1:15433      0.0.0.0:*

All four custom ports were listening: 19100 (tserver RPC), 19000 (tserver HTTP), 19042 (YCQL), and 15433 (YSQL/UI). This looked like a complete success. The YugabyteDB processes had started, bound to the correct ports, and appeared ready to accept connections.

But appearances were deceiving. The Docker health check remained stuck in "starting" status, never transitioning to "healthy." And when the assistant tried to connect to YSQL with ysqlsh -h 127.0.0.1 -p 15433, the connection hung indefinitely, eventually timing out after the tool's 120-second limit.

This created a confusing diagnostic picture: the ports were open, the processes were running, but the database wasn't accepting connections. Was it still bootstrapping? Was there a configuration error? Was the data directory corrupted from the previous failed attempts?

The `ps aux` Revelation

Message 1265 captures the moment the assistant decided to cut through the abstraction layers and look directly at the running processes inside the container. The command was simple:

docker exec test-cluster-yugabyte-1 ps aux 2>&1 | grep postgres | head -5

This is a classic systems debugging technique: when higher-level tools (health checks, connection attempts, status commands) give ambiguous results, go straight to the process table and see what's actually running.

The output revealed two critical pieces of information:

A stuck ysqlsh process: The first line showed ysqlsh running since 15:44, with the command ysqlsh postgresql://yugabyte:[REDACTED]@127.0.0.1:15433 -c \conninfo. This was a previous diagnostic attempt that had hung, still occupying a process slot. The S state indicated it was in interruptible sleep—waiting for a response from the server that would never come because... well, that was the question.
The PostgreSQL backend was actually running: The second line showed the actual PostgreSQL backend process (postgres -D /root/var/data/pg_data -p 15433 -h 127.0.0.1) that had started at 15:49, five minutes after the stuck ysqlsh. It was in R (running) state, with a full command line showing its data directory, port, and bind address. This was the breakthrough. The PostgreSQL backend was running. The database server itself was operational. The problem wasn't that YugabyteDB had failed to start—it was that something in the connection path was broken. The ysqlsh client couldn't reach the server, even though both were running on the same machine, bound to the same IP address and port.

The Hidden Clue: `yb_pg_metrics.node_name=biryani:9000`

One detail in the process listing deserves special attention: the PostgreSQL backend was started with the flag -c yb_pg_metrics.node_name=biryani:9000. The value "biryani:9000" is significant—it's the default node name generated by YugabyteDB, which typically includes the hostname and a port number. But "biryani" is not a standard hostname; it's a randomly generated container hostname assigned by Docker. The port 9000 is the default yb-tserver HTTP port, not the custom 19000 port that was configured.

This suggests that while the tserver flags were being passed correctly for some parameters (the port bindings worked), the yb_pg_metrics.node_name was still using default values. This inconsistency hints at a deeper configuration issue: the --tserver_flags parameter might not have been fully parsed or applied as expected. The node name referencing port 9000 (the default) rather than 19000 (the configured port) indicates that the tserver was partially configured with defaults, potentially causing a mismatch between the PostgreSQL backend and the tablet server it was trying to report to.

This kind of partial configuration failure is notoriously difficult to debug because the system appears to start correctly—ports bind, processes launch, logs show no errors—but internal communication between components fails silently.## Reasoning and Motivation: Why This Message Was Written

The assistant wrote this message to document a diagnostic checkpoint. After spending over an hour wrestling with port configuration, data directory cleanup, and waiting for health checks, the assistant needed to answer a fundamental question: Is the database actually running?

This question was the bottleneck for all further progress. If YugabyteDB wasn't running, the entire test cluster was dead in the water—the Kuri storage nodes couldn't initialize their schemas, the S3 proxy couldn't route requests, and the monitoring dashboard would show empty metrics. The assistant had reached a point where the standard tools (Docker health checks, yugabyted status, connection attempts) were all giving inconclusive or contradictory results. The health check was stuck on "starting," but the port bindings showed all services listening. Connection attempts timed out, but the process table showed the PostgreSQL backend in running state.

The motivation was diagnostic clarity. By running ps aux inside the container, the assistant could bypass the abstractions of Docker health checks and YugabyteDB's status reporting to see the raw process state. This is a classic debugging strategy: when the monitoring layer gives ambiguous results, go to the operating system layer.

Assumptions Made by the Assistant

Several assumptions are embedded in this diagnostic step:

The ps aux output would be authoritative: The assistant assumed that if the PostgreSQL backend process was visible in the process table, the database was "running" in a meaningful sense. This is generally true, but it doesn't guarantee that the database is accepting connections or that its internal state is consistent.
The problem was in the connection path, not the database itself: By concluding "YB processes are running!" and then investigating why ysqlsh was hanging, the assistant implicitly assumed that the database server was healthy and the issue was in the client-server connection. This assumption turned out to be partially correct—the server was running—but the hanging ysqlsh process suggested a deeper problem.
The stuck ysqlsh was a symptom, not the cause: The assistant noted that a previous ysqlsh command was still running (started at 15:44, still alive at 15:49+). This was correctly identified as a stuck diagnostic command, not the cause of the problem. However, the assistant didn't immediately consider that this stuck process might be holding a resource or interfering with subsequent connections.
Port configuration was correct: The assistant assumed that because the ports were listening, the configuration was correct. The yb_pg_metrics.node_name=biryani:9000 detail suggests otherwise—the tserver might have been running with a mix of configured and default values, potentially causing internal routing issues.

Mistakes and Incorrect Assumptions

The most significant mistake visible in this message is the over-reliance on port binding as a success indicator. The assistant had previously celebrated when ss -tlnp showed all four ports listening, interpreting this as evidence that YugabyteDB was fully operational. But port binding only means the process has opened a socket and called listen()—it doesn't mean the service is ready to accept connections, has completed initialization, or can route requests internally.

YugabyteDB is a distributed database with multiple components (yb-master, yb-tserver, PostgreSQL frontend) that must all reach a consistent state before the system is operational. The yb-master must elect a leader, the yb-tserver must register with the master, and the PostgreSQL backend must connect to the tserver's internal PostgreSQL compatibility layer. Port binding is the first step in a multi-stage initialization process, and the assistant had prematurely interpreted it as the last step.

Another subtle mistake was the failure to immediately recognize the significance of the node name mismatch. The yb_pg_metrics.node_name=biryani:9000 value contains port 9000, which is the default yb-tserver HTTP port, not the configured 19000. This mismatch suggests that the --tserver_flags parameter might have been partially applied—the port flags for --rpc_bind_addresses and --webserver_interface were correctly parsed (hence the ports bound to 19042, 19000, 19100), but other flags like --metrics_snapshotter_tserver_metrics_whitelist and --yb_pg_metrics.node_name retained their defaults. This kind of partial flag application can happen when flags are passed in an unexpected format or order, or when the yugabyted wrapper doesn't forward all flags correctly.

Input Knowledge Required

To fully understand this message, the reader needs:

YugabyteDB architecture knowledge: Understanding that YugabyteDB consists of multiple processes (yb-master, yb-tserver, PostgreSQL) that must all be running and registered with each other. The presence of the PostgreSQL backend doesn't guarantee the tserver is healthy.
Docker debugging skills: Knowing how to use docker exec to run commands inside a container, and how to interpret ps aux output to distinguish between user processes (ysqlsh) and system processes (postgres).
Unix process state understanding: The S and R state codes in ps aux output—S means interruptible sleep (waiting for something), R means running or runnable. The stuck ysqlsh in S state was waiting for a response that would never arrive.
YugabyteDB configuration familiarity: Recognizing that yb_pg_metrics.node_name=biryani:9000 contains a default port value (9000) rather than the configured custom port (19000), and understanding why this matters for internal metrics reporting.
The broader system architecture: Knowing that this YugabyteDB instance is part of a three-layer S3 storage system, and that the database must be operational before the Kuri storage nodes and S3 proxy can function.## Output Knowledge Created by This Message This message created several pieces of actionable knowledge:
Confirmed PostgreSQL process existence: The primary output was confirmation that the PostgreSQL backend was running inside the container. This ruled out a class of failures related to database startup and narrowed the search to connection-layer issues.
Identified a stuck diagnostic process: The hanging ysqlsh from a previous diagnostic attempt was visible in the process table. This explained why subsequent connection attempts might have failed—if the earlier ysqlsh had acquired a resource (like a connection slot or a lock) and was blocking new connections.
Revealed configuration inconsistency: The yb_pg_metrics.node_name value exposed a mismatch between configured and actual ports, suggesting that the --tserver_flags parameter was only partially effective. This became a lead for further investigation.
Established a diagnostic baseline: By documenting the exact state of the processes at this moment, the message created a reference point for future debugging. If the PostgreSQL process had later crashed or been restarted, the assistant would know the timeline.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the message itself. The opening line—"YB processes are running! The ysqlsh is also hanging."—shows the assistant processing two contradictory pieces of information simultaneously. The exclamation mark after "running" suggests a moment of relief or discovery, immediately tempered by the recognition that ysqlsh was still hanging.

The decision to check for the PostgreSQL process specifically (rather than just the yb-master or yb-tserver) shows systems-level thinking. The assistant knew that ysqlsh connects to the PostgreSQL frontend, not directly to the yb-tserver. If the PostgreSQL process wasn't running, that would explain the hanging connection. If it was running, the problem was elsewhere—perhaps in the network configuration, the authentication layer, or the internal routing between PostgreSQL and the tablet server.

The head -5 limit on the grep output shows practical debugging discipline. In a container running multiple processes, ps aux can produce dozens of lines. The assistant was looking for a specific pattern and didn't need to see every matching line. This is the mark of an experienced debugger who knows how to focus output on the relevant information.

Broader Implications for the Debugging Session

This message sits at a turning point in the debugging session. Before this message, the assistant was in a cycle of restarting containers, cleaning data directories, and waiting for health checks—all based on the assumption that YugabyteDB wasn't starting correctly. After this message, the focus shifted to understanding why a running database wasn't accepting connections.

The confirmation that PostgreSQL was running meant the problem was likely in one of three areas:

Network configuration: The PostgreSQL process was bound to 127.0.0.1:15433 inside the container, but the ysqlsh client was connecting to 127.0.0.1:15433 from the host. In bridge networking mode (which the assistant had reverted to after host networking caused port conflicts), the container's 127.0.0.1 is different from the host's 127.0.0.1. This is a classic Docker networking gotcha.
Authentication or authorization: The connection string included yugabyte:[REDACTED]@127.0.0.1:15433, suggesting password authentication was being attempted. If the password was incorrect or the user didn't have the right permissions, the connection would hang or fail.
Internal YugabyteDB state: The yb-tserver might not have fully registered with the yb-master, meaning the PostgreSQL backend couldn't route queries to the tablet storage layer. The node name mismatch (biryani:9000 vs biryani:19000) supports this theory. The assistant's next steps would likely involve checking network connectivity between the container and host, verifying authentication credentials, and examining the yb-tserver logs for registration errors.

Conclusion

Message 1265 is a masterclass in diagnostic discipline. When faced with contradictory signals from high-level monitoring tools (ports listening but health check stuck, processes running but connections hanging), the assistant dropped down to the operating system level to check the actual process state. The ps aux command—one of the oldest and most fundamental debugging tools in Unix—provided the clarity that all the abstraction layers had failed to deliver.

The message also reveals the importance of reading diagnostic output carefully. The yb_pg_metrics.node_name=biryani:9000 detail, easily overlooked in a quick scan of the process listing, contained a crucial clue about partial configuration application. In complex distributed systems, the devil is often in these small inconsistencies—a default value where a configured value was expected, a port number that doesn't match the architecture diagram, a process that started but never completed its initialization.

For anyone debugging distributed systems, this message offers a valuable lesson: when the monitoring dashboard says one thing and your connection attempts say another, go straight to the process table. The truth is always in the running processes.

The Diagnostic Pivot: How ps aux Revealed the Truth About a Stuck Database