The Four Words That Saved a Cluster: "Change all YB ports"

In a conversation spanning dozens of messages, hundreds of lines of code, and multiple architectural pivots, the most consequential decision sometimes arrives in the fewest words. Message 1213, sent by the user, contains exactly four words:

"Change all YB ports"

This terse directive came at a critical inflection point. The assistant had just spent several messages converting the entire test cluster from Docker bridge networking to host network mode, aiming to eliminate the Docker userland proxy bottleneck that was causing "connection reset by peer" errors during high-concurrency S3 load tests. The conversion touched the docker-compose.yml, the gen-config.sh configuration generator, and the README.md documentation. But when the assistant tried to start the newly configured cluster, it hit an unexpected wall: YugabyteDB refused to become healthy.

The Crisis That Preceded the Command

The host network conversion had gone smoothly for the application containers—kuri-1, kuri-2, and the S3 proxy all started without complaint. But the YugabyteDB container remained stubbornly "unhealthy," failing its health check repeatedly. The assistant's debugging revealed the root cause: ports 7000 and 7100 were already bound on the host machine. These are YugabyteDB's default internal ports for the yb-master HTTP and RPC interfaces. With host network mode, the container's ports map directly to the host's network namespace, so any pre-existing service listening on those ports creates an immediate conflict.

The assistant attempted to investigate further, running docker inspect to examine the health check logs, checking ss -tlnp to identify port usage, and even trying to exec into the container to run diagnostic commands—only to hit "OCI runtime exec failed: exec: 'ss': executable file not found in $PATH." The debugging was stalling. The container was stuck in a bootstrapping loop, unable to bind its required ports, and the assistant was running out of diagnostic options from outside the container.

The User's Intervention: A Decision, Not a Diagnosis

This is the moment the user stepped in with "Change all YB ports." The message is remarkable for what it does not contain. There is no analysis of which ports are conflicting, no request for more information, no suggestion to investigate what is using ports 7000 and 7100. The user bypassed the entire diagnostic phase and went straight to a solution: move every YugabyteDB port to a new range.

The reasoning behind this decision is worth unpacking. The user had several alternative paths available:

  1. Identify and stop the conflicting service — This would require discovering what was listening on 7000/7100, determining whether it was safe to stop, and potentially disrupting other workflows on the host.
  2. Revert to bridge networking — This would abandon the host network approach entirely, keeping the Docker proxy bottleneck that motivated the conversion in the first place.
  3. Use Docker's port mapping selectively — Map only the application ports through host networking while keeping YugabyteDB on a Docker network, adding complexity to the networking topology.
  4. Change all YugabyteDB ports — The chosen path. Simple, comprehensive, and definitive. The user's choice reflects a deep understanding of the system's architecture. YugabyteDB's port assignments are configurable through environment variables and command-line flags. The internal ports (7000, 7100, 9000, 9100) are used for inter-process communication between yb-master and yb-tserver processes within the same container, and for intra-cluster communication in multi-node deployments. In a single-node test cluster, these ports can be freely relocated as long as the configuration remains internally consistent. The user correctly assessed that there was no external dependency on specific port numbers—only internal consistency mattered.

The Assumptions Embedded in Four Words

The command "Change all YB ports" carries several implicit assumptions that are worth examining. First, it assumes that YugabyteDB's port configuration is fully parameterizable—that there are no hard-coded port expectations baked into the container image or the startup scripts. This is a correct assumption for YugabyteDB, which exposes --master_http_port, --master_rpc_port, --tserver_http_port, --tserver_rpc_port, --ysql_port, and --cql_port flags, plus the YB_MASTER_HTTP_PORT, YB_MASTER_RPC_PORT, YB_TSERVER_HTTP_PORT, YB_TSERVER_RPC_PORT, YSQL_PORT, and YCQL_PORT environment variables.

Second, the command assumes that the port conflicts are the only reason YugabyteDB is failing to become healthy. This is a reasonable inference from the evidence: the health check logs showed Connection refused on port 5433, and the ss -tlnp output showed that YugabyteDB had bound to port 15433 instead of 5433—a classic symptom of port conflict fallback behavior. However, there was a small risk that other issues (data directory permissions, filesystem space, kernel parameters) were also contributing. The user implicitly judged that the port conflict was the primary blocker and that fixing it would resolve the health check failure.

Third, the command assumes that changing all ports is safer than changing only the conflicting ones. This is a subtle but important engineering judgment. If the user had said "Change ports 7000 and 7100," the assistant might have shifted only those two ports while leaving the others at their defaults. But a partial shift could create inconsistencies—for example, if the yb-master HTTP port moves but the yb-tserver still tries to reach it on the old port, or if the health check script hard-codes certain port numbers. By saying "all," the user ensured a clean, consistent renumbering of the entire port space, eliminating any risk of cross-port mismatches.

The Execution That Followed

The assistant's response to the command was immediate and methodical. It stopped and removed the failing YugabyteDB containers, read the current docker-compose.yml, and systematically edited the file to apply a uniform port offset across all of YugabyteDB's services. The edits show the assistant updating the docker-compose.yml multiple times: first changing the YugabyteDB service's command to pass new port flags, then updating the db-init service's connection strings to point to the new YSQL port, then updating the kuri nodes' YCQL host references, and finally updating gen-config.sh to match. The key insight is that every component that referenced YugabyteDB ports—the database initialization container, the kuri storage nodes' configuration, and the health check—needed to be updated in lockstep. A single missed reference would leave the cluster unable to connect to its metadata store.

Input Knowledge Required

To understand the significance of "Change all YB ports," a reader needs to know several things:

  1. The host network mode context: The cluster had just been converted from bridge networking to host network mode, meaning container ports bind directly to the host's network interfaces rather than being mapped through Docker's NAT.
  2. YugabyteDB's port architecture: YugabyteDB uses multiple ports for different services (YSQL, YCQL, yb-master HTTP/RPC, yb-tserver HTTP/RPC), and these are configurable via environment variables or command-line flags.
  3. The port conflict evidence: The assistant had discovered that ports 7000 and 7100 were already in use on the host, causing YugabyteDB to bind to fallback ports and fail its health check.
  4. The cluster's configuration surface: The YugabyteDB port settings are referenced in the docker-compose.yml (for the yugabyte container's command), in the db-init service (for connecting to YSQL), and in the kuri nodes' configuration (for connecting to YCQL). Changing ports requires updating all three layers.

Output Knowledge Created

The message "Change all YB ports" created a clear, actionable directive that:

  1. Terminated the diagnostic phase: The assistant stopped investigating the port conflict's root cause and shifted to implementing the solution.
  2. Defined the scope of changes: "All" meant every YugabyteDB port, not just the ones known to conflict. This prevented a piecemeal approach that might miss some ports.
  3. Established the solution strategy: Port renumbering rather than conflict resolution, service termination, or network topology changes.
  4. Triggered a cascade of edits: The assistant went on to edit docker-compose.yml five times and gen-config.sh four times to propagate the new port assignments through every layer of the configuration.

The Thinking Process Visible in the Conversation

The user's thinking process is not directly visible—the message is too short to contain reasoning traces. But the timing and content reveal the thought pattern. The user observed the assistant struggling with a port conflict during the host network migration. Rather than letting the assistant continue down the diagnostic rabbit hole (checking what was on ports 7000/7100, trying to exec into the container, waiting for bootstrapping to complete), the user recognized that the fundamental issue was a namespace collision and that the cleanest fix was to move the database's port footprint to uncontested territory.

This is a classic systems-thinking move: when a component cannot occupy its default position because something else is already there, relocate the component rather than evict the existing occupant. The user implicitly understood that the services on ports 7000 and 7100 were probably unrelated to the test cluster and that moving YugabyteDB was less risky than stopping whatever was already running.

Conclusion

"Change all YB ports" is a masterclass in concise, high-leverage technical direction. In four words, the user communicated a diagnosis (the port conflict is the problem), a scope (all ports, not just the conflicting ones), a strategy (renumber rather than resolve), and a priority (do this now, stop debugging). The message demonstrates that effective technical communication does not require lengthy explanations when the recipient has sufficient context to interpret the intent. For the assistant, it was a clear pivot from investigation to execution. For the test cluster, it was the fix that finally allowed the host network migration to succeed, paving the way for the high-throughput load tests that had motivated the entire effort.