The Diagnostic Pivot: Recognizing Architectural Boundaries in Ansible Test Infrastructure

In the midst of a complex debugging session for a horizontally scalable S3 storage system built on YugabyteDB and Kuri storage nodes, a single assistant message at index 1542 marks a critical diagnostic turning point. The message reads:

[assistant] Several issues: 1. YugabyteDB doesn't have SSH (and doesn't need it - it's just the database) 2. The target containers are still booting (systemd startup)

Let me update the inventory to remove YugabyteDB from the SSH hosts and wait for systemd to fully boot: [read] /home/theuser/gw/ansible/test/docker/test-inventory/hosts.yml

At first glance, this appears to be a simple debugging note—two problems spotted, one file to edit. But beneath this terse summary lies a moment of genuine architectural insight. The assistant has just run the Ansible playbook test suite against a Docker-based test harness and received a stark failure: Ansible could not connect via SSH to the host at IP address 172.28.0.10. The error message was unambiguous: "Connection refused." The assistant's response is to step back, interpret the error in terms of the system architecture, and identify not one but two distinct failure modes that need correction.

The Context: A Test Harness Under Construction

To understand why this message matters, we must reconstruct the situation that produced it. The assistant had just committed a comprehensive set of Ansible deployment scripts for FGW (Filecoin Gateway) clusters, comprising seven roles, five playbooks, and a production inventory structure. To validate these scripts, the assistant created a Docker-based test harness with four containers: a YugabyteDB database container, three target hosts (two Kuri storage nodes and one S3 frontend proxy) running Ubuntu 24.04 with systemd and SSH, and an Ansible controller container. The test harness included setup, run, and cleanup scripts designed to simulate a production deployment end-to-end.

The first execution of run-tests.sh failed immediately at the connectivity check stage. The error showed Ansible attempting to SSH into 172.28.0.10—the IP address assigned to the YugabyteDB container—and receiving "Connection refused." This is where our subject message begins.

Issue One: The Database That Doesn't Need SSH

The assistant's first identified issue is the more architecturally significant one: "YugabyteDB doesn't have SSH (and doesn't need it - it's just the database)." This statement reveals a fundamental misunderstanding embedded in the original test inventory design. The inventory file listed YugabyteDB as an Ansible-managed host under a yugabyte group, complete with an ansible_host IP address. This implied that Ansible would manage the database container via SSH, just like the Kuri and S3 frontend hosts.

But the YugabyteDB container is not an SSH-accessible target. It is a database service container, running YugabyteDB's own initialization process (bin/yugabyted start). It exposes YSQL on port 5433 and YCQL on port 9042, but it does not run an SSH daemon. The database initialization playbook (setup-yb.yml) was designed to run against localhost (the Ansible controller) and use psql and cqlsh command-line tools to connect to the database over the network—not via SSH.

The assistant's insight here is a recognition of architectural boundaries. In infrastructure automation, not every component in a system is an Ansible target. Some components are services that are accessed over network protocols, not managed through SSH. Mixing these two categories in the inventory creates confusion: Ansible will attempt SSH connections to every host listed, and when those hosts don't run SSH, the entire playbook run fails before any useful work can begin.

This is a classic mistake in infrastructure-as-code design. The temptation is to list every component in the inventory for completeness, but the inventory should only contain hosts that Ansible will manage via SSH. The YugabyteDB container should be treated as an external dependency—something the playbooks connect to, not something they manage directly.

Issue Two: The Booting Containers

The second issue the assistant identifies is more operational: "The target containers are still booting (systemd startup)." The Docker target hosts run Ubuntu 24.04 with systemd as the init system. When these containers start, systemd goes through its boot sequence, which includes creating the /run/nologin file. This file causes PAM (Pluggable Authentication Modules) to reject SSH login attempts from non-root users with the message "System is booting up. Unprivileged users are not permitted to log in yet."

The assistant correctly recognizes that the connectivity failure might not be purely about the YugabyteDB issue—the Kuri and S3 frontend containers might also be unreachable simply because they haven't finished booting. This is a timing problem that manifests as an SSH failure, which could easily be mistaken for a configuration error.

The distinction between these two issues is important. The first is a design error (listing a non-SSH host in the inventory). The second is a timing issue (not waiting for systemd to finish booting). Both produce the same symptom—SSH connection failure—but require different fixes. The assistant's ability to separate these two causes demonstrates a systematic debugging approach: rather than treating the error as a single problem, the assistant decomposes it into its constituent parts.

The Thinking Process: What the Message Reveals

The subject message is notable for what it does not contain as much as for what it does. The assistant does not simply report the error and ask for guidance. Instead, the assistant presents a diagnosis: "Several issues," followed by a numbered list. This structure reveals an active analytical process.

The assistant has taken the raw error output—"ssh: connect to host 172.28.0.10 port 22: Connection refused"—and mapped it onto the system architecture. The IP address 172.28.0.10 corresponds to the YugabyteDB container. The assistant knows that this container runs YugabyteDB, not SSH. Therefore, the error is not a transient network issue or a configuration mistake; it is a fundamental mismatch between the inventory design and the actual system architecture.

Similarly, the assistant knows that the target containers use systemd, and that systemd-based containers take time to become fully operational. The assistant has seen this pattern before—containers that are "up" from Docker's perspective but not yet ready to accept SSH connections because systemd is still initializing.

This kind of diagnostic reasoning requires deep knowledge of the system's architecture, the container environment, and the behavior of the tools involved (Ansible, SSH, systemd, Docker). The assistant is not guessing; it is applying prior knowledge to interpret error signals.

Input Knowledge Required

To fully understand this message, a reader needs:

Ansible inventory mechanics: Understanding that Ansible uses ansible_host to determine where to connect via SSH, and that every host in the inventory is expected to be SSH-accessible unless explicitly configured otherwise.
Docker container architecture: Understanding that different containers serve different roles—some are SSH-accessible targets (like the Ubuntu-based Kuri and S3 hosts), while others are service containers (like YugabyteDB) that expose network ports but don't run SSH.
Systemd boot behavior: Understanding that systemd creates /run/nologin during boot to prevent non-root logins, and that this file must be removed (or systemd must finish booting) before SSH logins from unprivileged users will succeed.
The test harness design: Understanding the Docker Compose setup with four containers, their IP assignments, and the intended test flow (connectivity → database init → Kuri deployment → S3 frontend deployment → verification).
The FGW architecture: Understanding that YugabyteDB is the shared database layer, Kuri nodes are storage nodes, and S3 frontends are stateless proxies—each with different management requirements.

Output Knowledge Created

This message produces several important insights:

The inventory must be restructured: The YugabyteDB host must be removed from the Ansible-managed hosts list. Database initialization should happen via network connections from the controller, not via SSH into the database container.
The test script must wait for systemd: The connectivity check should include a wait loop that verifies systemd has finished booting before attempting SSH connections.
A reusable diagnostic pattern: The assistant demonstrates how to decompose a generic "connection refused" error into architecture-specific and timing-specific causes—a pattern applicable to many infrastructure debugging scenarios.

The Broader Significance

This message represents a boundary-drawing moment in infrastructure design. The assistant is learning—and teaching—where the lines of responsibility lie in a distributed system. Ansible manages hosts via SSH. Database containers provide services via network protocols. These are different interaction models, and mixing them in a single inventory creates confusion.

The fix that follows this message is straightforward: remove the YugabyteDB host from the inventory and adjust the test script to wait for systemd boot. But the insight that produces this fix is not trivial. It requires seeing through the surface-level error to the architectural assumptions beneath. Every "connection refused" error is a question: "What did you expect to find at this address, and why?" The assistant's answer reveals a deep understanding of the system being built.