The First Test Run: When Ansible Meets a Database Container Without SSH

The Message

The message is a terminal output captured at a critical juncture in the development of an Ansible-based deployment system for a horizontally scalable S3 storage architecture. It is the raw result of executing the very first automated test against a freshly built Docker-based test harness. The output reads:

[assistant] [bash] cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1
=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
[ERROR]: Task failed: Data could not be sent to remote host "172.28.0.10". Make sure this host can be reached over ssh: ssh: connect to host 172.28.0.10 port 22: Connection refused
Origin: <adhoc 'ping' task>

{'action': 'ping', 'args': {}, 'timeout': 0, 'async_val': 0, 'poll': 15}

yb-test | UNREACHABLE! => {
    "changed": false,
    "msg": "Task failed: Data could not be sent to remote host \"172.28.0.10\". Make sure this...

At first glance, this is simply a failed SSH connection. But within the broader context of the session, this message represents something far more significant: the moment a complex infrastructure-as-code project collides with reality for the first time, revealing a fundamental design assumption that had gone unnoticed through multiple rounds of planning and implementation.

Context and Motivation

To understand why this message was written, we need to step back and examine what had just been built. The assistant had spent the preceding messages creating a comprehensive Ansible deployment system for the Filecoin Gateway (FGW) project — a horizontally scalable S3-compatible storage architecture. The deployment system consisted of five Ansible playbooks, seven roles, and a production inventory structure, all committed in two git commits. Alongside the deployment code, the assistant had built an elaborate Docker-based test harness designed to validate the playbooks in a controlled environment before they ever touched real hardware.

The test harness was ambitious. It used Docker Compose to orchestrate five containers: a YugabyteDB database, two Kuri storage nodes, one S3 frontend proxy, and an Ansible controller. The target containers ran Ubuntu 24.04 with systemd and SSH, simulating production servers. The harness included setup scripts to build binaries, start containers, and install dependencies, plus run scripts to execute the playbooks and verify deployment.

The user's command was simple — "run the tests" (message 1518) — but the assistant had to work through a cascade of infrastructure issues before even getting to this point. The YugabyteDB container had a health check that failed because it was checking localhost instead of the container's hostname. The Docker volume mounts had read-only conflicts. The controller container needed sshpass, psql, and cqlsh installed. The target containers needed time for systemd to boot. Each of these issues was diagnosed and fixed in the messages leading up to message 1541.

Finally, after all that preparation, the assistant ran the tests. Message 1541 is the result.

The Failure and Its Meaning

The test failed immediately on the very first step: a connectivity check using Ansible's ping module. The target was yb-test at IP address 172.28.0.10 — the YugabyteDB container. The error message is unambiguous: SSH connection refused.

On the surface, this looks like a simple operational issue — perhaps the container wasn't ready, or SSH wasn't installed. But the deeper problem is architectural. The YugabyteDB container was never meant to be an Ansible-managed host. It was the database backend, running YugabyteDB's own yugabyted process. It didn't have an SSH server, it didn't have a user account for Ansible to connect to, and it didn't need any of the deployment steps that the playbooks would apply to the Kuri and S3 frontend nodes.

The test inventory had placed the YugabyteDB host (yb-test) in its own group under all.children, alongside the Kuri and S3 frontend groups. This made it a target for Ansible's default behavior: attempt to connect to every host in the inventory. The connectivity test script ran an ad-hoc ping against all hosts, and YugabyteDB was included.

Assumptions Made

This message reveals several assumptions that turned out to be incorrect:

Assumption 1: All containers in the test environment are Ansible targets. The test inventory was structured as a flat collection of hosts, with the YugabyteDB container listed alongside the actual deployment targets. The assistant assumed that the connectivity check would either skip the database host or that the database container would somehow be reachable via SSH.

Assumption 2: The test harness would work on the first attempt. Despite the careful construction of the Docker environment, the scripts, and the inventory, the first run revealed a fundamental flaw that had been present since the inventory was first written. The assistant had not mentally separated the database infrastructure from the application deployment targets.

Assumption 3: Ansible's default behavior (ping all hosts) was acceptable. The test script used ansible all -m ping or equivalent, which naturally tried to reach every host in the inventory. The assistant had not considered that the database host should be excluded from Ansible management entirely.

Assumption 4: The IP address scheme was sufficient for isolation. The test environment used a dedicated Docker network with static IPs (172.28.0.x), but this network-level isolation didn't prevent Ansible from attempting to manage the database container.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several domains:

  1. Ansible fundamentals: How Ansible uses SSH to connect to managed hosts, the concept of inventory groups, and the ping module for connectivity testing.
  2. Docker Compose networking: How containers are assigned IP addresses on custom networks, and how service names map to container hostnames.
  3. The FGW architecture: The three-layer design with YugabyteDB as the database backend, Kuri as storage nodes, and S3 frontend proxies as the stateless API layer. Understanding that YugabyteDB is infrastructure, not an application deployment target.
  4. The test harness structure: The setup.sh script that builds binaries and starts containers, the run-tests.sh script that executes playbooks, and the inventory structure that defines which hosts Ansible should manage.
  5. The preceding debugging session: The health check fix, the volume mount correction, the sshpass installation, and the workspace setup — all of which were necessary to get to this point.

Output Knowledge Created

This message generates several important pieces of knowledge:

  1. The test harness is functional enough to produce meaningful failures. The fact that the connectivity check ran at all means the Docker environment is correctly orchestrated, the controller container can reach the target network, and the basic infrastructure is operational.
  2. The inventory has a design flaw. The YugabyteDB host should not be an Ansible-managed target. This is the primary insight generated by this failure.
  3. The test script needs to be more selective. Rather than running against all hosts, the connectivity check should target only the hosts that Ansible will actually manage (Kuri and S3 frontend nodes).
  4. The debugging process has a clear next step. The assistant's response in the following messages (message 1542) confirms this: "Several issues: 1. YugabyteDB doesn't have SSH (and doesn't need it - it's just the database) 2. The target containers are still booting (systemd startup)."

The Thinking Process

The message itself is a raw command output, so the thinking process is visible not in the message text but in what it reveals about the assistant's debugging methodology. The assistant had just completed a multi-step setup process, each step fixing a different infrastructure issue. The decision to run the tests at this point reflects a "fail fast" approach — rather than trying to perfect every detail in advance, the assistant built a reasonable test harness and ran it to see what would happen.

The failure is clean and informative. It provides the exact error, the target host, the IP address, and the nature of the failure (SSH connection refused). This is the kind of failure that experienced developers welcome: it immediately points to a specific, fixable problem.

The assistant's thinking, visible in the subsequent messages, shows a rapid diagnostic process. The error message is parsed, the root cause is identified (YugabyteDB has no SSH), and a secondary issue is noted (target containers still booting). The fix is straightforward: remove YugabyteDB from the Ansible inventory's managed hosts and add a wait for systemd boot.

Mistakes and Incorrect Assumptions

The primary mistake in this message is not in the message itself — it's a test output, faithfully reported — but in the design that led to it. The test inventory should never have included the YugabyteDB host as an Ansible target. The database container exists to provide a service (YugabyteDB), not to be managed by Ansible. This is a classic infrastructure-as-code pitfall: treating all infrastructure components as uniform deployment targets when they have fundamentally different roles.

A secondary issue is the timing. Even if the inventory had been correct, the target containers (Kuri and S3 frontend) were still booting systemd. The connectivity check might have failed for them too, though for a different reason (SSH not yet accepting connections). The assistant addressed this in the subsequent fix by adding a systemd wait loop.

Conclusion

Message 1541 is a snapshot of a project at a specific moment: the first real test of a complex deployment automation system. It captures the inevitable gap between design and reality that every infrastructure project must cross. The failure is not a setback but a discovery — it reveals an assumption that was baked into the inventory design and never questioned until the moment of execution.

The message also demonstrates a healthy development practice: build a test harness early, run it often, and let failures guide the next iteration. The assistant could have spent more time reviewing the inventory structure in theory, but the practical test revealed the flaw faster and more concretely than any review would have. This is the essence of infrastructure-as-code development: treat your deployment code with the same rigor as application code, complete with automated testing and iterative debugging.