The Moment Connectivity Clicked: Debugging an Ansible Test Harness for Distributed S3 Infrastructure

The Message

The subject message is a terminal output from an Ansible test suite execution. It reads, in full:

[assistant] [bash] cd /home/theuser/gw/ansible/test && ./run-tests.sh 2>&1
=== FGW Ansible Playbook Tests ===


=== Test 1: Connectivity Check ===
s3-fe-01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
kuri-02 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
kuri-01 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

=== Test 2: YugabyteDB Initialization ===

=== Running: playbooks/setup-yb.yml  ===
[WARNING]: Deprecation warnings can be disabled by setting `deprecation_warnings=False` in ansible.cfg.
[DEPRECATION WARNING]: community.general...

At first glance, this appears to be a routine test run. But in the context of the session, it represents a hard-won milestone: the first time all three target hosts in the Docker-based test cluster respond successfully to Ansible's ping module. The three green SUCCESS responses for s3-fe-01, kuri-02, and kuri-01 are the culmination of an extensive debugging session that touched nearly every layer of the test infrastructure.

Why This Message Was Written: The Debugging Arc

This message was produced as part of an iterative test-and-fix cycle for validating Ansible deployment scripts. The assistant had just committed a comprehensive Ansible automation framework for a horizontally scalable S3 storage architecture built on YugabyteDB. The framework included five playbooks, seven roles, and a production inventory structure. To validate these scripts without touching real hardware, the assistant created a Docker-based test harness that simulated the production topology: a YugabyteDB container, three target hosts running Ubuntu 24.04 with systemd and SSH, and an Ansible controller container.

The user's instruction was simple: "run the tests." What followed was a cascade of failures that each revealed a hidden assumption or missing piece in the test environment. The assistant worked through them methodically:

  1. YugabyteDB health check failure: The database container bound to its own hostname (yugabyte) rather than localhost, causing the Docker health check to fail. Fixed by correcting the health check query host.
  2. Read-only volume mounts: The Ansible source directory was mounted read-only into the controller container, preventing file operations. Fixed by restructuring volume mounts to use a writable work volume.
  3. Missing sshpass: The controller needed sshpass for password-based SSH authentication to the target hosts. Initially installed via pip (which failed), then corrected to use apt-get.
  4. YugabyteDB in the SSH inventory: The test inventory listed the YugabyteDB container as an SSH target, but it doesn't run an SSH server. Fixed by removing it from the all children group and keeping it only as a data source.
  5. pam_nologin blocking SSH: Ubuntu's systemd creates /run/nologin during early boot, which prevents non-root SSH logins. The target containers were still booting when tests ran. Fixed by waiting for systemctl is-system-running and removing the nologin file.
  6. Missing Python 3 on targets: Ansible's ping module requires Python on the remote host. The Ubuntu 24.04 base image didn't have it installed. This was fixed in the Dockerfile by adding Python 3 to the target image build. Each of these fixes required the assistant to diagnose the failure, understand the root cause, and apply a targeted correction. Message 1551 is the first execution of the test suite after the cumulative effect of all these fixes — and it shows the connectivity test finally passing.## The Significance of the Three Green "SUCCESS" Lines The connectivity check output is deceptively simple. Each SUCCESS line represents an Ansible ping module execution against a target host, verifying that the controller can establish an SSH connection, authenticate, transfer the module payload, execute Python code on the remote host, and receive the result. For kuri-01, kuri-02, and s3-fe-01, all of these steps completed successfully. What makes this moment significant is what preceded it. In message 1549, the assistant ran the exact same test command and got a failure: The module interpreter '/usr/bin/python3' was not found. That error was caused by the target containers not having Python 3 installed — a critical detail because Ansible's default module execution relies on Python on the remote host. The assistant had to recognize that the Dockerfile for the target containers needed to include Python 3, rebuild the images, restart the containers, and re-run the setup before attempting the tests again. The message doesn't show that rebuild step explicitly (it happened in the background between messages), but the successful ping results confirm that the fix was applied correctly.

Assumptions Made and Mistakes Corrected

Several assumptions were embedded in the original test harness design, and the debugging process revealed each one:

Assumption 1: The YugabyteDB container could serve as an Ansible target. The original inventory listed yb-test under the all children group alongside the Kuri and S3 frontend hosts. This made sense from a logical grouping perspective — the database is part of the cluster — but failed operationally because the YugabyteDB image doesn't run an SSH server. The assistant corrected this by removing the Yugabyte host from the all group and keeping it only as a data source for database initialization tasks.

Assumption 2: Systemd would be ready for SSH connections immediately after container startup. The Dockerfile for target containers used Ubuntu 24.04 with systemd as the init system. Systemd creates /run/nologin during early boot to prevent non-root logins until the system is fully initialized. The assistant's initial approach was to wait a fixed number of seconds and then attempt connections, but the timing was unreliable. The fix involved explicitly checking systemctl is-system-running and removing the nologin file.

Assumption 3: The Ansible controller would have all required tools pre-installed. The controller image was based on python:3.11-slim, which includes Python and pip but not sshpass, psql, or cqlsh. The assistant had to install these tools after container startup, first attempting pip install sshpass (which failed because sshpass is a system package, not a Python package) and then correcting to apt-get install sshpass.

Assumption 4: Volume mounts from the host would be writable. The Docker Compose configuration mounted the project's ansible/ directory directly into the controller container. This directory is part of a Git repository and may have restrictive permissions. The assistant restructured the setup to copy files into a dedicated work volume instead of mounting them directly.

These assumptions were not unreasonable — they represent the gap between a conceptual design and a working implementation. The test harness was built to discover exactly these kinds of issues before deploying to production.## Input Knowledge Required to Understand This Message

To fully grasp what this message communicates, a reader needs familiarity with several domains:

Ansible fundamentals: Understanding that the ping module is not a network ICMP ping but an SSH connectivity and Python execution test. The SUCCESS output with "ping": "pong" is the standard Ansible ping response, confirming that the controller can push and execute modules on the remote host.

Docker networking and systemd: The test harness uses a custom Docker network with static IP assignments (172.28.0.21 for kuri-01, 172.28.0.22 for kuri-02, 172.28.0.31 for s3-fe-01). The target containers run Ubuntu 24.04 with systemd as PID 1, which introduces boot-time behaviors like pam_nologin that can interfere with SSH access.

Distributed S3 architecture context: The hosts being tested — kuri-01, kuri-02, and s3-fe-01 — correspond to the three-layer architecture: Kuri storage nodes (two instances) and an S3 frontend proxy. The YugabyteDB database is a separate service not managed via SSH. Understanding this topology explains why the inventory was structured the way it was and why the Yugabyte host was initially (and incorrectly) included in the SSH target group.

The debugging history: This message is meaningless without the context of the previous failures. The reader must know that the same test command failed minutes earlier with a Python interpreter error, and that the assistant had to rebuild the target Docker images with Python 3 installed to make the ping module work.

Output Knowledge Created by This Message

The message creates several forms of knowledge:

Operational confirmation: The three SUCCESS responses provide concrete evidence that the SSH infrastructure is working correctly. The controller can reach all three target hosts, authenticate, and execute Ansible modules. This is the foundational layer upon which all subsequent deployment tasks depend.

Validation of the Docker test harness design: The fact that the connectivity test passes confirms that the Docker Compose configuration, network setup, SSH key distribution, and user configuration are all correct. The test harness is now a viable tool for validating playbook changes without touching real infrastructure.

A baseline for further debugging: The message shows that Test 1 passes but Test 2 (YugabyteDB initialization) has begun and is producing deprecation warnings. The [DEPRECATION WARNING]: community.general... output signals that the setup-yb.yml playbook is running but may encounter issues — which indeed happens in subsequent messages when the controller lacks psql and cqlsh tools. This message thus marks the boundary between a solved problem (SSH connectivity) and an emerging one (database client tooling).

A debugging methodology: The sequence of fixes leading to this message demonstrates a pattern: identify the error, trace it to its root cause, apply the minimal fix, and re-run. This methodology is itself knowledge that can be applied to future infrastructure debugging.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the sequence of actions between messages. When the connectivity test failed in message 1549 with the Python interpreter error, the assistant didn't immediately re-run the test. Instead, it:

  1. Verified SSH at the transport layer: Before addressing the Python issue, the assistant ran sshpass -p fgwpass ssh ... echo SSH works to confirm that basic SSH connectivity was functional. This isolated the problem to the Ansible module execution layer rather than the network or authentication layer.
  2. Identified the correct fix: The error message explicitly stated that /usr/bin/python3 was not found on the remote host. The assistant recognized this as a missing package in the Docker image rather than a configuration issue.
  3. Applied the fix at the right level: Rather than installing Python 3 at runtime via Ansible (which would be circular — needing Python to install Python), the assistant updated the Dockerfile.target to include Python 3 in the base image, then rebuilt and restarted the containers.
  4. Excluded the non-SSH host: The assistant also recognized that the yb-test host (YugabyteDB) should not be part of the connectivity test at all, since it doesn't run SSH. This was corrected by editing the test script to limit the ping test to the kuri and s3_frontend groups. This diagnostic approach — isolate, identify, fix at the root, verify — is a model of systematic debugging. The assistant avoided the temptation to apply superficial fixes (like setting ansible_python_interpreter to a non-existent path) and instead addressed the actual missing dependency.

Conclusion

Message 1551 is a quiet victory in a long debugging session. It doesn't contain flashy code changes or architectural revelations. It is simply a terminal output showing three hosts responding "pong" to an Ansible ping. But that output represents the successful resolution of at least six distinct infrastructure issues, each of which would have blocked the entire deployment automation effort. The message is a testament to the value of building test harnesses early, iterating on them aggressively, and trusting the process of systematic debugging. Every "SUCCESS" line has a story behind it — and in this case, the story is one of methodical problem-solving across containers, networks, SSH configurations, package management, and system initialization.