The Test Harness That Caught Six Bugs: Validating Ansible Deployment Scripts for a Distributed S3 Architecture

Introduction

Infrastructure as code is built on a seductive promise: that deployment scripts, once written, can reliably reproduce complex environments across any number of machines. But this promise contains a hidden trap — the code must actually work. Between a well-structured Ansible role and a successfully deployed cluster lies a chasm of untested assumptions, missing dependencies, and subtle ordering constraints that only execution can reveal.

This article traces one such journey: the creation and iterative debugging of a Docker-based test harness for Ansible deployment scripts targeting a horizontally scalable S3 storage architecture for the Filecoin Gateway (FGW). The arc spans roughly 140 messages in a coding session, from the moment the Ansible scripts were committed to version control, through the construction of a comprehensive test environment, and into a debugging marathon that uncovered six distinct issues. The final fix — reordering two tasks in a Kuri Ansible role so that a configuration file is generated before the initialization command that depends on it — seems trivial in retrospect. But reaching that fix required understanding Docker networking, systemd boot sequencing, PAM authentication, Ansible variable resolution, and the internal architecture of the Kuri storage daemon.

The Foundation: Committing the Ansible Deployment Scripts

The story begins with the completion of a substantial body of work. Across dozens of preceding messages, the assistant had built an entire Ansible deployment system for FGW clusters from scratch. This included seven roles (common, wallet, yugabyte_init, kuri, s3_frontend, and supporting infrastructure), five playbooks (site.yml, deploy-kuri.yml, deploy-frontend.yml, setup-yb.yml, verify.yml), a complete inventory structure with group_vars for production, and a detailed specification document. In total, 30 files containing 1,708 lines of YAML, Jinja2 templates, and configuration were created.

But there was a critical gap: none of this code had ever been executed. The playbooks had never run against any target, the roles had never been applied, the templates had never been rendered. The assistant's summary at the end of the implementation phase included a "Usage" section showing commands like ansible-playbook playbooks/site.yml, but those commands were aspirational, not verified.

The user recognized this immediately. The instruction was concise and directive: "commit, create a docker-compose + bash/ansible harness for testing the ansible scripts." This was a demand for empirical validation. "Commit" meant preserve the work done so far. "Create a docker-compose + bash/ansible harness" meant build a laboratory where the untested deployment code could be exercised, broken, and fixed before it ever touched a real server.

The assistant's first action was to stage and commit the Ansible work. This process itself revealed a subtle issue: the project's root .gitignore contained a pattern kuri that matched the directory name of the Kuri role, causing git add ansible/roles/kuri/ to be silently ignored. The assistant had to use git add -f to force-add the directory. This interaction between past and present uses of the same name — kuri was originally a binary artifact, now also a directory — is now visible in the commit history as a piece of systems knowledge.

Building the Test Harness

With the Ansible scripts committed, the assistant turned to building the test harness. The design was ambitious: a Docker Compose environment that simulated the full production topology. The docker-compose.yml defined five containers: a YugabyteDB database instance, two Kuri storage node targets (kuri-01, kuri-02), one S3 frontend proxy target (s3-fe-01), and an Ansible controller container.

The target containers ran Ubuntu 24.04 with systemd and SSH, built from a custom Dockerfile.target. This was a deliberate choice: by using full systemd-based images, the test environment could validate the systemd service templates that the Ansible roles deployed. The controller container was based on python:3.11-slim and would execute the playbooks against the targets.

The test harness included three shell scripts: setup.sh (build binaries and start containers), run-tests.sh (execute playbooks in sequence and verify), and cleanup.sh (tear down the environment). A test inventory was created with host definitions and static IP addresses on a custom Docker network. Test wallet files were provided for authentication. A README documented the test flow.

The test flow defined in run-tests.sh showed methodical progression: connectivity check → YugabyteDB initialization → deploy Kuri nodes → deploy S3 frontend → verify all components → idempotency check (run site.yml twice to ensure no errors on the second pass). This layered approach meant that each test would only execute if the previous one succeeded, naturally isolating failures to the layer being validated.

The assistant committed this test harness as a second git commit and delivered a clean summary of everything created. The summary presented a well-structured directory tree and a simple three-step workflow: setup, run tests, cleanup. It conveyed confidence. The user's response was three words: "run the tests."

The Debugging Marathon Begins

The first test run failed immediately. The error was: ERROR: Test environment not running. Run ./setup.sh first. The setup script had built binaries and Docker images, but the containers were not in a running state when run-tests.sh attempted to execute. The assistant checked container status and found only YugabyteDB running; the target and controller containers needed to be started manually.

This initial failure triggered a cascade of discoveries. Each fix revealed the next assumption that had been made about the test environment, and each required a targeted correction before the tests could progress further.

Issue 1: The YugabyteDB Health Check Hostname

The first technical bug was subtle. The YugabyteDB container's health check in docker-compose.yml used ysqlsh -h localhost, but the YugabyteDB server was configured with --advertise_address=yugabyte, causing it to bind to its container hostname rather than localhost. The health check would fail because it was checking the wrong address. The fix was to change the health check to use -h yugabyte.

This is a classic container networking issue: services inside containers may not be available on localhost if they explicitly advertise on a different address. The assistant had assumed that localhost would work because the database process was running in the same container, but YugabyteDB's startup behavior overrode that assumption. The health check was silently failing, which meant the entire test suite was waiting for a database that would never become "healthy" according to Docker's monitoring.

Issue 2: Missing Tools in the Controller Container

When the test suite progressed to the YugabyteDB initialization playbook, it failed because the Ansible controller container lacked the command-line tools needed to execute database setup tasks. The controller was based on a minimal Python image and did not include psql (PostgreSQL client), cqlsh (Cassandra/CQL client), or sshpass (for SSH password authentication).

The assistant diagnosed this by examining the playbook's tasks and identifying which tools were required. The fix was to install postgresql-client via apt, cqlsh via pip, and sshpass as a system package. Notably, the assistant initially tried to install sshpass via pip, but quickly discovered it was a system package and corrected the approach. This ensured the controller had all necessary dependencies to execute the database initialization and SSH-based playbooks.

This issue highlights a fundamental tension in container design: minimal base images are efficient but often lack the tools needed for operational tasks. The controller container needed to be a "fat" container with all the tools required for deployment, not a slim Python runtime.

Issue 3: Missing Group Variables in the Test Inventory

The Kuri deployment playbook failed with an undefined variable error: fgw_config_dir was not defined. The assistant traced the variable's origin through Ansible's resolution chain: fgw_config_dir was defined in group_vars/kuri.yml in the production inventory, but the test inventory — a separate directory structure created specifically for the Docker harness — did not have this file.

The fix was straightforward: copy the production group_vars/kuri.yml and group_vars/s3_frontend.yml files into the test inventory's group_vars directory. Two cp commands resolved an entire class of failures. This moment illustrates a critical principle of infrastructure testing: test environments must mirror production faithfully. The most common cause of test failures that don't occur in production is environmental differences, and missing group_vars files are exactly this kind of structural gap.

The test inventory had been created as a simplified version of the production inventory, but the simplification had gone too far. By omitting the group_vars files, the test environment was testing a different configuration than what would actually be deployed. This is a subtle but dangerous form of test invalidity: the tests pass in the simplified environment but fail in production because the production environment has variables the tests never exercised.

Issue 4: Read-Only Volume Mounts

The next failure was more perplexing. The Ansible playbook attempted to deploy binaries to /opt/fgw/bin/ on the target hosts, but the directory was empty. When the assistant tried to copy binaries manually using docker cp, the error was unambiguous: mounted volume is marked read-only.

The Docker Compose file had configured the binaries:/opt/fgw/bin volume with :ro (read-only). This was an understandable choice — read-only mounts are a security best practice — but it conflicted with the Ansible deployment model, which required write access to set permissions and ownership. The fix was to edit the Docker Compose file to remove the :ro flag and then recreate the containers with docker compose down && docker compose up -d.

This tension between security and functionality is a classic infrastructure pitfall. The read-only mount was intended to protect binaries from accidental modification, but it inadvertently prevented the very deployment workflow the test harness was designed to validate. In production, the binaries would be downloaded fresh from a distribution URL (as the Ansible role's get_url task specified), so the read-only constraint was an artifact of the test environment that didn't match the production deployment model.

Issue 5: The pam_nologin Wall

Perhaps the most persistent issue was the pam_nologin mechanism. After container restart, the Ansible controller's SSH connections to the target hosts were rejected with: "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."

When a systemd-based Docker container boots, it creates a file called /run/nologin. The PAM (Pluggable Authentication Modules) framework checks for this file, and if it exists, the pam_nologin module blocks all non-root login attempts. This is a security feature designed to prevent users from logging in while the system is still initializing services. In a production server, this makes sense. But in a Docker test container, it becomes an obstacle: the containers are technically "booting" every time they start, and the Ansible controller tries to connect via SSH before systemd has finished its initialization.

The assistant tried multiple approaches: waiting longer, manually removing /run/nologin, checking systemctl is-system-running. The reliable fix was to use systemctl is-system-running --wait on each target host, which blocks until systemd reports that the system is fully operational. This ensured that SSH connections were only attempted after the boot sequence was complete.

However, even after systemctl is-system-running reported "running," the SSH connections sometimes still failed with the pam_nologin error. The assistant discovered that manually removing /run/nologin and /var/run/nologin resolved the issue immediately. This suggests that systemd's boot completion check and PAM's login check are not perfectly synchronized — there is a race condition where systemd considers the system "running" but PAM still sees the nologin file.

Issue 6: The Task Ordering Problem

With all infrastructure issues resolved, the test suite finally progressed to deploying Kuri storage nodes. And then it hit the most architecturally significant bug of the entire session.

The error message was: database "filecoingw" does not exist. But the YugabyteDB initialization playbook had created a database named filecoingw_kuri_01 — a per-node keyspace naming scheme implemented earlier in the project. Something was misaligned between the database initialization and the Kuri node's configuration.

The assistant's diagnosis was precise. The kuri init command, when executed, starts the full Kuri server process, which immediately attempts to connect to its database. The database connection parameters — host, port, database name, credentials — are stored in a file called settings.env. But the Ansible role was generating settings.env after running kuri init. The init command therefore had no database configuration and used defaults, connecting to a database named filecoingw that didn't exist.

In production, this would work because settings.env would already be present — either from a previous deployment or from manual setup. But in the test environment, the role was being applied from scratch, and the task ordering exposed the hidden dependency.

The fix was to reorder the tasks in the Kuri role's tasks/main.yml so that settings.env is generated first, and then kuri init is run with the environment properly configured. The assistant applied this edit in the final message of the session, and the edit was confirmed successful.

Architectural Insights from the Debugging Marathon

This debugging marathon, spanning six distinct issues across dozens of messages, reveals several fundamental truths about infrastructure automation.

Deployment Code Follows a Different Quality Regime

Application code can be unit-tested, integration-tested, and staged in environments that approximate production. Deployment code often lacks these safeguards because the infrastructure to test it is itself part of what needs testing. The Docker-based test harness was a solution to this bootstrapping problem: it simulated the target environment closely enough to validate the playbooks, while being cheap enough to destroy and recreate on every iteration.

The six bugs found in this session would have caused production failures of varying severity. The missing group_vars would have caused playbooks to fail with undefined variable errors. The read-only volume issue would have prevented binary deployment. The pam_nologin issue would have caused SSH connection failures during initial deployment. The task ordering bug would have caused database connection failures on fresh deployments. Each of these bugs was caught by the test harness before it ever touched a real server.

The Gap Between "Looks Correct" and "Works" Is Wide

The Ansible roles were carefully written, with proper variable definitions, template files, and task organization. Yet they contained multiple defects that only execution could reveal: the missing group_vars, the read-only volume conflict, the task ordering dependency. The user's demand for a test harness was not pedantic — it was essential. Without it, the deployment scripts would have failed in production, likely at the worst possible moment.

This is a recurring theme in infrastructure engineering. Code review can catch logical errors and style issues, but it cannot catch environmental mismatches or ordering dependencies that only manifest during execution. The only way to validate deployment code is to run it against a realistic target environment.

Systematic Debugging Is a Learnable Skill

The assistant's approach across all six issues followed a consistent pattern: observe the failure, parse the error message, trace the dependency chain to its root cause, formulate a hypothesis, apply a targeted fix, and retry. The assistant did not make random changes or guess at solutions. Each fix was grounded in evidence from the previous test run.

This systematic approach is visible in the diagnostic commands used throughout the session: checking container status with docker compose ps, examining logs with docker compose logs, testing connectivity with direct SSH commands, and verifying variable resolution by reading the Ansible role files. Each diagnostic step was chosen to narrow the search space and eliminate hypotheses.

The Most Impactful Fixes Are Often the Simplest

The final fix — reordering two tasks in a YAML file — was a change of perhaps three lines. But reaching that simple fix required understanding Docker networking, systemd boot sequencing, PAM authentication, Ansible variable resolution, the per-node database naming convention, and the Kuri daemon's startup behavior. The simplicity of the fix is inversely proportional to the depth of understanding required to identify it.

This is a pattern that appears repeatedly in complex systems: the most elegant solutions are often the simplest, but finding them requires deep knowledge of the system's architecture and behavior. The task ordering bug could have been "fixed" in many ways — by modifying the Kuri daemon to not connect to the database during init, by adding a retry loop, by hardcoding the database name — but the correct fix was to ensure the configuration was present before the command that needed it.

The Test Harness as a Living Artifact

One of the most interesting aspects of this session is how the test harness itself evolved through the debugging process. It was not a static artifact that was built once and then used unchanged. Instead, it was modified repeatedly as the debugging revealed gaps in its design.

The initial test harness had several assumptions that proved incorrect:

Conclusion

The arc of this session — from committing untested Ansible scripts, through building a comprehensive Docker test harness, to iteratively debugging six distinct issues — is a microcosm of infrastructure engineering. It demonstrates that writing deployment code is only half the work; the other half is validating that code against a realistic environment. The test harness caught bugs that would have caused production failures: missing dependencies, incorrect volume configurations, authentication timing issues, and task ordering errors. Each fix made the deployment scripts more robust, the test harness more reliable, and the overall system more trustworthy.

The final message in this chain — the task reordering fix — is not the end of the story. It is the point where the test harness had validated enough of the deployment pipeline that the remaining issues were genuine application logic problems rather than infrastructure scaffolding problems. The debugging marathon had succeeded in its purpose: it transformed untested code into validated infrastructure, one failure at a time.

The broader lesson is clear: infrastructure code must be tested with the same rigor as application code, and the test environment must mirror production as closely as possible. The Docker-based test harness described in this article is a template for how to achieve this validation — a laboratory where deployment scripts can be broken, debugged, and fixed before they ever touch a real server. In the world of infrastructure as code, this is not a luxury. It is a necessity.