From Commit to Validation: The Debugging Odyssey of an Ansible Deployment Test Harness

Introduction

Infrastructure as code is built on a seductive promise: that deployment scripts, once written, can reliably reproduce complex environments across any number of machines. But this promise contains a hidden trap — the code must actually work. Between a well-structured Ansible role and a successfully deployed cluster lies a chasm of untested assumptions, missing dependencies, and subtle ordering constraints that only execution can reveal. This article traces one such journey: the creation and iterative debugging of a Docker-based test harness for Ansible deployment scripts targeting a horizontally scalable S3 storage architecture for the Filecoin Gateway (FGW).

The arc of this chunk spans roughly 100 messages in a coding session, from the moment the Ansible scripts were committed to version control, through the construction of a comprehensive test environment, and into a debugging marathon that uncovered six distinct issues. The final fix — reordering two tasks in a Kuri Ansible role so that a configuration file is generated before the initialization command that depends on it — seems trivial in retrospect. But reaching that fix required understanding Docker networking, systemd boot sequencing, PAM authentication, Ansible variable resolution, and the internal architecture of the Kuri storage daemon. This is the hidden complexity of infrastructure testing, and this article unpacks it in full.

The Foundation: Committing the Ansible Deployment Scripts

The story begins with the completion of a substantial body of work. Across dozens of preceding messages, the assistant had built an entire Ansible deployment system for FGW clusters from scratch: seven roles (common, wallet, yugabyte_init, kuri, s3_frontend, and supporting infrastructure), five playbooks (site.yml, deploy-kuri.yml, deploy-frontend.yml, setup-yb.yml, verify.yml), a complete inventory structure with group_vars for production, and a detailed specification document [1][61]. In total, 30 files containing 1,708 lines of YAML, Jinja2 templates, and configuration were created.

But there was a critical gap: none of this code had ever been executed. The playbooks had never run against any target, the roles had never been applied, the templates had never been rendered. The assistant's summary at message 1486 included a "Usage" section showing commands like ansible-playbook playbooks/site.yml, but those commands were aspirational, not verified [84].

The user recognized this immediately. In message 1487, the instruction was concise and directive: "commit, create a docker-compose + bash/ansible harness for testing the ansible scripts" [54][85]. This was a demand for empirical validation. "Commit" meant preserve the work done so far. "Create a docker-compose + bash/ansible harness" meant build a laboratory where the untested deployment code could be exercised, broken, and fixed before it ever touched a real server.

The assistant's first action was to stage and commit the Ansible work [61]. This process itself revealed a subtle issue: the project's root .gitignore contained a pattern kuri that matched the directory name of the Kuri role, causing git add ansible/roles/kuri/ to be silently ignored [58][59]. The assistant had to use git add -f to force-add the directory. This interaction between past and present uses of the same name — kuri was originally a binary artifact, now also a directory — is now visible in the commit history as a piece of systems knowledge.

Building the Test Harness

With the Ansible scripts committed, the assistant turned to building the test harness [62][63]. The design was ambitious: a Docker Compose environment that simulated the full production topology. The docker-compose.yml defined five containers: a YugabyteDB database instance, two Kuri storage node targets (kuri-01, kuri-02), one S3 frontend proxy target (s3-fe-01), and an Ansible controller container [66].

The target containers ran Ubuntu 24.04 with systemd and SSH, built from a custom Dockerfile.target [64]. This was a deliberate choice: by using full systemd-based images, the test environment could validate the systemd service templates that the Ansible roles deployed. The controller container was based on python:3.11-slim and would execute the playbooks against the targets.

The test harness included three shell scripts: setup.sh (build binaries and start containers), run-tests.sh (execute playbooks in sequence and verify), and cleanup.sh (tear down the environment) [76][77]. A test inventory was created with host definitions and static IP addresses on a custom Docker network. Test wallet files were provided for authentication [72]. A README documented the test flow [79].

The test flow defined in run-tests.sh showed methodical progression: connectivity check → YugabyteDB initialization → deploy Kuri nodes → deploy S3 frontend → verify all components → idempotency check (run site.yml twice to ensure no errors on the second pass) [84]. This layered approach meant that each test would only execute if the previous one succeeded, naturally isolating failures to the layer being validated.

The assistant committed this test harness as a second git commit (message 1516) and delivered a clean summary of everything created [84]. The summary presented a well-structured directory tree and a simple three-step workflow: setup, run tests, cleanup. It conveyed confidence. The user's response was three words: "run the tests" [85].

The Debugging Marathon Begins

The first test run failed immediately. Message 1520 shows the output: ERROR: Test environment not running. Run ./setup.sh first. [87]. The setup script had built binaries and Docker images, but the containers were not in a running state when run-tests.sh attempted to execute. The assistant checked container status and found only YugabyteDB running; the target and controller containers needed to be started manually [88][89].

This initial failure triggered a cascade of discoveries. Each fix revealed the next assumption that had been made about the test environment, and each required a targeted correction before the tests could progress further.

Issue 1: The YugabyteDB Health Check Hostname

The first technical bug was subtle. The YugabyteDB container's health check in docker-compose.yml used ysqlsh -h localhost, but the YugabyteDB server was configured with --advertise_address=yugabyte, causing it to bind to its container hostname rather than localhost [90][91][94][95]. The health check would fail because it was checking the wrong address. The fix was to change the health check to use -h yugabyte.

This is a classic container networking issue: services inside containers may not be available on localhost if they explicitly advertise on a different address. The assistant had assumed that localhost would work because the database process was running in the same container, but YugabyteDB's startup behavior overrode that assumption.

Issue 2: Missing Tools in the Controller Container

When the test suite progressed to the YugabyteDB initialization playbook, it failed because the Ansible controller container lacked the command-line tools needed to execute database setup tasks [119]. The controller was based on a minimal Python image and did not include psql (PostgreSQL client), cqlsh (Cassandra/CQL client), or sshpass (for SSH password authentication).

The assistant diagnosed this by examining the playbook's tasks and identifying which tools were required [119][120]. The fix was to install postgresql-client via apt, cqlsh via pip, and sshpass as a system package (not a pip package, which was a correction from an earlier attempt) [104][105]. This ensured the controller had all necessary dependencies to execute the database initialization and SSH-based playbooks.

Issue 3: Missing Group Variables in the Test Inventory

The Kuri deployment playbook failed with an undefined variable error: fgw_config_dir was not defined [121][122]. The assistant traced the variable's origin through Ansible's resolution chain: fgw_config_dir was defined in group_vars/kuri.yml in the production inventory, but the test inventory — a separate directory structure created specifically for the Docker harness — did not have this file.

The fix was straightforward: copy the production group_vars/kuri.yml and group_vars/s3_frontend.yml files into the test inventory's group_vars directory [122][123]. Two cp commands resolved an entire class of failures. This moment illustrates a critical principle of infrastructure testing: test environments must mirror production faithfully. The most common cause of test failures that don't occur in production is environmental differences, and missing group_vars files are exactly this kind of structural gap.

Issue 4: Read-Only Volume Mounts

The next failure was more perplexing. The Ansible playbook attempted to deploy binaries to /opt/fgw/bin/ on the target hosts, but the directory was empty [125]. When the assistant tried to copy binaries manually using docker cp, the error was unambiguous: mounted volume is marked read-only [126][127].

The Docker Compose file had configured the binaries:/opt/fgw/bin volume with :ro (read-only). This was an understandable choice — read-only mounts are a security best practice — but it conflicted with the Ansible deployment model, which required write access to set permissions and ownership. The fix was to edit the Docker Compose file to remove the :ro flag and then recreate the containers with docker compose down && docker compose up -d [128].

This tension between security and functionality is a classic infrastructure pitfall. The read-only mount was intended to protect binaries from accidental modification, but it inadvertently prevented the very deployment workflow the test harness was designed to validate.

Issue 5: The pam_nologin Wall

Perhaps the most persistent issue was the pam_nologin mechanism [131][133][134]. After container restart, the Ansible controller's SSH connections to the target hosts were rejected with: "System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8)."

When a systemd-based Docker container boots, it creates a file called /run/nologin. The PAM (Pluggable Authentication Modules) framework checks for this file, and if it exists, the pam_nologin module blocks all non-root login attempts. This is a security feature designed to prevent users from logging in while the system is still initializing services. In a production server, this makes sense. But in a Docker test container, it becomes an obstacle: the containers are technically "booting" every time they start, and the Ansible controller tries to connect via SSH before systemd has finished its initialization.

The assistant tried multiple approaches: waiting longer, manually removing /run/nologin, checking systemctl is-system-running [132][134]. The reliable fix was to use systemctl is-system-running --wait on each target host, which blocks until systemd reports that the system is fully operational. This ensured that SSH connections were only attempted after the boot sequence was complete.

Issue 6: The Task Ordering Problem

With all infrastructure issues resolved, the test suite finally progressed to deploying Kuri storage nodes. And then it hit the most architecturally significant bug of the entire session [136][137].

The error message was: database "filecoingw" does not exist. But the YugabyteDB initialization playbook had created a database named filecoingw_kuri_01 — a per-node keyspace naming scheme implemented earlier in the project. Something was misaligned between the database initialization and the Kuri node's configuration.

The assistant's diagnosis was precise [136]. The kuri init command, when executed, starts the full Kuri server process, which immediately attempts to connect to its database. The database connection parameters — host, port, database name, credentials — are stored in a file called settings.env. But the Ansible role was generating settings.env after running kuri init. The init command therefore had no database configuration and used defaults, connecting to a database named filecoingw that didn't exist.

In production, this would work because settings.env would already be present — either from a previous deployment or from manual setup. But in the test environment, the role was being applied from scratch, and the task ordering exposed the hidden dependency.

The fix was to reorder the tasks in the Kuri role's tasks/main.yml so that settings.env is generated first, and then kuri init is run with the environment properly configured [137]. The assistant applied this edit in message 1571, and the edit was confirmed successful.

The Broader Significance

This debugging marathon, spanning six distinct issues across dozens of messages, reveals several fundamental truths about infrastructure automation.

First, deployment code follows a different quality regime than application code. Application code can be unit-tested, integration-tested, and staged in environments that approximate production. Deployment code often lacks these safeguards because the infrastructure to test it is itself part of what needs testing. The Docker-based test harness was a solution to this bootstrapping problem: it simulated the target environment closely enough to validate the playbooks, while being cheap enough to destroy and recreate on every iteration.

Second, the gap between "code that looks correct" and "code that works" is wide. The Ansible roles were carefully written, with proper variable definitions, template files, and task organization. Yet they contained multiple defects that only execution could reveal: the missing group_vars, the read-only volume conflict, the task ordering dependency. The user's demand for a test harness was not pedantic — it was essential. Without it, the deployment scripts would have failed in production, likely at the worst possible moment.

Third, systematic debugging is a skill that can be observed and learned. The assistant's approach across all six issues followed a consistent pattern: observe the failure, parse the error message, trace the dependency chain to its root cause, formulate a hypothesis, apply a targeted fix, and retry. The assistant did not make random changes or guess at solutions. Each fix was grounded in evidence from the previous test run.

Fourth, the most impactful fixes are often the simplest. The final fix — reordering two tasks in a YAML file — was a change of perhaps three lines. But reaching that simple fix required understanding Docker networking, systemd boot sequencing, PAM authentication, Ansible variable resolution, the per-node database naming convention, and the Kuri daemon's startup behavior. The simplicity of the fix is inversely proportional to the depth of understanding required to identify it.

Conclusion

The arc of this chunk — from committing untested Ansible scripts, through building a comprehensive Docker test harness, to iteratively debugging six distinct issues — is a microcosm of infrastructure engineering. It demonstrates that writing deployment code is only half the work; the other half is validating that code against a realistic environment. The test harness caught bugs that would have caused production failures: missing dependencies, incorrect volume configurations, authentication timing issues, and task ordering errors. Each fix made the deployment scripts more robust, the test harness more reliable, and the overall system more trustworthy.

The final message in this chain — the task reordering fix — is not the end of the story. It is the point where the test harness had validated enough of the deployment pipeline that the remaining issues were genuine application logic problems rather than infrastructure scaffolding problems. The debugging marathon had succeeded in its purpose: it transformed untested code into validated infrastructure, one failure at a time.## References

[1] "The Session Summary as Engineering Artifact: How One Message Captured an Entire Debugging Odyssey" — article on message 1434

[54] "The Test Harness Imperative: Validating Infrastructure Code Through Docker-Based Simulation" — article on message 1487

[58] "The .gitignore That Swallowed a Role" — article on message 1491

[59] "The Hidden Trap in .gitignore: When a Simple Pattern Silently Blocks Your Commit" — article on message 1492

[61] "The Commit That Cemented Infrastructure: Ansible Deployment for FGW Clusters" — article on message 1494

[62] "The Pivot Point: From Writing Ansible Scripts to Building a Test Harness" — article on message 1495

[63] "The First Brick: How a Single mkdir Command Launched an Ansible Test Harness" — article on message 1496

[64] "The Humble Dockerfile That Anchored an Entire Deployment Pipeline" — article on message 1497

[66] "The Pivotal Orchestration File: Writing docker-compose.yml for an Ansible Test Harness" — article on message 1499

[72] "The Test Wallet File: A Small File with Big Implications" — article on message 1505

[76] "The Final Piece: Why a Cleanup Script Matters in Infrastructure Testing" — article on message 1509

[77] "The Quiet Gatekeeper: Why chmod +x ansible/test/*.sh Matters More Than It Seems" — article on message 1510

[79] "The Documentation That Binds: How a README.md Anchored an Ansible Test Harness" — article on message 1512

[84] "The Summary That Preceded the Debugging Storm: Analyzing an Ansible Deployment Report" — article on message 1517

[85] "run the tests: The Three Words That Triggered a Debugging Marathon" — article on message 1518

[87] "The Error That Launched a Debugging Odyssey: A Single Bash Command's Hidden Depths" — article on message 1520

[88] "The Diagnostic Pivot: Reading Container Status in a Failing Test Harness" — article on message 1521

[89] "The Moment Between Failure and Fix: A Docker Diagnostic in an Ansible Test Harness" — article on message 1522

[90] "The Moment Before Insight: A Health Check Failure in the Debugging Process" — article on message 1523

[91] "The Moment of Discovery: Debugging a Database Connection in a Docker Test Harness" — article on message 1524

[94] "The Hostname That Wasn't There: A Debugging Breakthrough in Docker Networking" — article on message 1527

[95] "The Moment of Discovery: Fixing a Docker Health Check Hostname" — article on message 1528

[104] "The sshpass Correction: A Moment of Clarity in Infrastructure Debugging" — article on message 1538

[105] "The Quiet Glue: Installing Ansible in a Test Container" — article on message 1539

[119] "The Dependency Diagnosis: Installing Database Tools in an Ansible Test Harness" — article on message 1553

[120] "The Moment Connectivity Clicked: Debugging an Ansible Deployment Test Harness" — article on message 1554

[121] "The Missing Variable: Debugging Ansible Role Failures in a Distributed S3 Deployment" — article on message 1555

[122] "The Missing Link: How One File Copy Fixed an Ansible Deployment Pipeline" — article on message 1556

[123] "The Critical Copy: How a Single File Sync Unlocked Ansible Deployment Testing" — article on message 1557

[125] "The Empty Directory: A Pivotal Debugging Discovery in Ansible Deployment Automation" — article on message 1559

[126] "When Read-Only Volumes Block Deployment: Debugging Ansible Test Infrastructure" — article on message 1560

[127] "The Read-Only Volume That Broke the Deployment" — article on message 1561

[128] "The Read-Only Barrier: A Docker Compose Restart That Unblocked Ansible Deployment" — article on message 1562

[131] "The pam_nologin Wall: When Systemd Blocks SSH in Docker Containers" — article on message 1565

[132] "The Systemd Boot Race: Debugging Container SSH Failures in Ansible Test Infrastructure" — article on message 1566

[133] "The pam_nologin Wall: When Systemd Says 'Running' But SSH Says 'Not Yet'" — article on message 1567

[134] "The pam_nologin Problem: When 'System Is Running' Doesn't Mean 'Ready for SSH'" — article on message 1568

[136] "The Moment of Diagnosis: Debugging Task Order in Ansible Deployment" — article on message 1570

[137] "The Critical Task Ordering Fix: When kuri init Needed Its Environment Before It Could Run" — article on message 1571