The Pipeline That Wouldn't Deploy: Debugging a Cascade of Ansible Failures

Introduction

There is a particular kind of frustration unique to infrastructure-as-code development: the moment when a deployment pipeline that should work—every role is written, every template is rendered, every playbook is composed—fails not with a single clear error but with a cascade of seemingly unrelated problems, each one revealed only after the previous is fixed. This is the story of such a cascade. Over the course of a concentrated debugging session spanning more than a hundred messages, a developer working through an AI assistant systematically diagnosed and resolved seven distinct categories of failures in an Ansible-based deployment pipeline for Filecoin Gateway (FGW) clusters.

The bugs ranged from the trivial (a missing curl binary in a test container) to the architecturally significant (duplicate database migrations between infrastructure provisioning and application initialization). Each fix was a small victory; together, they transformed a broken, unreliable deployment process into a validated, repeatable pipeline that could deploy two Kuri storage nodes and an S3 frontend proxy onto target hosts with a single command. The session culminated in commit 806c370, encompassing 19 file changes with 258 insertions and 169 deletions—a permanent record of the transformation from chaos to order.

This article synthesizes that debugging arc. It traces the narrative of the session, examines each failure in its context, identifies the patterns in the debugging methodology, and extracts the lessons that apply beyond this specific project. The goal is to understand not just what went wrong and how it was fixed, but why these bugs emerged and what the debugging process reveals about the nature of building reliable infrastructure-as-code.

The System Under Deployment

To understand the debugging session, one must first understand the system being deployed. The Filecoin Gateway (FGW) is a horizontally scalable S3-compatible storage architecture built on three layers:

  1. S3 Frontend Proxies: Stateless HTTP servers that accept S3 API requests and route them to backend storage nodes. These are the entry point for clients and provide load balancing and request distribution.
  2. Kuri Storage Nodes: The core storage engine, responsible for managing IPFS-based data storage, Filecoin deal integration, and the RIBS (Remote Indexed Block Store) layer. Each Kuri node maintains its own wallet for Filecoin transactions and connects to a shared YugabyteDB database for metadata.
  3. YugabyteDB: A distributed SQL database that provides the metadata layer, storing information about stored objects, deals, and cluster state. The deployment pipeline, written in Ansible, needed to initialize the database, deploy Kuri nodes with their configuration and wallets, and deploy the S3 frontend proxy with knowledge of the backend nodes. The test harness used Docker containers running systemd as PID 1, simulating real Ubuntu servers with SSH access, to validate the Ansible playbooks before production use. The Ansible roles were organized with a clear separation of concerns: a common role for base system setup, a wallet role for secure key distribution, a yugabyte_init role for database provisioning, a kuri role for storage node deployment, and an s3_frontend role for proxy deployment. Playbooks like site.yml, deploy-kuri.yml, deploy-frontend.yml, setup-yb.yml, and verify.yml orchestrated these roles in the correct order. On paper, the architecture was sound. In practice, the gap between the design and the running system was filled with bugs.

The First Wall: Systemd Rejects Shell Syntax

The first major failure was also the most fundamental. The Kuri service was failing to start because systemd's EnvironmentFile directive silently rejected every line in the configuration file that began with export. The Jinja2 template (settings.env.j2) had been written using shell-style variable assignments:

export FGW_NODE_ID="kuri-01"
export FGW_LOG_LEVEL="*:*=debug"

This syntax works perfectly when a file is sourced by a shell (source settings.env or . settings.env). But systemd's EnvironmentFile parser does not invoke a shell—it reads the file directly and expects plain KEY=VALUE format. Every line with export was silently ignored, leaving the Kuri daemon with an empty environment. The daemon started, found no configuration, and crashed within 63 milliseconds. [6][7][8][9]

The error was insidious because systemd did not fail loudly. It logged a warning—"Ignoring invalid environment assignment"—and continued. The symptom looked like a code crash or network issue, but the root cause was a single word on each line of a template file. The developer had to inspect the systemd unit status, read the journal logs, and trace the failure back to the environment file to discover the problem. [6]

The fix was straightforward: remove the export prefix from the template. But the fix revealed a deeper tension: the same configuration file needed to serve two masters. The kuri init command sourced the file through a shell (which needed export), while systemd's EnvironmentFile parsed it directly (which rejected export). The resolution required generating the file without export and having the init command source it with . settings.env rather than relying on the file's own syntax. [9][10]

This bug is a classic example of a format boundary failure. Every time a configuration file crosses a system boundary—from developer workstation to deployment tool, from deployment tool to runtime environment, from one runtime to another—the format may be reinterpreted. The developer assumed that because the file worked when sourced by a shell, it would work everywhere. Systemd proved otherwise.

The Log Level Regex Trap

With the environment file fixed, the next failure appeared quickly. The log level configuration was invalid. The template used *:*=debug as a pattern, but the application's logging framework expected Java-style regex syntax: .*:.*=debug. The difference between a shell glob (*) and a regex (.*) caused the entire log level setting to be rejected. [23][24][25][26]

This bug is a subtle variation on the format boundary theme. The developer had written the log level pattern using the most natural wildcard syntax—the asterisk that everyone learns as "match anything" in file globbing. But the logging library, written in a language with a Java heritage, expected a different convention. The fix required updating both the test inventory (test-inventory/group_vars/all.yml) and the production defaults (inventory/production/group_vars/all.yml) to use the correct regex format. [27][28]

What makes this bug notable is that it was entirely invisible during development. The configuration file was syntactically valid. The Ansible template rendered without errors. The file was copied to the target host successfully. The failure only manifested at runtime when the application tried to parse the log level setting and silently fell back to a default. The developer had to inspect the application's startup logs to discover that the log level was not being applied.

The Wallet Dotfile Surprise

The wallet directory in the Ansible source tree contained a .gitkeep file—a common Git convention for preserving empty directories in version control. When the wallet role copied this directory to the target host using Ansible's copy module, it included the hidden file. The Kuri binary, during initialization, iterated over all files in the wallet directory and attempted to parse each one as a cryptographic key. The .gitkeep file was not a key, so parsing failed with a binary error. [17][18][19][20][21][22]

This bug is a boundary failure at the intersection of two systems with different assumptions. Git's convention treats .gitkeep as invisible metadata—a file that exists only to preserve the directory in version control, not as application data. Kuri's initialization logic treats every file in the wallet directory as application data—a cryptographic key that must be parsed. Neither system is wrong, but their interaction through the deployment pipeline creates a failure.

The fix evolved through several iterations, each revealing more about the system. The first attempt was to delete .gitkeep after copying—a point fix that addressed only one specific file. [22] The second attempt was to exclude all dotfiles from the wallet copy operation using a fileglob pattern that skipped hidden files. [41][42][43] The final approach was to leave the test wallet directory empty and let Kuri generate its own wallet during initialization, eliminating the need for pre-populated wallet files entirely. [39][40]

This progression from point fix to category fix to architectural fix is a pattern that appears throughout the session. The developer did not stop at the first working solution but continued to refine the approach until it addressed the root cause rather than the symptom.

The pam_nologin Wall

Perhaps the most frustrating bug in the session was the pam_nologin issue. The Docker test containers, running systemd as PID 1, created /run/nologin and /var/run/nologin files during the boot process. The PAM (Pluggable Authentication Modules) framework checks for these files and rejects SSH login attempts from non-root users while they exist. Since Ansible connected to the target hosts as the fgw user (an unprivileged account), every SSH connection was rejected with: "System is booting up. Unprivileged users are not permitted to log in yet." [54][55][56][57][58]

The assistant initially tried to prevent the nologin file from being created by removing the systemd-user-sessions.service from the Docker image. This did not work—the file is created by a different mechanism (PAM itself, not the systemd service). After multiple rebuild cycles and container recreations, the assistant pivoted to a workaround: manually removing the nologin files after container startup and adding a wait loop in the setup script to ensure SSH was available before running Ansible. [70][71][72][73][74][75][76][77][78]

This bug is a powerful reminder that "running" does not mean "ready." A container whose main process (systemd) has started is not necessarily in a state where it can accept SSH connections. The gap between process startup and system readiness is where many automation failures hide. The fix—a wait loop with a timeout—is a pragmatic acknowledgment that readiness must be explicitly checked, not assumed.

Duplicate Database Migrations

The yugabyte_init Ansible role was designed to initialize the YugabyteDB database by creating keyspaces and tables. However, the Kuri application also ran its own database migrations on startup, using CREATE TABLE statements (not CREATE TABLE IF NOT EXISTS). When both paths tried to create the same tables, the second attempt failed. [64][65][66][67]

The assistant identified two possible fixes: (1) stop creating tables in the yugabyte_init role and let Kuri handle all schema management, or (2) modify Kuri's migration code to use CREATE TABLE IF NOT EXISTS. The assistant chose option 1, which was faster to implement and established a cleaner separation of concerns: infrastructure roles handle infrastructure resources (keyspaces, users, connection configuration), while the application manages its own schema. [64]

This decision has lasting architectural implications. It means that schema changes in future versions of Kuri will be handled by the application's migration logic, not by Ansible playbooks. The deployment system is responsible for creating the database container (keyspace), and the application is responsible for filling it (tables). This separation of concerns reduces the coupling between the deployment pipeline and the application's internal schema, making both easier to evolve independently.

The Phantom Ansible Filter

The s3_frontend role used a custom Jinja2 filter called format_backend_url that did not exist in the Ansible environment. This caused the playbook to fail during template rendering. The filter was presumably intended to transform backend node names (like kuri-01) into formatted HTTP URLs (like http://kuri-01:8079), but it was never implemented. [83][84][85]

The assistant's fix was to eliminate the filter dependency entirely by building the backend URL list directly in the Jinja2 template. This was a pragmatic choice: implementing a custom filter plugin would require creating a Python module, registering it with Ansible, and ensuring it was available in all environments. Removing the dependency was simpler and less error-prone. [83]

This bug highlights a common pitfall in Ansible development: it is easy to write a template that references a custom filter or module during the design phase, but the implementation of that filter is a separate task that can be forgotten. The gap between the template and the filter is invisible until the playbook is actually executed. The safest approach is to minimize dependencies on custom extensions and to implement any required logic using standard Jinja2 constructs.

The Test Harness Gaps

After all the major bugs were fixed, the final verification step—a curl command against the S3 frontend's health endpoint—failed because curl was not installed in the ansible-controller container. [90] This is a trivial issue compared to the others, but it illustrates a recurring theme: the test harness itself had gaps that needed to be filled. Earlier in the session, the assistant had to install psql and cqlsh in the controller container to enable database verification. [60][61][62] The test harness was being built and validated alongside the deployment pipeline, and each missing tool was a discovery.

The test harness was not just a convenience—it was essential to the debugging process. Without it, each bug would have required deploying to real infrastructure, waiting for failures, and manually inspecting remote systems. The test harness allowed the assistant to:

The Commit as Artifact

The session culminated in commit 806c370, which contained 19 file changes with 258 insertions and 169 deletions. [92][94][96][97] The commit touched nearly every Ansible role and playbook: inventory configuration, the kuri role (tasks and templates), the s3_frontend role (tasks and templates), the wallet role, the yugabyte_init role, the setup playbook, and the test harness infrastructure.

The commit message, written in message 1669, listed each fix with its rationale:

ansible: fix issues found during test execution

- Fix settings.env format: remove 'export' prefix as systemd EnvironmentFile
  requires plain KEY=VALUE format
- Fix log level format: use '.*:.*=level' regex syntax instead of '*:*'
- Fix wallet role: remove .gitkeep and other dotfiles after copy, exclude
  hidden files from verification
- Remove table creation from yugabyte_init role: kuri handles CQL migrations
  itself, duplicate CREATE TABLE statements caused failures
- Fix s3_frontend role: remove broken format_backend_url filter, build
  backend list directly in template
- Update test harness:
  - Add nologin removal after SSH ready (systemd containers create it)
  - Rebuild Docker images with disabled systemd-user-sessions service
  - Make wallet directory properly empty (kuri creates wallet on init)
  - Install psql and cqlsh in ansible controller

This commit message is a model of clarity. Each bullet point identifies a problem, explains the fix, and provides enough context for a future reader to understand why the change was made. The assistant's careful handling of the commit—unstaging the s3-proxy binary artifact, reviewing the diff, writing a descriptive message—reflects an understanding that commits are communication artifacts, not just snapshots. [94][95]

The commit hash 806c370 serves as a permanent anchor point: the state of the system when the deployment pipeline was first validated. Any future developer can check out this commit and run the test suite to reproduce the working state. If a regression is introduced later, this commit provides a known-good baseline for comparison.

Assumptions Made and Broken

The debugging session is a catalog of incorrect assumptions, each of which taught a lesson about the system:

Assumption: systemd's EnvironmentFile accepts shell syntax. Broken. Systemd parses environment files directly, not through a shell, and rejects export prefixes. [6][7]

Assumption: Log level patterns use glob syntax. Broken. The application's logging framework uses Java-style regex syntax (.*:.*), not shell globs (*:*). [23][24]

Assumption: Hidden files in source directories are harmless. Broken. The .gitkeep file in the wallet directory was copied to the target and caused binary parsing errors. [17][18]

Assumption: Disabling systemd-user-sessions.service prevents nologin creation. Broken. The nologin file is created by PAM, not by that specific systemd service. [54][55]

Assumption: Infrastructure roles should create database tables. Broken. The application (Kuri) also runs migrations, causing duplicate CREATE TABLE failures. The correct separation is infrastructure creates keyspaces, application creates tables. [64]

Assumption: Custom Ansible filters exist if referenced. Broken. The format_backend_url filter was referenced in the role but never implemented. [83]

Assumption: The test controller container has common CLI tools. Broken. curl, psql, and cqlsh were all missing at various points. [60][61][90]

Each broken assumption represents a learning opportunity. The assistant's willingness to update its mental model of the system—rather than persisting with incorrect theories—is a hallmark of effective debugging. The assumptions were not unreasonable; they were based on prior experience with similar systems. But each system has its quirks, and the debugging process is the mechanism by which those quirks are discovered and accommodated.

Lessons for Infrastructure-as-Code

Several general lessons emerge from this session that apply beyond the FGW project:

1. Test assumptions about configuration format boundaries

Every time a configuration file crosses a system boundary—from developer workstation to deployment tool, from deployment tool to runtime environment, from one runtime to another—the format may be reinterpreted. The export prefix bug and the log level regex bug are both examples of format boundary failures. The lesson is to verify that configuration files are in the format expected by the consuming system, not just the format that works for the producing system.

2. "Running" is not "ready"

The pam_nologin bug is a vivid reminder that a process starting does not mean the system is ready to accept connections. Automation must account for initialization time, boot sequences, and service readiness. The fix—a wait loop with explicit readiness checks—is a pattern that should be applied broadly in deployment automation.

3. Hidden files are not invisible to deployment tools

Files like .gitkeep, .gitignore, and .DS_Store are invisible to developers but very visible to cp, rsync, and Ansible's copy module. Deployment roles must explicitly exclude non-application files, or better yet, keep source directories clean of such artifacts.

4. Separation of concerns applies to database initialization

Infrastructure provisioning should create infrastructure resources (keyspaces, users, network policies). Application deployment should manage schema and migrations. When both layers try to create the same tables, conflicts are inevitable. The clean separation established in this session—keyspaces in Ansible, tables in Kuri migrations—is a pattern worth replicating.

5. Custom filters and plugins are dependencies

Referencing a custom Ansible filter or module creates a dependency that must be satisfied in every environment. Before using a custom filter, consider whether the logic can be expressed in standard Jinja2 or Ansible constructs. If a custom extension is necessary, it should be implemented, tested, and documented before the template that references it is committed.

6. Iterative debugging requires a fast feedback loop

The Docker test harness was essential because it allowed the assistant to fix, rebuild, and retest in minutes. Without this rapid cycle, the debugging session would have taken much longer and been much more frustrating. Investing in a fast test environment is not a luxury—it is a productivity multiplier that pays for itself many times over during debugging.

From Debugging to Roadmap

The final messages of the session (1671–1674) mark a transition from debugging to planning. The assistant's summary message enumerates the passing tests and key fixes, providing closure on the debugging phase. [99] The user's response redirects effort toward the next milestones: Enterprise Grade features (metrics, monitoring, backup/restore, documentation, support AI agent), Persistent Retrieval Caches (per-node prefetch daemon), and Data Lifecycle management (garbage collection, deal extension, repair). [100]

This transition is significant. It acknowledges that the deployment pipeline is stable enough to build upon. The foundation is solid; now the team can add the features that make the system production-ready. The debugging session was not an end in itself—it was the necessary precondition for the work that follows.

The roadmap for Milestone 02 (Enterprise Grade) includes integrating Prometheus metrics into Kuri and s3-proxy, exposing /metrics endpoints, setting up Grafana dashboards, centralizing logs via Fluentd or Loki, implementing structured logging with correlation IDs, scripting YSQL/YCQL backups, writing deployment documentation, and building a RAG-based support AI agent. Milestone 03 (Persistent Retrieval Caches) involves designing a per-node prefetch daemon that predicts popular content and pulls it into a local cache. Milestone 04 (Data Lifecycle) covers garbage collection on Filecoin, automated deal extension, and data integrity repair.

Each of these milestones builds on the operational baseline established by the debugging session. Without a reliable deployment pipeline, adding monitoring, caching, and lifecycle management would be building on sand. With the pipeline validated, the team can proceed with confidence.

Conclusion

The debugging session documented in messages 1573–1674 is a microcosm of infrastructure-as-code development. It contains all the elements that make this work challenging and rewarding: subtle configuration mismatches, hidden assumptions about system behavior, the interplay between multiple layers of tooling, and the satisfaction of watching a complex system finally work as designed.

The seven categories of bugs—environment file syntax, log level format, wallet dotfiles, PAM nologin, duplicate migrations, missing Ansible filter, and test harness gaps—are not unique to this project. They are archetypes of infrastructure failures that appear in every deployment system. The specific manifestations differ, but the underlying patterns are universal.

What made this session successful was not the absence of bugs—bugs were abundant—but the methodology for finding and fixing them. The assistant's disciplined approach of observing, hypothesizing, investigating, fixing, and retesting, combined with a willingness to abandon incorrect theories and pivot to new approaches, turned a cascade of failures into a validated pipeline. The commit 806c370 is the permanent record of that transformation, but the real output is the knowledge embedded in the debugging process itself: knowledge about systemd's quirks, about PAM's boot-time behavior, about the separation of database concerns, and about the importance of testing assumptions at every boundary.

In infrastructure engineering, the goal is not to write bug-free code on the first attempt. The goal is to build a system where bugs can be found quickly, fixed safely, and prevented from recurring. The FGW debugging session is a case study in how to achieve that goal through iterative, evidence-based debugging. The pipeline that wouldn't deploy became the pipeline that deploys every time—not because the bugs were absent, but because each one was found, understood, and eliminated.

References