The Moment of Revelation: How One Ansible Playbook Run Exposed Three Hidden Bugs

In the middle of an intense debugging session for a distributed Filecoin Gateway (FGW) cluster deployment, one seemingly routine command execution became the crucible in which multiple subtle bugs were simultaneously exposed. The message at index 1588 in this coding session is a deceptively simple bash invocation: running an Ansible playbook against a single Kuri storage node in a Docker-based test harness. But this single execution surface — and the output it produced — served as the diagnostic linchpin that revealed three distinct configuration and deployment errors that had been silently lurking in the Ansible roles.

The Message in Full

The message reads:

[assistant] [bash] cd /home/theuser/gw/ansible/test/docker && docker compose exec -T ansible-controller bash -c "cd /ansible && ansible-playbook playbooks/deploy-kuri.yml -i /ansible/inventory/test/hosts.yml -l kuri-01 -v" 2>&1 | tail -200
    path: /home/fgw/.ribswallet/default
    pw_name: fgw
    rgrp: false
    roth: false
    rusr: true
    size: 33
    uid: 1001
    wgrp: false
    woth: false
    wusr: true
    xgrp: false
    xoth: false
    xusr: false
  - atime: 1769881677.6178234
    ctime: 1769881678.2098317
    dev: 75
    gid: 1001
    gr_name: fgw
    inode: 26992505
    isblk: false
    ischr: false
    isdir: false
    isfifo: false
    isgid: false
    islnk: false
    isreg: true
    issock: false
    isuid: fal...

At first glance, this appears to be nothing more than the tail end of an Ansible playbook execution, showing verbose file stat information about wallet files that were deployed to the target node. The output is truncated (the tail -200 pipeline captures only the last 200 lines, and the conversation data shows even less), but what's visible is a detailed stat result for a file at /home/fgw/.ribswallet/default — a wallet key file owned by the fgw user, 33 bytes in size, with specific permission bits set.

Why This Message Was Written

To understand why this message exists, we must trace the debugging journey that led to it. The assistant had been building and testing Ansible deployment scripts for a horizontally scalable S3-compatible storage architecture built on Filecoin. The architecture consists of three layers: stateless S3 frontend proxies, Kuri storage nodes (which handle IPFS and RIBS storage), and a shared YugabyteDB database.

The test harness — a Docker Compose environment with systemd-based target containers — had been failing repeatedly. Earlier in the session, the assistant discovered that the settings.env template contained export prefixes on every variable assignment. This is a critical incompatibility: systemd's EnvironmentFile directive expects plain KEY=VALUE format and silently ignores lines prefixed with export. The result was that the Kuri daemon started without any of its configuration variables — no IPFS_PATH, no database connection strings, no log level settings — causing it to look in the default home directory (/home/fgw/.ipfs) instead of the intended data directory (/data/fgw/kuri-01/ipfs).

The assistant had just fixed this issue by:

Rewriting the settings.env.j2 template to remove all export prefixes (message 1580)
Updating the kuri init task to explicitly use export when sourcing the file in a shell context (message 1582)
Resetting the IPFS state on both test nodes to ensure a clean retry (message 1583)
Copying the updated roles into the Ansible controller container (message 1584) Then came the failed attempt at message 1585, where the assistant ran the playbook with an incorrect inventory path (/ansible/inventory/hosts.yml instead of /ansible/inventory/test/hosts.yml). This caused Ansible to fail with "No inventory was parsed" warnings. The assistant corrected this in message 1587 by discovering the correct path through exploration. Message 1588 is therefore the first successful execution of the playbook with the correct inventory path after the export prefix fix. It represents a critical verification step — the moment when the assistant could finally see whether the core fix worked, and what other issues might surface.## The Contextual Backdrop: An Iterative Debugging Marathon This message did not occur in isolation. It was the product of an extended debugging session spanning multiple segments of work on the FGW Ansible deployment system. To appreciate its significance, one must understand the chain of failures that preceded it. The session had already uncovered and fixed several significant issues: The export prefix problem. The settings.env.j2 Jinja2 template was generating lines like export FGW_NODE_ID="kuri-01". While this is valid shell syntax for interactive use, systemd's EnvironmentFile directive does not support it. The export keyword is silently ignored, meaning every environment variable was effectively missing. This caused the Kuri daemon to fail immediately because it couldn't find its data directory. The pam_nologin blocking SSH. The Docker target containers were based on Ubuntu 24.04 with systemd, and during early boot, the pam_nologin module prevents non-root SSH logins. The Ansible controller, which connects as the fgw user, was consistently rejected with "System is booting up. Unprivileged users are not permitted to log in yet." The assistant had to manually remove /run/nologin files and later fixed the Dockerfile to disable this behavior. The database name mismatch. The kuri init command was trying to connect to a database named filecoingw (the default), but the YugabyteDB initialization role had created per-node databases named filecoingw_kuri_01 and filecoingw_kuri_02. The assistant had to reorder the tasks so that settings.env (which contains the correct database name) was generated before kuri init ran. The inventory path confusion. In message 1585, the assistant ran the playbook with -i /ansible/inventory/hosts.yml, which doesn't exist — the test inventory is at /ansible/inventory/test/hosts.yml. This is a simple but frustrating mistake that cost time and cognitive energy. Message 1588 was the payoff after all these fixes: the first clean invocation of the playbook with the correct inventory, the corrected template, and the properly ordered tasks. The output shown — a verbose file stat — indicates that the playbook progressed far enough to deploy wallet files and check their properties. This is significant progress compared to the earlier failures where the playbook couldn't even connect to hosts or where the Kuri daemon crashed immediately after startup.

What the Output Reveals: Three Bugs in One Snapshot

The truncated output visible in message 1588 shows an Ansible stat result for the wallet file /home/fgw/.ribswallet/default. The stat details include ownership (uid: 1001, gid: 1001, gr_name: fgw), size (size: 33), and permission flags. On its surface, this looks like a successful file deployment — the wallet was copied, permissions were set, and Ansible is verifying the result.

However, the messages that immediately follow (1589 onward) reveal that this execution surface exposed three distinct bugs that the assistant had to diagnose and fix:

Bug 1: Invalid Log Level Format

The first clue came from the Kuri daemon's error logs, visible in message 1589: "invalid log level: debug." The assistant had configured RIBS_LOGLEVEL: debug in the test inventory's group_vars/all.yml, but the application expected a more specific format like ribs:.*=debug,gw/.*=debug,ribs:rbdeal=info — a pattern-matching syntax for hierarchical loggers. The simple value debug was rejected outright.

This is a classic configuration pitfall: the developer (or in this case, the person writing the Ansible inventory) assumed a simple log level string would work, but the application's configuration parser required a structured format. The fix required looking at the actual Go source code (configuration/config.go) to understand what format the envconfig library expected.

Bug 2: The `.gitkeep` Wallet Contamination

The second issue was more subtle. The wallet directory in the Ansible source tree contained a .gitkeep file — a common convention for ensuring empty directories are tracked by Git. When the Ansible wallet role copied the entire directory to the target node, it included this hidden file. The Kuri binary, when initializing, tried to parse every file in the wallet directory as a key. The .gitkeep file, containing 218 bytes of non-key data, caused a binary parsing error.

This is a fascinating class of bug: a file that is semantically meaningless (a Git placeholder) but syntactically present in a directory that the application treats as homogeneous. The fix required either excluding .gitkeep from the copy operation (using the synchronize module with exclude patterns) or deleting it after copying. The assistant chose the latter approach — a pragmatic, if slightly brute-force, solution.

Bug 3: The Duplicate Table Creation

A third issue, visible in the broader context of the session (though not directly in message 1588's output), was that both the yugabyte_init role and the kuri init command were trying to create the same CQL tables. The yugabyte_init role created the S3 metadata tables during database initialization, but kuri init also attempted to create them as part of its startup routine. This caused duplicate table creation errors that were non-fatal but messy.

The assistant resolved this by removing table creation from the yugabyte_init role, delegating all schema management to the application's own initialization routine. This is a cleaner separation of concerns: the database initialization role handles database and keyspace creation (infrastructure concerns), while the application handles its own schema migrations (application concerns).## The Thinking Process: How the Assistant Diagnosed the Output

The reasoning visible in the messages surrounding message 1588 reveals a methodical diagnostic approach. When the assistant saw the playbook output, it didn't just declare victory and move on. Instead, it immediately checked the Kuri service status and journal logs (message 1578) to see whether the daemon actually started successfully. This is a critical habit: a playbook can report success (files copied, services enabled) while the actual application fails to run.

The assistant's thought process followed a clear pattern:

Verify the service is running. Check systemctl status to see if the daemon is active. In this case, it was in "auto-restart" loop — the process exited with code 1 and systemd kept retrying.
Examine the logs. Use journalctl to find the actual error messages. This revealed both the export prefix warnings (already fixed) and the "invalid log level: debug" error.
Trace the configuration source. The assistant searched the Go source code for how RIBS_LOGLEVEL is parsed, finding the format specification in configuration/config.go.
Check the inventory files. The assistant read both the test inventory and the production defaults to understand what values were being set and where the fix needed to be applied.
Apply the fix in the right layer. The log level format needed to change in the Ansible inventory (the data layer), not in the template or the application code.
Clean up and retry. Before re-running, the assistant reset the state on both nodes to ensure a clean test. This diagnostic loop — execute, check status, read logs, trace to source, fix, clean, retry — is the essence of infrastructure debugging. Each iteration narrows the gap between the desired state (a working cluster) and the actual state (a failing deployment).

Assumptions Made and Their Consequences

Several assumptions are embedded in this message and its surrounding context:

Assumption: The playbook output tells the whole story. The assistant initially ran the playbook and saw wallet file stats, which could have been misinterpreted as success. The decision to independently verify the service status was what saved the debugging session from going down a wrong path.

Assumption: debug is a valid log level. The person who wrote the inventory file (likely the assistant in an earlier session) assumed that RIBS_LOGLEVEL=debug would work, based on common logging conventions. The application's actual parser required a hierarchical pattern format. This is a reasonable assumption that happened to be wrong — a reminder that "obvious" configuration values often have hidden constraints.

Assumption: .gitkeep files are harmless. The Git convention of using .gitkeep to track empty directories is so common that it's easy to forget that application code might iterate over all files in a directory. The wallet directory is a security-sensitive location where every file is assumed to be a key — a .gitkeep file is not just harmless noise but a real operational hazard.

Assumption: The inventory path is consistent. The assistant initially used /ansible/inventory/hosts.yml (message 1585) based on a mental model of where the inventory should be. The actual path was /ansible/inventory/test/hosts.yml. This mismatch cost a full playbook execution cycle and required explicit directory listing to resolve.

Input Knowledge Required

To fully understand message 1588, a reader needs knowledge of:

Ansible playbook execution syntax: The -i flag for inventory, -l for host limiting, -v for verbose output, and the structure of playbook output (task names, changed/failed counts, stat results).
Docker Compose and container exec: The docker compose exec -T command for running commands inside a running container, with -T disabling pseudo-TTY allocation.
Systemd service management: How EnvironmentFile works, that it doesn't support export prefixes, and how to check service status with systemctl and journalctl.
The FGW architecture: The distinction between Kuri storage nodes and S3 frontend proxies, the role of YugabyteDB, and the wallet system for Filecoin deals.
Jinja2 templating: How Ansible uses .j2 templates to generate configuration files from variables.

Output Knowledge Created

This message, combined with the diagnostic messages that follow, created several important pieces of knowledge:

The export prefix fix was incomplete. Removing export from the template fixed systemd, but the kuri init task needed a separate mechanism to export variables in the shell context. This required a dual approach: plain KEY=VALUE for systemd, and explicit set -a; source ...; set +a for the init command.
The log level format is non-trivial. The application uses a hierarchical logger pattern (e.g., ribs:.*=debug,gw/.*=debug) rather than a simple level string. This needs to be documented and set correctly in both test and production inventories.
Wallet directories must be clean. Any non-key file in the wallet directory will cause the Kuri binary to fail during initialization. The deployment pipeline must explicitly exclude hidden files like .gitkeep.
The test harness is sensitive to boot timing. The pam_nologin issue required changes to the Dockerfile (systemd-user-sessions service) to ensure SSH logins work immediately after container startup.

Conclusion

Message 1588 is a powerful example of how a single command execution in a debugging session can serve as a diagnostic fulcrum — a point where multiple latent bugs are simultaneously revealed through careful observation of the output and independent verification of the results. The assistant's methodical approach of checking service status, reading logs, tracing configuration to source code, and applying targeted fixes demonstrates the discipline required for infrastructure-as-code debugging.

What makes this message particularly instructive is that the output itself appears mundane — a file stat result from a successful Ansible task. The real diagnostic value came not from what the playbook reported, but from what the assistant chose to investigate next. The decision to check systemctl status and journalctl transformed a seemingly successful playbook run into the discovery of three distinct bugs. In the world of distributed systems deployment, the playbook's green checkmark is never the final word — the real story is always in the logs.