The Clean Slate: How One Bash Command Marked a Turning Point in Debugging Infrastructure as Code

The Message

In the middle of a marathon debugging session for the Filecoin Gateway (FGW) Ansible deployment pipeline, the assistant issued what appeared to be a simple command:

Now let's reset and test again properly. This time I'll clean up all state and run the full test:
[bash] cd /home/theuser/gw/ansible/test && ./cleanup.sh 2>&1 || true
=== Cleaning up Ansible test environment ===
Stopping containers...
 Container ansible-test-controller Stopping 
 Container ansible-test-controller Stopped 
 Container ansible-test-controller Removed 
 Container ansible-test-s3-fe-01 Stopping 
 Container ansible-test-s3-fe-01 Stopped 
 Container ansible-test-s3-fe-01 Removing 
 Container ansible-test-s3-fe-01 Removed 
 Container ansible-test-kuri-02 Stopping 
 Container ansible-test-kuri-01 Stopping 
...

This message, message index 1616 in the conversation, is outwardly unremarkable: a developer running a cleanup script to tear down a Docker-based test environment. But in the context of the session, it represents a critical inflection point—a deliberate decision to abandon incremental patching and start fresh after a cascade of interrelated bugs had been discovered and fixed. Understanding why this moment matters requires examining the debugging odyssey that preceded it, the assumptions that shaped the decision, and the discoveries that followed.

The Context: A Cascade of Interrelated Bugs

The assistant had been working through a suite of Ansible playbooks designed to deploy a distributed Filecoin Gateway cluster. The architecture involved three tiers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB database. The test harness used Docker containers with systemd to simulate production Ubuntu servers, with an Ansible controller container orchestrating deployments.

The preceding messages (1579–1615) reveal a methodical bug-hunting expedition. Each issue discovered led to another, in a chain that demonstrates the fractal complexity of infrastructure-as-code:

Issue 1: The export Prefix Problem. Systemd's EnvironmentFile directive requires plain KEY=VALUE format, but the Jinja2 template for settings.env included export prefixes. The assistant had to split the approach: keep export out of the template for systemd compatibility, but explicitly use set -a; source ...; set +a in shell commands that needed the variables exported.

Issue 2: Invalid Log Level Format. The log level configuration used *:*=debug, but the application's regex parser rejected * as an invalid repetition operator. The fix required .*:.*=debug—a subtle but critical difference between shell glob patterns and regular expressions.

Issue 3: Wallet Dotfiles. The files/wallet/ directory contained a .gitkeep file (a common Git trick to track empty directories). When the wallet role copied this directory to the target nodes, kuri's binary tried to parse .gitkeep as a Filecoin wallet key file, producing cryptic parsing errors. The fix required excluding dotfiles from the copy operation.

Issue 4: Duplicate Table Creation. The yugabyte_init role created database tables, but kuri init also ran migrations that attempted to create the same tables. Since kuri used CREATE TABLE (not CREATE TABLE IF NOT EXISTS), the second run failed. The fix was to remove table creation from yugabyte_init and let kuri handle all migrations.

Issue 5: Non-Existent Ansible Filter. The s3_frontend role used a custom format_backend_url filter that didn't exist in Ansible's standard library, causing the frontend deployment to fail.

Each fix was individually correct. But the assistant recognized that applying patches to a running, partially-deployed system was accumulating state that could mask or interact with other issues. The decision to run cleanup.sh was an acknowledgment that the only way to validate the complete set of fixes was to start from a known-clean state.## The Reasoning Behind the Reset

The assistant's decision to run cleanup.sh was not arbitrary—it reflected a sophisticated understanding of testing methodology. There were several reasons a clean slate was necessary:

First, idempotency verification. Ansible playbooks are supposed to be idempotent: running them multiple times should produce the same result. But the assistant had been manually intervening—removing IPFS state, deleting specific files, and running ad-hoc commands. These interventions meant that subsequent test runs were testing a hybrid state, not the playbooks themselves. A clean restart would reveal whether the playbooks could independently reach a working state from scratch.

Second, the nologin problem. The Docker containers using systemd had a persistent issue: during boot, systemd creates /run/nologin and /var/run/nologin, which causes PAM to reject SSH logins with the message "System is booting up. Unprivileged users are not permitted to log in yet." The assistant had been manually removing these files after container startup, but this was a workaround, not a fix. A clean test would expose whether the Docker image build properly addressed this issue.

Third, the wallet directory cleanup. The test wallet directory had been emptied and replaced with a .keep file, but the setup script's cp -r /test-wallet/* /ansible/files/wallet/ would copy this file too. The assistant needed to verify that the wallet role's new dotfile-exclusion logic worked correctly.

Fourth, the cumulative effect of fixes. The log level fix, the export prefix removal, the table creation separation—each had been tested individually, but their interactions hadn't been validated together. For example, if the log level format was wrong, would the kuri init command fail silently, or would it produce a clear error? Would the systemd service file correctly reference the cleaned-up environment file? A full end-to-end test was the only way to know.

Assumptions Made and Their Validity

The assistant made several implicit assumptions when triggering the cleanup:

Assumption 1: The fixes were complete and correct. The assistant assumed that the five categories of bugs identified were the only ones preventing successful deployment. This assumption was tested and partially validated—the subsequent test runs did reveal additional issues (the format_backend_url filter, the pam_nologin problem in rebuilt images), proving the assumption was incomplete but productive.

Assumption 2: The cleanup script would produce a truly clean state. The cleanup.sh script stopped and removed containers, but it didn't necessarily remove Docker volumes or cached images. This became relevant later when the systemd-user-sessions.service removal didn't take effect because Docker was using cached images. The assistant had to explicitly force a --no-cache rebuild.

Assumption 3: The test infrastructure was reliable. The Docker Compose setup, the shared volumes, the network configuration—the assistant assumed these were stable and reproducible. When the pam_nologin issue persisted even after removing the systemd service, it revealed that the test infrastructure itself had nuances that needed addressing.

Assumption 4: Sequential fixes would compose cleanly. The assistant had made edits to multiple files: settings.env.j2, wallet/tasks/main.yml, yugabyte_init/tasks/main.yml, setup.sh, Dockerfile.target, and inventory files. The assumption was that these independent edits wouldn't conflict. This largely held true, though the wallet role's dotfile exclusion and the setup script's copy logic needed coordination.

The Knowledge Required to Understand This Message

To fully grasp what this message means, a reader needs:

Understanding of Ansible's architecture. The concept of roles, playbooks, inventory files, and the delegation model (where control node operations use delegate_to: localhost while remote operations run on target hosts) is essential. The EnvironmentFile vs shell sourcing distinction requires knowing how systemd unit files differ from shell scripts.

Knowledge of systemd container behavior. The pam_nologin mechanism, the systemd-user-sessions.service, and the boot sequence in Docker containers running systemd as PID 1 are non-trivial topics. Many developers are unaware that systemd containers have this SSH-blocking behavior.

Familiarity with Filecoin and IPFS concepts. The wallet system, base32-encoded addresses, the ribswallet directory, and the kuri init process all draw on Filecoin-specific knowledge. Understanding why a .gitkeep file would cause binary parsing errors requires knowing that wallet files are named after their public key addresses.

Awareness of regular expression syntax differences. The * vs .* distinction in log level configuration is a classic trap: shell globs use * as a wildcard, but Go's regexp package requires .* to match any character. This kind of cross-context knowledge is exactly what makes debugging infrastructure code so challenging.

What This Message Created

The immediate output of this message was a cleaned-up test environment. But the real output was the foundation for the subsequent successful test run. After the cleanup and rebuild:

  1. Connectivity checks passed for all three target hosts (kuri-01, kuri-02, s3-fe-01).
  2. YugabyteDB initialization succeeded with keyspace creation but no duplicate table errors.
  3. Kuri nodes deployed and initialized, creating their own wallets and passing health checks.
  4. Both kuri services showed active (running) with proper memory limits and configuration.
  5. The S3 frontend deployment was attempted and revealed the format_backend_url filter issue, which was then fixed. The cleanup also created a reproducible baseline. Subsequent debugging could now focus on the remaining issues (the frontend filter, the nologin workaround) without wondering whether leftover state from previous runs was causing problems.

The Thinking Process Revealed

The assistant's reasoning, visible in the surrounding messages, shows a methodical approach to debugging infrastructure code:

Pattern recognition. When the log level error appeared, the assistant immediately recognized the * vs .* issue—a pattern familiar from regex debugging. Similarly, the .gitkeep problem was recognized as a dotfile inclusion issue common in file-copy operations.

Root cause analysis. Rather than patching symptoms, the assistant traced each error to its source. The export prefix issue was traced from the systemd error message to the Jinja2 template. The duplicate table error was traced from the kuri init failure to the yugabyte_init role.

Separation of concerns. The decision to let kuri handle table creation while yugabyte_init only creates keyspaces reflects a clean architectural choice: the application should own its schema migrations, and infrastructure scripts should only set up the environment.

Incremental validation. Each fix was tested individually before the full cleanup. The assistant ran kuri init manually, checked systemd status, and verified wallet creation before declaring the fixes ready for a full test.

Conclusion

Message 1616 appears to be a mundane operational command—running a cleanup script. But in the context of this debugging session, it represents a deliberate strategic pivot. After discovering and fixing five distinct categories of bugs, the assistant recognized that incremental testing had reached its limits. The only way to validate the complete set of changes was to destroy the existing state and rebuild from scratch.

This is a lesson in infrastructure debugging that extends far beyond this specific project. When fixing complex deployment systems, individual fixes can be correct while their interactions remain untested. The clean slate approach—tear down, rebuild, and re-test from zero—is often the fastest path to a validated deployment pipeline. It's a technique that experienced infrastructure engineers reach for instinctively, and seeing it deployed here, in the middle of a marathon debugging session, marks the turning point where chaos gives way to order.