The .gitkeep That Broke the Cluster: A Case Study in Infrastructure-as-Code Debugging

In the iterative world of infrastructure-as-code, the smallest artifacts can cause the most bewildering failures. Consider the humble .gitkeep file — a zero-content placeholder used by developers to ensure Git tracks otherwise-empty directories. It is invisible to ls without flags, ignored by most tooling, and utterly harmless in a source repository. Yet, as this message from an Ansible deployment debugging session demonstrates, when .gitkeep crosses the boundary from development environment to production deployment, it can bring an entire distributed storage cluster to its knees.

The subject message, index 1593, captures the precise moment of diagnosis and the beginning of remediation:

[assistant] Yes, both the .gitkeep and the wallet files are there. The wallet copy role needs to exclude .gitkeep. Let me update the wallet role to exclude it: [read] /home/theuser/gw/ansible/roles/wallet/tasks/main.yml

This short message is a turning point in a much longer debugging session. It represents the crystallization of a chain of reasoning that began with a cryptic error — a Kuri storage node failing to start because it tried to parse a binary wallet key file that wasn't a key at all.

The Context: Deploying a Distributed S3 Storage Cluster

To understand the significance of this message, one must understand the system being built. The Filecoin Gateway (FGW) is a horizontally scalable S3-compatible storage architecture. The deployment pipeline uses Ansible playbooks to provision nodes with two roles: Kuri storage nodes (which run the core IPFS/RIBS daemon) and S3 frontend proxies (stateless API servers). The deployment had been tested through a Docker-based test harness, and the session leading up to message 1593 had already uncovered and fixed several issues: systemd's EnvironmentFile rejecting export prefixes in settings templates, invalid log level format strings, and database initialization ordering problems.

The test harness had progressed through connectivity checks and database initialization, but stalled at the Kuri node deployment step. The Kuri daemon failed to start, and the logs revealed two problems simultaneously: an invalid log level format (debug vs a required regex pattern) and a wallet file parsing error. The wallet error was the more subtle of the two.

Tracing the Root Cause: How .gitkeep Became a Wallet Key

The wallet role in Ansible is responsible for distributing cryptographic key files (ribswallet) to Kuri nodes. These are small binary files containing private key material. The source directory in the repository, ansible/files/wallet/, contained only a .gitkeep file — a common Git workaround to track an empty directory that will later hold wallet files generated by a separate initialization process.

The Docker test harness, however, had its own wallet generation step that created actual wallet files in a separate test-wallet/ directory. The test setup script then copied both the repository source (via /ansible-src/*, which included the .gitkeep) and the test-generated wallet files into the container's /ansible/files/wallet/ directory. The result was a directory containing both legitimate wallet key files AND the .gitkeep file.

When the Kuri daemon started, it scanned the wallet directory and attempted to parse every file as a cryptographic key. The .gitkeep file, containing whatever text or binary content it held (in this case, 218 bytes of content), failed to parse as a valid key. The daemon crashed with a binary parsing error, and systemd entered a restart loop.

The assistant's debugging process leading up to message 1593 was methodical. First, in message 1589, the assistant identified the dual issues from the Ansible playbook output: "There are two issues visible: 1. Invalid log level format: invalid log level: debug... 2. Wallet .gitkeep issue: The wallet directory has a .gitkeep file which the kuri binary tries to parse as a key." Then, the assistant checked the local wallet directory (ls -la /home/theuser/gw/ansible/files/wallet/) and found only .gitkeep — confirming that the source directory itself was empty of real wallet files. Next, the assistant inspected the test setup script and the test wallet directory, verifying that the test harness generated proper wallet files in a separate location. Finally, in message 1592, the assistant confirmed the merged state inside the container: both .gitkeep and wallet files were present in /ansible/files/wallet/.

The Reasoning Behind the Fix

Message 1593 represents the moment when the assistant transitions from diagnosis to treatment. The statement "The wallet copy role needs to exclude .gitkeep" is not merely a description of the problem — it is a design decision. The assistant could have chosen several approaches:

  1. Remove .gitkeep from the source repository entirely — but this would break the Git workflow for tracking the directory.
  2. Change the copy mechanism to be explicit about which files to copy — using a file list instead of a wildcard.
  3. Add a cleanup task after copy — deleting .gitkeep from the target.
  4. Use Ansible's synchronize module with exclusion patterns — leveraging rsync's --exclude flag.
  5. Filter out hidden files globally — a broader but riskier approach. The assistant's next action (message 1594) reveals the chosen approach: "I need to use synchronize module to exclude .gitkeep, or delete it after copying. The simplest fix is to delete it after copying since we're already using shell for permissions." This pragmatic choice — adding a cleanup step rather than restructuring the entire copy mechanism — reflects the iterative nature of infrastructure debugging. The goal is not architectural purity but a working deployment pipeline.

Assumptions and Their Consequences

This debugging episode reveals several assumptions that were baked into the original Ansible role design:

Assumption 1: All files in the wallet source directory are wallet key files. The copy task used a broad pattern that captured every file, including hidden dotfiles. This assumption held during initial development (when only wallet files existed) but broke when .gitkeep was introduced for Git directory tracking.

Assumption 2: The test harness and production deployment share identical file layouts. The test setup script merged two sources (repository files and test-generated files), creating a hybrid state that didn't exist in either environment alone. This assumption masked the bug during early testing.

Assumption 3: Dotfiles are irrelevant to application logic. While true for most deployment scenarios, the Kuri binary's wallet scanning logic explicitly iterates over all directory entries without filtering hidden files. This is arguably a design flaw in the binary itself — it should skip non-key files — but the deployment pipeline must work with the binary as it exists.

Input and Output Knowledge

The input knowledge required to understand this message is substantial. One must grasp: the Git convention of .gitkeep for tracking empty directories; Ansible's file copy and synchronize modules; the Kuri daemon's wallet initialization logic; Docker container filesystem layering; and the test harness architecture that merges multiple source directories. The output knowledge created by this message is the specific diagnosis that a hidden dotfile is causing a production service failure, and the strategic decision to exclude it from deployment rather than fixing it at the application layer.

The Broader Lesson

This message exemplifies a class of bugs that plague infrastructure-as-code systems: the boundary violation between development artifacts and deployment artifacts. .gitkeep, .gitignore, README.md, license files, CI configuration — all of these are essential for development but harmful in production. The fix implemented here — explicitly excluding .gitkeep from the wallet copy — is a tactical solution. A more strategic approach might involve separating "source of truth" wallet generation from deployment entirely, or implementing a more robust file filtering strategy in the Ansible role.

The debugging process visible in the surrounding messages demonstrates a disciplined approach to infrastructure debugging: observe the failure symptom, trace it to the specific file causing the error, verify the file's origin, understand why it shouldn't be there, and implement the minimal fix that restores the pipeline to working order. Message 1593 is the hinge point in that process — the moment of clarity where the root cause is confirmed and the path forward becomes clear.