The Pivot to Automation: How a Single User Question Transformed a Deployment Strategy

Introduction

In the middle of a complex multi-node QA cluster deployment for the Filecoin Gateway (FGW) distributed storage system, a single message from the user—"Why are you not doing this with ansible?"—triggered an immediate and decisive shift in approach. The assistant's response, message 2087 in the conversation, is the culmination of that pivot: a brief verification that the S3 frontend proxy had been successfully deployed using the proper infrastructure automation tooling. While the message itself is short, it represents a critical inflection point in the session where ad-hoc manual configuration gave way to reproducible, maintainable automation. This article examines the reasoning, decisions, assumptions, and knowledge embedded in that single verification message.

The Message

The subject message reads in full:

S3 frontend deployed successfully. Let me verify it's working:

>

`` [bash] ssh 10.1.232.82 "sudo systemctl status s3-frontend-s3_proxy_01 --no-pager" 2>&1 | tail -15 ● s3-frontend-s3_proxy_01.service - Filecoin Gateway S3 Frontend Proxy (s3_proxy_01) Loaded: loaded (/etc/systemd/system/s3-frontend-s3_proxy_01.service; enabled; preset: enabled) Active: active (running) since Sat 2026-01-31 22:37:21 UTC; 6s ago Docs: https://github.com/filecoin-project/filecoin-gateway Main PID: 21147 (s3-proxy) Tasks: 12 (limit: 23945) Memory: 7.0M (max: 8.0G available: 7.9G peak: 9.3M) CPU: 29ms CGroup: /system.slice/s3-frontend-... ``

At first glance, this appears to be a routine verification step—a developer checking that a service started correctly. But to understand its significance, we must examine the twenty messages that precede it and the architectural problem they were trying to solve.

The Problem That Led Here

The session had been focused on getting a three-node QA cluster operational. The architecture consisted of a head node running YugabyteDB (at 10.1.232.82) and two Kuri storage nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84). The Kuri nodes each ran an embedded S3-compatible API server that could serve objects stored in their local blockstores.

The critical flaw emerged during cross-node testing. When an object was written to kuri1 and then requested from kuri2, the read failed. The S3 object metadata was stored in a shared YugabyteDB CQL keyspace (filecoingw_s3), so kuri2 could see that the object existed and was owned by kuri_01. But the actual block data lived only on kuri1's local blockstore, and kuri2 had no mechanism to fetch it. The error message was explicit: "block was not found locally (offline): ipld: could not find [CID]."

The assistant explored several approaches to solve this. First, it attempted to use IPFS/libp2p bitswap to connect the two Kuri nodes directly, successfully establishing a swarm connection between them. But this didn't help—the S3 layer was not designed to retrieve blocks from peers; it served only from the local blockstore. The architecture required a different solution.

The correct approach, as defined in the project's roadmap, was to deploy an s3-proxy frontend—a stateless routing layer that sits in front of the Kuri nodes. When a GET request arrives, the proxy queries the shared CQL keyspace to determine which backend node owns the object, then routes the request to that node. This separates the stateless S3 API layer from the storage nodes, enabling horizontal scaling and cross-node reads.

The Manual Approach and the Pivot

The assistant began deploying the s3-proxy manually. It copied the binary to the head node via scp, created a configuration file (s3proxy.env) by writing it with a heredoc over SSH, and manually created a systemd service unit file. This approach worked in a narrow sense—the files were created and the service was started—but it revealed a deeper problem.

The initial manual configuration contained a subtle but critical error. The assistant set RIBS_S3_CQL_HOSTS="10.1.232.82" in the environment file, but the s3-proxy binary actually read from FGW_YCQL_HOSTS (as discovered moments later when the assistant grep'd the source code). The proxy started and immediately failed, connecting to [::1]:9042 (localhost) instead of the YugabyteDB host. The service crashed in a restart loop.

Then came the pivotal user message at index 2079: "Why are you not doing this with ansible?"

This question cut to the heart of a recurring tension in infrastructure work. The project already had a well-structured set of Ansible roles for deploying the s3_frontend component, complete with Jinja2 templates for environment files, systemd service templates, and proper variable management. The assistant had been ignoring this existing investment, opting instead for quick manual commands that were fragile, unreproducible, and error-prone.

The assistant's response was immediate and unambiguous: "You're right - I should be using ansible properly instead of manually SSHing and creating ad-hoc configs. Let me fix this properly with ansible." It stopped the manually-created service, read the existing Ansible roles to understand their structure, updated the QA inventory to add the head node to the s3_frontend group, and ran the deployment playbook.

What Message 2087 Actually Represents

Message 2087 is the verification that this pivot succeeded. The systemd status output shows the service s3-frontend-s3_proxy_01 is "active (running)" with a Main PID of 21147, using 7.0M of memory, and having been up for only 6 seconds at the time of the check. The service name follows the naming convention defined in the Ansible templates, confirming that the playbook correctly instantiated the configuration.

But the message conveys more than just a successful service start. It signals several things:

  1. The Ansible deployment pipeline works. The playbook executed without errors, the Jinja2 templates rendered correctly with the inventory variables, the systemd unit was properly installed and enabled, and the binary started without crashing.
  2. The configuration was correct this time. Unlike the manual attempt where RIBS_S3_CQL_HOSTS was the wrong variable name, the Ansible template presumably used FGW_YCQL_HOSTS (or the correct variable), allowing the proxy to connect to the YugabyteDB instance.
  3. The architecture is now aligned with the roadmap. The s3-proxy frontend on the head node (port 8078) will route requests to the appropriate Kuri backend (kuri1 on port 8079 or kuri2 on port 8079), enabling the cross-node reads that had been failing.
  4. The deployment is reproducible. Because it was done through Ansible, the same playbook can be run against any environment—development, staging, or production—with consistent results.

Assumptions and Decisions

Several assumptions underpin this message. The assistant assumes that the Ansible roles and templates are correctly designed for this deployment scenario—that the variables defined in the inventory will produce a valid configuration. It assumes that the s3-proxy binary, which was manually copied to the head node before the Ansible run, is compatible with the configuration the playbook generates. It assumes that the service starting successfully means it will also function correctly when handling actual S3 requests (a separate verification that follows in subsequent messages).

The key decision visible here is the choice to use the existing Ansible infrastructure rather than continuing with manual configuration. This decision carries implications: it means accepting the naming conventions, directory structures, and variable schemas defined in the playbooks rather than creating ad-hoc alternatives. It means the deployment is now documented in code rather than in the developer's memory. It means future changes go through the playbook rather than requiring SSH access to each node.

The Thinking Process

The assistant's thinking process in the messages leading up to 2087 reveals a pattern of iterative debugging followed by a moment of architectural clarity. The early attempts—bitswap peering, direct S3 reads, manual proxy configuration—were all reasonable technical explorations, but they lacked the discipline of using the project's established automation. The user's question reframed the problem not as a technical challenge (how to make cross-node reads work) but as a process challenge (how to make the deployment maintainable).

Message 2087 shows the assistant applying this lesson. The verification step is thorough: rather than assuming the Ansible playbook succeeded based on its exit code, the assistant independently checks the service status via SSH. This double-checking is characteristic of someone who has been burned by silent failures before. The 6-second uptime is noted but not treated as definitive—the assistant knows that a service can start and then crash moments later, which is why the verification command was run immediately after deployment.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

Conclusion

Message 2087 appears, on its surface, to be a mundane status check. But in the context of the session, it represents the successful resolution of two intertwined problems: the technical challenge of enabling cross-node S3 reads in a distributed storage system, and the process challenge of using proper infrastructure automation. The user's pointed question about Ansible forced a recognition that manual deployment, while faster in the moment, creates long-term costs in reproducibility, documentation, and reliability. The assistant's pivot—and the verification message that confirmed its success—demonstrates the kind of discipline that separates ad-hoc infrastructure from production-grade operations. In the world of distributed systems, how you deploy matters as much as what you deploy, and this message marks the moment when that lesson was fully absorbed.