Architecting a QA Test Environment for Distributed S3 Storage: A Deep Dive into Infrastructure Planning

Introduction

In the lifecycle of any distributed systems project, there comes a pivotal moment when development transitions into validation—when code that has been tested in isolation must prove itself in an environment that mirrors production. Message 1909 in this coding session captures exactly such a moment. It is a comprehensive, meticulously researched plan for deploying a Quality Assurance (QA) test environment for the Filecoin Gateway (FGW) distributed S3 storage system across three physical nodes. Written by the AI assistant in response to a user request for "best practices for an easy QA/test environment," this message is far more than a simple list of commands. It is a synthesis of delegated research, a negotiation of trade-offs, and a blueprint that encodes dozens of architectural decisions, assumptions about network topology, and operational philosophies about how infrastructure should be managed.

This article examines that single message in depth: why it was written, how its decisions were made, the assumptions it embeds, the knowledge it requires and creates, and the thinking process visible in its structure and content. By the end, we will see how a seemingly straightforward deployment plan reveals the complexity of bridging development and operations in a distributed storage system.

The Strategic Context: Why This Message Was Written

To understand why this message exists, we must look at what preceded it. The conversation leading up to message 1909 spans a massive development effort: the assistant had just completed and committed three major milestones—a multi-tier retrieval cache system (Milestone 03), a passive garbage collection system (Milestone 04), and enterprise-grade observability and operations (Milestone 02). The codebase had been tested, benchmarked, and committed. But code that works on a developer's workstation is not the same as code that works across three physical machines connected to the Filecoin mainnet.

The user's request—"Describe best practices for an easy QA/test environment. Aim for fairly small, e.g. 2x 1.25TiB nodes + 1x 200GiB S3+YB, ubuntu 24. Mainnet with cidg (supplied wallet/token). Include network reqs. Delegate agents to research anything you aren't 100% sure about"—signals a deliberate shift from development to validation. The user wants a plan that is practical, not theoretical. They specify exact storage sizes, operating system version, and the requirement to connect to the real Filecoin mainnet with real credentials. The instruction to "delegate agents to research anything you aren't 100% sure about" is particularly telling: it acknowledges that the assistant cannot afford to guess about infrastructure details, and that rigorous accuracy is required.

The assistant's response in message 1909 is the culmination of this research-driven approach. Before writing this plan, the assistant delegated four research tasks covering FGW deployment requirements, YugabyteDB minimal deployment, Filecoin mainnet requirements, and MinIO S3 requirements. It then asked the user three clarifying questions about deployment approach (Ansible vs. manual commands), CAR file staging (LocalWeb vs. MinIO), and observability (included vs. external). The answers to those questions—Ansible-based, LocalWeb on ports 7010/7011 without TLS, and external monitoring—directly shaped the final plan.

This message is thus the output of a multi-step research and clarification process. It is not the assistant's first draft; it is the refined, user-informed final plan. This is a critical point for understanding the message's authority: every line has been vetted against research and user preference.

Architecture and Design Decisions

The architecture diagram at the top of the plan is deceptively simple. Three nodes: two storage nodes (kuri-01 and kuri-02) with 1.25 TiB each, and one infrastructure node with 200 GiB running YugabyteDB. But beneath this simplicity lies a carefully reasoned set of design decisions.

Why three nodes? The two kuri nodes provide redundancy and allow testing of cross-node data access, failover scenarios, and distributed storage operations. The separate infra node isolates the database from the storage workload, preventing resource contention and allowing the database to be sized independently. This is a classic three-tier architecture pattern, but adapted for a distributed storage context.

Why colocate the S3 frontend on the kuri nodes? The plan places both the Kuri storage daemon and the S3 frontend proxy on the same physical nodes. This is a deliberate simplification for QA. In production, the S3 frontend would likely run on separate, stateless proxy nodes. But for testing, colocation reduces hardware requirements and network hops while still validating the critical path: client → S3 proxy → Kuri storage → YugabyteDB.

Why LocalWeb instead of MinIO? The user explicitly chose LocalWeb for CAR file staging, and the assistant respected that choice. LocalWeb is a built-in HTTP server within the FGW codebase that serves CAR files directly, eliminating the need for a separate MinIO deployment. By running it on ports 7010 and 7011 (one per node) without TLS, the plan keeps complexity low while still validating the CAR serving path that storage providers would use to retrieve data.

Why single-node YugabyteDB? The infra node runs a single-node YugabyteDB instance. This is a QA-appropriate choice: it validates the CQL and YSQL interfaces that Kuri uses, without the complexity of multi-node database replication. The assistant's research on YugabyteDB confirmed that a single-node deployment with 2 GB RAM and 2 cores is sufficient for testing, and the plan allocates 8 GB RAM and 4 cores—a comfortable margin.

The Research-Driven Approach

One of the most distinctive features of this message is that it does not appear in isolation. It is the product of a deliberate research methodology that the assistant executed before writing a single line of the plan. Understanding this methodology is essential to appreciating the message's depth.

The assistant delegated four research tasks, each targeting a specific knowledge gap:

FGW deployment requirements: Examining the Ansible roles, playbooks, and configuration files to understand what services need to be deployed and how they relate to each other.
YugabyteDB minimal deployment: Determining the minimum hardware, ports, and configuration for a single-node YugabyteDB instance suitable for testing.
Filecoin mainnet requirements: Understanding what network connectivity is needed for Filecoin mainnet communication, including the fact that FGW uses an external Lotus Gateway RPC endpoint (not a full Lotus node).
MinIO S3 requirements: Researching MinIO's minimum system requirements and whether it could coexist on the infra node. Each research task returned detailed findings that the assistant then synthesized into the plan. For example, the research on Filecoin mainnet confirmed that FGW connects via https://api.chain.love/rpc/v1—a public gateway that requires no full Lotus node. This finding directly appears in the plan's Ansible inventory as filecoin_api_endpoint: "https://api.chain.love/rpc/v1". The research also informed what was excluded from the plan. MinIO, which was initially considered for CAR file staging, was ultimately omitted because the user chose LocalWeb. The observability stack (Loki, Promtail) was excluded because the user chose external monitoring. These exclusions are as important as the inclusions—they represent deliberate scope management.

Network and Security Considerations

The network requirements section of the plan is remarkably detailed, specifying nine firewall rules, three outbound destinations, and three bandwidth minimums. This level of detail reveals several important assumptions and decisions.

The internal network assumption: The plan assumes that the three nodes can communicate over a private network with IP addresses in the 10.0.0.x range. The YugabyteDB ports (5433 for YSQL, 9042 for YCQL) are marked as "Internal" direction, meaning they should not be exposed to the public internet. This is a standard security practice for database traffic, but it requires that the infrastructure provider supports private networking.

The public-facing S3 and LocalWeb ports: Ports 8078 (S3 API), 7010, and 7011 (LocalWeb) are marked as inbound from "Clients" or "Internet." This means the kuri nodes need public IP addresses or a load balancer that can route external traffic to them. The plan's use of DNS names like kuri-01.qa.example.com reinforces this assumption.

The Prometheus scraping port: Port 2112 is opened for Prometheus metrics scraping. This is a security-relevant decision: the plan assumes an external Prometheus server that will scrape metrics from the kuri nodes. The scrape configuration provided later in the plan confirms this architecture.

Secret management: The plan uses Ansible Vault to encrypt the CIDgravity API token, showing awareness that secrets should not be stored in plaintext in version-controlled inventory files. The placeholder !vault | $ANSIBLE_VAULT;1.1;AES256 ... indicates where the encrypted token would go. This is a best practice that the assistant carried forward from earlier in the conversation, where the user had explicitly flagged that secrets must not be stored in plaintext.

Deployment Automation Philosophy

The plan's deployment procedure reveals a strong automation philosophy. Every step is expressed as an Ansible playbook command, not as manual SSH commands. The assistant provides five sequential steps:

Pre-requisites: Setting up the inventory, encrypting secrets, copying wallet files
System Preparation: Running prepare.yml for OS tuning, ulimits, Docker installation, time sync
Deploy YugabyteDB: Running yugabyte.yml targeting the infra node
Deploy FGW: Running kuri.yml and s3_frontend.yml targeting the kuri nodes
Verify Deployment: Running verify.yml and manual curl checks This sequence encodes a dependency ordering: the database must be running before Kuri starts, and the system must be prepared before any services are deployed. The assistant even provides a combined command (site.yml --skip-tags observability) for deploying everything at once, acknowledging that in practice, operators may want a single command. The Ansible inventory itself is a work of configuration design. It uses group variables for shared settings (Filecoin API endpoint, YugabyteDB connection details, storage limits) and host variables for per-node settings (IP addresses, ports, data directories). The s3_frontend group is a child of the kuri hosts, meaning the S3 proxy runs on the same machines as the storage daemons—a design choice that simplifies networking while still testing the proxy layer. One notable detail is the use of s3_auth_enabled: false for QA simplicity. This is a deliberate relaxation of security for testing purposes, acknowledging that authentication adds complexity without necessarily improving test coverage of the storage layer itself.

Configuration Trade-offs and Assumptions

The plan's configuration values reveal a series of trade-offs between production realism and QA practicality.

Storage limits: ribs_max_staging_group_count: 10 (approximately 300 GiB staging) and ribs_max_local_group_count: 30 (approximately 900 GiB data per node) are set to values that allow meaningful testing without filling the 1.25 TiB drives. These limits assume that each staging group consumes roughly 30 GiB and each local group consumes roughly 30 GiB—estimates that may need adjustment based on actual usage patterns.

Deal settings: ribs_minimum_replica_count: 3 and ribs_maximum_replica_count: 5 are "relaxed for testing." In production, these values might be higher to ensure data durability. The QA environment only has two storage nodes, so a minimum replica count of 3 means that some replicas will be stored on the same node or that the system will operate in a degraded state until more nodes are added. This is an acknowledged compromise for testing.

Deal check interval: ribs_deal_check_interval: "30s" is aggressive. In production, deal checking might run every few minutes or hours. The 30-second interval ensures that deal lifecycle events are visible quickly during testing, but it also generates more load on the Filecoin RPC endpoint and the database.

LocalWeb without TLS: The plan explicitly sets EXTERNAL_LOCALWEB_SERVER_TLS=false and uses HTTP URLs. This simplifies deployment (no certificate management, no HTTPS configuration) but means that CAR file transfers between storage providers and the gateway are unencrypted. For a QA environment connected to mainnet, this is a security trade-off that the user accepted when choosing LocalWeb over MinIO.

Cost estimates: The plan includes monthly cost estimates for AWS ($800-1000), GCP ($700-900), and Hetzner ($250-350). These estimates are based on specific instance types (e.g., AWS m6i.2xlarge, Hetzner AX52) and assume that the 1.25 TiB storage is provided as block storage (EBS gp3, SSD volumes). The wide range between cloud providers and Hetzner reflects the reality that bare-metal or dedicated server providers can offer significantly better price-to-performance ratios for storage-heavy workloads.

Monitoring and Operations

The plan's monitoring section is concise but complete. It provides a Prometheus scrape configuration that targets both kuri nodes on port 2112 and the YugabyteDB node on port 13000. It references five Grafana dashboards that were created during the enterprise-grade milestone: overview, S3 SLA, deals, storage, and financials.

The decision to use "external monitoring" (as specified by the user) means that the QA environment itself does not run Prometheus or Grafana. Instead, it exposes metrics endpoints that an existing monitoring infrastructure can scrape. This is a pragmatic choice for organizations that already have observability tooling—it avoids duplicating infrastructure while still providing visibility into the test environment.

The QA test scenarios section lists six tests that cover the critical paths: S3 upload/download, deal creation, CAR serving, failover, and metrics. These tests are not exhaustive, but they validate the core functionality that the system is designed to provide. The failover test—"Stop kuri-01, access via kuri-02"—is particularly important for a distributed storage system, as it validates that data is accessible even when one node is unavailable.

The Closing Question and Next Steps

The message ends with a question: "Shall I proceed with creating the QA-specific ansible inventory and any necessary role modifications for the HTTP-only LocalWeb configuration?" This is not a rhetorical flourish. It is a deliberate handoff point that acknowledges the plan is complete as a plan but that execution requires the user's approval.

This question also reveals an important assumption: that the existing Ansible roles may need modification to support HTTP-only LocalWeb. The plan notes that "the kuri role needs modification for HTTP-only LocalWeb," suggesting that the existing roles assume TLS is enabled. The assistant is offering to make those modifications as part of the next step, but only if the user confirms the plan.

Conclusion

Message 1909 is a masterclass in infrastructure planning for distributed systems. It demonstrates how to synthesize research from multiple domains into a coherent, actionable plan. It shows how to negotiate trade-offs between production realism and QA practicality. It encodes security best practices (Ansible Vault for secrets, private networking for databases) while acknowledging where security can be relaxed for testing (no TLS on LocalWeb, no S3 authentication). It provides enough detail to be immediately executable while leaving room for the user to make final decisions.

But perhaps most importantly, this message reveals the thinking process of an experienced infrastructure engineer. The assistant did not guess. It researched. It asked clarifying questions. It presented options and let the user choose. It documented assumptions explicitly. It provided cost estimates. It considered network topology, firewall rules, bandwidth requirements, and DNS naming. It thought about what could go wrong and built verification steps into the deployment procedure.

For anyone building a QA environment for a distributed storage system, this message is a template worth studying. It shows that a good infrastructure plan is not just a list of commands—it is a document that captures decisions, assumptions, and trade-offs, so that everyone involved understands not just what to do, but why it should be done that way.