The Moment Infrastructure Meets Architecture: Deploying the S3 Proxy on a Three-Node QA Cluster

Introduction

In the lifecycle of any distributed system, there comes a pivotal moment when the carefully designed architecture must prove itself against the messy reality of physical hardware. Message 2076 captures exactly such a moment in the development of the Filecoin Gateway (FGW) distributed S3 storage system. In this single message, an assistant attempts to deploy the critical s3-proxy frontend onto a three-node QA cluster, bridging the gap between a theoretically sound multi-node design and the operational demands of cross-node object retrieval. The message is deceptively simple—a single ssh command that creates a systemd service, starts it, and reports failure—but it encapsulates a rich story of architectural discovery, debugging under pressure, and the tension between manual expedience and infrastructure automation.

The Message in Full

The subject message executes a multi-part bash command over SSH to the head node (10.1.232.82):

ssh 10.1.232.82 "sudo tee /etc/systemd/system/s3-proxy.service > /dev/null << 'EOF'
[Unit]
Description=FGW S3 Proxy
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/fgw

EnvironmentFile=/opt/fgw/config/s3proxy.env

ExecStart=/opt/fgw/bin/s3-proxy
Restart=always
RestartSec=5

LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable s3-proxy && sudo systemctl start s3-proxy && sleep 3 && sudo systemctl status s3-proxy --no-pager"

The output reveals immediate failure:

Created symlink /etc/systemd/system/multi-user.target.wants/s3-proxy.service → /etc/systemd/system/s3-proxy.service.
● s3-proxy.service - FGW S3 Proxy
     Loaded: loaded (/etc/systemd/system/s3-proxy.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-01-31 22:35:55 UTC; 2s ago
    Process: 20266 ExecStart=/opt/fgw/bin/s3-proxy (code=exited, status=1/FAILURE)
   Main PID: 20266 (code=exited, status=1/FAILURE)
        CPU: 18ms

Why This Message Was Written: The Cross-Node Retrieval Problem

To understand the motivation behind this message, we must trace the debugging journey that preceded it. The assistant had successfully deployed two kuri storage nodes (on 10.1.232.83 and 10.1.232.84) and a single-node YugabyteDB instance (on 10.1.232.82). Both kuri daemons were running, and the cluster topology API showed both nodes as healthy. However, a critical architectural gap had emerged.

When the assistant ran a load test against kuri1, it worked. But when the user observed that "Only kuri1 getting traffic seems wrong," the deeper investigation began. The assistant tested cross-node access by writing an object to kuri1 and attempting to read it from kuri2. The read from kuri2 returned empty. The logs revealed the root cause: kuri2 could see the object metadata in the shared filecoingw_s3 CQL keyspace (the s3objects table stored the object's CID and node_id='kuri_01'), but it could not retrieve the actual block data because the blocks resided only on kuri1's local blockstore.

This is a fundamental architectural challenge in distributed storage systems: how do you serve objects that were written to one node when a client requests them from another node? The system had two potential solutions: peer-to-peer block exchange via IPFS/libp2p bitswap, or a routing proxy layer. The assistant first attempted the bitswap approach, connecting kuri1 to kuri2 via their IPFS swarm APIs. But the cross-node read still failed because the S3 server was designed to serve data from the local blockstore only—it was not performing network retrieval for S3 requests.

The correct architectural answer, as defined in the project's roadmap, was the s3-proxy frontend: a stateless routing layer that accepts all S3 requests, looks up the object's metadata in the shared CQL keyspace to determine which backend node owns the data, and proxies the request to that node. This is precisely what the assistant set out to deploy in message 2076.## The Decision-Making Process Visible in the Message

The subject message reveals several layers of decision-making, both explicit and implicit. At the surface level, the assistant is making a straightforward operational choice: deploy the s3-proxy as a systemd service on the head node. But the details of the service file encode architectural decisions that deserve scrutiny.

Choice of deployment target. The assistant chose to deploy the s3-proxy on the head node (10.1.232.82), which already hosted the YugabyteDB database. This is a sensible decision for a QA environment: the head node has direct, low-latency access to the CQL keyspace that the proxy needs to query for routing decisions. In a production deployment, the proxy layer would likely be deployed on separate nodes for horizontal scaling, but for a three-node QA cluster, colocating the proxy with the database minimizes network hops and simplifies the topology.

Choice of configuration mechanism. The assistant used an EnvironmentFile directive pointing to /opt/fgw/config/s3proxy.env. This file had been created in the immediately preceding messages (2074 and 2075) through a series of manual SSH commands. The environment file contained the critical routing configuration:

FGW_BACKEND_NODES="kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079"
RIBS_S3API_BINDADDR=":8078"
RIBS_S3_CQL_HOSTS="10.1.232.82"
RIBS_S3_CQL_PORT="9042"
RIBS_S3_CQL_KEYSPACE="filecoingw_s3"

The FGW_BACKEND_NODES variable is the heart of the routing layer: it maps node identifiers to their HTTP endpoints. The proxy uses this information, combined with the node_id stored in the s3objects table, to forward requests to the correct backend.

Choice of service configuration. The systemd unit file is straightforward but contains one notable design decision: Restart=always with RestartSec=5. This is a pragmatic choice for a QA environment where the service is expected to fail initially (as it indeed does) and should recover automatically once the underlying issue is resolved. The LimitNOFILE=1048576 setting anticipates the high file descriptor requirements of a proxy handling many concurrent connections.

The Failure and Its Implications

The message ends with a clear failure: the s3-proxy service exits with status 1 immediately after starting. The activating (auto-restart) state visible in the status output indicates that systemd has already begun the restart cycle, but the service continues to fail.

The subsequent messages (2077–2078) reveal the cause: the proxy is attempting to connect to [::1]:9042 (localhost) instead of 10.1.232.82:9042. The environment variable RIBS_S3_CQL_HOSTS is not being picked up by the application. Further investigation shows that the s3-proxy binary actually expects the environment variable FGW_YCQL_HOSTS, not RIBS_S3_CQL_HOSTS. This is a naming mismatch between the configuration file written by the assistant and the actual environment variables consumed by the application code.

This failure is instructive. It highlights a common pitfall in distributed systems development: configuration drift between components. The kuri nodes use RIBS_S3_CQL_HOSTS for their S3 configuration, but the s3-proxy binary, developed separately, uses FGW_YCQL_HOSTS. The assistant had written the environment file based on assumptions derived from the kuri configuration, without verifying the s3-proxy's actual configuration interface.

Assumptions Made and Their Consequences

Several assumptions underpin this message, some valid and some problematic:

Assumption 1: The environment file would be correctly parsed. The assistant assumed that the EnvironmentFile directive in the systemd unit would load the variables exactly as written. This is generally true for systemd, but the real issue was that the variable names in the file didn't match what the application expected.

Assumption 2: The head node had the necessary dependencies. The assistant assumed that the head node, which already ran YugabyteDB, would have the CQL driver dependencies needed by the s3-proxy. The failure mode—"unable to discover protocol version"—suggests a network connectivity issue rather than a missing library, but the assumption that the proxy could connect to the local database without additional configuration proved incorrect.

Assumption 3: Manual SSH deployment was acceptable. This assumption is challenged in the very next user message (2079), where the user asks, "Why are you not doing this with ansible?" The assistant had been operating in a "debug and fix" mode, iterating rapidly with manual commands. But the user's question reframes the task: the project already has a sophisticated Ansible-based deployment system with roles for s3_frontend, kuri, and yugabyte. By working outside that system, the assistant was creating configuration that would be invisible to the playbook-driven deployment pipeline, potentially causing drift between the QA environment and the automated deployment.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

The FGW architecture: The three-tier design with stateless S3 proxies, kuri storage nodes, and YugabyteDB for metadata. The proxy's role is to route S3 requests to the correct backend node based on object metadata.
Systemd service units: The structure of unit files, EnvironmentFile directive, Restart policies, and the systemctl command sequence.
CQL and YugabyteDB: The proxy connects to the CQL interface of YugabyteDB to query the filecoingw_s3 keyspace for routing decisions.
The preceding debugging context: The cross-node read failures, the bitswap attempt, and the realization that a proxy layer is necessary.

Output Knowledge Created

This message creates several artifacts:

A systemd unit file at /etc/systemd/system/s3-proxy.service on the head node, defining the proxy as a managed service.
A symlink enabling the service for multi-user.target.
A failed service state that triggers the subsequent debugging of the environment variable mismatch.
A record of the deployment attempt in the conversation log, which becomes the basis for the user's intervention ("Why are you not doing this with ansible?"). More broadly, the message creates knowledge about the gap between the intended configuration and the actual application requirements—knowledge that will be immediately applied in the following messages when the assistant pivots to using the Ansible roles.

The Thinking Process: A Study in Iterative Debugging

The reasoning visible in the surrounding messages reveals a methodical approach to problem-solving. The assistant's thinking proceeds through distinct phases:

Observation: Cross-node S3 reads fail; kuri2 returns empty responses.
Diagnosis: Logs show "block was not found locally (offline)"—the block exists only on kuri1.
Hypothesis testing: Attempt bitswap peering between nodes. Result: still fails.
Architectural recognition: The S3 server is designed for local-only block access; a proxy layer is required.
Solution design: Deploy s3-proxy on the head node with backend node list and CQL connection.
Implementation: Create environment file, systemd unit, start service.
Failure detection: Service exits immediately with status 1.
Root cause analysis (next messages): Environment variable name mismatch. This cycle—observe, diagnose, hypothesize, test, recognize, design, implement, fail, analyze—is the essence of distributed systems debugging. Each iteration narrows the gap between the mental model of how the system should work and how it actually works.

Conclusion

Message 2076 is a snapshot of infrastructure in motion: a single command that encapsulates hours of debugging, architectural reasoning, and operational decision-making. Its failure is not a mistake but a discovery—the kind of discovery that only happens when theory meets practice. The s3-proxy didn't start, but the attempt revealed the configuration mismatch that needed to be fixed. And the user's subsequent question about Ansible usage prompted a shift from ad-hoc debugging to disciplined infrastructure automation, ultimately leading to a properly deployed and verified multi-node cluster. In the end, the proxy was deployed correctly via Ansible, the load test showed both nodes receiving traffic, and cross-node reads worked as the architecture intended. The failure in message 2076 was not a setback but a necessary step on the path to a working system.