The Configuration That Wasn't: Debugging an S3 Proxy Deployment on a Distributed Storage Cluster

The Message

In a session dedicated to deploying and validating a QA test cluster for the Filecoin Gateway (FGW) distributed storage system, the assistant sent the following message:

[assistant] [bash] ssh 10.1.232.82 "sudo journalctl -u s3-proxy -n 20 --no-pager"
Jan 31 22:35:55 fgw-qa-head systemd[1]: Started s3-proxy.service - FGW S3 Proxy.
Jan 31 22:35:55 fgw-qa-head s3-proxy[20266]: Failed to connect to YCQL: create cql migrate session: gocql: unable to create session: unable to discover protocol version: dial tcp [::1]:9042: connect: connection refused
Jan 31 22:35:55 fgw-qa-head systemd[1]: s3-proxy.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 22:35:55 fgw-qa-head systemd[1]: s3-proxy.service: Failed with result 'exit-code'.
J...

At first glance, this appears to be nothing more than a routine diagnostic command — a developer checking why a freshly deployed service failed to start. But within this single message lies a microcosm of the entire debugging process: the moment when a carefully constructed configuration meets the unforgiving reality of what the code actually expects. This message captures the precise instant when an assumption about environment variable naming collapses, and the real work of diagnosis begins.

The Context: Building a Multi-Node Distributed Storage Cluster

To understand why this message matters, one must understand what was being built. The team was deploying a QA test environment for the FGW distributed storage system across three physical nodes: a head node at 10.1.232.82 running YugabyteDB (the distributed SQL/CQL database), and two storage nodes — kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84 — each running the "kuri" daemon that handles S3-compatible object storage.

The architecture followed a deliberate design: stateless S3 frontend proxies sitting in front of kuri storage nodes, which in turn connect to a shared YugabyteDB backend. This three-layer hierarchy (S3 proxy → Kuri nodes → YugabyteDB) was the result of earlier architectural corrections, where the team had realized that running kuri nodes as direct S3 endpoints violated the roadmap's requirement for separate stateless proxy nodes.

The immediate problem that led to this message was cross-node object reads. When a client wrote an object to kuri1, the object's metadata (including which node stored it) was recorded in the shared S3 CQL keyspace in YugabyteDB. But when a client tried to read that same object from kuri2, kuri2 could see the metadata — it knew the object existed and was on kuri1 — but could not retrieve the actual block data. The error message was telling: "block was not found locally (offline)." Each kuri node only served data from its local blockstore, with no mechanism to fetch blocks from peer nodes.

The solution was the s3-proxy: a frontend proxy that would sit on the head node, accept all S3 requests on port 8078, consult the shared S3 metadata to determine which backend node owned each object, and route the request accordingly. This proxy needed its own configuration — particularly the CQL connection details to read the shared metadata — and that configuration is precisely where things went wrong.

The Assumption and Its Failure

In the message immediately preceding the subject message (index 2076), the assistant had deployed the s3-proxy with careful ceremony: creating a configuration file at /opt/fgw/config/s3proxy.env, writing a systemd service file that loaded this environment file, enabling and starting the service. The configuration file included this critical line:

RIBS_S3_CQL_HOSTS="10.1.232.82"

This was the assumption: that the s3-proxy binary would read its CQL host configuration from the environment variable RIBS_S3_CQL_HOSTS. It was a reasonable assumption — the kuri nodes themselves used a similar naming convention (RIBS_S3_CQL_HOSTS appeared in their configuration). The assistant had even tested the binary earlier with RIBS_LOGLEVEL and observed it attempting to connect to CQL, confirming that environment variables were the mechanism.

But the subject message reveals the failure. The journalctl output shows:

Failed to connect to YCQL: create cql migrate session: gocql: unable to create session: unable to discover protocol version: dial tcp [::1]:9042: connect: connection refused

The critical detail is [::1]:9042 — IPv6 localhost, port 9042. Despite the configuration file specifying RIBS_S3_CQL_HOSTS="10.1.232.82", the s3-proxy was attempting to connect to localhost. The environment variable was being ignored entirely.

The Diagnostic Power of Journalctl

This message is a textbook example of why structured logging and centralized log management matter in distributed systems. The assistant didn't guess at the problem or blindly restart the service. Instead, they used journalctl — systemd's log collector — to retrieve the last 20 lines of the s3-proxy service's output. The -u s3-proxy flag filtered for that specific unit, -n 20 limited to the most recent entries, and --no-pager ensured the output was suitable for programmatic consumption.

The log output told a precise story in four lines:

1. Service started: systemd launched the s3-proxy process.
2. Connection failure: The process attempted to connect to YCQL (Yugabyte CQL) and failed with a specific error: connection refused on [::1]:9042.
3. Process exited: The main process terminated with exit code 1.
4. Service failed: systemd recorded the failure result. The connection refused error was particularly informative. It wasn't a network timeout or an authentication failure — it was a connection refused, meaning the TCP handshake was actively rejected. This could mean YugabyteDB wasn't running on the head node (it was), or that the s3-proxy was looking in the wrong place (it was). The [::1] address — IPv6 loopback — was the smoking gun. The s3-proxy was defaulting to localhost instead of using the configured host.

The Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

System administration: Understanding systemd units, service management (systemctl), and log inspection (journalctl) is essential. The message assumes familiarity with the concept of services that can fail, restart policies, and the distinction between a process exiting and a service being marked as failed.

Distributed database concepts: The error mentions "YCQL" — Yugabyte CQL, a Cassandra-compatible query language interface. Understanding that the s3-proxy needs to connect to a database to read routing metadata is crucial. The port 9042 is the standard CQL port for both Cassandra and YugabyteDB.

S3 architecture: The message sits within a larger narrative about building an S3-compatible storage system. The s3-proxy exists because of a deliberate architectural choice to separate the stateless routing layer from the stateful storage layer — a pattern familiar to anyone who has worked with distributed object stores.

Environment variable configuration: The entire debugging arc hinges on how the s3-proxy reads its configuration. The assistant assumed RIBS_S3_CQL_HOSTS was the correct variable, but the code actually reads FGW_YCQL_HOSTS. This is a subtle but critical distinction that requires understanding how Go applications typically read configuration from environment variables.

Network debugging: The ability to look at [::1]:9042 and immediately recognize it as an IPv6 localhost address — and to understand that this means the configured host was being ignored — is a skill born from experience with network diagnostics.

The Output Knowledge Created

This message, despite being a report of failure, created valuable knowledge:

Negative knowledge: The s3-proxy, as deployed, does not connect to the specified CQL host. This is a clear signal that either the configuration file is not being read correctly, or the environment variable name is wrong. The next message (index 2078) would confirm the latter: the code reads FGW_YCQL_HOSTS, not RIBS_S3_CQL_HOSTS.

System state knowledge: The head node at 10.1.232.82 has a running systemd service for s3-proxy, but it's in a failed state with auto-restart enabled (the status showed "activating (auto-restart)"). This means the service will keep trying and failing every 5 seconds, consuming minimal resources but generating log noise.

Architectural knowledge: The error confirms that the s3-proxy does attempt to connect to CQL at startup — it's not a lazy connection that only happens on request. This means the proxy validates its database connectivity at startup, which is a sensible design for a routing layer that cannot function without access to routing metadata.

The Thinking Process Revealed

The subject message reveals a methodical, hypothesis-driven debugging approach. The assistant didn't just deploy the service and move on — they immediately verified it was running, observed it was in a failed state, and then retrieved the logs to understand why. This is the scientific method applied to infrastructure: hypothesis (the s3-proxy will start and serve requests), experiment (deploy and start the service), observation (it failed), analysis (check logs).

The choice of journalctl over simply checking the service status is telling. systemctl status would have shown the service was failing, but only journalctl could reveal why. The assistant went straight for the diagnostic tool that would provide the most information, not the one that would confirm the obvious.

The log output itself tells a story of progressive narrowing. The first line establishes that the service started. The second line identifies the specific failure point: CQL connection. The third and fourth lines are the consequence — process exit and service failure. An experienced operator reads these four lines and immediately knows where to look next: the CQL connection configuration.

The Broader Significance

This message, for all its brevity, captures something essential about distributed systems engineering: the gap between what we intend to configure and what the system actually reads. The assistant wrote RIBS_S3_CQL_HOSTS=10.1.232.82 with the full intention of telling the s3-proxy where to find YugabyteDB. But the code read FGW_YCQL_HOSTS — a different variable entirely. The configuration file was correct in its content but wrong in its naming.

This is not a trivial mistake. It reflects the challenge of maintaining consistent naming conventions across a codebase where different components may have been written by different developers, at different times, with different assumptions about naming patterns. The kuri nodes used RIBS_* prefixes; the s3-proxy, it turned out, used FGW_* prefixes. These inconsistencies are the friction of real-world software development — the kind that automated tests catch late and that manual debugging catches only after a service fails to start.

The message also demonstrates the importance of fast feedback loops in infrastructure work. The assistant deployed the service and checked its status within seconds. The failure was detected immediately, not hours later when a user tried to access the S3 endpoint and found nothing responding. This rapid validation — deploy, verify, diagnose, fix — is the hallmark of effective infrastructure engineering.

Conclusion

A single journalctl command, four lines of log output, and a connection refused error on the wrong address. On its surface, message 2077 appears to be a routine diagnostic check — the kind of message that scrolls past in terminal output dozens of times a day. But within the context of building a distributed storage cluster across three physical nodes, with a carefully architected three-layer hierarchy and a complex configuration surface, this message represents the critical moment when an assumption was tested against reality and found wanting. It is the pivot point between a failed deployment and the discovery that would lead to a fix: changing RIBS_S3_CQL_HOSTS to FGW_YCQL_HOSTS. The message itself doesn't contain the fix — that comes in the next message — but it contains the discovery that makes the fix possible. In the archaeology of a debugging session, this is the layer where the truth first emerges.