The Missing Environment Variable: How a Cluster Topology Bug Revealed Architecture Tensions in a Distributed S3 System
Introduction
In distributed systems, the gap between architecture-as-designed and architecture-as-deployed often reveals itself through the most mundane of symptoms: an empty UI panel, a missing row in a dashboard, a silent null where data should be. This article examines a single message from an opencode coding session in which an assistant diagnosed and fixed a cluster topology visualization failure on a three-node Filecoin Gateway (FGW) distributed S3 storage cluster. The fix itself was trivial—adding an environment variable to two configuration files—but the reasoning behind it exposed deeper questions about architectural layering, configuration ownership, and the assumptions that accumulate when translating a roadmap into running infrastructure.
The Scene: A QA Cluster Comes to Life
The conversation leading up to this message documents the deployment of a QA test environment for FGW, a horizontally scalable S3-compatible storage system built on top of the Filecoin and IPFS ecosystems. The cluster spans three physical nodes: a head node running YugabyteDB (the distributed SQL and CQL database backing the system), and two "kuri" storage nodes that handle data ingestion, retrieval, and S3 API serving. The assistant had just resolved a series of "dirty migration" errors in the CQL schema migrations table—a common pain point when test suites leave database migrations in an inconsistent state—and both kuri daemons were finally running.
The user then reported a problem: the cluster topology visualization on the Web UI (accessible at port 9010) was not rendering. This was not a crash or an error message; the UI simply showed nothing, as if the cluster were empty.
The Subject Message: Diagnosis and Fix
The assistant's response, message index 2037, reads as follows:
I see - the ClusterTopology requires FGW_BACKEND_NODES environment variable to be set. This is typically set on S3 frontend proxies to know about backend kuri nodes. Since we're running kuri nodes directly (not separate frontends), we need to configure this.
>
Let me add the backend nodes configuration: `` ssh 10.1.232.83 "echo 'FGW_BACKEND_NODES=\"kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079\"' | sudo tee -a /data/fgw/config/settings.env && sudo systemctl restart kuri" ssh 10.1.232.84 "echo 'FGW_BACKEND_NODES=\"kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079\"' | sudo tee -a /data/fgw/config/settings.env && sudo systemctl restart kuri" ``
At first glance, this is a straightforward operational fix: identify a missing configuration parameter, apply it to both nodes, and restart the service. But the reasoning embedded in the first paragraph reveals a much richer story.
The Investigation Trail
To understand why this message was written, we must trace the investigation that preceded it. The user's initial report was terse: ":9010 cluster topology doesn't render". The assistant's first step was to probe the API directly, curling the Web UI's endpoint at http://127.0.0.1:9010/api/v0/topology. This returned HTML instead of JSON—a red flag suggesting the request was hitting a frontend route rather than an API handler. The assistant then tried the correct RPC endpoint, http://127.0.0.1:9010/rpc/v0, with a JSON-RPC payload requesting the RIBS.ClusterTopology method. This returned a valid JSON response, but the result was empty: {"proxies":[],"storageNodes":[],"dataFlows":[]}.
The assistant now had a precise symptom: the API was alive and responding correctly, but it had no data to report. The next step was to check the kuri logs for topology-related errors. The logs showed an unrelated error about S3 server construction, but nothing specifically about topology. At this point, the assistant made a key investigative decision: instead of guessing or restarting services, it read the source code.
Using a grep for ClusterTopology, the assistant located the implementation in /home/theuser/gw/rbstor/diag.go and read the relevant function. The code revealed that ClusterTopology() reads two environment variables: FGW_BACKEND_NODES and FGW_NODE_ID. If FGW_BACKEND_NODES is empty, the function returns an empty topology structure. This was the root cause.
The Reasoning: Architecture and Configuration Ownership
The assistant's comment—"This is typically set on S3 frontend proxies to know about backend kuri nodes. Since we're running kuri nodes directly (not separate frontends), we need to configure this"—is the intellectual core of the message. It reveals a critical architectural understanding.
The FGW system, as designed in its roadmap, has a three-layer architecture: stateless S3 frontend proxies that accept client requests, kuri storage nodes that manage data, and YugabyteDB as the shared metadata store. In this design, the FGW_BACKEND_NODES variable belongs on the proxy layer: it tells each proxy which kuri nodes exist so it can route requests to the correct backend. The kuri nodes themselves, in the canonical architecture, would not need this variable because they do not perform routing—they only serve their own local data.
However, the QA deployment had deviated from this architecture. Earlier in the session, the assistant had been running kuri nodes as direct S3 endpoints (each kuri exposed its own S3 API on port 8079), bypassing the proxy layer entirely. This was a pragmatic choice for a test environment, but it created a configuration gap: the cluster topology feature, which was designed to be populated by the proxy layer, had no data source when running on the kuri nodes directly.
The assistant's decision was to add FGW_BACKEND_NODES to the kuri nodes' settings anyway, effectively making each kuri node self-aware of the cluster. This was a workaround—a reasonable one for a QA environment—but it subtly changed the semantics of the configuration. The variable was now serving double duty: it was both a proxy routing table (its original purpose) and a cluster discovery mechanism for the topology UI.
Assumptions and Potential Mistakes
Several assumptions underpin this fix. The first is that the kuri nodes can meaningfully interpret the FGW_BACKEND_NODES variable in the same way a proxy would. The source code in diag.go reads the variable and uses it to build a topology response, but it does not use it for request routing. This means the topology UI would show both nodes, but cross-node S3 reads (where a client requests an object stored on the other node) would still fail unless additional routing logic was in place. The assistant had not yet addressed this cross-node routing problem—that would come later in the session when the s3-proxy was deployed.
A second assumption is that the environment variable approach is sufficient for cluster discovery. The topology function reads the variable at startup (via os.Getenv), meaning it captures a static snapshot of the cluster. If nodes are added or removed, the variable must be updated and the service restarted. In a production system, dynamic discovery via a shared registry (like the database itself) would be more robust.
A third, more subtle assumption is that the QA environment should mirror the production architecture. The assistant's comment "Since we're running kuri nodes directly (not separate frontends)" acknowledges the deviation but does not question whether this deviation is appropriate. In a QA context, running without proxies simplifies the deployment and reduces resource usage, but it also means the proxy layer's features (load balancing, request routing, caching) are not tested. This is a trade-off, not a mistake, but it is worth naming explicitly.
Input Knowledge Required
To understand this message, a reader needs several pieces of context. First, they need to know the FGW system's layered architecture: S3 proxies, kuri storage nodes, and YugabyteDB. Second, they need to understand that the cluster topology feature is a diagnostic visualization that shows which nodes are alive and healthy. Third, they need to know that environment variables are the primary configuration mechanism for this system, loaded from a settings.env file on each node. Fourth, they need to recognize the IP addresses: 10.1.232.83 is kuri1, 10.1.232.84 is kuri2, and port 8079 is the S3 API port on each node. Finally, they need to understand the JSON-RPC protocol used by the Web UI to communicate with the backend.
Output Knowledge Created
The message produced several concrete outputs. First, it updated the configuration on both kuri nodes by appending the FGW_BACKEND_NODES line to /data/fgw/config/settings.env. Second, it restarted both kuri daemons via systemctl restart, causing them to re-read their configuration. Third, it created a shared understanding (documented in the conversation) that the cluster topology feature depends on this variable and that the QA deployment's architecture differs from the canonical design.
The follow-up messages show that the fix worked: the topology API returned data with both nodes showing as healthy, and the Web UI rendered correctly. The assistant also verified that both kuri services were running properly after the restart.
The Thinking Process
The assistant's reasoning process, visible in the sequence of commands and comments, follows a classic debugging pattern. First, reproduce the symptom: curl the API endpoint and observe the empty result. Second, isolate the layer: determine that the API is responding but has no data, ruling out a connectivity or crash issue. Third, trace the data flow: read the source code to understand how the topology response is constructed. Fourth, identify the missing input: the environment variable FGW_BACKEND_NODES is checked at the top of the function. Fifth, apply the fix: add the variable to the configuration and restart.
What makes this reasoning noteworthy is the intermediate step where the assistant reads the source code rather than searching documentation or asking the user. This reflects a deep familiarity with the codebase and a debugging philosophy that prioritizes empirical evidence over speculation. The assistant also contextualizes the fix within the broader architecture, noting that the variable "is typically set on S3 frontend proxies" and explaining why the deviation exists.
Conclusion
Message 2037 is a small but revealing moment in a larger engineering effort. It shows how a missing environment variable—a trivial configuration detail—can expose the tension between an idealized architecture and a pragmatic deployment. The fix was correct and effective, but the reasoning behind it raises questions that deserve attention: Should the QA environment match the production architecture more closely? Should cluster discovery be dynamic rather than static? Should the topology feature work without the proxy layer? These are not bugs to be fixed but design decisions to be made, and the assistant's message provides the raw material for that conversation.
In the end, the cluster topology rendered, the user moved on to running load tests, and the assistant continued building. But the trace of reasoning left behind in this message—the source code reading, the architectural contextualization, the pragmatic workaround—is a small artifact of how distributed systems are built, one environment variable at a time.