The 8078-to-9010 Trap: Diagnosing a Silent Port Substitution Bug in Distributed Cluster Topology

Introduction

In the complex orchestration of a distributed storage system, few things are as deceptively fragile as the mechanism by which nodes learn about each other. This article examines a single message from a coding session for the Filecoin Gateway (FGW) project—a horizontally scalable S3-compatible storage layer built on top of the Filecoin network. The message, delivered by an AI assistant acting as a technical operator and debugger, captures the moment when a subtle but critical regression was diagnosed: a hardcoded port substitution in the cluster topology code was silently preventing kuri storage nodes from seeing each other's statistics.

The message is notable not because it fixes the bug—it doesn't—but because it demonstrates the precise, evidence-driven reasoning required to locate a failure that had no error logs, no crash reports, and no obvious symptoms beyond a single number showing "0" where it shouldn't have been. It is a masterclass in reading code against configuration, and it reveals how assumptions baked into early development can quietly break when deployed to a real multi-node cluster.

The Message in Full

The assistant wrote:

Found the issue! On line 263, the code tries to fetch remote stats:

``go statsURL := strings.Replace(nodeURL, ":8078", ":9010", 1) + "/api/stats" ``

But the kuri nodes use port 8079 (internal S3), not 8078. So the port replacement fails to work. Let me verify what ports are configured:

```bash # Check the FGW_BACKEND_NODES config on kuri nodes echo "=== kuri1 FGW_BACKEND_NODES ===" ssh 10.1.232.83 "grep FGW_BACKEND_NODES /data/fgw/config/settings.env"

echo "" echo "=== kuri2 FGW_BACKEND_NODES ===" ssh 10.1.232.84 "grep FGW_BACKEND_NODES /data/fgw/config/settings.env"

echo "" echo "=== Test /api/stats endpoint ===" curl -s http://10.1.232.83:9010/api/stats | jq . curl -s http://10.1.232.84:9010/api/stats | jq . ```

``` === kuri1 FGW_BACKEND_NODES === grep: /data/fgw/config/settings.env: Permission denied

=== kuri2 FGW_BACKEND_NODES === grep: /data/fgw/config/settings.env: Permission denied

=== Test /api/stats endpoint === { "nodeId": "kuri_01", "groupsCount": 1, "storageUsed": 2743416004, "requestsPerSecond": 0 } { "nodeId": "kuri_02", "groupsCount": 1, "storageUsed": 712452144, "requestsPerSecond": 1.4 } ```

Context: What Led to This Message

To understand why this message was written, we must trace the events that immediately preceded it. The user had deployed a three-node QA cluster for the FGW system: a head node running YugabyteDB and the S3 frontend proxy, plus two kuri storage nodes (kuri_01 and kuri_02). The cluster was functionally working—load tests passed, S3 reads and writes succeeded, and the web UI rendered a cluster topology page. But the user noticed something wrong: the "Storage Nodes" table on each node's dashboard showed zero storage used for the other node. Kuri_01 could see its own 2.7 GB of storage but reported 0 GB for kuri_02, and kuri_02 saw its own 712 MB but nothing for kuri_01.

This was described as a "regression"—implying that cross-node statistics had worked at some earlier point and had since broken. The user's observation was precise: "I still don't see kuri_1 stats from 2, and don't see kuri_1 stats from kuri_2." The assistant, after confirming the topology endpoint returned data for both nodes' local stats, began investigating the code path responsible for gathering remote node information.

The investigation led to rbstor/diag.go, which contains the ClusterTopology() diagnostic function. On line 263, the code constructs a URL to fetch statistics from a peer node by taking the node's S3 endpoint URL and replacing the port :8078 with :9010, then appending /api/stats. The assumption embedded in this line is that every node's S3 endpoint uses port 8078, and that the web UI (which serves the stats API) always lives on port 9010.

Why the Bug Exists: Architecture Mismatch

The root cause is a mismatch between the architecture assumed by the code and the architecture actually deployed. In the FGW system, there are two distinct tiers:

S3 Frontend Proxy (running on the head node, port 8078): This is the stateless public-facing endpoint that routes S3 requests to the appropriate kuri backend.
Kuri Storage Nodes (running on kuri_01 and kuri_02, internal S3 on port 8079): These are the actual storage servers that hold data and serve the web UI on port 9010. The FGW_BACKEND_NODES environment variable, which tells each node about its peers, contains URLs pointing to the kuri nodes' internal S3 endpoints—for example, http://10.1.232.83:8079. When ClusterTopology() tries to fetch remote stats, it takes this URL and performs strings.Replace(nodeURL, ":8078", ":9010", 1). Since the URL contains :8079 and not :8078, the replacement does nothing. The resulting URL is http://10.1.232.83:8079/api/stats—which hits the S3 endpoint, not the web UI, and likely returns an error or empty response that is silently ignored. The code was likely written and tested in an earlier phase of development when the architecture was simpler—perhaps when kuri nodes served both S3 and the web UI on the same port, or when the S3 proxy was not yet separated from the storage nodes. When the architecture was refactored to introduce the stateless S3 frontend (as documented in the session's earlier segments), the port substitution logic was not updated to match the new port assignments.

The Thinking Process Visible in the Message

The assistant's reasoning follows a clear diagnostic pattern. First, it identifies the exact line of code responsible for constructing the remote stats URL. Second, it articulates the mismatch: the code searches for :8078, but the kuri nodes use :8079. Third, it proposes to verify the hypothesis by checking the actual FGW_BACKEND_NODES configuration on both nodes and by testing the /api/stats endpoint directly.

The verification commands are carefully chosen. The grep for FGW_BACKEND_NODES would confirm what URLs the nodes hold for their peers. The direct curl calls to :9010/api/stats on both nodes serve a dual purpose: they prove that the stats endpoint is alive and returning correct data (kuri_01 shows 2.7 GB, kuri_02 shows 712 MB), and they implicitly demonstrate that the correct URL pattern is http://<ip>:9010/api/stats—not the mangled result of the failed port substitution.

Interestingly, the grep commands fail with "Permission denied" because the settings file is owned by the fgw user and not readable by the SSH session's default user. This is itself a data point: it confirms that the configuration files are properly restricted, but it also means the assistant cannot directly confirm the FGW_BACKEND_NODES value through this command. The assistant does not follow up with a sudo version of the grep—perhaps because the direct stats test already provides sufficient evidence for the diagnosis.

Assumptions and Their Consequences

The message reveals several assumptions, some correct and some incorrect:

Correct assumption: The /api/stats endpoint on port 9010 is the source of truth for per-node statistics. The curl tests confirm this—both nodes respond with valid JSON containing nodeId, groupsCount, storageUsed, and requestsPerSecond.

Correct assumption: The port substitution is failing silently. The strings.Replace function in Go, when the search string is not found, returns the original string unchanged. This means the code does not crash or log an error—it simply produces a wrong URL and gets a wrong (or empty) response.

Implicit assumption (potentially incorrect): The bug is solely in the port number. The assistant assumes that if the port were :8078 instead of :8079, the substitution would work correctly. However, there is a deeper architectural question: should the code be doing string replacement on URLs at all? A more robust approach would be to store the web UI port explicitly in the node configuration, or to derive it from a well-known base URL rather than attempting to mutate an S3 endpoint URL.

Implicit assumption: The regression is recent. The user described it as a regression, and the assistant accepts this framing. However, it is equally possible that cross-node statistics never worked in this deployment—that the port substitution had always been broken for the kuri nodes, and the user only noticed it now because they were actively monitoring the dashboard.

Input Knowledge Required to Understand This Message

A reader needs several pieces of context to fully grasp this message:

The FGW architecture: Understanding that there are S3 frontend proxies (port 8078) and kuri storage nodes (port 8079 for S3, port 9010 for web UI) is essential. Without this, the significance of the port numbers is lost.
The FGW_BACKEND_NODES environment variable: This variable tells each node about its peers. It contains URLs like http://10.1.232.83:8079 and http://10.1.232.84:8079. The code in diag.go iterates over these URLs to fetch remote stats.
Go's strings.Replace behavior: The function replaces occurrences of a substring. If the substring is not found, it returns the original string unchanged. This is why the bug is silent—no error, no panic, just a wrong URL.
The cluster topology feature: The web UI has a "Storage Nodes" table that should display aggregated statistics from all nodes. The user noticed that this table showed zeros for remote nodes.
The session history: Earlier in the session, the architecture was restructured to separate the S3 proxy from the kuri nodes. This restructuring changed port assignments but did not update all code paths that referenced ports.

Output Knowledge Created by This Message

This message produces several valuable pieces of knowledge:

A precise bug location: Line 263 of rbstor/diag.go is identified as the source of the regression. The specific code strings.Replace(nodeURL, ":8078", ":9010", 1) is quoted.
A clear explanation of the failure mode: The port substitution fails because kuri nodes use port 8079, not 8078. The replacement is a no-op, producing a URL that hits the S3 endpoint instead of the web UI stats endpoint.
Confirmation that the stats endpoints work: The curl tests prove that both nodes' /api/stats endpoints are functional and return correct data. The problem is purely in the discovery mechanism, not in the data source.
A configuration access issue: The grep commands fail with "Permission denied," revealing that the settings files are not readable by the default SSH user. This is a minor operational note—the assistant would need to use sudo to read those files.
A template for the fix: By implication, the fix is either to change the port in the substitution from :8078 to :8079, or to redesign the stats URL construction to use a dedicated configuration field rather than string manipulation. The message does not prescribe the fix, but it makes the path forward clear.

Broader Significance: Silent Failures in Distributed Systems

This bug is a textbook example of a class of failures that plague distributed systems: the silent data gap. No error is logged, no crash occurs, no timeout is reported. The system simply returns incomplete data—zeros where real numbers should be. A user monitoring the dashboard might not notice for days or weeks, assuming that the other node is idle or empty. Only a careful observer who knows what values to expect would spot the discrepancy.

The root cause—a hardcoded port number that diverged from the actual deployment—is equally archetypal. In any system that evolves over time, port numbers, file paths, and environment variable names are frequent sources of drift. The code was written when the architecture used port 8078 for kuri nodes. Later, the architecture was refactored to introduce a separate S3 proxy on 8078 and move the kuri S3 endpoint to 8079. The port substitution code was not updated because it was not part of the visible change—it was a hidden dependency.

This is why comprehensive integration tests and staging environments are critical. A unit test of the ClusterTopology() function might mock the HTTP calls and never catch the port mismatch. An integration test that deploys a real two-node cluster and checks the topology page would have caught it immediately—but only if someone thought to check that specific display value.

Conclusion

The assistant's message at index 2114 is a small but perfect example of diagnostic reasoning in a complex distributed system. It begins with a user observation ("I don't see stats from the other node"), traces the code path to a specific line, identifies the mismatch between the code's assumptions and the deployed reality, and verifies the hypothesis with direct endpoint tests. It does not fix the bug, but it provides everything needed for a fix: the location, the cause, and the evidence.

The message also serves as a cautionary tale about hardcoded values and silent failures. The strings.Replace call on line 263 of diag.go is innocuous in isolation, but in the context of an evolving architecture, it becomes a time bomb—waiting for a port number to change and silently breaking a feature that no one thinks to verify. The lesson for developers and operators is clear: when you change a port, a path, or a URL pattern, search for every place that value appears in code, configuration, and documentation. The silent bugs are the ones that hurt the most.