The Commit That Fixed a Silent Cluster Topology Bug

"Committed as af20e44" — a single message that marks the end of a diagnostic journey and the beginning of a properly functioning distributed storage cluster.

The Message

[assistant] Committed as `af20e44`:

fix: cluster topology stats URL port extraction for remote nodes

**Summary:**
- Fixed the bug where cross-node stats weren't showing in the cluster topology
- The issue was hardcoded port replacement (`:8078` → `:9010`) that didn't work for kuri nodes using `:8079`
- Now properly extracts host and appends the web UI port regardless of original port

At first glance, this appears to be a simple status update — a developer confirming that a code change has been committed to version control. But this message is the capstone of a much deeper narrative: the story of a subtle, silent bug in a distributed storage system that caused each node in a two-node cluster to see only its own statistics while remaining completely blind to its peer. The bug was invisible to most monitoring, caused no crashes, and produced no error logs. It simply made the cluster topology view in the web UI lie — and it took careful forensic work to uncover.

The Bug: A Hardcoded Assumption

The root cause was deceptively simple. In the file rbstor/diag.go, the ClusterTopology() method was responsible for gathering health and usage statistics from every storage node in the cluster and presenting them in a unified view. When the node being queried was a remote peer (not the local node), the code needed to fetch that peer's stats via an HTTP call to the peer's web UI endpoint. The web UI served a convenient /api/stats endpoint on port 9010. But the node URLs stored in the configuration — the FGW_BACKEND_NODES environment variable — pointed to the S3 API port, which was 8079 for the kuri storage nodes in this deployment.

The original author of this code had anticipated this mismatch and written a workaround:

statsURL := strings.Replace(nodeURL, ":8078", ":9010", 1) + "/api/stats"

The logic was straightforward: take the node URL (which was assumed to end with :8078), replace that port with :9010, and append the stats path. This would have worked perfectly if every storage node in every deployment used port 8078 for its S3 API. But in this particular cluster, the kuri nodes had been configured to use port 8079 — a one-digit difference that rendered the replacement a no-op. The strings.Replace call looked for :8078, found :8079 instead, performed zero substitutions, and returned the original URL unchanged. The code then appended /api/stats to http://10.1.232.83:8079, producing a URL that pointed to the S3 API endpoint, not the web UI. The S3 endpoint had no /api/stats handler, so the HTTP request either failed silently or returned garbage data. Either way, the remote node's stats were recorded as zero — zero storage used, zero groups, zero requests per second.

The cluster topology table in the web UI therefore showed each node's own data correctly (because the local node's stats were gathered directly from in-memory counters) but showed zeros for every other node. To a human operator looking at the dashboard, it would appear that the cluster was functioning but that the other node was empty — a misleading but not obviously broken state. No errors were logged, no crashes occurred, and the S3 proxy continued routing requests to both backends correctly. The only symptom was a silent gap in the observability layer.

The Diagnostic Journey

The user first noticed the anomaly not through automated alerting but through visual inspection of the web UI: "I still don't see kuri_1 stats from 2, and don't see kuri_1 stats from kuri_2." This was a regression — earlier in the session, cross-node stats had been visible. Something had broken.

The assistant's response was methodical. First, it confirmed the symptom by querying the ClusterTopology RPC endpoint from both nodes and comparing the results. From kuri_01's perspective, kuri_02 showed storageUsed: 0 and groupsCount: 0. From kuri_02's perspective, kuri_01 showed the same zeros. Each node could see itself perfectly but was blind to its peer. This ruled out a network-level issue — if both nodes could reach each other's RPC endpoint (port 9010) and both returned valid topology data, the problem had to be in how the remote stats were being fetched.

The assistant then traced the code path. It located the relevant source files — rbstor/diag.go for the topology implementation and integrations/web/rpc.go for the RPC layer — and read the critical section. The hardcoded port replacement jumped out immediately. The comment in the code even acknowledged the brittleness: "The web UI is served on port 9010 by default, but we need to use the same host. The nodeURL points to the S3 API port (8078), so we need to replace the port. For now, use the S3 API host but different port. Note: In production, nodes should expose stats on a known port."

The next step was to verify the actual port in use. The assistant checked the configuration on both kuri nodes by SSHing in and reading the FGW_BACKEND_NODES environment variable. The value confirmed the mismatch: kuri_01:http://10.1.232.83:8079,kuri_02:http://10.1.232.84:8079. Port 8079, not 8078. The hardcoded replacement was failing silently.

The Fix: Robust Host Extraction

The fix was not merely to change :8078 to :8079 in the replacement string — that would have been equally brittle and would break again if any future deployment used a different port. Instead, the assistant implemented a more robust approach: extract the host portion of the node URL (everything before the port, or the hostname/IP alone) and construct the stats URL by appending the known web UI port (:9010) and path. This way, the code works regardless of what port the S3 API is configured to use, as long as the web UI is consistently on port 9010.

The actual edit, visible in the git diff shown in message 2134, replaced the old comment-and-replacement block with a cleaner implementation. The old code had three lines of comments explaining the workaround and one line of actual logic. The new code removed the misleading comments and replaced the fragile strings.Replace call with a host extraction approach. The diff shows 9 insertions and 5 deletions — a small change with significant impact.

The Commit

After making the edit, the assistant rebuilt the binary using make kuboribs, deployed it to both kuri nodes via scp and systemctl restart, and verified that the fix worked by querying the ClusterTopology endpoint again. Both nodes now reported each other's stats correctly: kuri_01 showed 2.74 GB on kuri_02, and kuri_02 showed 712 MB on kuri_01. The cluster topology was finally accurate.

Only then did the assistant commit the change. The commit message — "fix: cluster topology stats URL port extraction for remote nodes" — follows the conventional commit format with a concise subject line and a multi-line body explaining the problem, the root cause, and the fix. The commit body reads:

The ClusterTopology endpoint was hardcoded to replace port 8078 with 9010 when fetching stats from remote nodes, but kuri nodes use port 8079. This caused cross-node stats to not be displayed in the web UI. Fix by extracting the host portion and appending the correct web UI port (9010) regardless of the original S3 API port.

This commit, af20e44, is now part of the project's history on the pgf-port branch, five commits ahead of the upstream magik/pgf-port branch.

Why This Message Matters

The message itself is brief — just three bullet points and a commit hash. But it represents the successful conclusion of a complete debugging cycle: symptom identification, hypothesis formation, code inspection, root cause confirmation, fix implementation, build and deployment, verification, and version control. Each of these steps required different kinds of knowledge and tooling.

The input knowledge required to understand this message includes: familiarity with Go's strings.Replace semantics, understanding of how distributed storage clusters expose health metrics, knowledge of the project's port conventions (8078 for S3 proxy, 8079 for kuri S3 API, 9010 for web UI), and familiarity with git workflows. Without this context, the message reads as a trivial port fix. With it, the message reveals itself as the resolution of a subtle architectural mismatch that could have undermined operator trust in the monitoring dashboard.

The output knowledge created by this message is both technical and procedural. Technically, the fix ensures that the cluster topology view in the web UI accurately reflects the state of all nodes, enabling operators to monitor storage utilization, request rates, and group counts across the entire cluster from any single node's dashboard. Procedurally, the commit creates a permanent, auditable record of the bug and its resolution, complete with a rationale that future developers can consult if a similar issue arises. The commit also serves as a boundary marker: before af20e44, the cluster topology was unreliable; after it, the topology is trustworthy.

Assumptions and Lessons

The original code made a brittle assumption: that all storage nodes in all deployments would use port 8078 for their S3 API. This assumption was baked into a single line of string replacement, with no validation, no fallback, and no error handling if the replacement produced no change. The comment in the code acknowledged the fragility — "Note: In production, nodes should expose stats on a known port" — but the implementation never addressed it.

The fix makes a different assumption: that the web UI port (9010) is consistent across all nodes. This is a safer assumption because the web UI port is typically configured centrally and is less likely to vary between nodes. But it is still an assumption. A truly robust solution might involve storing the stats endpoint URL explicitly in the configuration, or using a service discovery mechanism, or falling back to probing multiple ports. The fix chosen is pragmatic — it solves the immediate problem without over-engineering, and it is significantly more resilient than the original.

Conclusion

Message 2136 is a small message about a small commit, but it encapsulates the essence of debugging distributed systems: a silent symptom, a careful trace through unfamiliar code, a single wrong digit, and a fix that makes the system tell the truth again. The commit af20e44 will likely never be noticed by end users — they interact with the S3 API, not the cluster topology dashboard. But for the operators managing the cluster, the difference between seeing zeros and seeing real data is the difference between flying blind and having a functioning instrument panel. Sometimes the most important fixes are the ones that restore visibility.