The Verification That Changes Everything: Cross-Node Stats Communication in a Distributed S3 Architecture

Introduction

In the sprawling complexity of building a horizontally scalable S3-compatible storage system, the smallest verification steps often carry the most weight. Message 894 in this coding session is a deceptively simple exchange: two wget commands executed inside Docker containers, each fetching a JSON payload from a /api/stats endpoint. On its surface, it reads like a routine sanity check—the kind of "did it work?" moment that fills countless engineering logs. But beneath this brevity lies a pivotal architectural verification that validates weeks of design decisions, exposes the fundamental challenge of distributed monitoring, and confirms that a critical cross-node communication channel is operational.

This article examines that single message in detail, unpacking the reasoning that led to it, the assumptions it tests, the knowledge it produces, and the architectural principles it quietly confirms.

The Full Message

[assistant] The stats endpoint works. Now let me test cross-node communication: [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-1 wget -q -O- http://kuri-2:9010/api/stats 2>&1 {"nodeId":"kuri-2","groupsCount":1,"storageUsed":19524331141,"requestsPerSecond":0}

Context: The Problem That Led Here

To understand why this message exists, we must trace the problem it solves. The session's broader context is the construction of a three-layer horizontally scalable S3 architecture: stateless S3 frontend proxies at the top, independent Kuri storage nodes in the middle, and a shared YugabyteDB at the bottom. The assistant had built a monitoring dashboard in React that displayed a cluster topology—showing both S3 proxy nodes and Kuri storage nodes with live metrics.

The problem emerged when the assistant tested round-robin request distribution. Traffic was being evenly distributed between kuri-1 and kuri-2 (the logs confirmed alternating backend selections), yet the monitoring UI showed only kuri-1's metrics. The dashboard was effectively blind to half the cluster. This is a classic distributed systems monitoring challenge: each node only knows its own state, and without a mechanism to aggregate data from peers, the view is incomplete.

The root cause was in the ClusterTopology() method in rbstor/diag.go. The implementation only populated metrics for the local node—when nodeID == selfNodeID. For remote nodes, it returned zero values. The system had no mechanism to ask a peer "what are your stats?" The assistant recognized this and set out to build exactly that mechanism.

The Solution: A Lightweight Stats Endpoint

The assistant's approach was pragmatic. Rather than implementing a full distributed metrics aggregation system with push-based telemetry or a shared metrics database, they chose to add a simple HTTP endpoint to each node: /api/stats. This endpoint returns a small JSON payload containing four fields: nodeId, groupsCount, storageUsed, and requestsPerSecond. The implementation was straightforward—a handler registered on the existing web server mux in integrations/web/ribsweb.go.

This design choice reveals several assumptions and trade-offs. First, it assumes that HTTP is an acceptable protocol for inter-node communication within the Docker network. Given that the nodes are co-located in a Docker Compose environment, this is reasonable—latency is low, and the overhead of HTTP is negligible. Second, it assumes that the stats payload is small enough that serialization and network transfer costs are trivial. The four-field JSON response confirms this. Third, it assumes that each node knows the addresses of its peers—an assumption satisfied by the FGW_BACKEND_NODES environment variable already configured in the docker-compose file.

The assistant also had to work around an interface limitation. The RIBS interface in iface/iface_ribs.go did not expose a NodeID field. Rather than modifying the interface (which would require updating all implementations), the assistant read the FGW_NODE_ID environment variable directly using os.Getenv. This is a pragmatic shortcut, but it introduces a subtle coupling between the web handler and the deployment environment—a coupling that already existed elsewhere in the codebase.

The Verification

Message 894 is the verification step. The assistant first confirmed that the endpoint works locally on kuri-1 (in the preceding message, index 893), receiving:

{"nodeId":"kuri-1","groupsCount":1,"storageUsed":19446967194,"requestsPerSecond":0}

Then, in the subject message, the assistant tests cross-node communication by having kuri-1 fetch stats from kuri-2:

{"nodeId":"kuri-2","groupsCount":1,"storageUsed":19524331141,"requestsPerSecond":0}

Both requests succeed. The JSON is well-formed. The nodeId fields correctly identify each node. The storageUsed values differ slightly (19,446,967,194 bytes vs. 19,524,331,141 bytes), which is expected for independent storage nodes with their own data directories. The requestsPerSecond is zero on both, which makes sense—no test traffic was generated during this verification.

This is not merely a "it works" moment. It is a multi-layered verification:

  1. The endpoint compiles and runs: The code changes to ribsweb.go compiled successfully and the Docker image was rebuilt.
  2. The endpoint is reachable: The HTTP server on port 9010 correctly routes /api/stats to the handler, despite the catch-all / pattern also being registered.
  3. The handler returns valid JSON: The response is parseable and contains the expected fields with correct types.
  4. Cross-node networking works: kuri-1 can resolve the hostname kuri-2 and reach port 9010 inside the Docker network. This validates the Docker Compose service naming and internal networking.
  5. Each node reports its own identity: The nodeId values are correct, confirming that each node reads its own FGW_NODE_ID environment variable.
  6. The data is plausible: The storageUsed values are in the expected range (approximately 19.5 GB), consistent with the CAR files uploaded during earlier testing.

What This Enables

With this verification, the assistant can now update ClusterTopology() to call /api/stats on each remote node, aggregating the results into a complete cluster view. The monitoring dashboard will finally show metrics for all storage nodes, not just the one the user happens to be connected to. This transforms the dashboard from a single-node debug view into a genuine cluster monitoring tool.

The architectural pattern established here—a lightweight HTTP endpoint for node-local stats, consumed by a peer-aggregation mechanism—is a microcosm of how distributed systems solve the observability problem. Each node is an independent source of truth about its own state. Aggregation happens at query time, not at write time. There is no centralized metrics database, no push-based telemetry agent, no complex pipeline. Just a simple HTTP request and a JSON response.

Assumptions and Potential Pitfalls

The assistant's approach makes several assumptions that deserve examination:

Assumption 1: All nodes are mutually reachable via HTTP. In a Docker Compose environment with a shared network, this is reliable. In a production deployment with nodes spread across regions or cloud providers, this might require additional networking configuration, TLS, or authentication.

Assumption 2: The stats endpoint is fast enough to not block the dashboard. The ClusterTopology() method will make N HTTP requests (one per remote node) before returning. If any node is slow or unreachable, the dashboard could hang. The current implementation doesn't include timeouts or parallel requests—a potential improvement.

Assumption 3: The environment variable FGW_NODE_ID is always set. The handler reads this variable directly. If a node is started without it, the nodeId field would be empty, potentially causing confusion in the dashboard.

Assumption 4: The stats data is current. The endpoint returns instantaneous values. If a request is in flight when stats are fetched, it might not be reflected. For a monitoring dashboard polling every few seconds, this is acceptable, but it's not suitable for billing-accurate accounting.

The Thinking Process

The assistant's reasoning, visible in the preceding messages, follows a clear arc:

  1. Observation: The UI shows traffic only on kuri-1, but logs confirm round-robin distribution to both nodes.
  2. Diagnosis: ClusterTopology() only fills local stats; remote nodes show zeros.
  3. Design decision: Add a lightweight HTTP stats endpoint rather than implementing a full RPC-based metrics system.
  4. Implementation: Add handler to ribsweb.go, rebuild Docker image, restart containers.
  5. Verification (message 893): Test the endpoint locally on kuri-1.
  6. Verification (message 894): Test cross-node communication from kuri-1 to kuri-2. The decision to use HTTP rather than the existing WebSocket RPC system is notable. The assistant had previously used websocat to test RPC methods like RIBS.IOThroughput. But for simple stats retrieval, HTTP GET is simpler to implement, easier to debug (as evidenced by the wget commands), and doesn't require the WebSocket handshake overhead. This is a pragmatic engineering choice that prioritizes simplicity and debuggability over architectural purity.

Conclusion

Message 894 is a small but significant milestone. It confirms that a distributed system's components can communicate their individual states to each other, laying the groundwork for aggregated observability. The two JSON responses—each under 100 bytes—represent a successful test of networking, routing, serialization, and identity management across independent storage nodes. For the engineer reading the logs, those two lines of JSON are worth more than a thousand lines of unit tests: they prove that the system behaves correctly in its actual deployment environment, with real data, real networking, and real concurrency.

In the broader narrative of building a horizontally scalable S3 architecture, this message marks the transition from "each node is an island" to "the cluster can see itself." It is the moment the monitoring system graduates from a single-node debug tool to a genuine cluster observability platform. And it all starts with a simple wget command.