The WebSocket Epiphany: Debugging Cluster Topology Across Protocol Boundaries

Introduction

In the course of building a horizontally scalable S3 storage architecture, moments of quiet insight often matter as much as the loud, crashing bugs. Message 683 in this coding session captures one such moment—a brief, almost throwaway realization that the Remote Procedure Call (RPC) interface powering the cluster monitoring dashboard communicates over WebSocket, not plain HTTP. This single observation, followed by a targeted connectivity test, untangled a thread of confusion about why the cluster topology page appeared empty and set the stage for the next phase of debugging.

The message itself is deceptively simple:

Okay, the RPC uses websocket. The cluster monitoring page should use the JS frontend to call it. Let me check if the S3 proxy's healthz is accessible from kuri nodes (the ClusterTopology function checks /healthz): `` $ docker exec test-cluster-kuri-1-1 wget -q -O- http://kuri-1:8078/healthz 2>&1 OK ``

Beneath this terse output lies a cascade of reasoning, a protocol mismatch diagnosis, and a critical validation step that confirmed the cluster topology implementation could actually work. This article unpacks what happened in that message, why it was written, and what it reveals about the broader debugging process.

Context: The Cluster Monitoring Puzzle

To understand message 683, one must appreciate the events immediately preceding it. The user had requested two things: expose the second Kuri storage node's web UI on port 9011, and fix the cluster monitoring page, which stubbornly displayed "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology" despite the fact that the assistant had already added that environment variable to both Kuri nodes.

The assistant had dutifully implemented a ClusterTopology function in rbstor/diag.go that reads the FGW_BACKEND_NODES environment variable, parses the comma-separated list of peer addresses, performs HTTP health checks against each node's /healthz endpoint, and returns a structured topology with proxies and storage nodes. After rebuilding the Docker image and restarting the containers, the assistant tried to verify the RPC was returning data by curling the endpoint:

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' \
  http://localhost:9010/rpc/v0

The response was a curt "Bad Request."

This failure is the immediate trigger for message 683. The assistant had been operating under the assumption that the RPC endpoint accepted standard HTTP POST requests with JSON-RPC payloads—a reasonable assumption given that many JSON-RPC implementations support both HTTP and WebSocket transports. The "Bad Request" response, however, signaled that this particular endpoint was WebSocket-only.

The WebSocket Realization

The opening line of message 683—"Okay, the RPC uses websocket"—is the conclusion of a rapid diagnostic chain. The assistant had already checked the server-side routing code in integrations/web/ribsweb.go and seen that the RPC handler was registered with mux.Handle("/rpc/v0", rpc). The MakeRPCServer function, upon closer inspection, would reveal that it creates a WebSocket-based RPC server rather than an HTTP-based one. This is a deliberate architectural choice: WebSocket provides persistent bidirectional communication, which is ideal for a monitoring dashboard that needs real-time updates without the overhead of repeated HTTP handshakes.

The realization carries an implicit correction of the assistant's own earlier assumption. In message 681, the assistant had tried the same curl command and gotten "Bad Request," but hadn't yet connected the dots. By message 683, the pattern is clear: curl's HTTP POST is being rejected because the server is expecting a WebSocket upgrade handshake. The assistant now understands that the cluster monitoring page is designed to use the JavaScript frontend's WebSocket client to call this RPC—not a simple fetch request.

This is a classic debugging pattern: the tool you're using (curl) doesn't match the protocol the server expects, and the error message ("Bad Request") is generic enough that it doesn't immediately reveal the mismatch. The assistant had to cross-reference the server implementation, the frontend code, and the observed behavior to arrive at the correct diagnosis.## The Health Check Validation: A Strategic Pivot

Once the assistant identified the WebSocket requirement, the natural next step would have been to immediately test the RPC with a WebSocket client. But the assistant chose a different path: validating that the health check endpoint was reachable between containers. This is a fascinating strategic decision.

The ClusterTopology function, as implemented, performs HTTP GET requests against each backend node's /healthz endpoint to determine node health. If this inter-container communication were broken—due to Docker networking issues, port mapping problems, or the S3 proxy not exposing the health endpoint—then even a successful WebSocket RPC call would return stale or empty data. The assistant was verifying the data pipeline from the bottom up: confirm the health check works, then confirm the RPC returns it, then confirm the frontend displays it.

The command docker exec test-cluster-kuri-1-1 wget -q -O- http://kuri-1:8078/healthz is carefully constructed. It runs inside the kuri-1 container, using Docker's internal DNS resolution (the container name kuri-1 resolves to the container's IP on the test-cluster network). It targets port 8078, which is the S3 frontend proxy port—not the kuri node's own HTTP server. This confirms that the S3 proxy (which runs as a separate process or is embedded in the kuri node) is exposing the health check endpoint on the correct port. The response "OK" confirms that the health check is functional.

This validation is critical because the ClusterTopology implementation had been modified to check http://<node-id>:8078/healthz for each backend node. If the S3 proxy weren't running or the health endpoint weren't registered, the topology would show all nodes as "unhealthy" or "unreachable." By confirming the health check works from inside the container, the assistant eliminates a whole class of potential failures before moving on to the WebSocket test.

Assumptions and Their Corrections

Message 683 reveals several assumptions, some correct and some that would later need adjustment:

Assumption 1: The health check endpoint is the right way to probe node health. This is correct. The S3 proxy's /healthz endpoint is a lightweight, side-effect-free way to verify that the node is alive and responding. The assistant had added this endpoint in an earlier fix (message 647), so it was a recent addition that the assistant remembered and trusted.

Assumption 2: The cluster monitoring page uses the JS frontend to call the RPC. This is correct but incomplete. The frontend does use WebSocket, but the assistant would later discover that the React components need specific JSON field naming conventions (camelCase vs. the Go struct's PascalCase) to render the data correctly. This would require adding JSON tags to the Go struct definitions—a fix that happens in subsequent messages.

Assumption 3: The WebSocket RPC will return the topology data once the health checks work. This is correct in principle, but the assistant's initial implementation had a subtle bug: the ClusterTopology function was checking health against the wrong address format. The assistant had used http://kuri-1:8078 as the address, but the S3 proxy's health endpoint was at /healthz on that same host:port. The implementation appended /healthz correctly, so this worked.

Assumption 4: The "Bad Request" from curl meant the RPC was broken. This was the incorrect assumption that message 683 corrects. The assistant initially thought the RPC implementation had a bug, but the real issue was protocol mismatch. This is a valuable lesson: "Bad Request" can mean "I don't understand your protocol," not "your data is wrong."

Input Knowledge Required

To fully understand message 683, one needs:

  1. Docker networking basics: The docker exec command runs inside a container, and container names resolve via Docker's embedded DNS on the custom network. The fact that kuri-1 resolves to the container's IP is Docker-specific knowledge.
  2. WebSocket vs. HTTP: Understanding that WebSocket requires an upgrade handshake (HTTP 101 Switching Protocols) and that a standard HTTP POST will be rejected by a WebSocket-only endpoint. The "Bad Request" response is typical of WebSocket servers that don't recognize the incoming request as a valid upgrade.
  3. The project's architecture: Knowledge that the S3 frontend proxy runs on port 8078 and exposes a /healthz endpoint, and that the kuri nodes are separate containers with their own web UIs on port 9010.
  4. The debugging history: Awareness that the health endpoint was added in a previous fix (message 647) to resolve a different issue (the S3 proxy returning internal server errors), and that the ClusterTopology implementation was brand new, written in messages 666-671.
  5. Go's JSON-RPC libraries: The project uses a WebSocket-based JSON-RPC implementation where MakeRPCServer creates a handler that upgrades HTTP connections to WebSocket. This is a common pattern in Go web services that need real-time capabilities.

Output Knowledge Created

Message 683 produces several concrete pieces of knowledge:

  1. The RPC endpoint is WebSocket-only. This is the primary finding. Any testing or frontend development must use WebSocket, not HTTP POST.
  2. The health check endpoint is functional. The S3 proxy's /healthz returns "OK" when called from within the container network, confirming that the inter-container communication path works.
  3. The ClusterTopology implementation's health check logic is sound. The function checks http://<node>:8078/healthz, and this URL is reachable and responsive from the kuri-1 container.
  4. The frontend must be the testing vehicle. Since curl can't easily test WebSocket RPCs, the assistant will need to either install a WebSocket client (like websocat) or rely on the React frontend's behavior to verify the topology data flows correctly. This last point is crucial. In the messages immediately following 683 (684-690), the assistant installs websocat, tests both web UIs (:9010 and :9011), and confirms that the ClusterTopology RPC returns a full topology with both storage nodes showing as healthy. The WebSocket epiphany was the bottleneck—once the assistant understood the protocol requirement, the verification could proceed rapidly.

The Thinking Process

The reasoning visible in message 683 is a model of efficient debugging. The assistant starts with a conclusion ("the RPC uses websocket") that synthesizes multiple earlier observations: the "Bad Request" responses, the server-side routing code showing mux.Handle("/rpc/v0", rpc), and the knowledge that the project uses WebSocket-based RPC in other components.

Rather than immediately jumping to install a WebSocket client, the assistant asks a more fundamental question: "Is the health check accessible from kuri nodes?" This is a dependency check. The ClusterTopology function's correctness depends on the health check being reachable. If the health check fails, the topology data will be empty regardless of whether the RPC transport works. By validating the health check first, the assistant establishes a solid foundation.

The choice of wget over curl is also telling. The wget command is available inside the minimal Alpine-based Docker image, while curl might not be. The assistant had previously used docker exec with curl in message 676, but that was against the host's curl. Inside the container, wget is the safer bet.

The message ends with the health check result: "OK." This single word carries immense weight. It means the S3 proxy is running, the health endpoint is registered, the Docker network allows inter-container communication, and the port mapping is correct. The assistant can now proceed to test the WebSocket RPC with confidence that the underlying data pipeline is sound.

Conclusion

Message 683 is a masterclass in targeted debugging. It demonstrates how a single protocol mismatch can masquerade as a data or implementation bug, and how a disciplined approach—validate dependencies before testing the main path—can save hours of wasted effort. The assistant's realization that the RPC uses WebSocket, followed by the strategic health check validation, turned an empty monitoring dashboard into a solvable problem. Within minutes, the assistant would confirm that both Kuri nodes were reporting healthy status and the cluster topology was fully populated.

This message also highlights the importance of understanding your tools' protocol boundaries. Curl speaks HTTP; WebSocket servers speak a different dialect. Recognizing when you're using the wrong tool for the job is a skill that separates effective debuggers from those who chase symptoms. The assistant's quiet pivot from "the RPC is broken" to "I'm using the wrong protocol" is the kind of insight that comes from deep familiarity with the system's architecture—and the humility to question one's own assumptions.