The WebSocket Epiphany: A Debugging Turning Point in Distributed S3 Architecture

Introduction

In the midst of building a horizontally scalable S3-compatible storage cluster, a single message from an AI coding assistant captures a moment of critical insight that transformed a stalled debugging session into a breakthrough. The message is brief—barely a line of reasoning followed by a shell command—but it represents the pivot point where an incorrect assumption about protocol transport was corrected, unlocking the ability to verify the entire cluster monitoring system.

The message in question, from index 682 of a lengthy coding session, reads:

The RPC endpoint requires websocket. Let me check using a websocket client:

This simple declaration, following a series of failed HTTP requests to the cluster's RPC endpoint, marks the moment the assistant realized that the JSON-RPC interface was not served over plain HTTP POST as initially assumed, but rather over WebSocket connections. The realization seems obvious in retrospect, but the path to this understanding reveals deep layers of reasoning about distributed system architecture, protocol design, and the debugging process itself.

The Context: A Cluster in Need of Monitoring

To understand why this message matters, one must appreciate the context in which it was written. The assistant had been building a test cluster for a horizontally scalable S3 architecture—a system where stateless S3 frontend proxies sit in front of Kuri storage nodes, which in turn store data in a shared YugabyteDB database. The architecture follows a three-layer hierarchy: S3 proxy (port 8078) → Kuri storage nodes → YugabyteDB.

The user had recently requested that the cluster monitoring page be made functional. The assistant had implemented a ClusterTopology RPC method that reads the FGW_BACKEND_NODES environment variable, performs health checks on each storage node by hitting their /healthz endpoints, and returns a structured view of the cluster. The code had been written, the Docker images rebuilt, the containers restarted. Everything was in place—except the RPC appeared to be returning nothing.

Messages 676 through 681 show the assistant's escalating attempts to verify the RPC:

The Assumption That Failed

The assistant's initial assumption was that the JSON-RPC endpoint would accept standard HTTP POST requests. This is a reasonable assumption—JSON-RPC is commonly implemented over HTTP POST, and the assistant had been using curl throughout the session to test various endpoints. The S3 proxy on port 8078, the web UI on port 9010, the health check endpoints—all responded to plain HTTP requests.

But the RPC endpoint was different. The MakeRPCServer function in the Go codebase, registered via mux.Handle("/rpc/v0", rpc), was using a WebSocket-based transport. The Go net/http package's ServeMux can register any http.Handler, and the RPC handler was likely an upgraded WebSocket handler that rejected non-WebSocket connections at the protocol level.

The "Bad Request" response was not a routing error—it was a protocol mismatch. The server was saying, in effect, "I understand you're trying to reach /rpc/v0, but you're speaking HTTP and I only speak WebSocket."

The Epiphany: Message 682

Message 682 is the moment of synthesis. The assistant had been staring at "Bad Request" for six consecutive attempts. The evidence was:

  1. The endpoint exists in code (confirmed by grep and file read)
  2. The server is running and responding (confirmed by verbose curl showing HTTP response)
  3. The path is correct (confirmed by code inspection)
  4. The request format is valid JSON-RPC (confirmed by payload structure)
  5. Yet every request is rejected The only remaining variable was the transport protocol. The assistant realized: the RPC handler must be a WebSocket endpoint. This is a common pattern in Go services that need bidirectional communication—the server can push updates to clients (like real-time cluster metrics) without the client polling. The assistant's next action confirms this realization: instead of continuing to fight with curl, it pivots to checking the container logs. The docker logs test-cluster-kuri-1-1 command reveals the kuri node's startup sequence, including the RIBSWeb at http://127.0.0.1:9010 line confirming the web server is running. The assistant is now looking for a WebSocket client tool to properly test the RPC.

What the Message Reveals About the Thinking Process

The reasoning visible in this message and its surrounding context reveals several cognitive processes:

Hypothesis testing under constraint: The assistant systematically tested each variable—URL path, HTTP method, request format—before concluding the protocol itself was wrong. Each failed test eliminated a hypothesis and narrowed the search space.

Reading the code as documentation: When the endpoint refused HTTP requests, the assistant went to the source code (message 680) to understand how the handler was registered. The code showed mux.Handle("/rpc/v0", rpc) but didn't reveal the transport protocol—the rpc variable was the result of MakeRPCServer, whose implementation was elsewhere.

Protocol awareness: The assistant recognized that "Bad Request" from a Go HTTP server, when the path and method are valid, typically means the handler rejected the connection at a lower level. WebSocket upgrades require a specific HTTP upgrade handshake; a plain HTTP POST would be rejected before the JSON-RPC logic even ran.

Pivoting strategy: Rather than continuing to debug the HTTP approach (which was fundamentally impossible), the assistant accepted the new understanding and immediately shifted to finding a WebSocket client. This is visible in the subsequent messages where websocat is located and used successfully.

Input Knowledge Required

To fully understand this message, one needs:

  1. JSON-RPC over WebSocket: Knowledge that JSON-RPC can be transported over WebSocket rather than HTTP, and that this requires a different client approach.
  2. Go HTTP server patterns: Understanding that Go's http.Handler interface can wrap any protocol, and that WebSocket handlers typically reject non-upgrade requests with 400 status.
  3. Docker container inspection: Familiarity with docker logs to check container output for debugging.
  4. The cluster architecture: Awareness that the system has three layers (S3 proxy, Kuri nodes, YugabyteDB) and that each Kuri node runs a web UI with an RPC endpoint.
  5. The debugging history: The preceding six failed HTTP requests that established the pattern of "Bad Request" responses.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. The RPC endpoint requires WebSocket: This is the primary output—a corrected understanding of the transport protocol.
  2. The kuri-1 container is running and healthy: The log output confirms the node started successfully, with its PeerID, wallet address, and swarm listeners active.
  3. A non-critical error exists: The balance manager error about failing to resolve an address is visible but not blocking—the node is still operational.
  4. The debugging direction shifts: From HTTP debugging to WebSocket testing, which will succeed in the following messages.

Mistakes and Incorrect Assumptions

The primary mistake was the assumption that the JSON-RPC endpoint used HTTP POST. This assumption was reasonable given the prevalence of HTTP-based JSON-RPC in the ecosystem, and given that the assistant had been using curl successfully for all other endpoints in the cluster.

A secondary oversight was not checking the RPC handler implementation earlier. The MakeRPCServer function's return type would have revealed the WebSocket dependency if examined. However, the assistant was focused on the routing layer (mux.Handle) rather than the handler implementation.

The error in the logs—"balance manager: failed to get market balance"—is worth noting. The assistant does not comment on it in this message, but it represents a real issue: the Filecoin wallet address f1bkjjrmlkm4c5aii44gll2sevfcbauducfiveicq cannot be resolved on the network, likely because the test cluster is running in isolation without a connection to the Filecoin blockchain. This is a known limitation of test environments and does not prevent the storage and S3 functionality from working.

The Broader Significance

This message is a microcosm of distributed systems debugging. The pattern is universal: a component that should work doesn't, the developer tests every variable, and finally realizes the fundamental assumption about the interface is wrong. In distributed systems, protocol mismatches are common—a service expects gRPC but gets HTTP, a client sends JSON but the server expects Protobuf, a connection upgrade is required but not performed.

The assistant's response to the realization is also instructive. There is no frustration, no backtracking to rewrite the RPC handler to support HTTP. Instead, the assistant accepts the protocol as designed and adapts the testing approach. This is the mark of experienced debugging: work with the system as it exists, not as you assumed it to be.

Conclusion

Message 682 is a turning point disguised as a simple observation. In the span of a single sentence, the assistant corrected a fundamental misunderstanding about the cluster's RPC transport, pivoted the debugging strategy, and set the stage for successful verification of the cluster monitoring system. The subsequent messages show the payoff: websocat connects via WebSocket, the ClusterTopology RPC returns full data about both storage nodes, and the cluster monitoring dashboard becomes functional.

The message also reveals something about the nature of AI-assisted coding. The assistant's reasoning is visible in the sequence of failed attempts and the final insight. Each "Bad Request" was a data point, and the assistant processed them systematically until the only remaining explanation was the correct one. This is not magic—it is methodical problem-solving, made visible through the conversation transcript.

For anyone building or debugging distributed systems, this message serves as a reminder: when the server keeps saying "Bad Request," question your assumptions about how you're speaking to it. The protocol might not be what you think it is.