The "Bad Request" That Revealed a Protocol Mismatch

In the middle of debugging a distributed S3 storage cluster, a single two-word response — "Bad Request" — became the catalyst for a deeper understanding of how the system's components actually communicated. Message 677 in this coding session is deceptively simple on its surface:

[assistant] [bash] curl -s http://localhost:9010/rpc/v0 2>&1 Bad Request

But this brief exchange, consisting of a shell command and its output, represents a critical debugging pivot. The assistant had just spent significant effort implementing a ClusterTopology RPC method — modifying rbstor/diag.go to parse the FGW_BACKEND_NODES environment variable, perform health checks against each storage node's /healthz endpoint, and return structured topology data to the React frontend. The Docker image had been rebuilt, configurations regenerated, and containers restarted. The web UIs on both port 9010 and 9011 were confirmed serving HTML. Everything looked like it should work. But when the assistant tried to verify the RPC endpoint, it hit a wall.

The Debugging Context

To understand why this message was written, we need to trace the reasoning that led to it. In the immediately preceding message (index 676), the assistant had already attempted to call the cluster topology RPC using a proper JSON-RPC POST request:

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' \
  http://localhost:9010/rpc/v0

That also returned "Bad Request." Message 677 is the assistant's next logical step in the debugging process: strip the request down to its simplest form. Remove the POST method specification. Remove the JSON body. Remove the Content-Type header. Just send a plain GET to the same URL and see what happens. This is classic debugging methodology — isolate variables one at a time to identify the source of the failure.

The assistant's thinking here follows a clear pattern: "The POST with JSON body failed. Maybe the server doesn't like something about the request format. Let me try the absolute simplest HTTP request — a GET with no headers — to see if the endpoint responds at all." The fact that the GET also returns "Bad Request" tells the assistant something important: the problem isn't with the request formatting or the JSON payload. The endpoint itself, at least when accessed via plain HTTP, rejects all requests.

Assumptions and Their Consequences

This message reveals several implicit assumptions the assistant was operating under. The primary assumption was that the RPC endpoint at /rpc/v0 would respond to standard HTTP requests — either GET or POST — with a meaningful response. This assumption was reasonable given that the assistant had just seen the route registered in integrations/web/ribsweb.go as mux.Handle("/rpc/v0", rpc). A handler is a handler, and HTTP is HTTP.

But the "Bad Request" response, consistent across both GET and POST attempts, should have been a stronger signal. A properly configured HTTP endpoint that receives an unexpected method typically returns 405 Method Not Allowed or 400 Bad Request with an explanatory body. The fact that both methods failed identically suggested something deeper — perhaps the handler wasn't an HTTP handler at all.

The assistant also assumed that the nginx reverse proxy (which sat between the client and the kuri node's web server) was correctly forwarding all requests to /rpc/v0. The nginx config generated by gen-config.sh was a generic reverse proxy configuration, and while it worked for the main web UI routes, it might not have been handling the RPC path correctly — or the RPC handler itself might have required a protocol that nginx couldn't transparently proxy.

Input Knowledge Required

To fully understand this message, one needs to know several things about the system architecture:

1. The three-layer deployment: The test cluster consisted of an nginx reverse proxy (webui container) on port 9010, which proxied to kuri-1's internal web server on port 9010 inside the Docker network. This meant any request to localhost:9010/rpc/v0 first hit nginx, then was forwarded to the kuri node's RIBSWeb server.
2. The RPC implementation: The RIBSWeb server in integrations/web/ribsweb.go registered the RPC handler at /rpc/v0 using Go's standard http.ServeMux. The handler was created by MakeRPCServer, which the assistant had seen but not examined in detail.
3. The cluster topology feature: The assistant had just implemented ClusterTopology() in rbstor/diag.go to read FGW_BACKEND_NODES, parse comma-separated node addresses, perform HTTP health checks against each node's /healthz endpoint, and return structured iface.ClusterTopology data. This was the feature the assistant was trying to verify.
4. The React frontend expectations: The cluster monitoring page in the React frontend expected to receive cluster topology data via an RPC call, but the exact transport mechanism (WebSocket vs HTTP POST) was not yet clear to the assistant.

The Output Knowledge Created

Although message 677 itself only produced "Bad Request," it generated crucial knowledge for the assistant:

• The RPC endpoint rejects all plain HTTP requests. Both GET and POST failed identically, ruling out simple request formatting issues.
• The problem is at the protocol or handler level, not at the routing or nginx level. The nginx proxy was clearly receiving the request and forwarding it (otherwise the assistant would have seen a connection error or 502, not "Bad Request").
• The RPC handler requires a specific protocol that wasn't being used. The "Bad Request" response was coming from the RPC handler itself, not from nginx or the HTTP router. This knowledge directly drove the assistant's next steps. In subsequent messages, the assistant would discover that the RPC endpoint used WebSocket protocol rather than plain HTTP — a design choice that made sense for a real-time monitoring dashboard but wasn't obvious from the route registration alone. The assistant would go on to check the kuri node's logs (message 682), discover the WebSocket requirement, and ultimately use websocat to successfully query the cluster topology.

The Thinking Process in Action

What makes message 677 interesting is what it reveals about the assistant's debugging strategy. The assistant is systematically working through a troubleshooting checklist:

1. Verify the endpoint exists: Both web UIs return HTML on the root path, so the server is running.
2. Try the expected protocol: POST with JSON-RPC body (message 676) — fails with "Bad Request."
3. Simplify to isolate: Strip everything down to a bare GET request (message 677) — also fails.
4. Check server-side evidence: Look at container logs (message 682) to see what the server received.
5. Examine the source code: Read the route registration to understand how the handler is wired (message 679-680).
6. Try alternative paths: Test /api/rpc instead of /rpc/v0 (message 678).
7. Use verbose output: Add -v to see the full HTTP exchange (message 681).
8. Discover the protocol mismatch: Realize the RPC uses WebSocket, not HTTP POST. This progression shows disciplined debugging: each step narrows the possibilities. The "Bad Request" in message 677 was the clue that eliminated request formatting as the cause and pointed toward a deeper protocol issue. Without this step, the assistant might have spent time tweaking JSON payloads or HTTP headers, chasing a red herring.

Mistakes and Misconceptions

The assistant's primary mistake in this moment was an understandable one: assuming that a Go HTTP handler registered with http.ServeMux.Handle() would respond to standard HTTP requests. In Go's standard library, Handle() accepts any http.Handler, and many RPC implementations (including JSON-RPC over HTTP) use standard request/response patterns. The assistant had no reason to suspect WebSocket was involved until the "Bad Request" response persisted across multiple HTTP methods.

A secondary misconception was about the nginx proxy configuration. The assistant had set up nginx as a simple reverse proxy, but the WebSocket protocol requires specific proxy configuration (upgrading the connection from HTTP to WebSocket). The "Bad Request" could have been nginx refusing to forward a non-HTTP connection, though in this case it was the RPC handler itself rejecting the plain HTTP request.

The Broader Significance

Message 677 sits at a inflection point in the debugging session. Before it, the assistant was in "implementation mode" — writing code, rebuilding images, restarting containers. After it, the assistant shifted to "discovery mode" — examining logs, reading source code, and ultimately learning how the system actually worked at the protocol level. The "Bad Request" response was the signal that the assistant's mental model of the system was incomplete, and the subsequent investigation would fill in those gaps.

For anyone reading this conversation, message 677 is a reminder that the most informative responses are often the ones that say "no" — and that a good debugger listens carefully to what the system is telling them, even when the message is just two words.