The Debugging Pivot: When a 400 Bad Request Reveals a WebSocket Assumption

Introduction

In the midst of building a horizontally scalable S3 storage architecture with distributed Kuri storage nodes, a single debugging command can mark the boundary between two modes of thinking. Message 681 in this coding session is exactly such a boundary. It is the moment when the assistant, having just implemented a critical ClusterTopology RPC endpoint to populate an empty cluster monitoring dashboard, attempts to verify the fix using a straightforward HTTP POST request — and receives a 400 Bad Request in response. That seemingly mundane error is the pivot point that forces a reassessment of how the system's internal communication actually works, leading to the discovery that the RPC layer requires WebSocket transport rather than plain HTTP. This article examines that single message in depth: why it was written, the assumptions it reveals, the knowledge it produces, and the debugging trajectory it sets in motion.

The Context: A Cluster Monitoring Dashboard That Shows Nothing

The story leading to message 681 begins with a user observation. After the assistant had successfully exposed both Kuri node web UIs on ports 9010 and 9011, the user reported that the cluster monitoring page on port 9010 appeared empty, displaying the message: "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology." This was a clear signal that the ClusterTopology RPC — the backend function responsible for returning information about all nodes in the distributed cluster — was returning an empty or stub response.

The assistant traced the problem to rbstor/diag.go, where the ClusterTopology() method was a stub that returned empty slices for proxies and storage nodes. The implementation had a telling TODO comment: "Implement actual cluster monitoring when running in distributed mode." That moment had arrived. The assistant rewrote the function to read the FGW_BACKEND_NODES environment variable, parse the comma-separated list of backend addresses, perform HTTP health checks against each node's /healthz endpoint, and return a fully populated ClusterTopology struct with proxy and storage node information. The Docker image was rebuilt, the test cluster was restarted with --force-recreate, and the stage was set for verification.

Message 681: The Verification Attempt

Message 681 is the verification step. The full message reads:

Let me look at what the React frontend expects for the cluster monitoring:

`` curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' http://localhost:9010/rpc/v0 -v 2>&1 | head -30 ``

`` Note: Unnecessary use of -X or --request, POST is already inferred. * Host localhost:9010 was resolved. * IPv6: ::1 * IPv4: 127.0.0.1 * Trying [::1]:9010... * Established connection to localhost (::1 port 9010) from ::1 port 36614 * using HTTP/1.x > POST /rpc/v0 HTTP/1.1 > Host: localhost:9010 > User-Agent: curl/8.18.0 > Accept: */* > Content-Type: application/json > Content-Length: 68 > } [68 bytes data] * upload completely sent off: 68 bytes < HTTP/1.1 400 Bad Request < Server: ngi... ``

The assistant begins with the stated intention: "Let me look at what the React frontend expects for the cluster monitoring." This framing is important — the assistant is not testing the backend implementation in isolation; the assistant is trying to understand the interface contract that the frontend expects. The ClusterTopology RPC was just implemented in rbstor/diag.go, and the next logical step is to verify that it returns data in the format the React UI can consume.

The curl command is a textbook debugging pattern: implement a change, then test the backend directly before checking whether the frontend displays the data correctly. The assistant is bypassing the React UI to test the RPC endpoint in isolation. The curl command sends a JSON-RPC request — a standard protocol for remote procedure calls using JSON — to the path /rpc/v0 on the web UI server at port 9010. The -v flag requests verbose output, showing both request and response headers.

The response arrives quickly:

< HTTP/1.1 400 Bad Request
< Server: ngi...

A 400 Bad Request from an nginx server. The output is truncated by head -30, but the critical information is already visible: the server is rejecting the POST request.

The Assumptions Embedded in This Message

Every debugging action carries implicit assumptions, and message 681 is rich with them. The most fundamental assumption is that the RPC endpoint at /rpc/v0 accepts plain HTTP POST requests with a JSON-RPC body. This is a reasonable assumption — many JSON-RPC implementations do exactly this. The endpoint path /rpc/v0 strongly suggests an HTTP-based RPC interface. The assistant constructs a well-formed JSON-RPC request with the correct method name (RIBS.ClusterTopology), an empty parameter array, and a request ID. The Content-Type header is set to application/json. Everything looks correct from the client side.

A second assumption is that the endpoint is reachable at localhost:9010 — the web UI port. This is correct in terms of network routing; the nginx container on port 9010 does receive the request. But the assumption that nginx will proxy a POST to /rpc/v0 to the Kuri node's RPC handler is what fails. The nginx configuration, as later investigation reveals, is set up to serve the React frontend and proxy WebSocket connections — not to handle arbitrary HTTP POST requests to backend RPC endpoints.

A third, subtler assumption is that the ClusterTopology implementation is the only missing piece. The assistant assumes that once the backend function returns real data, the existing frontend code will automatically display it. But the 400 error introduces a new variable: perhaps the communication channel between frontend and backend also needs attention.

What the Message Does Not Yet Know

The beauty of message 681 is that it captures a moment of productive ignorance. The assistant does not yet know that:

The RPC endpoint requires WebSocket transport, not plain HTTP. The rpc/v0 handler in integrations/web/ribsweb.go is registered with mux.Handle("/rpc/v0", rpc), where rpc is a WebSocket-based handler created by MakeRPCServer.
The nginx reverse proxy in front of the Kuri node's web UI is configured to handle WebSocket upgrades but rejects plain POST requests to /rpc/v0 because it doesn't have a route for them — it expects those requests to be WebSocket connections.
The correct way to test the endpoint is with a WebSocket client like websocat or a browser-based WebSocket connection from the React frontend itself. The 400 Bad Request is not a sign that the ClusterTopology implementation is broken. It is a sign that the testing methodology is mismatched to the transport protocol. The assistant's backend code is likely correct; the verification strategy is what needs adjustment.

The Knowledge Produced by This Message

Even before the assistant fully interprets the result, message 681 produces valuable knowledge. The verbose curl output confirms that:

The nginx server at port 9010 is reachable and responding.
The server identifies itself as nginx (the Server: ngi... header).
The request reaches the server but is rejected at the HTTP level, before any application logic executes.
The rejection is a 400 Bad Request, not a 404 Not Found or a 503 Service Unavailable, which suggests the server understands the path but rejects the method or content type. This narrows the problem space considerably. The issue is not that the Kuri node is unreachable, or that the ClusterTopology function panics, or that the database connection is down. The issue is at the HTTP/nginx layer — a routing or protocol mismatch.

The Debugging Trajectory That Follows

Message 681 is the first step in a debugging chain that unfolds over the next several messages. The assistant immediately follows up (message 682) by checking the Kuri node's Docker logs, where the output shows "RIBSWeb at http://127.0.0.1:9010" — confirming the web server is running. Then the assistant tests whether the S3 proxy's health endpoint is accessible from within the container (messages 683-684), verifying that inter-container networking works. The key breakthrough comes when the assistant discovers websocat is available (message 687) and successfully calls the RPC endpoint via WebSocket (message 688), receiving the full ClusterTopology response with both storage nodes listed as healthy.

The contrast between message 681's 400 error and message 688's successful JSON response is instructive. The same RPC method, the same parameters, the same server — but transported over WebSocket instead of HTTP POST. The assistant's initial assumption about the transport protocol was the only thing standing between a failed test and a successful one.

Broader Lessons in Distributed Systems Debugging

Message 681 exemplifies a pattern that recurs throughout distributed systems development: the verification strategy must match the system's actual communication patterns, not the developer's assumptions about them. When building multi-service architectures with heterogeneous transport protocols — some services speaking HTTP, others WebSocket, others gRPC or raw TCP — it is easy to reach for the wrong testing tool. Curl is the universal hammer of HTTP debugging, but not every service accepts HTTP nails.

The assistant's choice to use curl was not a mistake; it was a reasonable first attempt. The mistake would have been to stop at the 400 error and conclude that the ClusterTopology implementation was broken. Instead, the assistant treated the 400 as a signal to investigate further, checking logs, testing connectivity, and ultimately finding the right tool for the job.

Conclusion

Message 681 is a single curl command that encapsulates the entire debugging process in miniature: an assumption, a test, a failure, and the beginning of a revised approach. It demonstrates that in complex distributed systems, the boundary between "the code is broken" and "the test is wrong" is often invisible until you cross it. The 400 Bad Request was not a dead end — it was a signpost pointing toward WebSocket, toward a deeper understanding of the system architecture, and ultimately toward a working cluster monitoring dashboard. For anyone debugging distributed systems, message 681 is a reminder that the first failed test is rarely the last word; it is usually the first sentence of a longer conversation with the system.