The Moment of Discovery: When a Cluster Topology RPC Returns "Bad Request"

A Single Curl Command That Exposed an Invisible Bug

In the midst of building a horizontally scalable S3 architecture with a distributed test cluster, a developer sends a single curl command to verify that a newly implemented cluster monitoring feature works. The response is two words: "Bad Request." This seemingly trivial error message represents a pivotal moment of discovery—a checkpoint where assumptions meet reality, and the debugging process must pivot.

The message under analysis is brief but consequential:

Both UIs are working. Let me check if the cluster topology RPC is returning data: `` curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' http://localhost:9010/rpc/v0 2>&1 Bad Request ``

To understand why this message was written, we must reconstruct the context that led to it. The developer had just spent considerable effort implementing a ClusterTopology RPC endpoint in the Kuri storage nodes. The original implementation in rbstor/diag.go was a stub that returned empty arrays—a placeholder marked with a TODO comment. The developer replaced that stub with a real implementation that reads the FGW_BACKEND_NODES environment variable, parses the comma-separated list of backend node addresses, performs health checks against each node, and returns a structured topology containing both S3 frontend proxies and Kuri storage nodes with their respective health statuses, request rates, and connection counts.

This implementation was part of a larger effort to address the user's request to expose both Kuri node web UIs (on ports 9010 and 9011) and to fix the cluster monitoring page, which was displaying the unhelpful message "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology." The developer had already rebuilt the Docker image, regenerated configuration files, and restarted the entire test cluster with docker compose up -d --force-recreate. The first verification step—checking that both web UIs return HTML—passed successfully. Both localhost:9010 and localhost:9011 served the RIBSWeb application.

The Reasoning Behind the Test

The decision to test the cluster topology RPC specifically reveals the developer's systematic debugging methodology. After confirming that the web UI containers were running and serving pages, the natural next question was: does the data pipeline work? The web UI's cluster monitoring page depends on the RIBS.ClusterTopology RPC method to populate its displays. If the RPC returns empty data or errors, the monitoring page would remain blank regardless of whether the UI itself loaded correctly.

The developer chose to test this at the HTTP level using curl rather than through the web UI's JavaScript frontend. This is a deliberate architectural decision: testing at the RPC boundary eliminates the React application, the state management layer, and the WebSocket connection as potential sources of failure. If the raw RPC call works, then any frontend issues must be in the JavaScript layer. If it fails, the problem is in the backend—either the RPC handler itself, the HTTP routing, or the infrastructure between them.

The choice of endpoint—http://localhost:9010/rpc/v0—is also significant. Port 9010 is the nginx reverse proxy that fronts kuri-1's web UI. The developer is testing through the same path that the browser would use, which means the test validates not just the RPC handler but also the nginx configuration, the proxy routing, and the upstream kuri node's HTTP server. This is an integration-level test, not a unit test.

Assumptions Embedded in the Test

Every test carries assumptions, and this one carries several. The developer assumes that the JSON-RPC protocol is correctly implemented on the server side, that the method name RIBS.ClusterTopology matches exactly what the server expects, that the HTTP POST with Content-Type: application/json is the correct way to invoke it, and that the server's RPC endpoint is mounted at /rpc/v0. The developer also assumes that the newly rebuilt Docker image was correctly deployed, that the configuration files were regenerated with FGW_BACKEND_NODES set, and that the kuri nodes have started up and initialized their RPC handlers before the test runs.

The "Bad Request" response shatters several of these assumptions simultaneously. It tells the developer that something in the request—the HTTP method, the headers, the URL path, the JSON body structure, or the method name—does not match what the server expects. But it does not say which. This is the classic ambiguity of a generic HTTP 400 error: it could be a routing problem, a parsing problem, or a validation problem.

What the Developer Knew (and Didn't Know)

To understand this message fully, we need to inventory the input knowledge the developer brought to this moment. The developer knew that the kuri nodes expose a JSON-RPC interface, that the ClusterTopology method had been implemented in rbstor/diag.go, that the nginx configuration proxies /rpc/v0 to the upstream kuri node, and that the web UI's cluster monitoring page depends on this RPC call. The developer also knew from prior work in this session that the kuri nodes had been crashing due to HTTP route conflicts (between HEAD / and GET /healthz), which had been fixed by replacing the standard ServeMux with a custom handler.

What the developer did not yet know was whether the custom HTTP routing correctly handles POST requests to /rpc/v0, whether the JSON-RPC handler is properly registered, whether the ClusterTopology method is exposed through the RPC registry, or whether the nginx proxy configuration correctly forwards POST requests with JSON bodies. The "Bad Request" response is the first signal that one or more of these unknowns is a problem.

The Output Knowledge Created

This message creates critical output knowledge: the cluster topology RPC is not working through the web UI proxy. This is negative knowledge—knowing what doesn't work—but it is immensely valuable because it narrows the search space. The developer now knows that the problem is not in the web UI's JavaScript (which hasn't been loaded yet) but in the backend RPC chain. The next debugging steps would logically involve testing the RPC endpoint directly on the kuri node (bypassing nginx), checking the kuri node's logs for errors, examining the HTTP routing code for the /rpc/v0 path, and verifying that the JSON-RPC handler is correctly initialized.

This moment also creates documentation knowledge. The fact that the developer chose to capture this test in the conversation history means that the failure mode is recorded for future reference. If the developer later discovers the root cause, the path from symptom to fix will be traceable.

The Thinking Process Visible in the Message

Even in this short message, we can see the developer's thinking process at work. The message begins with a positive confirmation: "Both UIs are working." This establishes a baseline—the infrastructure is running, the containers are healthy, the nginx proxies are serving pages. Then comes the pivot: "Let me check if the cluster topology RPC is returning data." This is a hypothesis-driven test: the developer has a specific question about a specific component and designs a minimal experiment to answer it.

The choice of curl command is itself a thinking artifact. The developer uses -s for silent mode (no progress meter), -X POST to specify the HTTP method, -H "Content-Type: application/json" to set the content type header, and -d with a JSON-RPC payload. The 2>&1 redirects stderr to stdout so that any error messages are captured. The JSON-RPC payload follows the standard format with jsonrpc, method, params, and id fields. This is not a random test—it is a carefully constructed probe designed to match the exact protocol that the server should understand.

The result, "Bad Request," is presented without commentary. The developer does not immediately jump to conclusions or propose fixes. Instead, the message simply reports the outcome, leaving the interpretation and next steps to the subsequent interaction. This restraint is itself a thinking strategy: before debugging, one must first confirm that the bug exists and characterize its symptoms.

Why This Message Matters

In the broader narrative of building a distributed S3 storage system, this message represents the moment when a newly implemented feature meets reality. The developer had written code, built an image, deployed containers, and verified basic connectivity. But the feature—cluster topology monitoring—remained unvalidated. This curl command was the first end-to-end test of that feature, and it failed.

This is the essence of incremental development in complex systems: each component is built, tested, and debugged in isolation before being integrated. The developer had already debugged and fixed the kuri node crashes, the web UI proxy configuration, the S3 proxy's missing database column, and the header injection for unsigned PUT requests. Each of those fixes was validated with a similar curl command. Now the cluster topology feature faces its moment of validation, and the "Bad Request" response signals that the debugging cycle must continue.

The message also illustrates a fundamental truth about distributed systems debugging: the error message is rarely the full story. "Bad Request" could mean a hundred different things, from a missing route handler to a JSON parsing error to an authentication failure. The developer's skill lies not in accepting the error at face value but in designing the next experiment to narrow the possibilities. The conversation does not show what happens next, but the pattern from prior interactions suggests a methodical drill-down: test the kuri node directly, check server logs, examine the HTTP router code, and verify the RPC handler registration.

Conclusion

A single curl command that returns "Bad Request" may seem like a trivial moment in a long coding session. But in the context of building a horizontally scalable S3 architecture with multiple interacting components—Docker containers, nginx proxies, Go HTTP servers, JSON-RPC handlers, YugabyteDB databases, and React frontends—this two-word response is a rich signal. It tells the developer that an assumption has failed, that a component is not wired correctly, and that the debugging journey must continue. The message captures not just a test result but a methodology: build, test, observe, and iterate. It is a snapshot of the scientific method applied to software engineering, one curl command at a time.