The Diagnostic Probe: Decoding a JSON-RPC Debugging Moment in a Distributed S3 Cluster

Introduction

In the course of building a horizontally scalable, three-layer S3 architecture—comprising stateless frontend proxies, Kuri storage nodes, and a shared YugabyteDB backend—a developer encounters countless small moments of debugging that reveal the true shape of the system. One such moment occurs in message 678 of a coding session, where the assistant executes a single curl command against a cluster monitoring endpoint and receives an unexpected response. This article examines that message in depth: a JSON-RPC probe that returns HTML instead of structured data, and the chain of reasoning, assumptions, and architectural knowledge that surrounds it.

The Message Itself

The subject message is a bash command and its output:

curl -s -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"RIBS.ClusterTopology","params":[],"id":1}' http://localhost:9010/api/rpc 2>&1

The response is not a JSON-RPC result but an HTML document:

<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/icon/ico_gr.png"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>RIBSWeb</title>...

The assistant receives the RIBSWeb frontend HTML page instead of a JSON-RPC response. This is the classic symptom of hitting a web application's default route rather than its API endpoint.

Why This Message Was Written: The Debugging Context

To understand why this particular curl command was issued, we must trace the debugging thread that led to it. The session's immediate context begins with the user asking about the web UI on port 9010 (message 648), followed by a question about parallel writes (message 650), and then a request to expose the second Kuri node's UI on port 9011 while also noting that the cluster monitoring page appears empty (message 654). The user reports seeing the message "No cluster nodes configured. Set up FGW_BACKEND_NODES to see cluster topology" and no data in any monitoring panels.

This triggers a substantial debugging and implementation effort. The assistant discovers that the ClusterTopology RPC method in rbstor/diag.go is a stub that returns empty arrays. The implementation reads FGW_BACKEND_NODES from the environment but had been returning nothing because the configuration wasn't being set. The assistant updates the Docker Compose configuration to add FGW_BACKEND_NODES to both Kuri nodes, implements a proper ClusterTopology function that parses the environment variable and performs health checks, rebuilds the Docker image, regenerates configuration files, and restarts the cluster.

After the cluster comes back up, the assistant verifies that both web UIs (ports 9010 and 9011) are serving HTML correctly. Then comes the critical moment: testing whether the cluster topology RPC actually returns data. The assistant first tries the endpoint at http://localhost:9010/rpc/v0 (messages 676–677) and receives "Bad Request." This is the immediate predecessor to message 678. The "Bad Request" response suggests a routing issue—the server received the request but couldn't parse it or match it to a handler.

The assistant then hypothesizes that the correct path might be /api/rpc instead of /rpc/v0, and issues the curl command in message 678 to test this theory. This is a classic debugging pattern: when one path fails, try a plausible alternative based on common API routing conventions.

Assumptions Embedded in the Probe

The assistant makes several assumptions in crafting this request. First, it assumes that the RPC endpoint exists on port 9010, which is the web UI port proxying to kuri-1. This assumption is reasonable—the cluster monitoring dashboard is served through the web UI, and the RPC endpoint that powers it should be accessible through the same port. However, this conflates two different services: the RIBS web frontend (which renders HTML) and the JSON-RPC API (which returns data). In many web architectures, these are served on different paths of the same HTTP server, but in this case, the nginx reverse proxy configuration may only forward certain paths to the backend.

Second, the assistant assumes the path is /api/rpc. This is a common convention in REST and JSON-RPC APIs—placing the RPC handler under an /api/ prefix. The previous attempt used /rpc/v0, which follows a versioned API pattern. The shift from /rpc/v0 to /api/rpc represents an educated guess about the server's routing table.

Third, the assistant assumes that a POST request with a JSON-RPC body is the correct way to invoke the method. The Content-Type: application/json header and the structured JSON-RPC payload (with jsonrpc, method, params, and id fields) follow the JSON-RPC 2.0 specification. This is correct for the protocol, but it assumes the server implements JSON-RPC 2.0 on this endpoint.

The Mistake: Incorrect Endpoint Path

The fundamental mistake revealed by the response is that /api/rpc is not the correct path. Instead of receiving a JSON-RPC response (which would look like {&#34;jsonrpc&#34;:&#34;2.0&#34;,&#34;result&#34;:{...},&#34;id&#34;:1}), the assistant receives the full HTML of the RIBSWeb application. This indicates that the request fell through to the web application's default route handler, which serves the single-page application's HTML shell.

This mistake is a natural consequence of debugging without complete documentation. The assistant does not know the exact routing configuration of the nginx proxy or the RIBS web server. It is probing in the dark, using common API path conventions. The "Bad Request" from /rpc/v0 and the HTML from /api/rpc together form a signal: neither path is correct, and the RPC endpoint must be somewhere else—perhaps on a different port entirely, or on a path that hasn't been tried yet.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains. First, familiarity with JSON-RPC 2.0 is essential—the structure of the request payload, the use of POST method, and the expected response format. Second, understanding of HTTP reverse proxying and nginx configuration helps explain why different paths can route to different backends. Third, knowledge of the specific architecture being built—the three-layer S3 system with Kuri storage nodes, S3 frontend proxies, and YugabyteDB—provides context for why cluster topology monitoring matters. Fourth, awareness of the debugging session's history explains why the assistant is probing this particular endpoint at this moment.

The response itself carries rich information. The HTML title "RIBSWeb" confirms that the request reached the RIBS web application. The presence of a React single-page application shell (with &lt;meta&gt;, &lt;link&gt;, and &lt;script&gt; tags for React and manifest) tells us the frontend is a modern JavaScript application. The HTML structure suggests the actual content is rendered client-side after JavaScript execution, which is why the response contains no monitoring data—the data would be fetched asynchronously by the React app.

Output Knowledge Created

This message produces several pieces of valuable knowledge. First, it definitively rules out /api/rpc as the correct path for the JSON-RPC endpoint on port 9010. Second, combined with the previous "Bad Request" on /rpc/v0, it narrows the search space—the RPC endpoint is not on either of these common paths. Third, it confirms that the web UI is serving correctly (the HTML is well-formed and contains the expected application shell), so the infrastructure for serving the frontend is working. Fourth, it reveals that the nginx proxy configuration likely only forwards specific paths to the Kuri backend, and generic API paths are not among them.

The debugging value extends beyond this single request. The assistant now knows it must investigate the actual routing configuration—either by reading the nginx config, examining the RIBS web server's route registration, or checking if the RPC endpoint lives on a different port (perhaps the Kuri node's internal API port like 7001 or 7002). This knowledge guides the next debugging steps.

The Thinking Process Visible in the Message

Although the message itself is just a bash command and its output, the thinking process is visible through the sequence of actions. The assistant is systematically testing hypotheses about the RPC endpoint location. The progression from /rpc/v0 (a versioned path) to /api/rpc (a generic API path) shows a search strategy based on common web API conventions. When the first attempt returns "Bad Request"—an HTTP-level error suggesting the server doesn't recognize the path or method—the assistant doesn't give up or assume the service is down. Instead, it tries an alternative path that might have different routing rules.

The choice of RIBS.ClusterTopology as the method name reveals another layer of thinking. The assistant has just implemented this method and knows it exists in the codebase. Testing it immediately after deployment is a verification step—confirming that the new code is actually running and accessible. The fact that the assistant tests via curl rather than through the web UI suggests a preference for direct, low-level verification before trusting the higher-level interface.

Broader Architectural Implications

This debugging moment illuminates an important architectural property: the separation between the web UI's HTML-serving path and the JSON-RPC data path. In a well-designed system, these might be unified under a single HTTP server with different route handlers, or they might be completely separate services. The fact that a POST request with JSON content-type returns HTML suggests the routing is not content-negotiation based but purely path-based. This has implications for how the monitoring dashboard fetches data—if the React app can reach the RPC endpoint but curl cannot on the same path, there may be CORS, nginx proxy_pass, or client-side routing considerations at play.

The empty cluster monitoring page that prompted this whole investigation is itself a symptom of the same routing issue. The React frontend likely makes a JSON-RPC call to populate the monitoring dashboard, and if that call fails (because the endpoint path is wrong or the backend returns HTML), the dashboard remains empty. Fixing the RPC endpoint routing would simultaneously fix both the curl test and the web UI's empty state.

Conclusion

Message 678 is a small but revealing moment in the larger narrative of building a distributed S3 cluster. A single curl command—carefully crafted with the correct JSON-RPC structure but directed at the wrong path—returns HTML instead of data, providing a clear signal that the routing configuration needs investigation. The message encapsulates the essence of systems debugging: forming hypotheses, testing them with precise probes, interpreting unexpected responses, and iterating toward understanding. It also demonstrates the importance of knowing your infrastructure's routing topology—a lesson that applies whether you're debugging a three-layer storage cluster or a simple web application.