The Curl That Revealed the Architecture Gap: Debugging Cross-Node S3 Reads in a Distributed Storage Cluster

In a single, deceptively simple bash command, a pivotal debugging moment unfolds. The message at index 2058 of this coding session is nothing more than a curl -v command — a verbose HTTP PUT request sent to a Kuri storage node in a three-machine QA cluster for the Filecoin Gateway's distributed S3 storage system. Yet this one command, and the reasoning that produced it, marks the critical turning point between an assumption that cross-node reads were broken and the discovery that the entire architecture needed a dedicated routing layer. It is a message that captures the essence of distributed systems debugging: the moment when a silent failure is dragged into the light, forcing a fundamental re-examination of how the system is supposed to work.

The Message

Here is the message exactly as it appeared in the conversation:

[assistant] [bash] # Check what's happening with the PUT
curl -v -X PUT -d "test-data" http://10.1.232.83:8079/testbucket/test.txt 2>&1 | head -30
  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed

  0      0   0      0   0      0      0      0                              0*   Trying 10.1.232.83:8079...
* Established connection to 10.1.232.83 (10.1.232.83 port 8079) from 10.0.0.1 port 42908 
* using HTTP/1.x
> PUT /testbucket/test.txt HTTP/1.1
> Host: 10.1.232.83:8079
> User-Agent: curl/8.18.0
> Accept: */*
> Content-Length: ...

The output is truncated by the head -30 pipe, cutting off before the server's response is shown. But the intent is clear: the assistant needs to see exactly what the server returns, not just whether the command succeeded or failed.

The Context: A Cross-Node Test That Returned Nothing

To understand why this message was written, we must look at what happened immediately before it. The session had been building toward a fully functional multi-node QA cluster. Two Kuri storage nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84) had been deployed alongside a single-node YugabyteDB instance on the head node (10.1.232.82). The cluster topology API was returning healthy status for both nodes. Everything looked good.

Then the user ran a load test against kuri1's S3 endpoint (port 8079) and noticed something suspicious: only kuri1 was receiving traffic. The user's observation — "Only kuri1 getting traffic seems wrong" — triggered a diagnostic chain. The assistant acknowledged the issue and ran a cross-node access test: write an object to kuri1, then try to read it from kuri2. The shared S3 CQL keyspace in YugabyteDB meant that both nodes should be able to see the object metadata. But the reads from both kuri1 and kuri2 returned empty. The write itself appeared to have succeeded (the output said "Written to kuri1"), but the subsequent reads produced nothing.

This is the critical moment. The assistant had assumed the PUT succeeded because the command didn't produce an error. But in distributed systems, silent failures are the norm. A curl command with -s (silent mode) suppresses progress output and error messages, and a non-200 response code can easily be missed. The assistant needed to see the raw HTTP exchange to understand what was really happening.## The Reasoning: Why This Curl Command Was Necessary

The assistant's reasoning at this point is a textbook example of systematic debugging. The sequence of events reveals a clear thought process:

  1. Observation: Cross-node reads return empty. The shared S3 metadata index should allow any node to locate objects, but the actual data blocks aren't being served.
  2. Hypothesis: The PUT request might be failing silently. The earlier test used curl -s -X PUT, which suppresses all progress and error output. A 400-level or 500-level response code could easily go unnoticed in a shell script that doesn't check exit codes.
  3. Diagnostic step: Run the same PUT request with verbose output (-v flag) to see the full HTTP request/response exchange. This is the message we're analyzing.
  4. Anticipation: The assistant expects to see either a successful 200 OK response (confirming the write worked, which would mean the read failure is a separate issue) or an error response (explaining why the object wasn't stored). The head -30 pipe is also telling. The assistant is being conservative about output volume, grabbing just enough of the response to see the HTTP status line and headers without flooding the conversation with the full response body. This is a pragmatic choice in a terminal-based interaction where screen real estate matters.

The Assumptions Embedded in This Message

Every debugging step carries assumptions, and this message is no exception. Several assumptions are visible:

Assumption 1: The PUT is the right thing to test. The assistant assumes that the cross-node read failure might be caused by a write failure. This is reasonable — if the object was never stored, of course reads will fail. But it's also possible the write succeeded and the read path is broken. The verbose curl will help distinguish these cases.

Assumption 2: The S3 endpoint is listening on port 8079. The assistant assumes that the Kuri daemon's S3 server is running and accessible on port 8079. Earlier in the session, the Kuri nodes had been configured with RIBS_S3API_BINDADDR=":8079", so this is a well-founded assumption. But it's still an assumption — the service could have crashed, the port could be firewalled, or the configuration could have been overwritten.

Assumption 3: The S3 API doesn't require AWS SigV4 signing. The PUT request uses a bare curl -X PUT -d "test-data" without any AWS authentication headers. The assistant is assuming the server accepts unsigned requests. This turns out to be incorrect — the subsequent message (index 2060) reveals a 500 error with "invalid content sha256" because the S3 server requires the x-amz-content-sha256: UNSIGNED-PAYLOAD header. This is a significant finding: the S3 implementation is stricter about request validation than the assistant anticipated.

Assumption 4: The test bucket exists. The PUT targets /testbucket/test.txt, but there's no explicit bucket creation step visible in the preceding messages. The assistant assumes the S3 server will create the bucket on demand or that it already exists from earlier testing. If the bucket doesn't exist, the PUT would fail with a "NoSuchBucket" error.

The Mistake: Silent Mode and Trusting Exit Codes

The most significant mistake visible in the context around this message is the earlier use of curl -s for the cross-node test. The -s (silent) flag suppresses the progress meter and error output, making it easy to miss a non-200 response. The output from message 2057 shows:

Written to kuri1
Reading from kuri2:

Reading from kuri1:

The "Written to kuri1" message comes from an echo statement in the shell script, not from curl. It's printed regardless of whether the PUT succeeded. The empty lines after "Reading from kuri2:" and "Reading from kuri1:" suggest the GET requests returned nothing — but without verbose output, we can't tell if that's a 404, a 500, or an empty 200 response.

This is a classic debugging trap in shell scripting: assuming that because a command ran without visible errors, it succeeded. The assistant's decision to re-run with -v is the correction of this mistake.

The Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the FGW architecture: The Filecoin Gateway's distributed S3 system consists of Kuri storage nodes (which serve S3 API directly on port 8079) and a separate S3 proxy frontend (which routes requests to the correct backend). At this point in the session, the proxy hasn't been deployed yet — the assistant is testing against raw Kuri nodes.
  2. Knowledge of the QA cluster topology: Three physical nodes (head at 10.1.232.82, kuri1 at 10.1.232.83, kuri2 at 10.1.232.84) with YugabyteDB on the head node providing shared CQL and SQL databases.
  3. Knowledge of HTTP and S3 protocols: Understanding what curl -v shows, what the various HTTP headers mean, and how S3 signing works.
  4. Knowledge of the session history: The earlier debugging of dirty migration states, the configuration of FGW_BACKEND_NODES for cluster topology, and the load test that revealed the traffic imbalance.

The Output Knowledge Created

This message produces several pieces of knowledge:

  1. The raw HTTP exchange for a PUT to the Kuri S3 endpoint, showing the request headers, connection details, and (after the truncated output) the response status code.
  2. Confirmation that the TCP connection succeeds — the "Established connection" line proves the Kuri daemon is listening and accepting connections on port 8079.
  3. Evidence of the request format — the assistant can see exactly what headers curl sends by default and what the server receives.
  4. The basis for the next debugging step — once the response is visible, the assistant can determine whether the write succeeded or failed, and proceed accordingly.

The Thinking Process Revealed

The assistant's thinking process, visible through the sequence of commands and their outputs, follows a classic scientific method:

  1. Observation: Cross-node reads return empty.
  2. Question: Is the write succeeding? Is the read path broken? Is there a fundamental architectural issue?
  3. Hypothesis: The write might be failing silently due to missing S3 headers or authentication requirements.
  4. Prediction: A verbose curl will show either a 200 OK (write works, read path is broken) or an error (write fails, explaining the empty reads).
  5. Test: Run the verbose PUT command.
  6. Analysis: Examine the response to determine the next step. This is not just debugging — it's knowledge construction. Each step builds on the previous one, and the assistant is effectively building a mental model of the system's behavior in real time. The message we're analyzing is the test step in this cycle.

The Architectural Discovery That Followed

The verbose curl in message 2058 reveals a 500 error with "invalid content sha256" (visible in the subsequent message 2060). This leads to the discovery that the S3 server requires x-amz-content-sha256: UNSIGNED-PAYLOAD for unsigned requests. Once the assistant adds this header, the PUT succeeds, and the cross-node test reveals the real problem: kuri2 can see the object metadata in the shared CQL keyspace but cannot serve the actual data blocks, which are stored on kuri1.

This is the architectural gap. The Kuri nodes are designed to serve data from their local blockstores. They share metadata through YugabyteDB, but the actual block data is not replicated or fetched across nodes. The system needs a routing layer — the S3 proxy frontend — that reads the node_id from the S3 object metadata and forwards the request to the correct backend node.

The assistant then deploys the S3 proxy on the head node (port 8078), configures it with the backend node list and CQL connection details, and verifies that cross-node reads work correctly. The load test against the proxy shows both Kuri nodes receiving traffic evenly.

Conclusion

Message 2058 is a small command with large consequences. A single curl -v command, born from the recognition that silent mode had hidden a failure, sets off a chain of discoveries that fundamentally reshape the deployment architecture. It reveals that the S3 implementation requires specific headers, that the Kuri nodes cannot serve each other's blocks, and that the system needs a dedicated routing proxy to function as a multi-node cluster.

The message is a masterclass in distributed systems debugging: start with the simplest possible diagnostic, question your assumptions, and let the evidence guide you to the architectural truth. In a conversation spanning hundreds of messages across multiple sessions, this one curl command is the pivot point where the cluster stops being a collection of independent nodes and becomes a true distributed system.