The Silent Failure: Debugging a Phantom S3 Write in a Distributed Storage Cluster

The Message

[assistant] The object wasn't written to the shared S3 keyspace. Let me check the kuri node's S3 endpoint directly:
[bash] curl -s http://localhost:8078/test-bucket/hello.txt -v 2>&1 | head -20
* Host localhost:8078 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8078...
* Established connection to localhost (::1 port 8078) from ::1 port 54934 
* using HTTP/1.x
> GET /test-bucket/hello.txt HTTP/1.1
> Host: localhost:8078
> User-Agent: curl/8.18.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Sat, 31 Jan 2026 00:59:35 GMT
< Content-Length: 10
< 
{ [10 byt...
[bash] docker logs test-cluster-s3-proxy-1 2>&1 | tail -10
S3 Frontend Proxy started on :8078 (node: proxy-1)
Backend nodes: 2 configured

Context and Motivation

Message 627 arrives at a critical juncture in the development of a horizontally scalable S3-compatible storage system built on top of the Filecoin Gateway (FGW) architecture. The assistant has been working through a gauntlet of infrastructure and code issues: HTTP route conflicts in Go 1.22's strict ServeMux, missing database columns in YugabyteDB, broken container networking, and configuration mismatches between stateless S3 frontend proxies and Kuri storage backend nodes. Each fix has been a step toward a working three-layer architecture: S3 proxy (port 8078) → Kuri storage nodes → YugabyteDB metadata store.

In the immediately preceding messages, the assistant had finally gotten the cluster to a seemingly healthy state. Both Kuri nodes were running. The S3 proxy's /healthz endpoint returned ok. The web UI at port 9010 was serving React content. Everything looked green. Then came the moment of truth: a real S3 operation. The assistant issued a PUT request to store a simple text object ("Hello from test cluster!") at the path /test-bucket/hello.txt through the S3 proxy on port 8078. The command returned with no output—ambiguous, but not an obvious error. The subsequent GET request, however, returned a crisp 404 Not Found. The object had vanished.

Message 626, the immediate predecessor, confirmed the worst: the database was empty. A SELECT * FROM filecoingw_s3.S3Objects returned zero rows. The PUT had apparently been accepted by the proxy but had never materialized in the shared metadata store. This is the moment message 627 is written—the moment when the assistant realizes that the cluster's "green" status was deceptive, and that a silent, invisible failure mode exists somewhere in the data path between the curl command and the YugabyteDB table.

The Reasoning and Thinking Process

The message opens with a declarative diagnosis: "The object wasn't written to the shared S3 keyspace." This is not a guess; it's a conclusion drawn from direct evidence. The assistant had just queried the database and found nothing. The statement serves as a thesis for the investigation that follows. The assistant then articulates a plan: "Let me check the kuri node's S3 endpoint directly."

This plan reveals a specific hypothesis about where the failure might lie. The assistant suspects that the S3 proxy—the stateless frontend layer—might be the point of failure. By testing the Kuri node's S3 endpoint directly (which, confusingly, also listens on port 8078 within the container network), the assistant hopes to isolate whether the problem is in the proxy's routing logic or deeper in the Kuri node's own S3 handler. If the direct endpoint also returns 404, the issue is likely in the Kuri node's handling or the database layer. If the direct endpoint works, the proxy is the culprit.

The first command executed is a verbose GET request to the same path through the same proxy. The output is identical to the previous GET: 404 Not Found. This is expected—if the object wasn't written, it can't be read. The verbose output does, however, confirm that the proxy is responding at all, that HTTP/1.1 is working, and that the response headers are well-formed. The proxy is alive but has no data to serve.

The second command checks the S3 proxy's logs. The output is striking in its brevity and silence:

S3 Frontend Proxy started on :8078 (node: proxy-1)
Backend nodes: 2 configured

There are no error messages. No "failed to route to backend." No "connection refused." No "database query failed." The proxy logged its startup and then... nothing. This is deeply informative in its own way. The absence of error logs tells the assistant that the proxy did not crash, did not encounter a database error, and did not fail to connect to a backend. The PUT request was handled—or at least, it didn't produce any log output. This narrows the search space considerably. The proxy either silently dropped the request, routed it to a backend that silently dropped it, or the PUT succeeded at the proxy level but the data never reached the database for some other reason.

Assumptions and Their Implications

The assistant makes several assumptions in this message, some explicit and some implicit.

The first assumption is that the PUT request actually reached the S3 proxy. The curl command echo "Hello from test cluster!" | curl -s -X PUT --data-binary @- http://localhost:8078/test-bucket/hello.txt returned no output, which in curl's default mode means no HTTP response body was received—but it also means no error message was printed to stderr. The assistant implicitly assumes this means the request was accepted, but in reality, a silent failure could have occurred at the TCP level, at the HTTP level (a 500 with no body), or at the application level. The assistant does not re-run the PUT with verbose output to confirm the HTTP status code.

The second assumption is that the database query is authoritative. The assistant ran SELECT * FROM filecoingw_s3.S3Objects and got zero rows. This assumes that the correct keyspace and table are being queried, that the CQL connection is working, and that the data would have been written to this table if the PUT had succeeded. Given that the assistant had just manually created the S3Objects table with the node_id column (messages 609-611), this is a reasonable assumption, but it's worth noting that the table schema had been changed multiple times during this session, and there could be a mismatch between what the application code expects and what the database actually contains.

The third assumption is that the proxy logs would contain evidence of a failure. The assistant checks docker logs test-cluster-s3-proxy-1 and finds only startup messages. The assumption is that if the proxy had encountered an error during the PUT operation, it would have logged it. This is a reasonable assumption for a well-instrumented system, but it's possible that the error occurs at a layer that doesn't produce logs—for example, a silent panic recovery, a dropped connection, or a routing decision that silently discards the request.

The fourth assumption is that the Kuri node's S3 endpoint is accessible and would behave differently from the proxy. The assistant says "Let me check the kuri node's S3 endpoint directly," but then runs the curl command against the same localhost:8078 address. This is actually the proxy again, not the Kuri node directly. To reach the Kuri node directly, the assistant would need to use the Docker internal network address (e.g., http://kuri-1:8078/). This is a subtle but important mistake—the assistant is testing the same endpoint and getting the same result, which doesn't isolate the problem.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains. First, an understanding of the S3 protocol and its HTTP semantics: that PUT creates or overwrites an object, GET retrieves it, and 404 means the object doesn't exist. Second, familiarity with distributed storage architectures: the concept of a stateless proxy layer that routes to storage backends, and a shared metadata database (YugabyteDB, a distributed SQL database compatible with Cassandra's CQL). Third, Docker container networking: the assistant is running a multi-container setup where services communicate over an internal Docker network, and the S3 proxy exposes port 8078 to the host. Fourth, the Go programming language and its HTTP server patterns, particularly the ServeMux routing behavior that had caused earlier panics. Fifth, the YugabyteDB CQL shell (ycqlsh) and the schema of the S3Objects table, which the assistant had just modified to include a node_id column.

The reader also needs to understand the broader architectural context established earlier in the session: that the S3 proxy is a stateless frontend that routes requests to Kuri backend nodes based on some routing strategy, that each Kuri node has its own local storage but shares metadata through YugabyteDB, and that the system is designed to be horizontally scalable with multiple proxy instances and multiple Kuri nodes.

Output Knowledge Created

This message produces several valuable pieces of knowledge. First, it confirms that the S3 proxy is operational and responsive—it returns well-formed HTTP responses with proper headers and status codes. Second, it establishes that the proxy's logging is not capturing any errors related to the failed PUT operation, which is itself a diagnostic finding. Third, it narrows the investigation to the data path between the proxy's HTTP handler and the database write—the proxy accepted the request, but the data never reached the database. Fourth, it reveals that the proxy's backend routing configuration ("Backend nodes: 2 configured") is active, meaning the proxy believes it has two healthy backends to route to.

The message also implicitly creates knowledge about what doesn't work: testing the proxy endpoint from the host with curl is not the same as testing the Kuri node directly. The assistant's attempt to "check the kuri node's S3 endpoint directly" by curling localhost:8078 actually tests the proxy again. This is a methodological error that will need to be corrected in subsequent messages (which it is—the assistant later uses docker exec to test from within the container network).

Mistakes and Incorrect Assumptions

The most significant mistake in this message is the failure to re-run the PUT operation with verbose output. The original PUT command used -s (silent mode), which suppresses progress meters and error messages but also suppresses the HTTP response body and status code. The assistant never confirmed what HTTP status code the PUT returned. It could have been a 200 OK (which would make the 404 on GET even more puzzling), a 500 Internal Server Error (which would immediately indicate a backend problem), or a 201 Created (which would suggest the proxy thinks it succeeded but the database disagrees). Without this information, the assistant is debugging blind.

The second mistake is the logical leap from "the database is empty" to "the object wasn't written." While this is the most likely explanation, it's also possible that the object was written to a different keyspace, a different table, or that the database query itself failed silently. The assistant does not check the database logs or verify the CQL connection from the proxy's perspective.

The third mistake is the assumption that the proxy logs would contain evidence of a failure. The assistant checks the logs and finds nothing, then moves on without considering that the proxy might not log successful operations, or that the error might occur before the logging layer is reached. In subsequent messages, the assistant discovers that the PUT actually returned 500 Internal Server Error (message 631), which means the proxy did encounter an error but didn't log it. This is a significant gap in the proxy's instrumentation.

The fourth mistake is the incorrect direct-endpoint test. The assistant says "Let me check the kuri node's S3 endpoint directly" but then curls localhost:8078, which is the proxy's host port mapping. The Kuri node's internal endpoint would be something like http://kuri-1:8078/ from within the Docker network. This mistake delays the diagnosis by one round-trip.

The Deeper Significance

Message 627 is a classic example of debugging a distributed system where each component reports "green" but the system as a whole is broken. The proxy reports "Backend nodes: 2 configured." The Kuri nodes report "Daemon is ready." The database reports "Databases initialized successfully." Yet an S3 PUT operation produces no persisted data. This is the kind of failure that haunts distributed systems: the absence of errors is not the same as the presence of correctness.

The message also illustrates the importance of following the data. The assistant traces the path from curl to proxy to database, checking each hop for evidence. The proxy logs are silent, which is itself a finding. The database is empty, which is another finding. The GET returns 404, which confirms the database is consistent with the proxy's behavior. The assistant is methodically building a chain of evidence, even if some of the investigative steps are imperfect.

In the broader narrative of this coding session, message 627 is the turning point where the assistant realizes that the cluster's apparent health is deceptive. The proxy starts, the backends connect, the health endpoint returns "ok"—but real S3 operations fail silently. This realization drives the next several messages, where the assistant discovers that the PUT returns a 500 error, that the S3 handler requires a Content-SHA256 header that curl doesn't send by default, and that the proxy needs to auto-inject this header for unsigned requests. The silent failure turns out to be a missing header validation, not a routing or database issue at all.

Conclusion

Message 627 captures a moment of diagnostic clarity in the midst of complex distributed systems debugging. The assistant correctly identifies that a PUT operation has failed to persist data, formulates a hypothesis about the failure location, and begins testing. The investigative steps reveal important boundary conditions: the proxy is alive but logging nothing, the database is empty, and the direct backend test is inadvertently testing the proxy again. These findings, even the mistaken ones, drive the investigation forward toward the root cause. The message is a testament to the iterative, hypothesis-driven nature of debugging distributed systems, where each answer generates new questions, and where the absence of evidence is itself a form of evidence.