The Silent 200: Debugging an S3 Proxy That Said "OK" But Didn't Store Anything
In distributed systems debugging, the most dangerous response is a silent success. A 200 OK that doesn't actually mean the operation completed is a trap that can send developers down rabbit holes for hours. This is precisely the situation captured in message 628 of an opencode coding session, where an assistant building a horizontally scalable S3 architecture encounters a PUT request that returns success but a subsequent GET that returns 404 — and must decide where to look for the bug.
The Message
The target message is brief but revealing:
The PUT went through but the GET didn't find it. The problem is likely the backends aren't being reached. Let me check backend connectivity:
>
`` [bash] docker exec test-cluster-s3-proxy-1 wget -q -O- http://kuri-1:8078/healthz 2>&1 OK ``
In just three lines, the assistant articulates a hypothesis, designs a diagnostic test, executes it, and gets a result. But the real story lies in what this message doesn't yet know — and how the reasoning visible here sets the stage for the actual discovery.
The Context: A Cluster in Recovery
To understand this message, one must appreciate the state of the test cluster at this moment. The assistant had just emerged from a grueling debugging session. The Kuri storage nodes had been crashing with a Go 1.22 HTTP route conflict between HEAD / and GET /healthz. The web UI container was broken. The S3 proxy was returning Internal Server Error because the S3Objects table in YugabyteDB lacked a required node_id column. The database schema had to be manually dropped and recreated. A healthz endpoint had to be added to the S3 frontend proxy.
By message 628, all of these fires had been extinguished. The Kuri nodes were running. The web UI was serving pages. The S3 proxy was responding. The assistant had just performed a triumphant end-to-end test: a PUT to http://localhost:8078/test-bucket/hello.txt and a subsequent GET of the same URL. The PUT returned what appeared to be success. The GET returned 404 Not Found.
This is the classic distributed systems puzzle: data that was supposedly written cannot be read.
The Hypothesis: Backend Connectivity
The assistant's reasoning is clearly stated: "The problem is likely the backends aren't being reached." This is a logical first hypothesis given the architecture. The system has three layers: an S3 frontend proxy on port 8078, two Kuri storage nodes (kuri-1 and kuri-2), and a shared YugabyteDB. The proxy is supposed to route requests to the backends. If the PUT succeeded at the proxy level but the object wasn't found on GET, perhaps the proxy never forwarded the PUT to a backend at all — or perhaps the proxy can't reach the backends for the GET request.
The diagnostic test is elegantly simple: exec into the s3-proxy container and use wget to hit the kuri-1 backend's healthz endpoint. This tests both network connectivity (can the proxy container resolve and reach the kuri-1 hostname?) and backend responsiveness (is the backend's HTTP server alive?). The result is a clean OK.
What the Test Actually Proved
The healthz response of "OK" disproves the connectivity hypothesis. The backends are reachable. But this creates a deeper mystery: if the backends are reachable, why did the GET return 404 for an object that was supposedly just written?
The answer, which the assistant would discover in subsequent messages, was more subtle than a connectivity issue. The Kuri backend nodes implement proper AWS S4-compatible signing, which requires the x-amz-content-sha256 header on PUT requests. When curl sends a PUT without this header, the backend rejects the request. However, the S3 proxy was returning a 200 OK status code to the client without actually forwarding the request to the backend successfully — or perhaps the proxy was returning 200 based on its own logic without verifying that the backend accepted the data.
This is a critical architectural insight: the proxy's success response was a lie. The 200 OK meant "the proxy received your request and processed it according to its own logic," not "the data has been durably stored on a backend node." The GET then correctly returned 404 because nothing was ever stored.
Assumptions and Their Consequences
The assistant made several assumptions in this message, some explicit and some implicit. The explicit assumption is that the backends might be unreachable — a reasonable first guess that was quickly disproven. The implicit assumptions are more interesting.
First, the assistant assumed that the PUT "went through" — that a response with no error message and a 200 status code meant the data was stored. In many HTTP APIs, this is a safe assumption. But in a distributed system with a proxy layer, the proxy can return success before the backend has confirmed persistence. The assistant's mental model treated the S3 proxy as a transparent pass-through, but it was actually behaving more like a buffer that acknowledged receipt before verifying delivery.
Second, the assistant assumed that the healthz endpoint on the backend was sufficient to test end-to-end request routing. A healthz check confirms the server is running and accepting connections, but it doesn't exercise the full request path — authentication, header parsing, body handling, database writes. The assistant would later discover that the real issue was in the header forwarding logic, not in connectivity.
The Thinking Process Visible in the Message
The message reveals a methodical debugging approach. The assistant observes a symptom (PUT succeeds, GET fails), forms a hypothesis (backends unreachable), designs a minimal test (wget to backend healthz), and executes it. The test is well-chosen: it's the simplest possible check that would confirm or refute the hypothesis. Using wget -q -O- suppresses verbose output and just returns the response body, making the result immediately readable.
The choice to test from inside the s3-proxy container is also significant. Rather than testing from the host machine (which might have different network routing), the assistant execs into the container that's actually experiencing the problem. This eliminates variables like Docker networking, port mappings, and host-level firewall rules.
The Broader Significance
This message captures a universal moment in distributed systems development: the moment when a hypothesis is tested and found insufficient. The healthz check returning "OK" doesn't solve the problem, but it narrows the search space. The assistant must now look elsewhere — at the request forwarding logic, at the header handling, at the backend's request validation.
The debugging journey that follows this message leads to the discovery that the S3 proxy needs to inject the x-amz-content-sha256 header for unsigned requests. The fix involves modifying the proxy's proxyRequest method to add UNSIGNED-PAYLOAD as the content SHA256 when the client hasn't provided one. This is a protocol-level fix, not a connectivity fix — a different category of bug entirely.
Conclusion
Message 628 is a snapshot of a developer in the act of debugging, thinking aloud about what might be wrong and testing a hypothesis. The hypothesis turns out to be wrong, but the test is clean, fast, and definitive. The "OK" response from the backend healthz closes one line of investigation and opens another. In the broader narrative of building a horizontally scalable S3 architecture, this message represents the transition from infrastructure debugging (containers crashing, databases missing columns) to protocol debugging (S3 signing requirements, header forwarding). It's a small but instructive moment in the life of a distributed system — and a reminder that in distributed systems, a silent 200 is often the most deceptive response of all.