The 404 That Revealed Everything: A Debugging Crossroads in a Distributed S3 Architecture

Introduction

In the long and intricate process of building a horizontally scalable S3-compatible storage system, there comes a moment when all the infrastructure pieces are finally in place—the containers are running, the health checks pass, and the architecture looks correct on paper. Then you run a simple test: upload a file, download it back. And you get a 404.

This article examines a single message from an opencode coding session—message index 625—where the assistant, having just completed a marathon debugging session spanning HTTP route conflicts, missing database columns, and container orchestration fixes, runs a straightforward curl command to verify that a freshly uploaded S3 object can be retrieved. The response is HTTP/1.1 404 Not Found. On the surface, this is the correct response for a missing object. But the object should exist—the assistant had just uploaded it moments earlier. This 404 is therefore not a routine "object not found" but a symptom of a deeper failure in the write path of a three-layer distributed system.

The Message Itself

The subject message is deceptively simple—a single bash command with its output:

[assistant] [bash] curl -sv http://localhost:8078/test-bucket/hello.txt 2>&1
* Host localhost:8078 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8078...
* Established connection to localhost (::1 port 8078) from ::1 port 56934 
* using HTTP/1.x
> GET /test-bucket/hello.txt HTTP/1.1
> Host: localhost:8078
> User-Agent: curl/8.18.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Sat, 31 Jan 2026 00:59:27 GMT
< Content-Length: 10
< 
{ [10 byt...

The assistant uses curl -sv—the -s flag for silent mode (suppressing progress output) and the -v flag for verbose output, which displays the full HTTP request and response headers. This is a deliberate diagnostic choice. The verbose output reveals that the TCP connection succeeded, the HTTP request was sent correctly, and the server responded promptly. The problem is not at the network or transport layer; it is at the application layer. The S3 proxy is alive, accepting connections, and returning valid HTTP responses—but it cannot find the object.

Why This Message Was Written: The Debugging Context

To understand why this message exists, one must appreciate the debugging marathon that preceded it. The assistant had been building a test cluster for a horizontally scalable S3 architecture consisting of three layers:

  1. S3 Frontend Proxy (port 8078) — a stateless HTTP server that accepts S3 API requests and routes them to backend storage nodes.
  2. Kuri Storage Nodes (ports 7001, 7002) — the actual storage engines that hold data and metadata.
  3. YugabyteDB — the shared metadata database storing object locations and attributes. In the messages immediately preceding this one (indices 596–623), the assistant had fixed a cascade of failures: - HTTP route conflict: Go 1.22's ServeMux rejected registering both GET / and /healthz because the former is method-specific with a wildcard path while the latter is path-specific with wildcard methods. The assistant replaced the standard mux with a custom handler that manually routes /healthz before falling through to method-based dispatch. - Missing database column: The S3Objects table in YugabyteDB lacked the node_id column required by the S3 proxy's lookup queries. The assistant dropped and recreated the table manually. - Missing healthz endpoint: The S3 proxy had no /healthz endpoint at all, causing requests to that path to be treated as S3 object lookups. The assistant added one. - Web UI proxying: The web UI container was fixed to use Nginx as a reverse proxy to the Kuri nodes. After rebuilding the Docker image, restarting containers, and verifying that curl http://localhost:8078/healthz returned ok, the assistant was ready for the ultimate validation: an end-to-end S3 upload and download. Message 624 shows the PUT command: echo &#34;Hello from test cluster!&#34; | curl -s -X PUT --data-binary @- http://localhost:8078/test-bucket/hello.txt. Message 625 is the GET that follows—the verification step that should confirm the object was stored successfully. The 404 response is therefore the moment of truth, and it is a moment of failure.## Input Knowledge Required to Understand This Message To grasp the significance of this 404 response, the reader must understand several layers of context:
  4. The three-layer architecture: The S3 proxy on port 8078 does not store data itself. It is a stateless routing layer that forwards PUT requests to one of the Kuri backend nodes and records the object's location (which node holds it) in YugabyteDB. A GET request must first query YugabyteDB to discover which Kuri node owns the object, then fetch the data from that node. This means a 404 could originate from any of these layers.
  5. The recent schema fix: The S3Objects table had just been manually recreated with the node_id column. Before the fix, the S3 proxy was crashing with "Undefined Column" errors. The fix was applied by dropping the old table and creating a new one via ycqlsh—but this was done outside the db-init container's automated migration process. This means the schema fix was ephemeral; if the database were ever re-initialized from scratch, the old schema would return.
  6. The PUT command's silence: In message 624, the assistant ran echo &#34;Hello from test cluster!&#34; | curl -s -X PUT --data-binary @- http://localhost:8078/test-bucket/hello.txt. The -s flag suppresses response output. The assistant did not see the HTTP status code of the PUT. This is a crucial detail: the PUT might have succeeded (returning 200) or failed (returning 500), but the assistant did not verify. The GET in message 625 is the first verification attempt, and it reveals that whatever happened during the PUT, the object did not end up retrievable.
  7. The proxy's backend routing logic: The S3 proxy maintains a BackendPool of Kuri nodes. For PUT requests, it selects a backend node (likely using a hash of the bucket/key or a round-robin strategy), forwards the data, and writes the node assignment to YugabyteDB. For GET requests, it queries YugabyteDB for the node_id, then fetches from that node. If the PUT's database write failed silently, or if the backend selection logic is broken, the GET would correctly return 404.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message, some of which turn out to be incorrect:

Assumption 1: The PUT succeeded. The assistant does not check the PUT response before proceeding to the GET. This is a natural workflow—you upload, then verify—but it means the GET's 404 could be caused by a failed PUT that the assistant never noticed. The subsequent messages (626–631) reveal that the PUT did fail: the S3 proxy returned 500 Internal Server Error for subsequent PUT attempts. The assistant's debugging then shifts to investigating backend connectivity.

Assumption 2: The S3 proxy is correctly routing to backends. The assistant assumes that because the proxy's /healthz endpoint returns ok, the proxy can reach the Kuri nodes. But a health check endpoint is a simple "am I alive?" signal; it does not validate the complex routing logic that selects a backend node, forwards the HTTP request body, and writes metadata to YugabyteDB. The 404 reveals that at least one of these steps is failing.

Assumption 3: The database schema is correct. The assistant manually dropped and recreated the S3Objects table, but the db-init container's startup script was not updated to include the new schema. This means any future restart of the cluster from a clean state would recreate the old, broken schema. More immediately, it means the assistant is relying on a manually applied schema fix that is not part of the automated deployment—a maintenance burden that will need to be addressed.

Assumption 4: The object key is correct. The GET request uses the exact same path (/test-bucket/hello.txt) as the PUT. This is correct, but the assistant does not consider the possibility that the S3 proxy might be transforming or normalizing keys in a way that makes the GET path mismatch the PUT path.

The Thinking Process Visible in the Message

The message itself is a single command with its output, but the thinking process is visible in the choice of command and flags. The assistant uses curl -sv rather than a simple curl -s. The -v flag is diagnostic: it shows the full HTTP request and response headers, confirming that the TCP connection succeeded, the HTTP request was well-formed, and the server responded with a proper HTTP response. The assistant is systematically ruling out lower-level failures before investigating the application logic.

The output shows HTTP/1.1 404 Not Found with Content-Length: 10. The body is truncated in the output ({ [10 byt...), but the assistant can infer that the body is likely the text "Not Found" (10 characters). This is the S3 proxy's standard response for missing objects. The response headers include X-Content-Type-Options: nosniff, which is a security header that prevents MIME sniffing—this confirms the response is coming from the Go HTTP server (which sets this header by default), not from some other middleware.

The Date header shows Sat, 31 Jan 2026 00:59:27 GMT. This is approximately 57 seconds after the PUT command was issued (the PUT was at approximately 00:58:27 based on the previous message's timestamps). The 57-second gap is reasonable for a manual test sequence where the assistant typed the GET command after seeing the PUT complete.

Output Knowledge Created by This Message

This message produces several pieces of knowledge that drive the subsequent debugging:

  1. The write path is broken. The object was not stored, despite the PUT command completing without an error message. The assistant now knows that the S3 proxy's write path has a defect.
  2. The read path works correctly for missing objects. The S3 proxy correctly returns 404 for objects that don't exist, rather than crashing or returning a 500 error. This is actually a positive sign—the database query logic works; it's the data that's missing.
  3. The network layer is functional. The TCP connection, DNS resolution, and HTTP protocol handling all work correctly between the client and the S3 proxy.
  4. The health check endpoint is not sufficient validation. The fact that /healthz returns ok does not guarantee that the full request pipeline works. The assistant will need to test the backend routing and database write paths directly.

The Broader Significance

This message, for all its apparent simplicity, represents a classic moment in distributed systems debugging: the point where the system looks healthy but behaves incorrectly. The containers are running, the health checks pass, the logs show no errors—yet the fundamental operation (store and retrieve data) fails. This is the difference between infrastructure health and application correctness.

The assistant's response to this 404 is instructive. Rather than assuming a transient error or restarting containers, the assistant immediately queries the database (message 626) to check if the object was stored. When the database shows zero rows, the assistant narrows the investigation to the PUT path, testing direct backend access and discovering that the S3 proxy returns 500 for PUT requests. This systematic narrowing—from end-to-end test, to database inspection, to component isolation—is the hallmark of effective debugging in distributed systems.

In the end, this single 404 response triggers a chain of investigation that will uncover deeper issues in the S3 proxy's backend routing logic, ultimately leading to a more robust and correctly functioning system. The message is a testament to the importance of verification in software engineering: the PUT "succeeded" silently, but only the GET revealed the truth.