The "Not Found" That Meant Everything: Decoding a Debugging Breakthrough in Distributed S3 Architecture

The Message

In a debugging session spanning dozens of iterations across a horizontally scalable S3 storage cluster, one short message stands as a quiet turning point. After fixing HTTP route conflicts, missing database columns, and broken container proxies, the assistant writes:

"Not Found" is good - that means the lookup succeeded but the object doesn't exist (which is expected for /healthz since it's not an S3 path). Let me test actual S3 operations:

>

``bash curl -s http://localhost:8078/ 2>&1 Not Found ``

Seven words of output — "Not Found" — and the assistant declares it good. To an outside observer, this might look like failure. A service returning 404 is rarely cause for celebration. But in the context of a distributed system being assembled piece by piece, this "Not Found" represents a fundamental shift: the system has crossed the boundary from broken to operational but empty. This article unpacks why that single response mattered, what it reveals about the debugging process, and how the assistant's reasoning transformed a seemingly negative result into a validation of architectural correctness.## The Context: A Cluster in Crisis

To understand why "Not Found" was cause for celebration, we need to understand the state of the cluster moments before this message. The assistant was deep in the trenches of debugging a three-layer distributed S3 architecture built on the Filecoin Gateway project. The architecture followed a clear roadmap: stateless S3 frontend proxies (running on port 8078) that route requests to Kuri storage nodes (each with their own LocalWeb endpoints), which in turn store data and reference metadata in a shared YugabyteDB database.

Just minutes earlier, the cluster was in shambles. Both Kuri storage nodes were crashing on startup with a Go 1.22 HTTP route conflict — the standard ServeMux could not disambiguate between HEAD / and GET /healthz, causing a panic. The web UI container (port 9010) was a dead placeholder that printed a message instead of proxying to the actual Kuri web interface. The S3 proxy (port 8078) was returning "Internal Server Error" on every request because the S3Objects table in YugabyteDB was missing the node_id column — the database schema had been created from an earlier migration that predated the column addition, and the db-init container was idempotent (it only created keyspaces, not tables), so the stale schema persisted silently.

The assistant had been methodically working through these issues. The HTTP route conflict was resolved by replacing the standard ServeMux with a custom handler that manually inspected the request path and method. The web UI was fixed by replacing the placeholder with an Nginx reverse proxy configuration. The database schema was corrected by manually dropping and recreating the S3Objects and MultipartUploads tables with the node_id column included. Each fix was verified, the Docker image rebuilt, the cluster restarted, and the cycle repeated.

The Moment of Verification

After rebuilding the Docker image, restarting the cluster, and manually fixing the database schema, the assistant ran two curl commands. The first tested the health endpoint:

curl -s http://localhost:8078/healthz

The response was "Not Found." The second tested the root S3 path:

curl -s http://localhost:8078/

Again, "Not Found."

This is where the message in question occurs. The assistant immediately recognized that "Not Found" was not an error — it was the correct behavior for an empty S3-compatible storage system. The S3 proxy was now successfully connecting to the database, performing lookups, and returning proper HTTP responses. The previous "Internal Server Error" had been replaced by a semantically meaningful HTTP status code.

The Reasoning Behind the Interpretation

The assistant's reasoning reveals a deep understanding of how the system is supposed to work. The key insight is that the S3 proxy performs a lookup on every request: it queries the S3Objects table to find which Kuri storage node owns a particular object. When the table is empty (as it would be in a freshly initialized cluster), the lookup correctly returns zero results, and the proxy responds with "Not Found" (HTTP 404). This is the expected behavior for a GET request on a non-existent object.

The assistant explicitly articulates this reasoning: "that means the lookup succeeded but the object doesn't exist (which is expected for /healthz since it's not an S3 path)." This statement encodes several layers of understanding:

  1. The database connection is working. The proxy can reach YugabyteDB, authenticate, and execute CQL queries. The earlier "Undefined Column" error is gone.
  2. The query logic is correct. The proxy is constructing proper SELECT statements, the database is parsing them, and the result set is empty — not erroring.
  3. The HTTP routing is functional. The custom handler is correctly dispatching requests to the appropriate handler functions based on path and method.
  4. The response semantics are correct. The proxy returns 404 for missing objects, which is the standard S3 behavior. This last point is particularly important. The assistant had just fixed a bug where the proxy was returning "Internal Server Error" (HTTP 500) for every request, including the health check. A 500 error indicates a server-side failure — something is broken. A 404 indicates a correct operational state: the server is working, it just doesn't have what you're asking for. The transition from 500 to 404 is the transition from "broken" to "working but empty."## Assumptions Embedded in the Message The assistant's interpretation of "Not Found" rests on several assumptions, most of which are well-founded but worth examining: Assumption 1: The database schema is now correct. The assistant had manually dropped and recreated the S3Objects table with the node_id column. However, the db-init container in the Docker Compose setup still only creates keyspaces, not tables. If the cluster were restarted from scratch without the manual intervention, the old schema would reappear. The assistant implicitly assumes that the manual fix is sufficient for the current debugging session, but the underlying automation gap remains — a point that would need to be addressed in a production setup. Assumption 2: The S3 proxy is routing correctly. The "Not Found" response confirms that the proxy can talk to the database, but it doesn't confirm that the proxy can route requests to the Kuri storage nodes for actual object storage. The proxy's architecture is that it looks up the node_id from the database and then forwards the request to the appropriate Kuri node. Since there are no objects in the database yet, the routing layer hasn't been exercised. The assistant acknowledges this implicitly by saying "Let me test actual S3 operations" — recognizing that the health check alone isn't sufficient validation. Assumption 3: Both Kuri nodes are healthy. The assistant had confirmed earlier that both Kuri nodes were running (not crashed), but hadn't verified that they could accept connections from the S3 proxy. The "Not Found" response doesn't involve the Kuri nodes at all — it's purely a database lookup. The end-to-end path (proxy → database lookup → proxy → Kuri node → response) remains untested. Assumption 4: The HTTP handler is correct. The custom handler that replaced the standard ServeMux was written to avoid the Go 1.22 route conflict. The assistant assumes that this handler correctly dispatches all HTTP methods (GET, PUT, DELETE, HEAD) to the appropriate internal functions. The "Not Found" response only tests GET on the root path, which is a minimal subset of the handler's logic. These assumptions are reasonable for a debugging session — you can't test everything at once, and each successful step validates a layer of the stack. The assistant is following a classic debugging strategy: fix the most fundamental issues first (crashes, connectivity, schema), then verify each layer incrementally.

What the Message Reveals About the Thinking Process

The message is remarkable for what it doesn't say as much as what it does. The assistant doesn't celebrate, doesn't run a victory lap, doesn't declare the cluster "fixed." Instead, the tone is measured and analytical: "Not Found is good." This is the language of a developer who has been burned by false positives before, who knows that a single passing test doesn't mean the system is correct.

The assistant's next action is telling: "Let me test actual S3 operations." This indicates a clear mental model of the testing hierarchy. The assistant has verified:

  1. L0: Process health — Containers are running (confirmed earlier)
  2. L1: HTTP endpoint responsiveness — The proxy responds to HTTP requests (confirmed now)
  3. L2: Database connectivity — The proxy can query the database (confirmed now)
  4. L3: Correct HTTP semantics — The proxy returns appropriate status codes (confirmed now)
  5. L4: S3 protocol compliance — The proxy handles PUT/GET/DELETE correctly (not yet tested)
  6. L5: End-to-end object flow — Objects can be stored and retrieved through the full stack (not yet tested)
  7. L6: Multi-node routing — Objects are correctly distributed across Kuri nodes (not yet tested) The assistant has just cleared levels 1-3 and is preparing to test level 4. This structured approach to verification is characteristic of experienced systems engineers who understand that distributed systems fail in layers and must be validated in layers.

Input and Output Knowledge

Input knowledge required to understand this message: The reader needs to understand the architecture of the system (S3 proxy → Kuri nodes → YugabyteDB), the HTTP semantics of S3 (what "Not Found" means in the S3 protocol), the previous state of the cluster (broken with Internal Server Error), and the debugging steps that led to this point (fixing the route conflict, fixing the web UI, fixing the database schema). Without this context, "Not Found" looks like a bug rather than a breakthrough.

Output knowledge created by this message: The message confirms that the S3 proxy's database layer is operational, that the schema fix was applied correctly, that the HTTP handler is functional for GET requests, and that the proxy returns semantically appropriate responses. It also establishes a baseline for further testing: the assistant now knows that if subsequent S3 operations fail, the problem is likely in the routing layer or the Kuri nodes, not in the database or the basic HTTP infrastructure.## Mistakes and Missed Opportunities

While the assistant's interpretation of "Not Found" was correct, there are a few aspects worth critiquing:

The db-init automation gap. The assistant fixed the database schema manually by running ycqlsh commands directly against YugabyteDB. This is fine for a debugging session, but it creates a hidden dependency: if someone restarts the cluster from scratch (without the manual fix), the old schema will reappear and the proxy will break again. The assistant updated the docker-compose.yml db-init section earlier, but the logs showed it was still only creating keyspaces. A more robust fix would have been to ensure the db-init container runs the CQL migration files, making the schema creation idempotent and automated.

The missing end-to-end test. The assistant declared "Not Found is good" and moved on, but didn't immediately run a PUT request to create an object and then GET it back. This would have validated the full path through the proxy, database, and Kuri node. The assistant's next step was to "test actual S3 operations," but the message ends before that happens. In a production debugging scenario, the temptation to pause at a partial victory is strong, but the most valuable debugging happens when you push through to a complete end-to-end validation.

The assumption about Kuri node health. The assistant had confirmed earlier that both Kuri nodes were running (not crashed), but hadn't checked whether they were actually serving requests. A quick curl to the Kuri nodes' LocalWeb endpoints (ports 7001 and 7002) would have confirmed that the storage layer was operational. Without this check, the "Not Found" response from the proxy only validates half the system.

These are minor critiques in the context of a complex debugging session. The assistant's approach was methodical and effective, and the decision to validate incrementally rather than all at once is a sound strategy for distributed systems debugging.

Conclusion

The message "Not Found is good" captures a moment that every distributed systems engineer knows intimately: the moment when a system transitions from "definitely broken" to "probably working." It's a fragile moment, full of assumptions and untested paths, but it's also a moment of genuine progress. The assistant's ability to recognize this transition, to interpret a seemingly negative result as positive validation, and to articulate the reasoning behind that interpretation, demonstrates a deep understanding of how distributed systems behave.

The "Not Found" response was not the end of the debugging session — there were more issues to find and fix, more layers to validate, more tests to run. But it was a necessary milestone, a checkpoint that allowed the assistant to move from "why is everything broken?" to "what's the next thing to test?" In the messy, iterative process of building and debugging distributed systems, sometimes "Not Found" is exactly what you want to hear.