The Quiet Victory: How a "Not Found" Response Proved a Distributed S3 Architecture Was Finally Working

The Message

[assistant] S3 proxy is working. Now let me test the kuri backends directly: [bash] curl -s http://localhost:8078/healthz 2>&1 Not Found

At first glance, this message from a debugging session for a horizontally scalable S3 architecture seems almost trivial—a two-line exchange confirming that a health endpoint returns "Not Found." But in the context of the preceding hour of intense debugging, this brief message represents a pivotal moment: the transition from frantic problem-solving to methodical verification. It is the sound of a system that was broken, then fixed, now behaving predictably. Understanding why this message matters requires unpacking the cascade of failures that preceded it and the careful reasoning that led to this moment of confirmation.

The Context: A Cluster in Crisis

To appreciate the significance of this message, one must understand what was happening in the moments before it was written. The assistant had been building and debugging a test cluster for a horizontally scalable S3 architecture—a system with three layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. The architecture was ambitious: clients would connect to an S3 proxy on port 8078, which would route requests to one of two Kuri storage nodes, which in turn stored data and metadata in YugabyteDB.

But the cluster was broken. Multiple failures had surfaced simultaneously:

  1. The Kuri nodes were crashing with a panic caused by an HTTP route conflict. Go 1.22's new ServeMux implementation was stricter about pattern matching, and registering a bare HEAD / alongside GET /healthz created an unresolvable conflict. The error message was telling: pattern "GET /" conflicts with pattern "/healthz".
  2. The web UI container was a placeholder—it didn't actually proxy to the Kuri nodes' web interfaces. Users would see an echo message instead of the monitoring dashboard.
  3. The S3 proxy was returning "Internal Server Error" because the S3Objects table in YugabyteDB was missing the node_id column. The database initialization script had created the keyspace but hadn't applied the migration that added this critical column. Every query to the S3 proxy failed with Undefined Column. Column doesn't exist. Each of these issues was individually fixable, but together they represented a systemic failure. The assistant had to diagnose each problem, understand its root cause, and apply the correct fix—all while maintaining a mental model of how the three-layer architecture was supposed to work.

The Fixes: A Systematic Approach

The assistant's debugging process reveals a methodical, hypothesis-driven approach. Rather than guessing at solutions, the assistant read logs, examined code, and traced each failure to its source.

For the HTTP route conflict, the assistant initially tried to register /healthz with a more specific pattern, but this failed because Go 1.22's router considers method-qualified patterns and bare patterns as conflicting when one is more general. The solution was to abandon the standard ServeMux entirely and use a custom handler function that manually inspected the request path and method, routing to the appropriate handler without relying on the router's pattern matching. This was a pragmatic decision: the standard library's router was causing more problems than it solved for this particular use case.

For the missing node_id column, the assistant discovered that the database initialization script (db-init) was creating keyspaces but not running the CQL migrations that added columns to the tables. The migration file existed (1754293669_init_s3_object_index.up.cql) and contained the correct schema with node_id, but it was never executed. Rather than restructuring the entire initialization pipeline, the assistant took a pragmatic approach: manually dropping the old table and recreating it with the correct schema using ycqlsh, then updating the docker-compose.yml to ensure future deployments would create the correct schema automatically.

For the web UI, the assistant replaced the placeholder container with an Nginx reverse proxy configuration that forwarded requests to the Kuri node's web interface on port 7001.

Each fix was tested independently. The Kuri nodes were restarted and confirmed running. The web UI was verified returning HTML. The S3 proxy was restarted after the schema fix. And then came the moment of truth.

The Significance of "Not Found"

When the assistant ran curl -s http://localhost:8078/healthz and received "Not Found," this was not a failure—it was a triumph. Here's why.

Previously, every request to the S3 proxy returned "Internal Server Error" (a 500 status code). This was happening because the proxy's first action on receiving any request was to query the S3Objects table to check if the requested object existed. That query was failing because the node_id column didn't exist in the database schema. The proxy couldn't even determine that the object didn't exist—it couldn't run the query at all.

After the schema fix, the same query succeeded. The proxy could now look up objects in the database. When it looked up the path /healthz, it found nothing—because /healthz is not an S3 object path. The proxy correctly returned 404 Not Found, which is the proper HTTP response for a nonexistent resource.

The "Not Found" response was proof that:

  1. The database connection was working—the proxy could establish a CQL session and execute queries against YugabyteDB.
  2. The schema was correct—the node_id column existed and the query compiled successfully.
  3. The routing logic was functional—the proxy received the request, processed it through the handler chain, and returned an appropriate response.
  4. The error handling was working—rather than panicking or returning a generic error, the proxy returned a semantically correct HTTP status code. This is a classic debugging pattern: the difference between a 500 Internal Server Error and a 404 Not Found is the difference between "the system is broken" and "the system is working correctly, but the requested resource doesn't exist." The former requires immediate intervention; the latter is normal operation.

The Thinking Process: Verification Before Celebration

The message also reveals the assistant's thinking process. Note the structure: "S3 proxy is working. Now let me test the kuri backends directly." This is a checkpoint. The assistant has confirmed one component is functional and is planning the next verification step.

The assistant could have celebrated—the S3 proxy was finally working after multiple failed attempts. But instead, the focus immediately shifted to the next test: verifying the Kuri backends. This reflects a disciplined approach to debugging where each component is verified in isolation before testing the integrated system. The assistant is building confidence incrementally, layer by layer.

The choice of /healthz as the test endpoint is also revealing. The assistant didn't test with an actual S3 object path like /bucket/key—that would require creating an object first, which introduces additional dependencies. Instead, the assistant chose the simplest possible test: send a request to a path that should not exist and verify that the response is a proper 404. This is a minimal, high-signal test that validates the core functionality without requiring a multi-step setup.

Assumptions and Knowledge

This message relies on several pieces of knowledge that the reader (or the assistant) must have:

  1. Understanding of HTTP semantics: Knowing that 404 Not Found is the correct response for a nonexistent resource, and that it indicates the server is functioning correctly.
  2. Knowledge of the architecture: Understanding that the S3 proxy is a stateless frontend that routes to Kuri storage nodes, and that /healthz is not a valid S3 path.
  3. Context of previous failures: Knowing that the proxy was previously returning 500 Internal Server Error, and that the fix involved correcting the database schema.
  4. Familiarity with the debugging workflow: Understanding that component-by-component verification is a systematic way to build confidence in a complex distributed system. The assistant's assumption that "Not Found" is the correct response for /healthz on the S3 proxy is correct. The S3 proxy handles S3 API requests (object GET, PUT, DELETE, etc.), not health check endpoints. Health checks are performed on the Kuri nodes directly. If the assistant had received a 200 OK for /healthz, that would actually have been suspicious—it would mean the proxy was treating a health check path as a valid S3 object path.

The Broader Lesson

This message, in its brevity, encapsulates a fundamental truth about debugging distributed systems: progress is measured not by the absence of errors, but by the quality of errors. A 404 Not Found is a good error—it means the system is processing requests correctly and returning semantically appropriate responses. A 500 Internal Server Error is a bad error—it means the system cannot process requests at all.

The assistant's quiet "S3 proxy is working" followed by a verification command and a "Not Found" response is a moment of professional satisfaction. It's the debugging equivalent of a surgeon closing an incision after a successful operation—the dramatic part is over, and the routine verification confirms that everything is as it should be.

This message also demonstrates the importance of incremental verification in complex systems. Rather than fixing all issues and then testing the entire system at once, the assistant fixed each issue, verified it independently, and only then moved to the next component. This approach minimizes the risk of cascading failures and makes it easier to isolate new problems when they arise.

In the end, "Not Found" was exactly what the assistant needed to find.