The Routing Decision: Why DELETE and HEAD Must Follow the Object, Not the Load Balancer

In the architecture of a horizontally scalable S3-compatible storage system, the simplest decisions often carry the deepest architectural weight. This is the story of one such decision — a single message in an opencode coding session where an AI assistant realizes that two HTTP handler methods, handleDelete and handleHead, must be updated to use directed routing instead of round-robin distribution. The message appears unremarkable at first glance: a brief reasoning note followed by a file read. But within it lies the entire philosophy of the distributed storage system being built.

The Message

The subject message, sent by the assistant at index 91 of the conversation, reads:

## Agent Reasoning Let me also update handleDelete and handleHead to use the router for routing to the correct node, similar to handleGet. [read] /home/theuser/gw/server/s3frontend/server.go <file> 00111| 00112| func (s FrontendServer) handlePut(w http.ResponseWriter, r http.Request) { 00113| params := r.URL.Query() 00114| 00115| // Check if this is a multipart part upload 00116| if params.Has("partNumber") && params.Has("uploadId") { 00117| // Multipart part upload - round robin to distribute load 00118| s.proxyRoundRobin(w, r) 00119| return 00120| } 00121| 00122| // Regular PUT object - round robin for write distribution 00123| s.proxyRoundRobin(w, r) 00124| } 001...

The assistant reads the file to examine the handlePut method as a reference point, understanding that PUT operations use round-robin distribution while GET operations (already updated in preceding messages) use the ObjectRouter to find the specific backend node holding the requested object. The realization is that DELETE and HEAD — operations that must target the specific node where an object resides — were still using the default round-robin behavior, which would be incorrect.

Why This Message Was Written: The Reasoning and Context

To understand why this message exists, we must trace the architecture being built. The assistant is implementing a horizontally scalable S3-compatible storage system for the Filecoin Gateway. The architecture follows a clean separation of concerns:

  1. Stateless S3 Frontend Proxies — These are lightweight HTTP servers that accept S3 API requests from clients. They maintain no state of their own, making them horizontally scalable by simply adding more instances.
  2. Kuri Storage Nodes — These are the backend nodes that actually store object data in RIBS blockstores. Each node maintains its own independent data, and objects are distributed across nodes without replication.
  3. Shared YCQL Database (YugabyteDB) — A distributed CQL-compatible database that tracks which node holds each object. This is the coordination layer that enables the frontend proxies to route requests correctly. The fundamental pattern is: writes can go anywhere, but reads must go to the right place. When a client PUTs an object, the frontend proxy can forward the request to any Kuri node via round-robin — the node that receives the write stores the object and records its location in the shared YCQL database. When a client GETs an object, the frontend must first query YCQL to discover which node holds the object, then forward the request specifically to that node. This pattern was established in the preceding messages. The assistant had already created the ObjectRouter (in messages 76-84) and updated handleGet to use it for directed routing. The handlePut method, as seen in the file read, correctly uses round-robin for both regular PUTs and multipart part uploads. But what about DELETE and HEAD? These operations share a critical property with GET: they must target the specific node that holds the object. A DELETE request sent to the wrong node would silently fail to remove the object. A HEAD request sent to the wrong node would return incorrect metadata or a 404. The assistant recognized this gap and initiated the fix.

How the Decision Was Made

The decision process visible in this message is a textbook example of incremental architectural reasoning. The assistant had already:

  1. Created the ObjectRouter with a LookupObjectNode method that queries YCQL
  2. Updated handleGet to use the router for directed reads
  3. Updated the Start function and fx.go to wire the router into the server The file read of handlePut serves as a reference — the assistant is checking the existing handler signatures and patterns before applying the same treatment to handleDelete and handleHead. The reasoning is explicit: "similar to handleGet." This is pattern-matching at the architectural level: any operation that requires locating an existing object must use directed routing. The decision also reveals an implicit understanding of S3 semantics. In the S3 API: - PUT creates or overwrites an object. The client specifies the key, and the system can choose where to store it. - GET retrieves an object. The system must find where it was stored. - HEAD retrieves object metadata. Like GET, it must find the object. - DELETE removes an object. Like GET, it must find the object. PUT is the only operation where the storage location is flexible. All other operations are location-dependent. The assistant correctly identified that DELETE and HEAD belong to the location-dependent category.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Explicit assumption: The router's LookupObjectNode method works correctly for DELETE and HEAD operations, not just GET. The assistant assumes that the YCQL query (SELECT node_id FROM S3Objects WHERE ...) returns the correct node regardless of which HTTP method will be used against that node. This is a reasonable assumption since the routing decision depends only on the object's location, not on the operation type.

Implicit assumption: The backend Kuri nodes handle DELETE and HEAD requests uniformly through the same S3 API endpoint. The assistant assumes that forwarding a DELETE or HEAD request to a Kuri node's S3 port will work the same way as forwarding a GET request. This is consistent with the S3 protocol, where all methods are handled through the same HTTP endpoint.

Implicit assumption: The proxyToNode method (used for directed routing) accepts the same request format for DELETE and HEAD as it does for GET. The assistant assumes no method-specific proxying logic is needed — the proxy simply forwards the HTTP request verbatim to the correct backend.

Architectural assumption: Round-robin for writes and directed routing for reads is the correct consistency model. This assumes that the YCQL database provides strong enough consistency (QUORUM level) to guarantee that a write recorded in YCQL is immediately visible to subsequent read routing queries. The assistant had configured QUORUM consistency in the router's lookup query.

Potential Mistakes and Incorrect Assumptions

While the core insight is correct, there are subtle considerations worth examining:

The DELETE consistency problem: There is a potential race condition where a DELETE request is routed to a node, but a concurrent PUT for the same key is routed to a different node. The YCQL record might be updated by the PUT after the DELETE has already checked the database, leading to a "lost update" scenario. The assistant does not address this in this message, though it may be handled at a different layer (e.g., conditional updates or object versioning).

The HEAD response problem: A HEAD request should return the same metadata that a GET would return, but without the body. If the frontend proxy simply forwards the HEAD request to the correct backend node, this should work correctly. However, if the frontend proxy were to intercept HEAD requests and generate responses locally (which it doesn't), it would need access to the metadata. The assistant's approach of proxying the request avoids this issue.

Multipart uploads and DELETE: The handlePut method shows special handling for multipart part uploads (checking for partNumber and uploadId query parameters). The assistant does not mention whether DELETE of multipart objects requires special routing. In the S3 API, aborting a multipart upload requires knowing which node holds the upload record. This may be a gap that is addressed later in Phase 4.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The S3 API semantics — Understanding what PUT, GET, HEAD, and DELETE mean in the context of object storage, and which operations are location-dependent.
  2. The distributed storage architecture — The separation between stateless frontend proxies and stateful backend storage nodes, and the role of the shared YCQL database as a coordination layer.
  3. The round-robin vs. directed routing distinction — Round-robin distributes load evenly across all backends but cannot guarantee finding a specific object; directed routing queries the database to find the correct backend.
  4. The ObjectRouter component — Created in preceding messages, this component encapsulates the YCQL lookup logic and provides the LookupObjectNode method.
  5. The codebase structure — The FrontendServer struct with its handleGet, handlePut, handleDelete, and handleHead methods, and the proxyRoundRobin and proxyToNode helper methods.

Output Knowledge Created

This message creates several forms of knowledge:

Immediate code change: The assistant will update handleDelete and handleHead to use the router for directed routing, completing the integration of the ObjectRouter into all location-dependent HTTP handlers.

Architectural precedent: The message establishes a pattern for determining which operations use round-robin vs. directed routing: operations that create or overwrite objects (PUT) use round-robin; operations that must find existing objects (GET, HEAD, DELETE) use directed routing.

Design documentation: The reasoning in this message serves as implicit documentation of the routing strategy. A future developer reading the code can understand why handlePut uses proxyRoundRobin while handleGet uses the router — the architectural rationale is embedded in the implementation history.

Validation of the router design: By extending the router's use to DELETE and HEAD, the assistant validates that the LookupObjectNode abstraction is general enough to serve all location-dependent operations. If the router had been designed only for GET, this extension would have required refactoring.

The Thinking Process Visible in Reasoning

The assistant's reasoning reveals a systematic, incremental approach to software construction. The thought process follows a clear pattern:

  1. Identify the gap: The assistant recognizes that handleDelete and handleHead have not been updated to use the router, while handleGet has.
  2. Reference existing code: The assistant reads handlePut to understand the existing handler patterns and confirm the routing logic for different operation types.
  3. Apply the pattern: The assistant decides to apply the same treatment to DELETE and HEAD as was applied to GET — use the router for directed routing.
  4. Execute the change: The next message (index 92) shows the edit being applied. This is not a flash of insight but a methodical sweep — the assistant is checking each HTTP handler to ensure consistent routing behavior. The reasoning is driven by a mental model of the architecture: "any operation that needs to find an existing object must use the router." The assistant is essentially running a static analysis in its head, identifying all code paths that violate this invariant. The file read of handlePut is particularly instructive. The assistant reads the PUT handler not because it needs to change it, but because it serves as a reference for the handler pattern — the function signature, the parameter extraction, the proxy call pattern. This is how experienced developers work: they read surrounding code to understand the conventions before making changes, ensuring consistency.

Conclusion

Message 91 appears, on its surface, to be a minor cleanup task — updating two HTTP handlers to use a router that was already integrated for GET requests. But beneath this surface lies the entire architectural philosophy of the distributed S3 system. The distinction between round-robin writes and directed reads is the fundamental design pattern that enables horizontal scalability without data replication. Every handler must correctly implement this distinction, or the system breaks.

The assistant's systematic approach — identifying the gap, referencing existing patterns, and applying consistent changes — reflects sound engineering practice. In distributed systems, the most dangerous bugs are not in the complex algorithms but in the simple inconsistencies: a handler that routes to the wrong node, a timeout that isn't handled, a consistency level that isn't applied uniformly. By methodically ensuring that all location-dependent operations use directed routing, the assistant closes one such gap.

The message also illustrates a broader truth about software architecture: the simplest decisions — whether to route a DELETE request through a router or a round-robin pool — encode the deepest design principles. Every line of code is a decision about how the system should behave, and every decision reflects an understanding of the system's fundamental nature.