The Moment the Cluster Worked: Verifying Round-Robin Routing in a Distributed S3 Architecture

Introduction

In the middle of a lengthy debugging session for a horizontally scalable S3 storage system, a single message captures the moment when months of architectural planning and hours of iterative debugging finally paid off. The message, sent by an AI assistant to a developer working on the Filecoin Gateway's distributed S3 architecture, is deceptively brief: a simple shell loop that verifies round-robin object distribution across two storage nodes. But behind those few lines of bash lies a critical milestone—the confirmation that the three-layer architecture (S3 frontend proxy → Kuri storage nodes → shared YugabyteDB) was functioning as designed, with requests correctly routing to the appropriate backend node and objects being retrievable from the node that stored them.

The Message

The assistant writes:

Round-robin is working - objects are distributed between kuri-1 and kuri-2. Let me verify GET routes to the correct node:

>

``bash # GET objects and check X-Node-Id header for i in 1 2 3 4; do node=$(curl -sI http://localhost:8078/test-bucket/obj$i.txt | grep -i X-Node-Id) echo "obj$i.txt: $node" done obj1.txt: X-Node-Id: kuri-1 obj2.txt: X-Node-Id: kuri-2 obj3.txt: X-Node-Id: kuri-1 obj4.txt: X-Node-Id: kuri-2 ``

This is the culmination of a long chain of debugging that began with HTTP route conflicts causing server panics, missing database columns breaking object lookups, and authentication header requirements blocking unsigned requests.

Why This Message Was Written: The Motivation and Context

To understand why this verification step was necessary, one must understand the architecture being tested. The system under development is a horizontally scalable S3-compatible storage gateway built on top of IPFS/Kubo nodes (called "Kuri" nodes). The architecture follows a three-layer design: stateless S3 frontend proxies accept client requests and route them to backend Kuri storage nodes, which in turn store data in IPFS and record metadata in a shared YugabyteDB (a distributed SQL database compatible with Cassandra's CQL protocol).

The critical design requirement is that the S3 proxies must be stateless—they cannot store any data locally. Instead, they must route requests to the appropriate backend node. For write operations (PUT), the proxy selects a backend using a round-robin algorithm and forwards the request. For read operations (GET), the proxy must look up which backend node stores the object by querying the shared database, then route the request to that specific node.

This message was written because the assistant had just finished fixing a series of bugs that prevented this routing from working. Earlier in the session, the assistant had discovered that:

  1. The S3 proxy was crashing due to an HTTP route conflict between HEAD / and GET /healthz in Go 1.22's strict ServeMux routing.
  2. The S3Objects database table was missing the node_id column, making it impossible for the proxy to determine which backend stored an object.
  3. The Kuri backend nodes required an x-amz-content-sha256 header on PUT requests, but the proxy wasn't injecting it for unsigned requests from tools like curl. Each of these issues had been fixed in the preceding messages: the route conflict was resolved by switching to a custom handler, the database schema was corrected by dropping and recreating the table with the node_id column, and the proxy code was modified to auto-inject UNSIGNED-PAYLOAD as the content SHA256 header when the client didn't provide one. After rebuilding the Docker image and restarting the containers, the assistant had successfully performed a single PUT and GET cycle, confirming that the basic flow worked. But one successful transaction does not prove a distributed system works correctly. The assistant needed to verify two things:
  4. Round-robin distribution: Are PUT requests being distributed evenly across both Kuri nodes?
  5. Correct GET routing: When reading an object, does the proxy route the request to the node that actually stored it? This message represents the systematic verification of both properties.

How Decisions Were Made

The assistant's approach to verification reveals a methodical testing philosophy. Rather than writing a complex test script or using a formal testing framework, the assistant chose a simple, transparent approach that any developer could understand and reproduce.

For distribution verification, the assistant used a shell loop to PUT four objects (obj1.txt through obj4.txt) to the S3 proxy endpoint, then queried the shared YugabyteDB to inspect which node_id was recorded for each object. The database query (visible in the preceding message at index 643) showed a perfect alternating pattern: obj1.txt went to kuri-1, obj2.txt to kuri-2, obj3.txt to kuri-1, and obj4.txt to kuri-2. This confirmed that the round-robin algorithm in the BackendPool was functioning correctly.

For GET routing verification, the assistant used a different technique. Instead of querying the database again, it issued HEAD requests (using curl -sI) for each object and inspected the X-Node-Id response header. This header is set by the S3 proxy to indicate which backend node served the request. By checking this header, the assistant could verify that the proxy's lookup logic was correctly mapping each object to its storage node.

The choice of four objects was deliberate: with two backend nodes, four iterations produces two complete round-robin cycles, making any misrouting immediately obvious. The alternating pattern (kuri-1, kuri-2, kuri-1, kuri-2) confirms both that the round-robin algorithm is alternating correctly and that the database lookup for GET requests returns the correct node.

Assumptions Underlying the Verification

Every test makes assumptions, and this one is no exception. The assistant implicitly assumed that:

  1. The round-robin algorithm is deterministic and alternating: The test assumes that the first request goes to kuri-1, the second to kuri-2, and so on. If the algorithm used a different selection strategy (e.g., random selection or least-loaded), the pattern might differ, but the test would still detect whether distribution was happening at all.
  2. The X-Node-Id header is set by the proxy: The verification relies on this custom header being present in the response. If the header were missing or misnamed, the grep -i X-Node-Id would produce empty output, and the test would fail to confirm correct routing.
  3. The database state is consistent: The test assumes that the objects stored in the previous step are still present in the database and that the node_id values correctly reflect which backend handled each PUT.
  4. Both backend nodes are healthy: The round-robin algorithm might skip unhealthy nodes. The test implicitly assumes both kuri-1 and kuri-2 are operational, which was confirmed earlier by checking their logs.
  5. The proxy's backend pool has exactly two nodes: The alternating pattern only makes sense with an even number of nodes. If a third node were added, the pattern would shift. These assumptions were reasonable given the state of the test cluster at this point, but they highlight an important aspect of distributed systems testing: a passing test does not prove the system is correct in all circumstances, only that it behaves as expected under the tested conditions.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

Architecture knowledge: Understanding the three-layer design—S3 frontend proxies as stateless routing layer, Kuri nodes as storage backends, and YugabyteDB as the shared metadata store. The reader must understand why round-robin distribution matters for horizontal scalability and why GET requests must route to the specific node that stores an object.

Protocol knowledge: Familiarity with HTTP headers, particularly the X-Node-Id custom header used for debugging. Understanding of S3's authentication model and the x-amz-content-sha256 header requirement that caused earlier failures.

Database knowledge: Understanding that the S3Objects table includes a node_id column that records which backend stored each object, enabling the proxy to route GET requests correctly.

Testing methodology: Appreciation for systematic verification—testing PUT distribution separately from GET routing, using multiple iterations to confirm patterns, and inspecting response headers rather than just checking status codes.

Operational knowledge: Familiarity with Docker Compose for container orchestration, bash scripting for test automation, and the specific tools used (curl, grep, docker exec).

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. Empirical confirmation of round-robin distribution: The database query (from the preceding message) shows that objects are evenly distributed between kuri-1 and kuri-2 in strict alternating order. This confirms that the BackendPool.SelectAny() method is working as implemented.
  2. Empirical confirmation of correct GET routing: The X-Node-Id headers in the response confirm that the proxy's lookup logic correctly identifies which backend stores each object and routes the request accordingly. obj1.txt (stored on kuri-1) is served by kuri-1, obj2.txt (stored on kuri-2) is served by kuri-2, and so on.
  3. Validation of the stateless proxy design: The proxy correctly uses the database as the source of truth for object location, rather than maintaining any local state. This is a fundamental requirement of the architecture.
  4. A reproducible test procedure: Any developer can run the same commands to verify the cluster's behavior after changes. The test is simple, transparent, and produces immediately interpretable output.
  5. Confidence in the fix chain: The series of fixes (route conflict, missing column, header injection) has produced a correctly functioning system. The round-robin verification proves that the fixes work together correctly.

The Significance of This Milestone

In the broader context of the development session, this message represents the transition from debugging to validation. The earlier messages in the session were reactive—identifying failures, diagnosing root causes, and applying fixes. This message is proactive—systematically verifying that the system behaves correctly under normal operation.

The round-robin verification is particularly significant because it tests the core value proposition of the architecture: horizontal scalability. A single-node S3 gateway is trivial to implement; the entire point of the three-layer design is to distribute load across multiple storage nodes. If round-robin distribution fails, or if GET requests cannot find the correct node, the architecture provides no benefit over a simpler single-node design.

Furthermore, the verification demonstrates that the stateless proxy pattern works in practice. The proxy has no local cache of object locations; it queries the database for each GET request. This design choice means that backend nodes can be added, removed, or fail without affecting the proxy's ability to route requests—as long as the database remains consistent.

Conclusion

The message at index 644 is a quiet triumph in a debugging session filled with errors, crashes, and misconfigurations. A simple bash loop, four curl commands, and a grep for a custom header—these humble tools confirm that a complex distributed system is working as designed. The round-robin distribution is correct, the GET routing is accurate, and the stateless proxy architecture is validated.

For the developer reading the assistant's output, this message provides the confidence to move forward. The foundational routing layer works. The next challenges—adding more nodes, implementing health checks, building monitoring dashboards, and tuning performance—can proceed on a solid foundation. The cluster is no longer a collection of containers struggling to communicate; it is a functioning distributed system.

In the history of this project, this message marks the moment when the architecture stopped being a design on paper and became a working reality. The round-robin works. The cluster is alive.