The Moment of Verification: Confirming Round-Robin Distribution in a Horizontally Scalable S3 Architecture

Introduction

In the course of building a horizontally scalable, distributed S3-compatible storage system—one that separates stateless frontend proxies from backend storage nodes—there comes a pivotal moment when theory meets practice. Message 643 in this coding session captures exactly that moment. After an extended debugging session involving HTTP route conflicts, missing database columns, content-sha256 header injection, and container orchestration issues, the assistant finally arrives at a clean verification: the S3 proxy correctly distributes objects across backend storage nodes in a round-robin fashion, and both PUT and GET operations work end-to-end through the three-layer architecture.

This message is not merely a log of successful commands. It is a deliberate validation of the core architectural premise that motivated the entire session. Understanding why this message was written, what assumptions it tests, and what knowledge it produces reveals the deeper structure of how distributed systems are built and verified in practice.

The Message Itself

The assistant writes:

Excellent! Both PUT and GET are working through the S3 proxy! Let me run a few more tests to verify routing:

``` # PUT several objects to see round-robin distribution for i in 1 2 3 4; do echo "Object $i content" | curl -s -X PUT --data-binary @- http://localhost:8078/test-bucket/obj$i.txt -H "X-Amz-Content-Sha256: UNSIGNED-PAYLOAD" echo "" done

# Check which nodes got which objects docker exec test-cluster-yugabyte-1 bin/ycqlsh yugabyte -e "SELECT bucket, key, node_id FROM filecoingw_s3.S3Objects;"

bucket | key | node_id -------------+----------+--------- test-bucket | obj1.txt | kuri-1 test-bucket | obj2.txt | kuri-2 test-bucket | obj3.txt | kuri-1 test-bucket | obj4.txt | kuri-2 test-bucket | proxy.txt | kuri-2

(5 rows) ```

The message is concise, but every element carries significance. The initial exclamation—"Excellent!"—signals that a critical threshold has been crossed. The assistant then immediately pivots from celebration to systematic verification, running a multi-object upload test and querying the database to confirm the distribution pattern.

Why This Message Was Written: The Reasoning and Motivation

The message was written to answer a single, essential question: Does the round-robin routing mechanism actually work? This question sits at the heart of the horizontally scalable S3 architecture. The entire design—stateless frontend proxies routing to multiple Kuri storage nodes—depends on the proxy's ability to distribute objects across backends. If the proxy always routed to the same node, the system would gain no scalability from adding more storage nodes.

The motivation stems from the architecture's fundamental premise. The roadmap for this project explicitly calls for separate stateless frontend proxy nodes that can route requests to any of several Kuri storage backends. Earlier in the session, the assistant had made a significant architectural error by running Kuri nodes as direct S3 endpoints, violating this separation. That error was corrected, and the proper three-layer hierarchy was established: S3 proxy (port 8078) → Kuri nodes → YugabyteDB. But correction alone is not enough—verification is required.

The assistant's reasoning follows a clear pattern: first achieve basic functionality (a single PUT and GET work), then test the distributed behavior (multiple PUTs to verify routing), then inspect the system state (query the database) to confirm the expected distribution. This is the scientific method applied to system building: hypothesis, experiment, observation, conclusion.

How Decisions Were Made

Several decisions are embedded in this message, some explicit and some implicit.

The decision to test with four objects is deliberate. With two backend nodes, uploading four objects creates a clear pattern: if round-robin works, each node should receive exactly two of the four new objects (or approximately so). The assistant could have tested with two objects, but four provides a more convincing demonstration of alternating distribution. The resulting pattern—kuri-1 gets obj1 and obj3, kuri-2 gets obj2 and obj4—is exactly what round-robin with two nodes should produce.

The decision to include the X-Amz-Content-Sha256: UNSIGNED-PAYLOAD header reflects a lesson learned from earlier debugging. In message 631, a PUT request without this header returned a 500 Internal Server Error because the Kuri backend's S3 handler requires the content-sha256 header. The assistant had previously modified the proxy to auto-inject this header for unsigned requests (message 637), but in this test, the header is explicitly provided in the curl command. This suggests the assistant is being thorough—testing both the explicit header path and, by implication, the proxy's auto-injection path.

The decision to query the database directly rather than rely on proxy responses shows a preference for ground truth. The proxy could report success for all four PUTs, but only by inspecting the database can the assistant confirm that objects were actually stored on the correct nodes. This reflects an engineering mindset: trust the data, not the interface.

Assumptions Made

The message rests on several assumptions, most of which are justified but worth examining.

The assumption that round-robin distribution is the correct behavior. The assistant assumes that the BackendPool.SelectAny() method (seen in the proxy code at message 636) implements round-robin selection. This is a reasonable assumption given the architecture, but the test itself serves to validate it. If the distribution had been random or biased toward one node, the test would have revealed it.

The assumption that node_id in the database accurately reflects which backend stored the object. The proxy's PUT handler selects a backend and forwards the request; the backend then writes the object and records its node ID in the database. If the proxy forwarded to kuri-1 but kuri-1 recorded itself as kuri-2 due to a configuration error, the database would show misleading data. The assistant implicitly trusts the node's self-identification.

The assumption that the test cluster is isolated and deterministic. Running in Docker Compose on a single machine, the assistant assumes no external interference and that the round-robin counter starts from a known state. The presence of proxy.txt (from the earlier test in message 640) assigned to kuri-2 confirms that the counter was at kuri-2 before the loop started, which is why obj1 goes to kuri-1 (the next in sequence).

The assumption that five rows constitute sufficient evidence. The assistant does not run a statistical test or repeat the experiment. The alternating pattern is visually clear, and in the context of a test cluster, this is considered adequate proof. In a production environment, more rigorous validation might be warranted.

Mistakes or Incorrect Assumptions

While the message itself is correct, it is worth noting what it does not test.

The message does not test GET routing. The assistant verified that PUT distributes objects, but GET requests must also be routed correctly. The proxy's GET handler uses a different routing strategy: it looks up the object's node_id in the database and routes the request to that specific backend. This was implemented earlier but is not explicitly verified in this message. The earlier single-object GET test (message 642) worked, but the assistant does not verify that all four new objects can be retrieved through the proxy.

The message does not test failure scenarios. What happens if kuri-1 goes down? Does the proxy correctly route new objects to kuri-2 only? Can it still serve existing objects on kuri-1? These are important questions for a horizontally scalable system, but they are outside the scope of this verification message.

The message assumes the database schema is correct. The node_id column was added manually in message 609 after the initial schema lacked it. The assistant updated the docker-compose.yml db-init to include the column going forward, but the manual fix was necessary because the keyspace wasn't cleaned up. This is a pragmatic decision, but it means the automated provisioning path might not match the manually corrected state.

Input Knowledge Required

To understand this message, one needs knowledge of several domains.

Distributed systems architecture: The concept of separating stateless proxies from stateful storage nodes, and the use of round-robin for request distribution, is fundamental. Without this context, the test appears to be a simple database query rather than a validation of architectural correctness.

The S3 protocol: Understanding that S3 uses HTTP PUT for uploads and GET for downloads, and that the x-amz-content-sha256 header is part of AWS's signature v4 authentication, is necessary to interpret the curl commands.

Docker and container orchestration: The assistant uses docker exec to query YugabyteDB, runs commands inside containers, and relies on Docker Compose for service management. Knowledge of container networking (why localhost:8078 reaches the proxy but the proxy reaches backends by container name) is implicit.

YugabyteDB and CQL: The query uses Cassandra Query Language (CQL) against YugabyteDB, a distributed SQL database. The SELECT bucket, key, node_id query targets the shared keyspace that both Kuri nodes write to.

Go programming and the project's codebase: The assistant has been reading and modifying Go source files (server.go, handlers.go). The BackendPool and its SelectAny() method are part of this codebase. Understanding the test requires knowing that this method implements round-robin selection.

Output Knowledge Created

This message produces several pieces of valuable knowledge.

Verified round-robin distribution: The primary output is empirical confirmation that the proxy distributes PUT requests across backend nodes in an alternating pattern. This validates the core architectural design.

Confirmation of end-to-end functionality: The S3 proxy → Kuri node → YugabyteDB pipeline works for both write and read operations. The earlier debugging (missing columns, header injection, route conflicts) has been successfully resolved.

A reproducible test procedure: The shell loop and database query form a repeatable test that can be run after any configuration change to verify that distribution still works. This is a form of living documentation.

Evidence of node identity correctness: Both kuri-1 and kuri-2 correctly identify themselves in the database. The node_id column shows "kuri-1" and "kuri-2" as expected, confirming that the per-node configuration (which was a major focus of earlier work) is correct.

Baseline performance data: While not explicitly measured, the test demonstrates that the system can handle multiple concurrent PUT operations without error. The response times and success rate are implicitly validated.

The Thinking Process Visible in the Message

The assistant's thinking process is visible in the structure of the message itself. It follows a clear pattern:

Celebrate the milestone: "Excellent! Both PUT and GET are working through the S3 proxy!" This acknowledges that the basic functionality is confirmed, providing emotional closure to the debugging phase.
Identify the next question: "Let me run a few more tests to verify routing." The assistant recognizes that basic functionality is necessary but not sufficient—the distributed behavior must be tested separately.
Design the experiment: The shell loop with four iterations, explicit header provision, and database query shows deliberate experimental design. The assistant could have tested with one object, but four provides a pattern.
Execute and observe: The commands are run, and the output is captured inline. The assistant does not interpret the results in prose—the alternating pattern speaks for itself.
Draw the conclusion: The conclusion is implicit in the presentation. The assistant does not say "round-robin works" because the evidence is clear. The five rows show exactly the expected distribution: kuri-1, kuri-2, kuri-1, kuri-2. This thinking pattern—verify basics, then verify distribution, then inspect ground truth—is characteristic of experienced systems engineers. It reflects an understanding that distributed systems fail in subtle ways, and that surface-level success (a 200 OK response) can mask deeper problems (all objects on one node).

Conclusion

Message 643 represents the culmination of a significant debugging and implementation effort. It is the moment when the horizontally scalable S3 architecture transitions from a theoretical design to a working system. The round-robin distribution test confirms that the three-layer hierarchy—S3 proxy, Kuri storage nodes, YugabyteDB—functions as intended.

But the message is more than a verification log. It is a window into the engineering mindset: the systematic progression from basic functionality to distributed behavior, the preference for database-level ground truth over API-level responses, and the design of experiments that produce unambiguous evidence. For anyone building or debugging distributed systems, this message exemplifies the discipline of verification that separates working systems from merely plausible ones.

The test cluster is now a functional, observable distributed S3 gateway. The round-robin distribution works. The objects are where they should be. And the assistant can move forward to the next challenge: monitoring, metrics, and production readiness.