The Silent Asymmetry: When Horizontal Scalability Fails at the First Test

"There is traffic visible on Frontend Proxies table, but all of it goes only to kuri-1, none to kuri-2"

This single sentence, delivered by the user at message 849 of a lengthy coding session, cuts to the heart of what makes distributed systems so difficult to build. It is not a question, not a bug report filed with logs and stack traces, not a demand for a fix. It is an observation—quiet, precise, and devastating. The user has looked at the monitoring dashboard they just helped deploy, seen the traffic metrics flowing, and noticed something wrong: the system is not doing what it was designed to do.

The Context: A System Built for Balance

To understand why this message matters, we must understand what was supposed to be happening. The conversation leading up to this moment spans a multi-session effort to build a horizontally scalable S3-compatible storage system for a Filecoin Gateway. The architecture follows a three-layer design: stateless S3 frontend proxies at the top, independent Kuri storage nodes in the middle, and a shared YugabyteDB database at the bottom. The frontend proxy is supposed to distribute writes across the Kuri nodes using round-robin logic, while reads are routed by looking up object metadata in the shared database. This is the fundamental promise of horizontal scalability: add more Kuri nodes, get more throughput.

In the messages immediately preceding this one (indices 812 through 848), the assistant had been deep in deployment and verification mode. The test cluster had been brought up after fixing several bugs—a missing container here, a configuration error there, a database table that didn't exist. By message 835, all containers were running. By message 836, the S3 health endpoint returned "ok" and the web UI was serving HTML. By message 840, the IOThroughput RPC endpoint was returning metrics with actual non-zero values. The assistant generated test traffic—ten files uploaded, ten files downloaded—and confirmed that the monitoring endpoints were all functional. The ClusterTopology RPC showed both storage nodes. Everything looked green.

Then the user looked at the actual dashboard and saw the truth that the aggregate metrics obscured: all the traffic was flowing to kuri-1. Kuri-2 sat idle.

What the Message Reveals About the Observer

This message reveals a user who understands the architecture deeply and is testing it against its own design goals. They are not running a simple "is it up?" check. They are running a behavioral test: does the system actually distribute load as advertised? The user has enough context to know what "Frontend Proxies table" refers to—the monitoring UI's cluster topology view that shows per-node metrics. They have enough patience to look past the "everything is green" veneer and notice the asymmetry. This is the kind of observation that only comes from someone who holds a mental model of the desired system behavior and is actively comparing it against observed behavior.

The user's assumption, implicit in the message, is that traffic should be distributed. The message only makes sense as a problem report if the expected behavior is balanced load. This assumption is correct—the architecture document and the implementation plan both call for round-robin write distribution across Kuri nodes. The user is not introducing a new requirement; they are holding the system accountable to its own specification.

The Input Knowledge Required

To understand this message, a reader needs to know several things that were established earlier in the conversation. First, the architecture: there are two Kuri storage nodes (kuri-1 and kuri-2), each with its own independent RIBS keyspace in YugabyteDB (filecoingw_kuri1 and filecoingw_kuri2), and a shared S3 keyspace (filecoingw_s3) that maps object keys to the node that owns them. Second, the frontend proxy is supposed to implement round-robin write distribution—this was explicitly stated in earlier messages as a design goal. Third, the monitoring UI has a "Frontend Proxies" table that shows per-node request rates, and the user has been looking at this table.

The user also assumes that the test traffic they generated (or that was generated by the assistant) should have been split between the two nodes. This is a reasonable assumption given the round-robin design, but it depends on the implementation being correct—which, as the observation reveals, it is not.

The Mistake That Wasn't Stated

The message does not identify why traffic is going only to kuri-1. It does not speculate about root cause. It simply reports the symptom. This restraint is notable—the user could have said "the round-robin logic is broken" or "kuri-2 isn't receiving writes," but they didn't. They stated the observable fact and let the asymmetry speak for itself.

The likely root cause, which the assistant would need to investigate after this message, is a bug in the frontend proxy's routing logic. Possible explanations include: the round-robin counter always returning the same index; the health check marking kuri-2 as unhealthy and the proxy skipping it; the node selection logic defaulting to the first node when no selection strategy is configured; or the write path not actually implementing round-robin at all despite being intended to. The message itself does not answer this—it only creates the need to investigate.

Output Knowledge Created

This message creates actionable knowledge. Before it, the assistant believed the system was working correctly—all containers up, all RPC endpoints returning data, test traffic flowing. After it, the assistant knows that the system is operationally correct but functionally incorrect. The difference is crucial: the cluster runs, but it does not scale.

The message also creates a new debugging task. The assistant must now trace the write path through the frontend proxy, find where the node selection decision is made, and determine why it always picks kuri-1. This is a fundamentally different kind of debugging than the earlier work (missing tables, container crashes, configuration errors). It is a logic bug, not an infrastructure bug.

The Deeper Significance

This message illustrates a pattern that recurs throughout distributed systems development: the gap between "the system works" and "the system works as designed." The assistant had verified connectivity, API responses, and metric collection—all the signs of life. But the user's observation revealed that the system was not fulfilling its primary design goal. Horizontal scalability was aspirational, not actual.

The message also demonstrates the value of user observation in the debugging loop. The assistant, focused on getting the cluster running and verifying endpoints, might not have immediately noticed the traffic imbalance. The user, looking at the system from a different angle—the monitoring dashboard rather than the RPC responses—saw the pattern that didn't fit. This is a reminder that monitoring dashboards exist precisely to make these patterns visible, and that the human eye is still one of the best anomaly detectors in the loop.

In the broader narrative of the coding session, this message marks the transition from deployment to optimization. The cluster is built, the containers are running, the database is initialized, the frontend is rendering. Now the real work begins: making the system actually do what it was designed to do.