Reading the Load Distribution Signal: A Diagnostic Deep-Dive into S3 Proxy Routing

Introduction

In the course of deploying and validating a distributed S3-compatible storage system built on the Filecoin Gateway (FGW) architecture, one critical question inevitably arises: is the traffic actually being distributed across the backend nodes? The answer is rarely a simple yes or no—it lives in the numbers, ratios, and patterns that emerge under load. Message 2098 in this coding session captures the exact moment when an engineer, having just completed a successful load test, turns to the metrics to answer that question and finds something worth investigating.

This message is a diagnostic pivot point. It is not a bug fix, not a feature implementation, and not a deployment action. It is a moment of analysis—a reading of the system's vital signs—that reveals a 62/38 load imbalance between two storage nodes and prompts a chain of investigation into routing strategy, configuration, and system behavior. Understanding this message requires understanding what came before it, what the engineer was trying to prove, and what the numbers actually mean in the context of a hash-based distributed system.

The Message in Full

Here is the exact text of message 2098, written by the assistant (the AI coding agent) in response to the user's prompt to continue with next steps:

[assistant] The metrics show load distribution across both nodes:

| Metric | kuri_01 | kuri_02 | Ratio | |--------|---------|---------|-------| | PUT (writes) | 9,932 | 6,017 | 62% / 38% | | GET (reads) | 16,896 | 9,153 | 65% / 35% | | DELETE | 9,930 | 6,016 | 62% / 38% |

Load is distributed but not evenly - kuri_01 is getting ~62% of writes. This could be due to: 1. Hash-based routing favoring one node 2. Round-robin not perfectly balanced 3. One node being slightly faster

Let me check the proxy's routing strategy and cluster health:

[bash] # Check S3 proxy logs for routing info ssh 10.1.232.82 "sudo journalctl -u s3-frontend-s3_proxy_01 --no-pager -n 30 2>/dev/null | grep -iE 'backend|route|node|balance' || echo 'No routing logs found'" Jan 31 22:37:19 fgw-qa-head s3-frontend-s3_proxy_01[21057]: S3 Frontend Proxy started on :8078 (node: s3_proxy_01) Jan 31 22:37:19 fgw-qa-head s3-frontend-s3_proxy_01[21057]: Backend nodes: 2 configured Jan 31 22:37:21 fgw-qa-head s3-frontend-s3_proxy_01[21147]: S3 Frontend Proxy started on :8078 (node: s3_proxy_01) Jan 31 22:37:21 fgw-qa-head s3-frontend-s3_proxy_01[21147]: Backend nodes: 2 configured Jan 31 22:38:14 fgw-qa-head s3-frontend-s3_proxy_01[21147]: 2026-01-31T22:38:14.152Z ER...

[bash] # Check cluster topology from proxy perspective curl -s http://10.1.232.82:8078/cluster/topology 2>/dev/null || echo "No topology endpoint on proxy"

# Check the s3 proxy config ssh 10.1.232.82 "cat /data/fgw/s3-proxy/s3_proxy_01.yaml 2>/dev/null || echo 'Config not at expected path'" Not Found Config not at expected path

Why This Message Was Written: Context and Motivation

To understand why message 2098 exists, we need to trace the narrative arc of the preceding minutes. The session had been focused on deploying and validating a three-node FGW cluster: one head node running YugabyteDB and the S3 proxy, and two kuri storage nodes handling actual data persistence. The user had explicitly asked the assistant to "continue if you have next steps" (message 2094), and the assistant had responded with a comprehensive session summary (message 2093) that listed three pending items: verifying load distribution, setting up internet port mappings, and updating LocalWeb URLs.

The assistant then began executing on item one: load distribution verification. It first queried the YugabyteDB CQL store directly to count objects per node, finding only 5 objects (3 on kuri_01, 2 on kuri_02)—clearly not representative of real traffic. So it ran a fresh load test using ritool loadtest with 5 concurrent workers over 15 seconds, which produced 1,689 writes and 1,573 reads with zero data corruption. But the load test cleaned up its objects at the end, so the CQL count approach wouldn't work for the new data.

This is where the critical decision was made: instead of running another load test without cleanup, the assistant pivoted to querying the Prometheus metrics endpoints on each kuri node directly. This was a smart move—the metrics counters (fgw_s3_http_requests_total) accumulate over the lifetime of the process and include all historical traffic, including the just-completed load test. By scraping localhost:2112/metrics on both nodes, the assistant could compare request counts without needing to preserve test objects.

The result was message 2098: a data table showing that kuri_01 had handled 9,932 PUT requests, 16,896 GET requests, and 9,930 DELETE requests, while kuri_02 had handled only 6,017 PUTs, 9,153 GETs, and 6,016 DELETEs. The ratios were consistently around 62/38 in favor of kuri_01.

The motivation for writing this message was twofold. First, the assistant needed to communicate this finding to the user—to present the data transparently and invite interpretation or further instruction. Second, the assistant needed to begin its own diagnostic process: having spotted an imbalance, it immediately started investigating possible causes by checking proxy logs, topology endpoints, and configuration files. The message is both a report and the first step of a debugging workflow.## How Decisions Were Made: The Diagnostic Path

The decision-making in this message reveals a methodical diagnostic approach. The assistant had three possible avenues for measuring load distribution:

CQL object counting — querying the shared filecoingw_s3 keyspace to count rows per node_id. This was attempted first but the load test's cleanup routine had deleted all objects, leaving only 5 pre-existing entries.
Running another load test without cleanup — a viable option but time-consuming and potentially disruptive.
Prometheus metrics scraping — querying the /metrics endpoint on each node's port 2112, which exposes cumulative counters that persist across object lifecycles. The assistant chose option three, and this was the correct call. Prometheus counters are designed for exactly this kind of analysis: they are monotonically increasing, process-lifetime accumulators that survive object deletion. By scraping both nodes immediately after the load test, the assistant got a complete picture of all traffic the proxy had ever routed to each backend, including the 1,689 writes and 1,573 reads from the just-completed test. The decision to present the data as a ratio table (62/38, 65/35, 62/38) was also deliberate. Raw numbers like 9,932 vs 6,017 are hard to interpret without context—they could reflect a brief burst or a sustained imbalance. Converting to percentages normalizes the data and makes the pattern immediately visible: the imbalance is consistent across all three operation types (PUT, GET, DELETE), which strongly suggests a systemic routing bias rather than a transient condition. The assistant then made a second set of decisions about what to investigate next. It checked three things: - Proxy logs for routing-related messages (backend configuration, routing decisions) - A topology endpoint on the proxy itself (which returned "Not Found") - A YAML config file at a guessed path (/data/fgw/s3-proxy/s3_proxy_01.yaml), which also didn't exist These choices reveal an assumption about where proxy configuration would live. The assistant expected a YAML file based on common patterns in similar systems, but the actual configuration was stored as a systemd EnvironmentFile at /opt/fgw/config/s3_proxy_01/settings.env—a fact that would be discovered in the very next messages (2099–2102). The "Not Found" responses are not failures; they are information-gathering steps that narrow the search space.

Assumptions Embedded in the Analysis

Every diagnostic rests on assumptions, and message 2098 contains several worth examining.

Assumption 1: 62/38 is an imbalance worth investigating. In a two-node cluster with hash-based routing, perfect 50/50 distribution only occurs when keys are uniformly distributed across the hash space. The load test used keys generated by ritool loadtest, which likely uses sequential or timestamp-based naming patterns. If the hash function maps these keys unevenly onto the two backend slots, a 62/38 split is entirely within normal expectations. The assistant's framing ("Load is distributed but not evenly") carries an implicit judgment that 62/38 is suboptimal, but for many hash-based systems this is considered acceptable.

Assumption 2: The three listed hypotheses are exhaustive. The assistant offered three possible explanations: hash-based routing favoring one node, imperfect round-robin, or one node being slightly faster. The third hypothesis (speed imbalance) is actually unlikely to cause a persistent write imbalance in a hash-based system, because routing decisions are typically made before the request reaches the backend—the proxy doesn't know which node will respond faster. A more likely explanation not listed is that the routing hash ring has an uneven partition of the key space, which is a property of the hash function and the number of virtual nodes, not of backend performance.

Assumption 3: The proxy config exists at a YAML path. The assistant tried cat /data/fgw/s3-proxy/s3_proxy_01.yaml, which failed. This was a reasonable guess based on common conventions, but the actual config was an environment file managed by Ansible. This assumption was corrected in subsequent messages when the assistant found the systemd unit file and traced the EnvironmentFile directive to the correct path.

Assumption 4: The proxy has a /cluster/topology endpoint. The curl to http://10.1.232.82:8078/cluster/topology returned nothing. This endpoint may not exist on the proxy (it exists on the kuri nodes at port 9010 via RPC). The assistant was testing a hypothesis that the proxy would expose cluster topology information directly, which turned out to be incorrect.

Input Knowledge Required to Understand This Message

A reader needs significant context to fully grasp what message 2098 is doing:

The FGW architecture: Understanding that the system has a stateless S3 proxy frontend that routes requests to backend "kuri" storage nodes, which in turn store data in YugabyteDB. The proxy is the routing layer; the kuri nodes are the data plane.
Prometheus metrics conventions: Knowing that fgw_s3_http_requests_total is a cumulative counter labeled by HTTP method and status code, and that scraping localhost:2112/metrics gives a point-in-time snapshot of all activity since process start.
Load testing methodology: Understanding that ritool loadtest generates objects, verifies read-after-write consistency, and optionally cleans up. The cleanup phase explains why CQL object counting failed—the objects were deleted.
Hash-based routing: Knowing that distributed systems often route requests by hashing the object key and mapping the hash to a backend node. This means key distribution directly affects load distribution, and perfectly even distribution requires either uniform keys or consistent hashing with virtual nodes.
The session's history: The preceding messages show the S3 proxy being deployed via Ansible, the load test being run, and the initial CQL count attempt. Without this context, the pivot to Prometheus metrics seems arbitrary.

Output Knowledge Created

Message 2098 produces several distinct pieces of knowledge:

Quantitative load distribution data: The exact request counts and ratios for PUT, GET, and DELETE operations across both nodes. This is the first time in the session that actual traffic distribution has been measured at scale.
Confirmation that both backends are receiving traffic: The proxy logs confirm "Backend nodes: 2 configured" and both nodes show non-zero metrics. The system is working as a distributed cluster, not accidentally routing all traffic to one node.
A negative result about the topology endpoint: The /cluster/topology endpoint does not exist on the proxy, which is useful negative knowledge—it tells the operator that topology information must be obtained from the kuri nodes directly.
A negative result about the YAML config path: The config is not at /data/fgw/s3-proxy/s3_proxy_01.yaml, prompting the search that eventually finds it at /opt/fgw/config/s3_proxy_01/settings.env.
A diagnostic framework: The three hypotheses (hash bias, round-robin imperfection, speed difference) provide a mental model for thinking about load imbalance that can guide further investigation.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure of the message itself. It follows a clear pattern: observe → quantify → hypothesize → investigate.

The observation phase is the table of metrics. The quantification phase is the ratio calculation. The hypothesis phase is the three-item list. The investigation phase is the three shell commands that follow.

What's particularly interesting is the implicit reasoning about which metrics to compare. The assistant chose to compare absolute request counts rather than, say, request rates or error rates. This is a deliberate choice: absolute counts are simpler and more reliable for a point-in-time comparison, while rates would require knowing the exact time windows. The consistency of the ratio across all three operation types (PUT, GET, DELETE) is itself a finding—it suggests the imbalance is structural rather than operation-specific.

The assistant also implicitly reasoned that the imbalance is worth reporting but not necessarily alarming. The tone is neutral and analytical: "Load is distributed but not evenly." There's no panic, no immediate action to rebalance. The next steps are investigative, not corrective. This is the hallmark of mature system debugging—first understand, then act.

The failed config path lookup reveals another layer of reasoning: the assistant guessed a path based on naming conventions (/data/fgw/s3-proxy/s3_proxy_01.yaml). When that failed, it didn't give up—it moved to checking the systemd unit file (in subsequent messages) to trace the actual config location. The "Not Found" responses are treated as data, not as dead ends.

Conclusion

Message 2098 is a masterclass in distributed systems diagnostics. It takes raw Prometheus counters, transforms them into actionable ratios, presents them transparently, and immediately begins investigating the causes. The 62/38 load distribution it reveals is not a bug—it's a feature of hash-based routing with a finite key set. But by measuring it, quantifying it, and reporting it, the assistant creates a baseline against which future changes can be measured. The next time load distribution is checked, the operator will know whether the ratio has improved, worsened, or stayed the same.

This message also demonstrates the importance of choosing the right measurement tool. The assistant could have wasted time running another load test without cleanup, or trying to query CQL for objects that no longer existed. Instead, it recognized that Prometheus metrics—already being collected for observability—contained exactly the data needed. This is the essence of operational intelligence: knowing where to look for the signal.