Verifying Load Distribution in a Distributed S3 Cluster: Metrics-Driven Validation Under Pressure

Introduction

In the lifecycle of deploying a distributed storage system, few moments are as revealing as the first real load test. The message under analysis captures exactly such a moment: an engineer has just deployed a horizontally scalable S3-compatible proxy layer atop a multi-node storage backend, and is now attempting to verify that traffic is being distributed evenly across the cluster. What makes this message particularly instructive is not the load test itself, but the adaptive reasoning that follows when the initial verification strategy hits an unexpected obstacle.

The message, sent by an AI assistant operating as a DevOps engineer, documents the results of a load test against a newly deployed S3 proxy frontend, followed by a pivot to Prometheus metrics-based verification when the load test tool's built-in cleanup behavior invalidated the original plan. This article examines the reasoning, assumptions, decisions, and knowledge embedded in this single operational message.

The Message

The assistant writes:

The loadtest ran successfully with: - 1689 writes, 1573 reads - 0 data corruption - ~112 writes/sec, ~105 reads/sec

But it cleaned up the objects at the end. Let me run another test without cleanup to check distribution, or check the metrics endpoints for per-node stats:

``bash # Check metrics from both kuri nodes for request counts echo "=== KURI1 Metrics ===" ssh 10.1.232.83 "curl -s localhost:2112/metrics 2>/dev/null | grep -E 'http_requests_total|s3_requests_total|ribs_' | head -20" echo "" echo "=== KURI2 Metrics ===" ssh 10.1.232.84 "curl -s localhost:2112/metrics 2>/dev/null | grep -E 'http_requests_total|s3_requests_total|ribs_' | head -20" ``

The output shows kuri1's metrics: `` fgw_s3_http_requests_total{code="200",method="get"} 16896 fgw_s3_http_requests_total{code="200",method="put"} 9932 fgw_s3_http_requests_total{code="204",method="delete"} 9930 fgw_s3_http_requests_total{code="404",method="get"} 1 fgw_s3_http_requests_total{code="500",method="get"} 1 fgw_s3_http_requests_total{code="500",method="put"} 9 ``

Notably, the kuri2 metrics output is absent from the message, an omission that carries its own significance.

Context and Motivation: Why This Message Was Written

To understand why this message exists, one must trace the preceding twenty-two messages in the conversation. The assistant had just completed deploying an S3 proxy frontend on a head node (10.1.232.82) that routes requests to two backend Kuri storage nodes (10.1.232.83 and 10.1.232.84). This architecture follows a stateless proxy pattern: the S3 frontend handles authentication, routing, and protocol translation, while the Kuri nodes manage actual data storage and retrieval.

The deployment had been rocky. Earlier attempts to configure the proxy manually via SSH had failed because the environment variables were not being read correctly—the code was looking for FGW_YCQL_HOSTS instead of FGW_S3_CQL_HOSTS. The user rightly pushed back: "Why are you not doing this with ansible?" This led to a proper Ansible-based deployment that succeeded.

With the proxy running and basic read/write tests passing, the natural next step was to verify the system's core architectural promise: that the proxy distributes traffic across both backend nodes. Without this verification, the cluster is not truly distributed—it might be routing all traffic to a single node, defeating the purpose of horizontal scaling.

The assistant's first attempt at verification used a CQL query to count objects per node, but this failed because YugabyteDB does not support GROUP BY in CQL. A quick workaround using Python's Counter showed only 5 objects—leftover from earlier manual tests. The assistant then ran a proper load test using the ritool loadtest command, which brings us to the subject message.## The Reasoning Process: Adaptive Verification Under Constraints

The message reveals a sophisticated reasoning chain. The assistant has just received the load test results: 1689 writes, 1573 reads, zero corruption, respectable throughput of ~112 writes/sec and ~105 reads/sec. These numbers are encouraging—they demonstrate that the proxy is functional under load. But the assistant immediately identifies a problem: "But it cleaned up the objects at the end."

This is a critical observation. The ritool loadtest command, by default, deletes all objects it creates after the test completes. This means the assistant cannot simply query the database to count objects per node, because the objects no longer exist. The original verification strategy—counting stored objects per node via CQL—is now unavailable.

The assistant's response is a textbook example of adaptive troubleshooting. Rather than running the load test again with different flags (which would take time), the assistant proposes two alternatives in rapid succession: "Let me run another test without cleanup to check distribution, or check the metrics endpoints for per-node stats." The second option is immediately executed, suggesting the assistant recognized that metrics endpoints offer a more elegant solution: they provide cumulative request counts without needing to create persistent objects.

This decision reflects an understanding of the system's architecture. The Kuri nodes expose Prometheus metrics on port 2112, including fgw_s3_http_requests_total with labels for HTTP method and response code. By querying these metrics on both nodes, the assistant can compare request counts and infer distribution patterns—all without creating any test data.

Assumptions Made

Several assumptions underpin this message:

First, the assistant assumes that the load test traffic was routed through the S3 proxy and that the proxy distributed requests across both Kuri nodes. This is the hypothesis being tested, not a known fact. The metrics output will either confirm or refute this.

Second, the assistant assumes that the Prometheus metrics counters on each Kuri node accurately reflect only the traffic routed through the proxy. This is a reasonable assumption given the architecture—the Kuri nodes' S3 ports (8079) are internal and not directly exposed to the load test tool, which targets the proxy's public endpoint (8078). However, if any previous direct testing had hit the Kuri nodes' S3 ports, those requests would be included in the counters.

Third, the assistant assumes that curl -s localhost:2112/metrics will work from the SSH session. This requires that the Prometheus HTTP endpoint is listening on localhost, that the port is accessible, and that the Kuri process is running. The assistant has previously verified that all services are active, so this assumption is well-founded.

Fourth, the assistant assumes that the metrics labels (code="200", method="put", etc.) are sufficient to distinguish load test traffic from other requests. This is reasonable given that the load test is the primary source of S3 traffic at this moment.

Mistakes and Incorrect Assumptions

The most notable issue in this message is the missing kuri2 metrics output. The assistant queries both nodes but only displays kuri1's results. The kuri2 output is absent, and the assistant does not comment on this absence. This could mean:

The command failed silently (e.g., SSH connection issue, curl timeout, or the metrics endpoint wasn't responding on kuri2).
The output was identical and the assistant chose not to display it for brevity.
The assistant inadvertently omitted it. Given the context of the conversation—where the assistant has been meticulously displaying command outputs—the most likely explanation is that something went wrong with the kuri2 query, and the assistant either didn't notice or intended to address it in a follow-up. This is a minor but notable gap in the reasoning process. Additionally, there is a subtle assumption about what the metrics actually prove. Even if both nodes show similar request counts, this does not guarantee that the proxy is distributing traffic evenly in real-time. The metrics are cumulative over the lifetime of the process. If kuri1 had been running longer or had processed requests from earlier tests, its counters would be higher regardless of the load test distribution. The assistant does not account for this baseline offset.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Distributed systems architecture: Understanding the stateless proxy pattern, where a frontend routes requests to backend storage nodes, is essential. Without this, the purpose of checking per-node metrics is unclear.

Prometheus metrics exposition format: The output shows # HELP and # TYPE lines followed by metric names with label-value pairs. Familiarity with this format is needed to interpret the data.

S3 protocol semantics: The metrics distinguish GET, PUT, and DELETE methods with HTTP status codes. Understanding that 200 means success, 204 means success with no content (typical for DELETE), and 500 means server error provides context for evaluating system health.

Command-line tooling: The use of ssh with remote command execution, curl for HTTP requests, and grep for filtering demonstrates standard DevOps practices.

The specific project architecture: Knowing that Kuri nodes expose Prometheus on port 2112, that the S3 proxy runs on port 8078, and that ritool loadtest is the project's load testing tool is necessary to connect the dots.## Output Knowledge Created

This message produces several valuable outputs:

Load test results: The assistant confirms that the S3 proxy can handle ~112 writes/sec and ~105 reads/sec with 5 concurrent workers over 15 seconds, with zero data corruption. This is the first quantitative validation of the proxy's performance in the QA environment.

Metrics-based verification data: The kuri1 metrics show 16,896 GET requests with 200 status, 9,932 PUT requests with 200 status, and 9,930 DELETE requests with 204 status. These numbers are far higher than the load test's 1,689 writes and 1,573 reads, indicating that the metrics counters include all requests processed since the Kuri process started—not just the latest load test. This is itself a valuable discovery: the metrics are cumulative, not session-based.

Error rate insight: The metrics reveal 9 PUT requests returning 500 (server error) and 1 GET returning 500. While the error rate is low (roughly 0.05% for PUT, 0.006% for GET), these errors warrant investigation. The assistant does not comment on them in this message, but the data is now available for future analysis.

Architectural validation pathway: The message establishes a repeatable method for verifying load distribution without creating persistent test objects. This is a reusable operational pattern.

The Thinking Process Visible in Reasoning

The assistant's reasoning is visible in several ways:

The "But" pivot: The sentence "But it cleaned up the objects at the end" is the fulcrum of the message. It shows the assistant recognizing an obstacle and immediately recalibrating. This is not a failure—it is adaptive intelligence at work.

The parallel proposal: "Let me run another test without cleanup to check distribution, or check the metrics endpoints for per-node stats" presents two options simultaneously. The assistant then chooses the second option without waiting for user input, demonstrating autonomous decision-making within established boundaries.

The grep pattern selection: The choice of http_requests_total|s3_requests_total|ribs_ as grep patterns reveals an understanding of the metrics naming conventions. The assistant knows which metric families are relevant to the question at hand and filters aggressively to avoid information overload.

The head -20 limit: By limiting output to 20 lines, the assistant acknowledges that metrics endpoints can return hundreds of lines and that only a subset is needed for this verification.

Broader Significance

This message sits at the intersection of deployment validation and operational monitoring. It represents the moment when a system transitions from "it works" to "it works correctly under load." The assistant's ability to pivot from a blocked verification strategy (CQL queries after cleanup) to an alternative strategy (Prometheus metrics) demonstrates the kind of flexible thinking that distinguishes effective operations from rote execution.

The missing kuri2 output is a reminder that even in automated operations, human oversight is necessary. A fully autonomous system might have proceeded without noticing the gap. The assistant's message, by including only kuri1's output, implicitly flags that something may be amiss with kuri2—whether intentionally or not.

Conclusion

The subject message captures a critical moment in the deployment lifecycle: the transition from functional validation to performance validation under load. The assistant successfully runs a load test, identifies a limitation in the verification approach, adapts on the fly, and begins collecting metrics-based evidence of load distribution. The message is a microcosm of the entire session's theme: iterative debugging, adaptive problem-solving, and the relentless pursuit of architectural verification.

The 1,689 writes and 1,573 reads with zero corruption are more than just numbers—they are proof that the stateless proxy architecture is working. The metrics from kuri1, showing tens of thousands of requests, provide the foundation for deeper analysis. And the unanswered question about kuri2's metrics sets the stage for the next iteration of debugging, which is exactly how complex distributed systems are built: one verified assumption at a time.