Pushing the Limits: Diagnosing Throughput Bottlenecks in a Distributed S3 Load Test

Introduction

In the course of building a horizontally scalable, three-layer S3 architecture for the Filecoin Gateway, a critical moment arrived when the system had to be tested under realistic load. The subject message—index 1177 in the conversation—captures a pivotal diagnostic step: the decision to run a 1000-worker load test after observing performance degradation at 100 concurrent connections. This message, brief as it appears, is a window into the engineering mindset of systematic bottleneck hunting, where each test result informs the next hypothesis, and the boundary between "working correctly" and "working at scale" is probed with deliberate, escalating pressure.

The message reads in full:

[assistant] The "context canceled" errors suggest the clients are timing out while waiting for responses. The batcher is working, but the system is overloaded at 100 concurrent connections.

>

Let me run the 1000 worker test to see the limits: [bash] cd /home/theuser/gw && echo "=== 1000 WORKERS ===" && go run ./integrations/ritool/... loadtest run --duration 30s --concurrency 1000 --cleanup=false http://localhost:8078 2>&1 | tail -80 === 1000 WORKERS === [W25] VERIFY READ ERROR loadtest/w25/obj93-1769873374835758839: execute request: Get "http://localhost:8078/loadtest/loadtest/w25/obj93-1769873374835758839": read tcp [::1]:45462->[::1]:8078: read: connection reset by peer [W368] VERIFY READ ERROR loadtest/w368/obj81-1769873375016231390: execute request: Get "http://localhost:8078/loadtest/loadtest/w368/obj81-1769873375016231390": read tcp [::1]:44728->[::1]:8078: read: connection reset by peer [W502] VERIFY READ ERROR loadt...

This article unpacks the reasoning, assumptions, decisions, and knowledge embedded in this single message, treating it as a case study in performance investigation within a complex distributed system.

Context: What Led to This Message

To understand why this message was written, one must trace the arc of the preceding session. The assistant had just implemented a CQLBatcher—a write-path optimization that collects individual CQL INSERT calls and flushes them in batches using a worker pool with exponential backoff retries. This batcher was designed to reduce database contention under high concurrency, a necessary improvement for the S3 proxy layer that routes requests to Kuri storage nodes backed by YugabyteDB.

Earlier in the session, the assistant had confirmed that the batcher eliminated false corruption warnings that had plagued earlier tests. At 10 concurrent workers, the system delivered ~115 MB/s with zero corruption. At 100 workers, however, a new class of errors appeared: "connection reset by peer" during verify-read operations. The kuri-1 node logs showed repetitive "syncing group 101" messages, hinting at internal congestion.

The assistant's immediate inference, stated in the subject message, was that "context canceled" errors indicated client-side timeouts—the workers were waiting too long for responses and giving up. The batcher was functioning correctly, but the system as a whole was hitting a throughput ceiling at 100 concurrent connections. The natural next question: where exactly is the ceiling? The decision to jump to 1000 workers was a deliberate stress-test strategy: push far beyond the known failure point to characterize the failure mode comprehensively.

The Reasoning: Why 1000 Workers?

The decision to run a 1000-worker test rather than incrementally tuning at 100 or 200 workers reveals several layers of reasoning. First, the assistant wanted to "see the limits"—not just confirm that 100 workers caused problems, but to understand the shape of the degradation. Would errors increase linearly? Would the system completely lock up? Would a different failure mode emerge at higher concurrency? Running a test an order of magnitude above the known trouble point is a classic engineering technique: it transforms a vague "something is slow" observation into a concrete, reproducible failure pattern.

Second, the assistant was testing a hypothesis about the bottleneck's location. If the bottleneck were in the Kuri nodes themselves (e.g., the batcher's worker pool being saturated, or YugabyteDB struggling with write load), then 1000 workers would likely cause severe degradation or crashes on the backend. If the bottleneck were in the Docker networking layer—specifically Docker's userland proxy, which translates between host ports and container ports—then the errors might manifest as connection resets regardless of backend health. The assistant did not yet know which was the culprit, but the 1000-worker test would generate enough data to differentiate.

Third, there was an implicit assumption that the load test tool itself could sustain 1000 concurrent goroutines without becoming the bottleneck. The ritool/loadtest tool had been previously optimized with pre-allocated buffers and zero-allocation data generation, so the assistant had confidence that the client side would not artificially limit results.

Assumptions Embedded in the Message

Every diagnostic message rests on assumptions, and this one is no exception. The assistant assumed that the "context canceled" errors were purely a symptom of client-side timeouts caused by server overload. This was a reasonable inference—when a server cannot process requests fast enough, clients' context deadlines expire and they cancel their operations. However, the subsequent conversation reveals that this assumption was incomplete: the user immediately suggested "Might be docker-proxy issues?", pointing to a bottleneck at the Docker userland proxy level rather than the application layer.

The assistant also assumed that running a 1000-worker test was safe and informative. This assumption carried risk: a massive concurrency spike could potentially crash the YugabyteDB container, corrupt data, or leave the cluster in an unrecoverable state. The assistant mitigated this by using the --cleanup=false flag, which meant objects would persist after the test, but the test itself was bounded to 30 seconds. The assumption was that the system would degrade gracefully rather than fail catastrophically.

Another assumption was that the "connection reset by peer" errors seen at 100 workers would scale proportionally at 1000 workers. In reality, connection resets at Docker proxy level often exhibit threshold behavior: below a certain number of concurrent connections, the proxy handles them fine; above that threshold, connections are abruptly terminated. The 1000-worker test would reveal whether the error rate plateaued (suggesting a fixed resource limit) or grew linearly (suggesting per-connection overhead).

What Went Wrong: The Mistake in the Diagnosis

The most significant incorrect assumption in this message was the localization of the bottleneck. The assistant attributed the overload to "the system" broadly—the batcher, the Kuri nodes, or the database. The phrase "the batcher is working, but the system is overloaded at 100 concurrent connections" hedges the diagnosis without pinpointing the actual constrained resource. In reality, as the user suspected and the assistant subsequently confirmed in message 1179, the bottleneck was Docker's userland proxy (docker-proxy), which has known limitations under high concurrency.

The assistant's mistake was not in running the 1000-worker test—that was a sound engineering decision—but in prematurely concluding that the system was overloaded in a generic sense. The "connection reset by peer" error is a classic signature of a proxy or load balancer closing connections before they complete, not necessarily of the backend being unable to handle the load. By jumping to 1000 workers without first isolating the proxy layer, the assistant introduced a confounding variable: the test results would mix proxy-level failures with genuine backend failures, making it harder to interpret.

However, this "mistake" was also a productive one. The 1000-worker test produced dramatic evidence of connection resets, which, combined with the user's Docker proxy hypothesis, led directly to the solution: switching to Docker host networking mode to bypass the userland proxy entirely. The assistant's willingness to stress the system hard enough to expose the proxy's limits was, in retrospect, the fastest path to the correct diagnosis.

Input Knowledge Required

To fully understand this message, a reader needs substantial context about the system architecture. The three-layer design consists of:

Output Knowledge Created

This message produced several valuable outputs. First, it generated empirical data about the system's behavior at extreme concurrency: the 1000-worker test showed that connection resets were the dominant failure mode, with verify-read operations failing not due to checksum mismatches (actual corruption) but due to TCP connections being torn down mid-request. This output directly informed the decision to investigate Docker networking.

Second, the message established a baseline for future optimization. The assistant now knew that the system could handle 10 workers cleanly (~115 MB/s), struggled at 100 workers (connection resets appearing), and was essentially non-functional at 1000 workers. This three-point curve would later be used to measure the improvement from host networking mode.

Third, the message implicitly validated the batcher's correctness. Despite the connection resets, the errors were all "VERIFY READ ERROR" with "connection reset by peer"—not "CHECKSUM MISMATCH" or "CORRUPTION DETECTED." This distinction was crucial: it meant the batcher was writing data correctly, and the read-after-write verification was failing only because the read itself could not complete due to network issues. The data integrity path was sound.

The Thinking Process: A Window into Engineering Judgment

The structure of the message reveals the assistant's real-time thought process. It begins with a diagnosis ("context canceled errors suggest the clients are timing out"), then states a conclusion about the batcher's status, and finally proposes an experiment. This is classic scientific reasoning: observe a phenomenon, form a hypothesis, design a test that will produce discriminating evidence.

The choice to use tail -80 rather than capturing the full output is telling. The assistant expected the output to be voluminous—at 1000 workers, even a 30-second test generates thousands of log lines—and knew that the relevant information would be in the tail: the final error summary, the throughput numbers, and the last few failure messages. This is a practical optimization born from experience with large-scale testing.

The message also shows restraint. The assistant does not immediately blame the batcher, does not restart containers, and does not tweak parameters. Instead, the response is to gather more data. This is the hallmark of a mature debugging approach: resist the urge to fix until you understand.

Conclusion

Message 1177 is a small but instructive moment in a larger engineering narrative. It captures the transition from "the system works" to "the system works at scale"—a transition that every distributed system must undergo. The assistant's decision to run a 1000-worker test was not random escalation but a calculated diagnostic move, grounded in the assumption that pushing past the failure point reveals more than tuning at the edge. The message's implicit mistake (attributing overload to the backend rather than the Docker proxy) was corrected by the user's insight and the assistant's subsequent investigation, leading to the host networking fix that would unlock significantly higher throughput.

In the end, this message teaches a lesson about performance debugging: stress the system until it breaks, then examine the breakage pattern. The pattern—connection resets, not corruption—told the story. The batcher was fine. The database was fine. The Docker proxy was the bottleneck. And the 1000-worker test, chaotic as it seemed, was exactly the right experiment to prove it.