The 100-Worker Threshold: When a Load Test Reveals Infrastructure Bottlenecks

In the middle of a high-stakes debugging session focused on optimizing a horizontally scalable S3 storage architecture, a single message from the assistant marks a critical inflection point. The message is brief, almost casual in its delivery:

Good, both nodes are healthy now. Let me run 100 workers:

```bash cd /home/theuser/gw && echo "=== 100 WORKERS ===" && go run ./integrations/ritool/... loadtest run --duration 30s --concurrency 100 --cleanup=false http://localhost:8078 2>&1 === 100 WORKERS === ═══════════════════════════════════════════════════════════════ S3 LOAD TEST ═══════════════════════════════════════════════════════════════

Endpoint: http://localhost:8078 Bucket: loadte... ```

This is message 1174 in a long conversation between a developer (the user) and an AI assistant working together to build and debug a distributed S3-compatible storage system. The message appears at the precise moment when the system transitions from "working correctly at low concurrency" to "breaking under realistic load." Understanding why this message matters requires unpacking the entire context that led to it, the assumptions baked into its execution, and the revelations that followed.

The Road to This Moment

To understand message 1174, one must first understand what came immediately before it. The session had been consumed with two major threads: a false corruption investigation and a YCQL write path optimization.

Earlier in the session, load tests had produced alarming "verify errors" — checksum mismatches suggesting data corruption when reading back objects that had just been written. This is the kind of bug that keeps storage engineers awake at night. The assistant and user had invested significant effort adding better error classification to the load test tool, distinguishing between genuine checksum mismatches and context deadline timeouts. The conclusion was a relief: no actual corruption was occurring. The "verify errors" were simply timeouts at the end of test runs, where the verification read raced against the write still being committed.

With corruption ruled out, attention turned to performance. The assistant implemented a CQLBatcher in the database/cqldb package — a mechanism that collects individual CQL INSERT calls and flushes them in batches of up to 15,000 entries or within 10–30 milliseconds. This batcher used a worker pool of 8 goroutines with exponential backoff retries and blocked callers until the batch was committed, preserving read-after-write consistency. The batcher was integrated into ObjectIndexCql.Put() by adding a Session() method to the Database interface.

But then the cluster wouldn't start. A configuration validation error surfaced: RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1. The default value of RetrievableRepairThreshold was 3, but the test cluster configuration set MinimumReplicaCount to 1. The assistant fixed the gen-config.sh script to export the missing environment variable, rebuilt the Docker image, and performed a clean restart of the entire test cluster.

After the restart, both Kuri storage nodes came online. The assistant ran a preliminary 10-worker load test (message 1171) which completed successfully. The user noted that kuri-2 was initially not healthy, but then confirmed "It is now." This sets the stage for message 1174.

Why This Message Was Written

Message 1174 is the logical next step in a systematic benchmarking protocol. The assistant had just confirmed three things:

Both nodes are healthy — the S3 proxy's backend pool reports kuri-1 and kuri-2 as available backends.
The 10-worker test passed — no corruption, no timeouts, clean results at low concurrency.
The system is operational — the S3 proxy on port 8078 accepts PUT requests and routes them to Kuri nodes. The natural progression in any load testing regimen is to increase concurrency until the system breaks, then diagnose the failure mode. The assistant chose 100 workers as the next step, following a logarithmic scaling pattern (1, 10, 100, 1000) that is standard practice for finding throughput ceilings. The --cleanup=false flag was retained from the 10-worker test, ensuring that written objects remain available for verification reads — a critical design choice that allows the test to detect silent data loss or corruption. The message is driven by a straightforward motivation: validate that the CQLBatcher optimization and configuration fixes actually improve throughput under realistic concurrency. The 10-worker test showed the system working. The 100-worker test would show whether it scales.

Assumptions Embedded in the Action

Every load test carries assumptions, and message 1174 is no exception. The most significant assumptions include:

That the Docker networking stack would handle 100 concurrent connections without becoming a bottleneck. This assumption proved incorrect. The test environment used Docker Compose with default bridge networking, which routes traffic through the Docker userland proxy — a known bottleneck at high concurrency. The "connection reset by peer" errors that appeared in the subsequent message (1175) were directly attributable to this proxy being overwhelmed.

That the CQLBatcher would scale with concurrency. The batcher was designed to reduce database contention by batching writes, but its behavior under 100 concurrent workers was untested. The batcher's 8-worker pool and 15,000-entry batch size might behave differently when 100 goroutines are all trying to insert objects simultaneously.

That the S3 proxy's backend pool would distribute load evenly. The proxy uses a round-robin or health-aware strategy to route requests to kuri-1 and kuri-2. At 100 workers, any imbalance in routing could cause one node to be overwhelmed while the other remains underutilized.

That the verification reads would succeed. The load test writes objects and then reads them back to verify checksums. At high concurrency, the verification phase itself becomes a significant load on the system, potentially interfering with ongoing writes.

The Thinking Process Visible in the Message

The message's reasoning content is compressed into a single sentence: "Good, both nodes are healthy now." This statement reveals a multi-step verification process that happened off-screen:

The assistant checked the S3 proxy logs to confirm both backends were registered as healthy.
The assistant cross-referenced the user's observation that kuri-2 had transitioned from "not healthy" to healthy.
The assistant made a judgment call that the system was stable enough to proceed with higher concurrency testing. The choice of 100 workers (rather than, say, 50 or 200) reflects an understanding of the system's expected capabilities. The assistant had previously seen the system handle 10 workers cleanly. Jumping to 100 is an order-of-magnitude increase — aggressive enough to find the breaking point, but not so aggressive as to be pointless (if 1000 workers had been tried first and failed catastrophically, the failure mode would be harder to isolate).

What the Message Reveals About the System

The truncated output in message 1174 — ending with "Bucket: loadte..." — is itself informative. The load test output was cut off because the tool's output was still streaming when the message was captured. This is typical of long-running commands where the assistant captures the initial banner but the actual test results appear later. The full results, visible in message 1175, tell a different story:

[W49] VERIFY READ ERROR loadtest/w49/obj504-...: 
  read tcp [::1]:42784->[::1]:8078: read: connection reset by peer
[W73] VERIFY READ ERROR loadtest/w73/obj518-...: 
  read tcp [::1]:43210->[::1]:8078: read: connection reset by peer

"Connection reset by peer" errors at 100 workers revealed that the infrastructure — specifically the Docker networking layer — was the bottleneck, not the application logic or the database. This is a classic finding in distributed systems: the first bottleneck is often not the code you just optimized, but the infrastructure you assumed would "just work."

Input Knowledge Required

To fully understand message 1174, one needs:

Knowledge of the S3 proxy architecture: The test cluster uses a three-layer design — stateless S3 frontend proxies on port 8078, Kuri storage nodes on internal ports, and a shared YugabyteDB for metadata. The load test targets the proxy layer, which routes to Kuri nodes.
Understanding of the load test tool: ritool loadtest run is a custom Go tool that writes objects to an S3-compatible endpoint, then reads them back to verify checksums. The --concurrency flag controls the number of parallel workers, --duration sets the test length, and --cleanup=false preserves objects for verification.
Awareness of the previous debugging context: The false corruption investigation, the CQLBatcher implementation, and the configuration fix for RetrievableRepairThreshold are all prerequisites for understanding why this particular test matters.

Output Knowledge Created

Message 1174 and its aftermath produced several valuable data points:

Throughput ceiling identified: The system could handle 10 workers cleanly but showed connection resets at 100 workers, placing the practical throughput ceiling somewhere between these values under the current Docker networking configuration.
Bottleneck localization: The "connection reset by peer" errors pointed to the Docker userland proxy rather than the application or database. This guided the subsequent fix: switching to host networking mode.
Validation of the batcher's correctness: Even under the stress of 100 workers, no genuine checksum mismatches appeared — only connection errors. This confirmed that the CQLBatcher was not introducing data corruption, which was the primary concern after the earlier false corruption investigation.
A clear next step: The failure mode at 100 workers directly motivated the Docker networking optimization that followed in the session, where the assistant rewrote docker-compose.yml to use host networking and removed the unnecessary nginx proxy.

The Broader Significance

Message 1174 is a textbook example of how load testing should work in practice. The assistant did not jump straight to maximum concurrency. Instead, it followed a disciplined progression: verify the system is healthy, test at low concurrency, confirm correctness, then escalate. When the system broke at 100 workers, the failure mode was clean and diagnosable — connection resets rather than silent corruption or mysterious crashes.

The message also illustrates a crucial insight about distributed systems optimization: the first bottleneck is rarely where you expect it. The assistant had invested significant effort optimizing the database write path with the CQLBatcher, yet the system broke at the Docker networking layer — a component that had been running without modification throughout the session. This is the nature of performance work: you optimize one layer only to discover the next bottleneck waiting beneath it.

In the end, message 1174 is not just a load test invocation. It is the pivot point where the session's focus shifted from "does the batcher work?" to "can the infrastructure handle the traffic?" — a shift that would lead to the host networking fix and ultimately to the clean throughput scaling that the architecture was designed to deliver.