From False Corruption to Host Networking: Optimizing the YCQL Write Path and Eliminating Docker Proxy Bottlenecks in a Distributed S3 Storage System

Introduction

In the ongoing development of a horizontally scalable, three-layer S3-compatible storage system, a single coding session unfolded across nearly 200 messages that together represent a masterclass in distributed systems performance engineering. The session began with an alarming discovery—apparent data corruption during load testing—and ended with a fundamental re-architecture of the test cluster's networking layer. Along the way, the team implemented a sophisticated write-path batching engine, diagnosed a configuration validation bug that silently prevented storage nodes from starting, and identified Docker's userland proxy as a bottleneck that had been masquerading as data corruption.

This article traces that journey, examining the key decisions, assumptions, and insights that transformed a performance crisis into a series of architectural breakthroughs. The narrative is not linear; it is a spiral. Each layer of investigation revealed a deeper problem, and each fix uncovered the next bottleneck. The false corruption alarm led to the CQLBatcher, which led to load testing that revealed connection resets, which led to the Docker proxy diagnosis, which led to the host networking migration. Understanding how these threads interconnect reveals the essential nature of distributed systems debugging: every answer generates a new question, and the system's true architecture is only visible after you have traced every path from symptom to root cause.

The False Alarm: When "Data Corruption" Was Just a Timeout

The session's opening crisis was a set of "verify errors" reported by the S3 load test tool. Objects written to the cluster could not be read back correctly, with checksum mismatches suggesting that data was being corrupted in flight. In a storage system, this is an existential threat. The assistant immediately launched a deep investigation, tracing the write path through the codebase and identifying that the ObjectIndexCql.Put() method in the Kuri storage nodes was performing individual YCQL INSERT statements without any batching [1].

Under high concurrency, this pattern creates enormous database contention: each INSERT opens a separate network round-trip to YugabyteDB, competing for connections and overwhelming the database's ability to keep up. The initial load test at 10 concurrent workers had shown 2 verification failures out of 7,292 writes—a small percentage but an unacceptable result for a storage system claiming data integrity [1].

The solution seemed clear: implement a batching layer that collects individual INSERT calls and flushes them in bulk. The assistant designed and built a CQLBatcher in the database/cqldb package—a worker-pool-based engine with configurable batch sizes (default 15,000 entries), idle timeouts (10ms), max latency bounds (30ms), 8 concurrent workers, exponential backoff retries, and a blocking Submit() API that guarantees read-after-write consistency. The batcher was integrated into ObjectIndexCql.Put() by adding a Session() method to the Database interface, exposing the underlying gocql.Session [1].

But before the batcher could be tested, the assistant ran a baseline load test against the existing (pre-batcher) cluster. The output contained something unexpected. The error wasn't a checksum mismatch at all—it was context deadline exceeded on a VERIFY READ. This was the moment of truth. The assistant's response—"Interesting! The error was context deadline exceeded on a VERIFY READ—that's a timeout, not actual data corruption"—captures the pivot. The load test tool was using a single context with a duration timeout for both writes and verification reads. When the test was ending, the context was being cancelled, causing any in-flight verification reads to fail with context deadline exceeded. These timeouts were being counted as "verify errors" indistinguishable from actual checksum mismatches.

This discovery reframed the entire investigation. The system was not corrupting data; the test harness was creating its own errors. The assistant immediately added better error classification to distinguish between actual checksum mismatches and read timeouts, introducing a separate verifyTimeouts counter and updating the summary output to show timeouts separately from corruption events. The CQLBatcher, while still a valuable optimization for high-concurrency scenarios, was not the fix for the immediate "corruption" problem. The real fix was much simpler: separate the verification read context from the test duration context.

The CQLBatcher: Batching YCQL Writes for Performance

With the false corruption alarm resolved, the team turned to the genuine performance bottleneck: the write path to YugabyteDB. The ObjectIndexCql.Put() method was performing individual INSERT statements for every S3 object write, creating a one-to-one correspondence between HTTP requests and database round-trips. Under high concurrency, this pattern guarantees that the database connection pool becomes the system's limiting factor.

The CQLBatcher architecture is worth examining in detail because it reflects several important design decisions for distributed systems:

Worker pool with bounded concurrency. The batcher uses 8 goroutine workers that consume batches from a shared channel. This limits the number of concurrent database connections and prevents the batcher from overwhelming the database with simultaneous batch commits. The worker count is configurable but defaults to a conservative value that matches typical database connection pool sizes.

Configurable batch sizing. Each batch accumulates up to 15,000 individual INSERT entries before being flushed. This threshold is configurable and represents a trade-off between latency and throughput. Smaller batches reduce the time any single INSERT waits before being committed, but increase the overhead of batch management. Larger batches maximize throughput at the cost of increased latency for individual operations.

Idle and max-latency flush triggers. The batcher does not rely solely on the entry count threshold. It also flushes batches after 10ms of idle time (no new entries arriving) or after 30ms of maximum latency since the first entry in the batch was submitted. These time-based triggers ensure that the batcher does not introduce unbounded latency during low-throughput periods, while still aggregating aggressively during high-throughput bursts.

Exponential backoff retries. When a batch commit fails, the batcher retries with exponential backoff. This is critical for handling transient database failures without dropping writes or requiring the application layer to implement its own retry logic. The backoff prevents the batcher from hammering the database with retries during a degradation event.

Blocking Submit API. The Submit() method blocks the caller until the batch containing the entry has been committed. This guarantees read-after-write consistency: a client that writes an object and immediately reads it back will always find the data, because the write is not considered complete until the database confirms persistence. This design choice sacrifices some throughput (callers block waiting for batch completion) in exchange for a strong consistency guarantee.

The integration into ObjectIndexCql.Put() required adding a Session() method to the Database interface. This exposed the underlying gocql.Session to the batcher, allowing it to create and execute batch operations directly. The interface change rippled through the codebase, but the impact was minimal because the Database interface is only implemented by the YugabyteDB CQL driver.

The Configuration Bug That Silenced the Cluster

With the batcher implemented and the code compiling cleanly, the team prepared for high-concurrency testing. But when the assistant tried to restart the cluster, a new problem emerged: the S3 proxy returned "Service Unavailable - No healthy backends." The Kuri storage nodes, which should have been serving S3 requests on port 8078, were not reachable.

The assistant embarked on a systematic investigation. Checking the proxy logs showed that backends (kuri-1 and kuri-2) were being registered but marked unhealthy. Direct inspection of the Kuri containers revealed that port 8078 was not listening—the S3 server had never started. The logs contained a cryptic error: Configuration load failed: %w RetrievableRepairThreshold greater than MinimumReplicaCount: 3 > 1.

This was a configuration validation error. The Kuri binary, upon startup, parsed its environment variables and checked that RetrievableRepairThreshold (default 3) was not greater than MinimumReplicaCount (set to 1 in the generated config). The validation failed, and the configuration loading returned an error. Critically, this error did not crash the process—it printed a message and continued, but the S3 server initialization was gated on successful configuration loading. The S3 server never started.

The assistant initially suspected their own batcher changes had caused the problem, stashed the modifications, rebuilt the Docker image, and ran a clean restart. But the error persisted. This was the critical moment: the problem was pre-existing, not caused by the recent optimization work. The assistant had to shift from debugging their own code to understanding the configuration validation logic in the Kuri node itself.

The fix was straightforward: add RIBS_RETRIEVALBLE_REPAIR_THRESHOLD="1" to the generated configuration files and update the gen-config.sh script to ensure this parameter was always set consistently with MinimumReplicaCount. This one-line change, applied to the configuration generation layer rather than patching individual settings files, resolved the startup failure and allowed the cluster to come online.

This episode illustrates a recurring theme in distributed systems engineering: configuration management is a first-class engineering problem. In a system with multiple services, each with its own validation logic and default values, configuration becomes a combinatorial challenge. The gen-config.sh script is a reasonable approach, but it must be kept in sync with the codebase. Every time a new configuration parameter with validation is added, the generation script must be updated. The fact that this bug was pre-existing—not introduced by the current session's changes—makes it even more instructive. It had been silently preventing the cluster from functioning correctly, masked by the fact that earlier tests had used different configuration paths or had not exercised the S3 API endpoint.

Load Testing with the Batcher: 10 Workers, 100 Workers, 1000 Workers

With the cluster healthy and the batcher integrated, the assistant ran a systematic load testing protocol. The results at different concurrency levels told a revealing story.

10 concurrent workers: The system performed flawlessly. Write throughput reached 115 MB/s (231 operations per second), read throughput reached 117 MB/s (234 operations per second), and verification showed zero corruption events. This was the first clean result since the batcher was introduced, proving that the write path was correct and the configuration fixes were effective. The batcher was aggregating individual INSERT calls into efficient batch commits, and the database was keeping up without contention.

100 concurrent workers: Throughput scaled impressively to 334 MB/s for writes (674 ops/s) and 333 MB/s for reads (671 ops/s)—a 3x improvement over the 10-worker baseline. However, 958 "corruption" events appeared. Crucially, the assistant had already added the error classification that distinguished between actual data mismatches and connection failures. The "corruption" was actually read: connection reset by peer errors. The data was not corrupted; the TCP connections were being dropped before verification could complete.

1000 concurrent workers: The situation worsened to 3,735 connection failures while throughput plateaued at 337 MB/s for writes and 376 MB/s for reads. Throughput had stopped scaling, and the error rate was increasing linearly with concurrency.

The pattern was unmistakable. At 10 workers, zero errors. At 100 workers, connection resets. At 1000 workers, more connection resets and a throughput ceiling. The bottleneck was not in the application logic or the database—it was in the network transport layer.

The user's observation—"Might be docker-proxy issues?"—identified the culprit with surgical precision. Docker's default bridge networking mode uses a userland proxy process (docker-proxy) to forward traffic from the host port to the container port. This proxy is a userspace process that handles port mapping by accepting connections on the host side and forwarding them to the container. At high connection counts, this proxy becomes a significant bottleneck: it exhausts file descriptors, hits epoll limits, and eventually starts resetting connections.

The assistant confirmed the hypothesis by attempting to bypass the proxy, testing directly against a Kuri node's internal Docker IP address (172.22.0.3:8078). Direct access worked for individual requests, but running the full load test against the internal IP failed because internal IPs are only reachable from within the Docker network. The diagnostic direction was clear: the Docker bridge network's port mapping layer was the bottleneck.

The Host Networking Migration: Eliminating the Proxy

The user's next instruction was decisive: "Rewrite the test-cluster to use host network." This was not a suggestion to experiment—it was a directive to fundamentally change the network architecture of the test cluster.

Docker's host networking mode bypasses the userland proxy entirely by binding containers directly to the host's network stack. Containers share the host's IP address and port space, eliminating the translation layer. The trade-off is reduced network isolation, but for a performance-testing cluster, the throughput gains justified the change.

The assistant responded with a systematic, multi-file transformation. First, it read the existing docker-compose.yml to understand the current port allocation and architecture. The compose file revealed a carefully planned port layout:

The Architecture of a Debugging Session

What makes this session remarkable is not any single insight but the structure of the investigation itself. The team cycled through multiple layers of the system, each time discovering that the problem was not where they initially looked:

  1. Application layer: The initial hypothesis was data corruption in the storage engine. The investigation traced the write path, identified the batching gap, and led to the CQLBatcher implementation.
  2. Test infrastructure layer: The "corruption" turned out to be a testing artifact—a shared context in the load test tool that caused timeouts to be misclassified as verification failures.
  3. Configuration layer: The cluster wouldn't start because of a validation rule mismatch between a default value and a configured value, silently preventing the S3 server from initializing.
  4. Network infrastructure layer: At high concurrency, Docker's userland proxy became the bottleneck, dropping connections and creating errors that looked like application failures. Each layer revealed a different kind of problem, and each fix uncovered the next bottleneck. This is the essential nature of distributed systems debugging: the system is only as strong as its weakest link, and strengthening one link reveals the next.

Lessons for Distributed Systems Engineering

Several enduring principles emerge from this session:

Instrumentation is not neutral. The categories you choose for counting errors shape what you can see. A single verifyErrors counter conflates timeouts with corruption, making it impossible to distinguish between a slow network and a broken database. The fix—separate counters for timeouts and mismatches—is trivial once you know to look for it, but the original design didn't anticipate this distinction.

Configuration management is a first-class engineering problem. In a distributed system with multiple services, each with its own validation logic and default values, configuration becomes a combinatorial challenge. The gen-config.sh script is a reasonable approach, but it must be kept in sync with the codebase. Every time a new configuration parameter with validation is added, the generation script must be updated. The RetrievableRepairThreshold bug had been silently preventing the cluster from functioning correctly, masked by the fact that earlier tests had not exercised the S3 API endpoint.

The first bottleneck is rarely where you expect it. The assistant invested significant effort optimizing the database write path with the CQLBatcher, yet the system broke at the Docker networking layer—a component that had been running without modification throughout the session. This is the nature of performance work: you optimize one layer only to discover the next bottleneck waiting beneath it.

Test infrastructure is part of the system. When diagnosing production-like failures, it's easy to trust the test harness and look for bugs in the application. But the test harness itself can be the source of errors. The load test tool was using a single context for both test timing and individual operations—a design choice that created false positives under any scenario where operations outlasted the test duration.

Architectural decisions cascade. The decision to switch to host networking—driven by a performance bottleneck at high concurrency—did not stop at the docker-compose.yml file. It rippled into the configuration generation scripts, the service dependencies, and the mental model of how the cluster operates. Removing nginx was not an independent choice; it was a logical consequence of a deeper change.

Batch everything. The CQLBatcher pattern—collecting individual operations and flushing them in bulk with configurable thresholds, time-based triggers, and exponential backoff retries—is a general-purpose optimization that applies to any system where the cost of individual operations is dominated by round-trip latency. The batcher's blocking Submit API, which guarantees read-after-write consistency, demonstrates that batching does not have to sacrifice correctness for performance.

Conclusion

The session that began with a false corruption alarm and ended with a host networking migration is a microcosm of distributed systems engineering. It demonstrates that the most elusive bugs are often not in the hot path—the batching code, the health check logic, the routing algorithm—but in the cold path of configuration loading, initialization sequences, and infrastructure assumptions. A service that fails to start correctly will never serve a single request, no matter how well-optimized its runtime code is. A network proxy that silently drops connections at high concurrency will make a perfectly correct application look broken.

The team's success came from a willingness to question every layer, to read raw error output instead of relying on summary metrics, and to make fundamental architectural changes when incremental fixes proved insufficient. The CQLBatcher, the configuration fix, and the host networking migration each addressed a different bottleneck, and each was necessary for the system to achieve its potential throughput. Together, they transformed a cluster that couldn't sustain 100 concurrent connections without errors into one ready for high-performance S3 storage.

The load test results tell the final story. At 10 workers with the batcher, the system delivered 115 MB/s with zero corruption. At 100 workers, throughput scaled to 334 MB/s—a 3x improvement—before hitting the Docker proxy bottleneck. With host networking, the proxy bottleneck is eliminated, and the system is positioned to scale to much higher concurrency levels. The path from false corruption to host networking is not just a debugging story; it is a blueprint for systematic performance optimization in distributed systems.

References

[1] "The Architecture of a Debugging Breakthrough: From False Corruption to Host Networking in a Distributed S3 Storage System" — The previous segment's article covering the initial false corruption investigation and the early stages of the CQLBatcher implementation.