The Loadtest Specification: Defining Quality Gates for a Distributed S3 Architecture

Message Analysis

Subject Message (index 925, role: user): "Write a loadtest utility in ritool, util that writes objects, incl multiparts, into the endpoins, X to Y kb/mb per object, some read/write ratio. The tool should test throughput and correctness of read-after-write guarantee."

This single message, appearing at message index 925 in a long coding session, represents a critical inflection point in the development of a horizontally scalable S3-compatible storage system for the Filecoin Gateway. After dozens of messages spent building, debugging, and refining the three-layer architecture—stateless S3 frontend proxies routing to independent Kuri storage nodes backed by shared YugabyteDB—the user pivots from building the system to proving the system works. The request for a loadtest utility is not merely a feature addition; it is an implicit demand for objective evidence that the architecture meets its performance and correctness requirements.

The Context That Demanded This Message

To understand why this message was written, one must appreciate the state of the project at that moment. The assistant had just completed a grueling debugging session spanning segments 0 through 3 of the conversation. The architecture had been fundamentally restructured after the user identified a critical flaw: the assistant had been running Kuri nodes as direct S3 endpoints, violating the roadmap's requirement for separate stateless frontend proxy nodes. This led to a complete redesign with proper separation of concerns—S3 proxies on port 8078, Kuri storage nodes with independent configurations, and a shared metadata layer in YugabyteDB.

By message 924, the assistant had committed six logical commits covering JSON tag fixes, I/O throughput monitoring, cluster-wide stats aggregation, round-robin logging, and visual UI improvements. The test cluster was operational: two Kuri nodes, an S3 frontend proxy, nginx-based web UIs, and real-time monitoring dashboards showing request throughput and latency distributions. But operational is not the same as verified. The user could see the cluster running, but lacked quantitative evidence that it could handle realistic workloads, that objects written could be read back correctly, and that the system could sustain throughput under load.

This is the precise moment when the user says, in effect: "Stop building features. Build the tool that will tell us if the features work."

The Requirements Embedded in the Request

The user's message is deceptively dense. In a single sentence, it encodes a complete specification:

"Write a loadtest utility in ritool" — The tool must follow the existing patterns of the ritool CLI, a command-line interface for "ribs repository manipulation commands." This constrains the implementation to the same Go codebase, the same urfave/cli command registration pattern, and the same build system. The user is not asking for a standalone script; they are asking for a committed, maintainable part of the project's tooling.

"util that writes objects, incl multiparts" — The tool must exercise both simple PUT operations and the S3 Multipart Upload API. This is significant because multipart uploads are a critical feature for large objects (typically >5MB) and involve a multi-step protocol: InitiateMultipartUpload, UploadPart (multiple times), and CompleteMultipartUpload. Testing multipart reveals whether the S3 frontend proxy correctly handles the stateful coordination required for multi-part operations.

"into the endpoins" — Note the typo ("endpoins" instead of "endpoints"), but the meaning is clear: the tool must target the S3 API endpoint (port 8078), not the Kuri nodes directly. This reinforces the architectural separation—the loadtest exercises the system through the stateless frontend layer.

"X to Y kb/mb per object" — Object sizes must be configurable, with a range. This is crucial for realistic testing: small objects stress metadata operations and request overhead, while large objects stress I/O throughput and buffer management. The user wants to probe behavior across the size spectrum.

"some read/write ratio" — The workload must mix reads and writes in a configurable proportion. A pure write test would miss read latency and correctness issues; a pure read test requires pre-existing data. A configurable ratio allows the user to simulate realistic workloads (e.g., 80% reads, 20% writes for a storage gateway) or stress-test specific paths.

"test throughput and correctness of read-after-write guarantee" — This is the core requirement. Throughput measures how much data the system can move per unit time (MB/s, ops/sec). Correctness of read-after-write means: after a successful PUT response, a subsequent GET of the same object must return identical content. This is the fundamental contract of any storage system, and the user is demanding automated verification of it under load.

Assumptions Embedded in the Request

The user makes several assumptions that are worth examining:

Assumption 1: The S3 endpoint supports multipart uploads. This assumption proved partially incorrect during subsequent testing. When the assistant ran the loadtest with multipart enabled, it encountered numerous write errors, suggesting that the S3 frontend proxy's multipart implementation had gaps or bugs. The user's request implicitly assumed multipart was functional; the loadtest itself became the mechanism to discover it was not.

Assumption 2: The existing ritool CLI patterns are suitable for a loadtest command. The ritool was originally designed for repository inspection commands (carlog analysis, claims extension, group management). Adding a network-intensive load testing tool to this CLI is a departure from its original purpose. The user assumes the CLI framework can accommodate this new category of command without architectural friction.

Assumption 3: Read-after-write correctness can be verified with MD5 checksums. The assistant's implementation used MD5 hashes computed client-side before upload, stored as the object's content, and verified on read. This assumes the S3 endpoint preserves object content bit-exactly and that MD5 is a sufficient checksum. While reasonable, this assumption would fail if the S3 implementation performed server-side transformations (e.g., compression, encryption, or format conversion).

Assumption 4: The test cluster can sustain meaningful load. The user's request assumes the Docker Compose-based test cluster with two Kuri nodes and a shared YugabyteDB instance can handle concurrent load without becoming the bottleneck itself. In practice, the test cluster's resource constraints (running on a single machine) would limit the maximum throughput the loadtest could measure.

Input Knowledge Required

To fully understand this message, the reader needs to know:

  1. The architecture: The three-layer design with S3 frontend proxies (stateless, port 8078), Kuri storage nodes (independent, ports 7001/7002), and shared YugabyteDB for metadata. The loadtest targets the S3 API layer.
  2. The ritool codebase: The existing CLI commands in integrations/ritool/ use urfave/cli for command registration, with each command defined as a package-level variable (e.g., headCmd, carlogCmd). The main entry point registers commands in a slice.
  3. The S3 API semantics: Understanding of PUT, GET, DELETE, and Multipart Upload operations, including the HTTP request/response formats, error codes, and the MD5 ETag convention.
  4. The project's performance concerns: Earlier segments revealed that the assistant had been optimizing a data generator with pre-allocated buffers to avoid allocation bottlenecks. The user's later message (index 950) explicitly asks for buffer pool optimizations and shard-based random data generation, indicating that performance is a first-class concern.
  5. The test cluster configuration: The endpoint URL http://localhost:8078, the bucket naming convention, and the fact that the cluster runs on a single development machine with resource constraints.

Output Knowledge Created

This message generated a specification that the assistant would implement across the next 24 messages (926-949). The output knowledge includes:

  1. The loadtest command structure: A new loadtestCmd registered in main.go, with subcommands including run for executing tests.
  2. Configuration parameters: --bucket, --concurrency, --duration, --min-size, --max-size, --read-ratio, --multipart-threshold, --verify, --cleanup, --verbose.
  3. The implementation pattern: A worker pool architecture with configurable concurrency, each worker generating random objects, uploading via PUT or multipart, optionally reading them back for verification, and reporting statistics.
  4. The metrics pipeline: Collection of operation counts, latency histograms, throughput measurements, and error rates, with percentile reporting (p50, p95, p99).
  5. The verification protocol: Client-side MD5 computation before upload, comparison with server response, and re-verification on read to detect data corruption or inconsistency.

The Thinking Process Visible in the Request

The user's thinking reveals a methodical approach to quality assurance. Having spent the previous segments building and debugging the system, the user now asks: "How do we know it works?" The request is not for a simple smoke test or a one-off script—it is for a committed, reusable, parameterized testing tool that can be run repeatedly as the system evolves.

The inclusion of multipart support is particularly telling. The user understands that simple PUT operations are not sufficient to validate the S3 frontend proxy's correctness. Multipart uploads involve multiple HTTP requests, state management across the proxy layer, and eventual consistency considerations. If the system can handle multipart uploads correctly under load, it demonstrates a higher level of maturity.

The read-after-write correctness requirement reveals the user's deep understanding of distributed storage guarantees. In a system with multiple storage nodes and eventual consistency, it is possible for a write to succeed on one node while a subsequent read is routed to a different node that hasn't yet received the update. The loadtest must detect this scenario.

The configurable read/write ratio shows the user's intent to model realistic workloads. A pure write test might miss read-side performance issues; a pure read test requires pre-seeded data. By making the ratio configurable, the user can explore the system's behavior across the spectrum from write-heavy (ingest) to read-heavy (retrieval) workloads.

Conclusion

Message 925 is a turning point in the coding session. It transitions the conversation from construction to validation. The user is not satisfied with a system that merely starts up and shows green health checks—they demand a quantitative, repeatable, automated assessment of throughput and correctness. The loadtest utility becomes the objective arbiter of whether the distributed S3 architecture actually works.

The message also reveals the user's engineering philosophy: build the test infrastructure alongside the system itself. The loadtest is not an afterthought or a separate project; it is a committed part of the ritool codebase, subject to the same code review, version control, and maintenance as the production code. This integration of testing into the core tooling is a hallmark of professional software engineering, and it is this standard that the user enforces with this single, dense, specification-packed message.