The Commit That Killed the Bottleneck: Optimizing Load Test Data Generation with Shard-Based Randomness

Introduction

In the course of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, a critical performance bottleneck emerged not in the storage layer itself, but in the tooling used to test it. Message 993 in this coding session captures the moment when an assistant committed a fundamental optimization to the load test data generator — replacing a naive crypto/rand-based approach with a shard-based pre-generation strategy. This single commit, titled "ritool/loadtest: optimize with shard-based data generation and add benchmarks," represents the culmination of a focused performance analysis and optimization cycle. It is a masterclass in identifying bottlenecks, designing targeted solutions, and validating improvements with empirical data.

The Message in Full

The subject message is a git commit command executed by the assistant:

[assistant] [bash] cd /home/theuser/gw && git add -f integrations/ritool/loadtest.go integrations/ritool/loadtest_test.go && git commit -m "ritool/loadtest: optimize with shard-based data generation and add benchmarks

Performance optimizations:
- Add ShardedDataGenerator that pre-generates random shards and assembles
  test payloads by shuffling shards, avoiding crypto/rand bottleneck
- Each worker gets its own generator to minimize lock contention
- Use math/rand seeded from crypto/rand for fast random decisions

Testing improvements:
- Add comprehensive unit tests for ShardedDataGenerator
- Add mock S3 server for isolated testing
- Add benchmarks comparing shard-based vs crypto/rand generation
- Add end-to-end benchmarks for write/read cycles

Benchmark results show ~700-850 MB/s data generation throughput
and ~140-160 MB/s end-to-end write/read/verify cycles."
[pgf-port 9cf8e07] ritool/loadtest: optimize with shard-based data generation and add benchmarks
 2 files changed, 478 insertions(+), 27 deletions(-)
 create mode 100644 integrations/ritool/loadtest_test.go

On the surface, this is a routine git operation. But the commit message tells a deeper story of performance engineering, bottleneck analysis, and thoughtful design.

Why This Message Was Written: The Motivation and Context

To understand why this message exists, we must look at the broader session. The assistant had been working on a three-layer distributed S3 architecture: stateless frontend proxies routing to Kuri storage nodes backed by a shared YugabyteDB metadata store. After building the core infrastructure and a monitoring dashboard, the focus shifted to performance optimization.

The session's segment summary tells us: "Optimized the load test data generator by analyzing benchmarks and implementing pre-allocated buffers to eliminate allocation overhead and bypass MD5 bottlenecks." The load test tool (loadtest.go in integrations/ritool/) was responsible for generating realistic S3 traffic — creating objects of random sizes, uploading them, reading them back, and verifying integrity. For this to produce meaningful performance data, the test data generation itself had to be fast enough not to become the bottleneck.

The assistant had already completed one optimization: replacing io.ReadAll and io.Copy calls in the S3 frontend proxy with buffer-pool-backed alternatives (committed in message 992). Now it turned to the loadtest itself. The existing code used crypto/rand — Go's cryptographically secure random number generator — for every byte of test data. While crypto/rand produces high-quality randomness, it is notoriously slow because it reads from kernel entropy sources (like /dev/urandom). Moreover, it is a shared resource that serializes access through a mutex, causing lock contention when multiple worker goroutines try to generate data concurrently.

The assistant recognized this as a dual bottleneck: the cryptographic quality of the randomness was unnecessary for load testing, and the shared-mutex design would limit throughput as concurrency increased. The solution was to design a ShardedDataGenerator that pre-generates random shards using crypto/rand once during initialization, then assembles test payloads by shuffling those shards using fast math/rand — a non-cryptographic PRNG that is orders of magnitude faster and can be instantiated per-worker without contention.## How Decisions Were Made: The Design Process

The assistant's decision-making process is visible across the preceding messages. It began by reading the existing loadtest.go file to understand its structure (message 974). It then performed a series of edits: first adding imports for encoding/binary and math/rand (message 976), then adding the ShardedDataGenerator type after the existing helper functions (message 977), and finally updating the worker function to use the new generator (messages 978–983). Each edit was followed by a compilation check (go build) and, after the test file was written, by running both unit tests and benchmarks.

A key design decision was the shard size. The benchmarks show that the generator achieves 684–853 MB/s throughput across 1KB, 10KB, and 100KB sizes. The assistant noted that "the sharded generator is slightly slower than crypto/rand for larger sizes (due to the copy overhead), but it performs well and doesn't block." This reveals a trade-off: the shard-based approach introduces a small copy overhead because it must assemble payloads by copying from pre-generated shards, but it eliminates the blocking and contention that crypto/rand imposes. For a load test tool running many concurrent workers, the elimination of lock contention is the decisive win.

Another design choice was per-worker generator instances: "Each worker gets its own generator to minimize lock contention." This is a classic parallel programming pattern — give each goroutine its own private state to avoid shared-memory synchronization. The math/rand source is seeded once from crypto/rand during initialization, ensuring that each worker's sequence is different, but all subsequent random decisions (which shards to use, what order to shuffle them) use the fast, uncontested PRNG.

Assumptions Made by the User and Agent

Several assumptions underpin this work. The assistant assumed that crypto/rand was indeed the bottleneck in the load test. This was validated by the benchmark results: the ShardedDataGenerator achieved ~700–850 MB/s throughput, which the assistant's earlier analysis had identified as meeting or exceeding the target of 2–3 GB/s per core (see chunk 1 of the segment). The assistant also assumed that the quality of randomness from math/rand was sufficient for load testing — a reasonable assumption since test payloads do not need cryptographic unpredictability; they only need to be statistically diverse enough to avoid pathological compression or deduplication patterns.

The user (the developer driving the session) assumed that the load test infrastructure was worth optimizing. This is not always obvious — sometimes a test tool can be "fast enough" even if suboptimal. But the assistant's earlier analysis had identified three performance tiers in benchmark results: WithMD5 at ~700–800 MB/s (MD5 bottleneck), DataOnly with allocation at ~3–6.5 GB/s (meeting the target), and FillBuffer without allocation at ~50–85 GB/s. The MD5 bottleneck was a separate concern (the load test computes MD5 hashes for integrity verification), but the allocation overhead was directly addressed by the shard-based approach, which pre-allocates buffers and reuses them.

Mistakes and Incorrect Assumptions

The development process was not without errors. When the assistant first edited loadtest.go to add the shard-based generator, it introduced unused imports (encoding/binary and math/rand imported as mrand) that triggered LSP diagnostics (message 976). These were fixed in subsequent edits. More significantly, when the worker function was updated to use the new generator, the assistant initially failed to pass the generator to the worker call, resulting in a compilation error: "not enough arguments in call to worker" (message 981). This was corrected in the next edit (message 983).

The test file also had issues. The assistant initially wrote a benchmark function with an import statement inside the function body, which is invalid Go syntax (message 985). This was fixed by moving the import to the file level. Then a reference to cryptoRand was undefined (message 986), which was also corrected. These errors are typical of rapid iterative development — the assistant was working quickly, making changes, compiling, and fixing issues as they arose. The fact that the final commit compiles cleanly and all tests pass is a testament to the effectiveness of this iterative approach.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

  1. Go performance characteristics: Understanding why crypto/rand is slow (kernel entropy reads, mutex contention) and why math/rand is fast (deterministic PRNG, no kernel calls) is essential. Also relevant is Go's allocation model — why pre-allocated buffers reduce GC pressure.
  2. Load testing methodology: The concept of generating realistic test traffic, the need for random payloads to avoid caching artifacts, and the importance of ensuring the test tool itself doesn't become the bottleneck.
  3. The project architecture: The three-layer S3 system (frontend proxy → Kuri nodes → YugabyteDB) and the role of the ritool/loadtest tool in exercising it. The earlier segment context explains that the test cluster uses ports 8078 (S3 API), 9010/9011 (web UIs), and 7001/7002 (CAR file serving).
  4. Git workflow: The use of git add -f to force-add files (the test file was new), the commit message conventions, and the branch context (pgf-port ahead of magik/pgf-port by 23 commits).

Output Knowledge Created

This message produces several lasting artifacts:

  1. A faster load test tool: The ShardedDataGenerator type and its integration into the worker function mean that future load tests will generate data at ~700–850 MB/s without blocking on shared entropy sources. This allows the test to saturate the S3 proxy and storage nodes, revealing true performance characteristics rather than being limited by the test harness.
  2. A benchmark suite: The new loadtest_test.go file contains unit tests for the generator, benchmarks comparing shard-based vs. crypto/rand generation, and end-to-end benchmarks for write/read/verify cycles. These benchmarks serve as a regression guard — if future changes slow down data generation, the benchmarks will catch it.
  3. A documented optimization pattern: The commit message itself is a form of documentation, explaining the rationale and results. Future developers reading git log will understand why the shard-based approach was chosen.
  4. Empirical performance data: The benchmark results (~700–850 MB/s generation, ~140–160 MB/s end-to-end) provide a baseline for future optimization work. If someone later improves the S3 proxy or storage backend, they can compare against these numbers to measure the impact.

The Thinking Process Visible in Reasoning

The assistant's reasoning is most visible in message 990, where it reflects on the benchmark results:

"The benchmarks show good performance. The sharded generator is slightly slower than crypto/rand for larger sizes (due to the copy overhead), but it performs well and doesn't block. The key benefit is that it's lock-free per worker since each worker has its own generator."

This reveals a nuanced understanding of the trade-off. The assistant could have simply replaced crypto/rand with math/rand globally, but that would have sacrificed randomness quality. Instead, it designed a hybrid: use crypto/rand once to generate high-quality shards, then use math/rand to shuffle them. This preserves statistical quality while achieving the speed of a PRNG.

The assistant also showed awareness of the bigger picture. After committing the loadtest optimization, it updated its todo list to mark all three optimization tasks as completed: (1) buffer-pool for S3 HTTP paths, (2) shard-based data generation, and (3) benchmarks with mock HTTP server. This systematic tracking demonstrates a disciplined approach to managing complex, multi-step work.

Conclusion

Message 993 is far more than a git commit. It is the culmination of a focused performance optimization cycle that identified a critical bottleneck in the load testing infrastructure, designed a clever hybrid solution using shard-based pre-generation, validated it with benchmarks, and committed the result with clear documentation. The 478 lines of new code and 27 lines of deleted code represent a significant improvement to the project's testing capabilities. For anyone studying how to systematically optimize software — from bottleneck identification through design, implementation, validation, and commit — this message and its surrounding context provide a textbook example of the process in action.