The Art of Profiling: How One Developer Uncovered a Hidden MD5 Bottleneck and Pushed Data Generation from 700 MB/s to 85 GB/s

In a single, deceptively simple message, an assistant wrote: "Let me add a test for FillBuffer and GenerateData." On its surface, this looks like a routine maintenance step — adding test coverage for two new methods. But this message sits at the culmination of a furious optimization sprint that began when a user challenged the assistant's benchmark results as "extremely slow," expecting 2–3 GB/s per core. What followed was a textbook case of performance profiling, bottleneck identification, and surgical optimization that ultimately revealed a three-tier performance hierarchy ranging from a disappointing 700 MB/s all the way to an astonishing 85 GB/s.

The Context: A Load Test That Wasn't Fast Enough

The story begins in the broader context of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway. The architecture involved stateless S3 frontend proxies routing requests to independent Kuri storage nodes, with a shared YugabyteDB for metadata. To test this system under realistic conditions, the assistant had built a load testing utility (ritool/loadtest.go) with a ShardedDataGenerator — a component designed to produce random test payloads without becoming a bottleneck itself.

The original design pre-generated 256 random shards of 4 KB each, then assembled test payloads by shuffling and sampling from these shards. Each worker got its own generator to avoid lock contention. The initial benchmarks showed throughput around 700–850 MB/s, which seemed respectable. But the user knew better: "The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core."

The Profiling Revelation

The assistant responded by running Go's CPU profiler on the benchmark, and the results were stark. The pprof output showed that nearly 50% of all CPU time was spent in crypto/md5.block — the MD5 checksum computation. Another 15% went to runtime.memclrNoHeapPointers, the memory clearing that happens when allocating new byte slices. Together, these two operations consumed two-thirds of the CPU time, leaving only a fraction for actual data generation.

This was the critical insight: the Generate method was computing an MD5 checksum on every call, bundling data generation with checksumming into a single operation. For a load testing tool that only needs random bytes to throw at an S3 endpoint, computing MD5 on every payload was pure overhead — a tax that served no purpose in the test scenario.

The Refactoring: Separating Concerns

Armed with this knowledge, the assistant refactored the ShardedDataGenerator in three key ways:

  1. Separated data generation from checksum computation. The original Generate method returned both data and its MD5 checksum. The new design split this into GenerateData (data only) and kept a separate Generate method for when checksums were actually needed.
  2. Added a FillBuffer method that writes into a pre-allocated buffer. Instead of allocating a new byte slice for every call — which triggered the expensive memclrNoHeapPointers — this method reused an existing buffer, eliminating allocation overhead entirely.
  3. Switched to mrand.Uint32() for faster random index selection. The original code used Intn() which involves modulo operations; Uint32() with masking is significantly faster.

The Subject Message: Adding Tests for the Optimized Path

This brings us to the subject message. After running the updated benchmarks and seeing the dramatic results — FillBuffer without allocation achieving 50–85 GB/s, DataOnly with allocation still hitting 3–6.5 GB/s, and only the MD5-included path stuck at 700–800 MB/s — the assistant needed to update the loadtest worker to use these optimized methods. But before changing production code, the prudent step was to add tests for the new methods.

The message reads in full:

[assistant] Let me add a test for FillBuffer and GenerateData: [edit] /home/theuser/gw/integrations/ritool/loadtest_test.go Edit applied successfully.

This is the moment where the assistant transitions from discovery to validation. The benchmarks had already proven the performance characteristics in isolation, but unit tests serve a different purpose: they verify correctness across edge cases, ensure the methods handle various sizes correctly, and guard against regressions when the code is modified later.

Input Knowledge Required

To understand what this message accomplishes, one needs to know several things:

Output Knowledge Created

This message produces several tangible outputs:

  1. A new test function for FillBuffer that verifies the method correctly fills a pre-allocated buffer with random data of the requested size, without allocating new memory.
  2. A new test function for GenerateData that verifies the allocation-based path produces correct-sized, unique data without computing MD5.
  3. Validation that the optimized methods work correctly before they're integrated into the loadtest worker. This is a safety net — the benchmarks showed raw throughput, but the tests confirm functional correctness.
  4. Documentation of the API contract for future developers. Tests serve as executable documentation, showing exactly what inputs each method expects and what outputs it produces.

Assumptions and Their Validity

The assistant made several assumptions in this work:

Assumption: MD5 is unnecessary for load testing. This is correct for the primary use case — generating random payloads to stress-test an S3 endpoint. The S3 PUT API requires a Content-MD5 header for integrity verification, but that can be computed by the HTTP client layer rather than baked into the data generator. However, if a future test scenario needed to verify data integrity after read-back, the MD5 would be needed. The design preserves the MD5 path in Generate for exactly this purpose.

Assumption: Pre-allocated buffers won't cause correctness issues. The FillBuffer method writes exactly size bytes into the provided buffer. If the caller passes a buffer that's too small, there's a risk of panic or silent corruption. The tests need to verify that the method handles buffer sizing correctly.

Assumption: The benchmark results on a 96-core Threadripper PRO generalize. The benchmarks were run on an AMD Ryzen Threadripper PRO 7995WX with 192 logical cores. The 50–85 GB/s for FillBuffer is impressive, but this reflects the memory bandwidth of a high-end workstation CPU. On more modest hardware, the numbers would be lower, though the relative performance tiers should hold.

The Thinking Process

The assistant's reasoning follows a clear arc:

  1. Accept the user's challenge: Rather than defending the original 700 MB/s, the assistant immediately profiles to find the real bottleneck.
  2. Interpret profiling data: The pprof output shows MD5 at 50% and memory clearing at 15%. The assistant correctly identifies these as the targets for optimization.
  3. Design the fix: Separate checksum from generation, add a pre-allocated buffer path, use faster random selection. Each change targets a specific bottleneck.
  4. Benchmark to validate: Run the updated benchmarks to confirm the optimization works, producing the three-tier hierarchy.
  5. Test before integrating: Before updating the loadtest worker to use the new methods, add tests to ensure correctness. This is the subject message.
  6. Proceed to integration: The next logical step (implied but not shown in this message) is to update the worker function to use FillBuffer with pre-allocated buffers, eliminating allocation overhead in the hot loop.

Mistakes and Incorrect Assumptions

The most significant mistake was the original design that bundled MD5 checksumming with data generation. This wasn't a bug — it was a design choice that made the API convenient (one call gives you data and its checksum) but imposed a 50% performance tax on every call. The refactoring corrected this by separating the concerns.

A subtler issue: the original benchmarks only measured the combined path, masking the true cost of MD5. It took the user's prodding and a profiling session to reveal the bottleneck. This is a common pitfall in performance work — measuring the wrong thing, or measuring too coarsely.

The memory clearing overhead (15%) was another hidden cost. Go zero-initializes all allocated memory, which is a safety feature — it prevents accidental data leaks. But in a load test generator that immediately overwrites every byte, this zeroing is wasted work. The FillBuffer method eliminates it entirely by reusing buffers.

Conclusion

This single message — "Let me add a test for FillBuffer and GenerateData" — is the quiet bridge between discovery and delivery. It represents the moment when raw performance data is translated into verified, production-ready code. The tests it adds are the foundation for the next step: integrating the optimized data generation into the loadtest worker, where pre-allocated buffers will eliminate allocation overhead from the hot loop, and the absence of MD5 computation will free 50% of CPU time for actual data generation.

In the broader narrative of building a horizontally scalable S3 system, this optimization might seem like a minor detail. But load testing infrastructure is the bedrock of performance validation — if the test harness itself is a bottleneck, you can never accurately measure the system under test. By pushing data generation from 700 MB/s to 85 GB/s, the assistant ensured that the load test would never be the limiting factor, no matter how many concurrent workers were thrown at the S3 proxy.