Profiling to Victory: How Benchmark Analysis Uncovered a 100x Performance Gain in Data Generation

"The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core."

This blunt challenge from the user landed at a critical juncture in the development of a horizontally scalable S3 storage system for the Filecoin Gateway. The assistant had just committed an optimized loadtest utility with a ShardedDataGenerator designed to avoid the crypto/rand bottleneck during high-throughput testing. The initial benchmarks showed a respectable ~700–850 MB/s data generation throughput. But the user knew better: on a 96-core AMD Ryzen Threadripper PRO 7995WX, that number was orders of magnitude too low. The expectation was 2–3 GB/s per core. Something was deeply wrong.

The subject message — message 1006 in the conversation — is the moment of reckoning. It is the assistant's analysis of benchmark results after a targeted profiling and optimization cycle, and it reveals a dramatic three-tier performance breakdown that transformed a mediocre data generator into one capable of 50–85 GB/s. This message is a masterclass in performance debugging: identify the real bottleneck, isolate it, measure the impact of each optimization, and then act on the findings.

The Full Text of the Subject Message

[assistant] Excellent! Now we see the real performance:

- WithMD5: ~700-800 MB/s (MD5 is the bottleneck) - DataOnly (with allocation): ~3-6.5 GB/s - FillBuffer (no allocation): ~50-85 GB/s

The FillBuffer without allocation is extremely fast (50+ GB/s), and even with allocation it's 3-6 GB/s which meets the target of 2-3 GB/s per core.

Now let me update the loadtest worker to use pre-allocated buffers where possible. But first, I need to update the test that uses Generate: [read] /home/theuser/gw/integrations/ritool/loadtest_test.go

The Reasoning and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the chain of events that led to it. The assistant had been building a comprehensive load testing infrastructure for an S3-compatible distributed storage system. The architecture involved three layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB metadata store. To validate performance, the assistant needed a data generator that could produce random test payloads at high throughput without becoming a bottleneck itself.

The initial approach used a ShardedDataGenerator that pre-generated 256 random shards of 4KB each, then assembled test payloads by shuffling and sampling these shards. This clever design avoided the well-known bottleneck of crypto/rand, which serializes through a global mutex and becomes a contention point under high concurrency. The initial benchmarks showed ~700–850 MB/s, which seemed reasonable at first glance.

But the user's challenge — "profile why it's so extremely slow" — forced a deeper investigation. The assistant ran a CPU profile using Go's pprof tool, and the results were eye-opening: 50% of all CPU time was spent in crypto/md5.block. The MD5 checksum calculation, not the data generation itself, was the dominant bottleneck. Another 15% went to runtime.memclrNoHeapPointers, the zeroing of newly allocated byte slices. The data generation logic itself was barely visible in the profile.

This discovery drove a refactoring effort. The assistant made three key changes:

Separated data generation from checksum computation — the Generate method had been computing an MD5 hash of every payload, which was unnecessary for many use cases.
Added a FillBuffer method that writes data into a pre-allocated buffer, avoiding allocation and zeroing overhead entirely.
Used mrand.Uint32() instead of mrand.Intn() for faster random index selection, a micro-optimization that reduced per-operation overhead. The benchmark results in the subject message are the payoff of that refactoring. They reveal three distinct performance tiers that tell a compelling story about where the real costs lie.## The Three Performance Tiers: A Diagnostic Breakdown The three numbers in the subject message are not just raw performance figures — they are a diagnostic fingerprint of where CPU cycles are being spent. Tier 1: WithMD5 (~700–800 MB/s). This is the original benchmark result that triggered the investigation. The CPU profile had already revealed that crypto/md5.block consumed 50% of execution time. The MD5 hash computation is a fixed cost per byte — Go's standard library implementation processes data in 64-byte blocks, and on this AMD Ryzen processor, it achieves roughly 1.4–1.6 GB/s per core. Since the benchmark was also doing data generation (shard assembly + copying), the combined throughput settled at ~700–800 MB/s. The MD5 was the ceiling. Tier 2: DataOnly with allocation (~3–6.5 GB/s). By removing the MD5 computation, the assistant isolated the cost of data generation alone. The 4–8x speedup over the MD5-included case confirms that the hash was indeed the primary bottleneck. But even at 3–6.5 GB/s, the benchmark still showed allocation overhead: runtime.memclrNoHeapPointers accounted for 15% of the original profile. Every call to make([]byte, size) forces the Go runtime to zero the memory, which is a linear-time operation proportional to the allocation size. The variance in this tier (3 to 6.5 GB/s) likely reflects different sizes hitting different allocation paths in the Go memory allocator. Tier 3: FillBuffer with no allocation (~50–85 GB/s). This is the "cheating" benchmark — and it's the most informative one. By pre-allocating a buffer once and reusing it across benchmark iterations, the assistant eliminated both allocation and zeroing costs. The resulting throughput of 50–85 GB/s is essentially the speed of memory bandwidth: the shard copy operations (memcpy) and the random index selection. At these speeds, the bottleneck has shifted from the CPU to the memory subsystem. This number is an upper bound on what the hardware can sustain for a pure copy workload.

Assumptions Made and Mistakes Corrected

The subject message reveals several implicit assumptions that were either validated or corrected during this investigation.

Assumption 1: The shard-based approach was already optimal. The initial design of ShardedDataGenerator assumed that avoiding crypto/rand was sufficient. The benchmarks showed ~700 MB/s and the assistant accepted this as "good performance." The user's challenge revealed that this assumption was wrong — the real bottleneck was MD5, not the random number generation.

Assumption 2: Computing MD5 in the data generator was useful. The original Generate method returned both (data []byte, checksum string) because the loadtest needed checksums for verification. But the benchmark was measuring the generator in isolation, and the MD5 cost was invisible until profiled. The assistant's mistake was bundling an expensive operation into what should have been a lightweight data generation primitive.

Assumption 3: Allocation overhead was negligible. The 15% spent on memclrNoHeapPointers was a significant fraction of CPU time. The assistant's initial optimization of using pre-generated shards addressed the randomness bottleneck but didn't address allocation. The FillBuffer method corrected this by accepting a pre-allocated buffer, allowing the caller to control allocation frequency.

Mistake: The benchmark was measuring the wrong thing. The original benchmark BenchmarkShardedDataGenerator_Generate was testing the Generate method which included MD5. The assistant didn't realize that the benchmark was conflating data generation with checksum computation. The user's expectation of 2–3 GB/s per core was for data generation alone, not data generation plus hashing. Once the MD5 was separated, the DataOnly numbers (3–6.5 GB/s) exceeded the user's target.

Input Knowledge Required

To fully understand this message, the reader needs familiarity with several domains:

Go benchmarking conventions: The testing.B type, b.SetBytes(), b.ResetTimer(), and the -benchmem flag. The benchmark results format (ns/op, MB/s, B/op, allocs/op) is standard Go testing output.
CPU profiling with pprof: The assistant used go test -cpuprofile=cpu.prof followed by go tool pprof -top to identify the MD5 bottleneck. Understanding that crypto/md5.block consuming 50% of samples means half the CPU time was spent in that function.
Memory allocation costs in Go: The runtime.memclrNoHeapPointers function zeros newly allocated memory. In Go, make([]byte, n) always zeroes the buffer, which is O(n) and can be a bottleneck at high throughput.
The Threadripper PRO 7995WX architecture: This is a 96-core workstation CPU with 8-channel DDR5 memory. Achieving 50–85 GB/s for buffer filling is consistent with the memory bandwidth of this platform (~100+ GB/s theoretical).
MD5 throughput characteristics: Go's MD5 implementation processes ~1.4–1.6 GB/s per core on modern AMD CPUs. This is a well-known performance characteristic of cryptographic hash functions.

Output Knowledge Created

This message creates actionable knowledge for the next phase of development:

The loadtest worker should use FillBuffer instead of Generate. The subject message explicitly states "Now let me update the loadtest worker to use pre-allocated buffers where possible." This means the worker should allocate reusable buffers once per goroutine and call FillBuffer for each payload, avoiding allocation and MD5 overhead during the test loop.
MD5 checksums should be computed separately when needed. The Generate method that returns both data and checksum is still useful for setup/verification phases, but the main test loop should use the allocation-free path.
The benchmark suite should include all three variants. The three-tier breakdown is itself a valuable diagnostic tool. Future optimizations can be evaluated by comparing against these baselines.
The target of 2–3 GB/s per core is achievable. The DataOnly numbers (3–6.5 GB/s) confirm that the data generation meets the user's expectation. The FillBuffer numbers show that the hardware is capable of much more, suggesting that the real bottleneck in end-to-end tests will be elsewhere (network I/O, S3 proxy processing, or the Kuri storage backend).

The Thinking Process Visible in the Message

The subject message is concise, but it reveals a clear analytical process. The assistant doesn't just report numbers — it interprets them. The phrase "Now we see the real performance" acknowledges that the earlier benchmarks were misleading. The three-tier categorization is a deliberate diagnostic structure: each tier isolates a different cost (MD5, allocation, pure copy).

The parenthetical "(MD5 is the bottleneck)" in the first bullet shows that the assistant has internalized the profiling result and is now using the benchmark numbers to confirm the diagnosis. The statement that "even with allocation it's 3-6 GB/s which meets the target of 2-3 GB/s per core" shows a goal-oriented evaluation: the assistant is measuring against the user's stated requirement.

The immediate action — "Now let me update the loadtest worker to use pre-allocated buffers where possible" — demonstrates that the assistant treats benchmarks as actionable intelligence, not just academic measurements. The read of the test file that follows is the first step in implementing the optimization.

The message also shows a subtle awareness of benchmark methodology. The assistant had previously fixed a duplicate loop in the benchmark code (message 1004), ensuring that the benchmark was measuring the right thing. The "FillBuffer" benchmark name suggests a deliberate design choice: the method fills a pre-allocated buffer rather than allocating a new one, and the benchmark name reflects this contract.

Conclusion

Message 1006 is a turning point in the loadtest optimization effort. It transforms a disappointing benchmark result into a clear, actionable performance model. The three-tier breakdown — WithMD5, DataOnly, FillBuffer — serves as both a diagnostic tool and a roadmap for optimization. The assistant correctly identifies that the real work is now to update the loadtest worker to use the allocation-free path, and the subsequent messages in the conversation show this implementation proceeding.

The broader lesson is that performance optimization requires relentless measurement and isolation of bottlenecks. The user's challenge to "profile why it's so extremely slow" was the catalyst that turned a superficial optimization (shard-based generation) into a deep one (eliminating MD5 and allocation overhead). The 100x gap between the worst case (700 MB/s with MD5) and the best case (85 GB/s with pre-allocated buffers) is a dramatic illustration of how much performance can be hidden by the wrong measurement.