From "Good Enough" to 85 GB/s: A Performance Optimization Journey Through Profiling, Bottleneck Identification, and Pre-Allocated Buffers

Introduction

In the course of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, a routine optimization task turned into a textbook case of performance debugging. The system under development featured a three-layer architecture: stateless S3 frontend proxies routing requests to independent Kuri storage nodes, with object metadata shared via YugabyteDB. After establishing a working test cluster, debugging the monitoring dashboard, and fixing critical bugs, the team turned its attention to performance — specifically, to the load testing utility that would validate the system's throughput and correctness [1].

What began as a straightforward implementation of a shard-based data generator — intended to avoid the crypto/rand bottleneck — ended with a 120x performance improvement, from 700 MB/s to 85 GB/s. But this dramatic number is not the result of clever algorithm design or low-level assembly optimizations. It is the result of something far more fundamental: the willingness to challenge assumptions, the discipline to profile before optimizing, and the courage to admit that "good enough" is rarely good enough when the hardware is capable of so much more.

This article traces the full arc of that optimization journey, from the initial specification through the profiling revelation, the refactoring that separated concerns, the three-tier benchmark hierarchy that emerged, and the silent acknowledgment of success that closed the loop. Along the way, it examines the specific technical decisions — pre-allocated buffers, lazy checksum computation, faster random selection — and the broader methodology that turned a mediocre data generator into one that operates at the limits of memory bandwidth.

The Optimization Directive: Three Specific Targets

The optimization phase began with a clear directive from the user at message 950, which laid out three specific tasks:

Integrate go-buffer-pool into all S3 HTTP paths to eliminate allocation overhead from io.Copy and io.ReadAll calls, using at least 256KB buffers for copy operations.
Implement shard-based random data generation in the loadtest to avoid the crypto/rand bottleneck, by pre-generating smaller random shards and assembling test payloads through shuffling.
Add unit-test benchmarks with a mock HTTP server to validate performance in isolation, without requiring the full test cluster [2]. This was not a vague request for "make it faster." It was a precise, actionable specification targeting specific components and specific optimization techniques. The user's message revealed deep architectural thinking about where bottlenecks would emerge in a high-throughput distributed system. The buffer-pool directive targeted Go's standard io.Copy and io.ReadAll calls, which allocate fresh 32KB buffers on every invocation. In a high-throughput S3 proxy handling concurrent uploads and downloads, these allocations accumulate rapidly, pressuring the garbage collector and causing latency spikes. The go-buffer-pool library, already a dependency in the project's carlog and rbdeal packages, provided a pool of reusable byte buffers that could eliminate this allocation overhead entirely. The shard-based data generation directive was equally insightful. The naive approach to generating test data — calling crypto/rand for every byte of every payload — would become CPU-bound on the random number generator long before the S3 proxy reached its throughput limits. The user's proposed solution was elegant: pre-generate a pool of random shards during initialization, then assemble test payloads by shuffling and concatenating these shards using the much faster math/rand source. This moved the expensive cryptographic randomness to initialization time, making per-request data generation essentially a fast shuffle-and-copy operation. The third directive — adding benchmarks with a mock HTTP server — closed the feedback loop. Without isolated benchmarks, the team could not measure whether their optimizations were actually working. A mock server that simply read and discarded request bodies would allow the benchmarks to run without the full test cluster, making them fast, reproducible, and suitable for regression testing [2].

Initial Implementation: The Shard-Based Generator

The assistant began by surveying the codebase to understand existing patterns. A grep for io.Copy, io.ReadAll, and ioutil.ReadAll across the S3-related code revealed that the main targets were in server/s3frontend/server.go, specifically in the proxyRequest function that forwarded HTTP requests to backend Kuri nodes [3]. The function read entire request bodies into memory with io.ReadAll, then forwarded them — a pattern that would benefit significantly from buffer pooling.

The assistant also discovered that go-buffer-pool was already used elsewhere in the project, making it a natural fit. The implementation involved replacing io.ReadAll with pool.Get(estimatedSize) to obtain a pre-allocated buffer, and using io.CopyBuffer with a 256KB pooled buffer for copying response bodies back to clients [3]. This was not a complex change — 26 insertions and 5 deletions across one file — but it required understanding the existing buffer-pool infrastructure and the specific I/O patterns in the proxy.

For the loadtest optimization, the assistant read the existing loadtest.go and identified the core problem: every test payload was generated by calling crypto/rand.Read, which is designed for cryptographic security and runs at approximately 700–800 MB/s on the test machine's AMD Ryzen Threadripper PRO 7995WX processor [3]. While this sounds fast, it was the bottleneck compared to what the rest of the system could sustain.

The solution was a ShardedDataGenerator structure that pre-generated a pool of random data shards using crypto/rand once during initialization, then assembled test payloads by selecting and concatenating shards using the much faster math/rand source. Each worker received its own generator instance, eliminating lock contention. The implementation hit several snags — unused imports, a misplaced import statement in the test file, and a worker function signature mismatch — but each was caught by LSP diagnostics and fixed iteratively [3].

With the optimizations implemented and committed, the assistant ran the benchmarks and reported data generation throughput of approximately 700–850 MB/s. The commit message read like a victory lap: "Performance optimizations," "Benchmark results show ~700-850 MB/s data generation throughput." The todo list was marked complete. The assistant was ready to move on [4].## The Performance Crisis: When "Good Enough" Isn't

Then the user read those numbers and responded with a single sentence that changed everything: "The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core" [5].

This was a moment of reckoning. The assistant had accepted 700 MB/s as "good" because it had no reference point for what was achievable. The user, drawing on deep domain knowledge about what a 96-core AMD Ryzen Threadripper PRO 7995WX could deliver, recognized that this was barely a fraction of the hardware's potential. The instruction was precise: don't guess, don't tweak — profile.

The assistant responded by running go test with CPU and memory profiling enabled, capturing a cpu.prof file, and then using go tool pprof -top to analyze it [6]. The output was unambiguous:

49.75% of CPU time was spent in crypto/md5.block — the MD5 checksum computation
15.58% was spent in runtime.memclrNoHeapPointers — zeroing newly allocated memory The bottleneck was not in the random number generation at all. It was in the MD5 checksum that the Generate method computed on every payload, presumably for read-after-write verification. The assistant had inadvertently coupled two independent operations — data production and integrity verification — into a single function, and the benchmark was measuring the combined cost rather than isolating the data generation performance [6][7]. This is a classic profiling pitfall. When you measure a composite operation and find it slow, the instinct is to optimize the whole thing. But profiling reveals which part is slow, and often the answer is that a seemingly minor sub-operation is consuming the majority of resources. The MD5 hash, while fast in absolute terms, operates on every byte of data and is fundamentally CPU-bound. When your goal is to measure raw data generation throughput, including an MD5 computation is like measuring a car's top speed while driving with the parking brake engaged. The memclrNoHeapPointers finding was equally important. Every time Go allocates a new byte slice with make([]byte, size), the runtime must zero that memory to ensure safety. For large allocations — the benchmark was generating payloads up to 10 MB — this zeroing operation consumes significant CPU time. At 15.58% of total CPU time, it was the second-largest bottleneck, and it was entirely avoidable through buffer reuse [6].

The Refactoring: Separating Concerns

Armed with this knowledge, the assistant made a strategic decision: separate data generation from checksum computation [7]. The refactoring involved three key changes:

Adding a GenerateData method that produces random data without computing a checksum, using a pre-allocated buffer to avoid allocation overhead
Adding a FillBuffer method that writes directly into an existing buffer, eliminating both allocation and the MD5 computation entirely
Keeping the original Generate method for cases where the checksum is actually needed, but ensuring the benchmarks could measure each path independently The assistant also replaced mrand.Intn() with mrand.Uint32() for faster random index selection in the shard shuffling logic — a micro-optimization that targeted the shard selection itself, anticipating that after eliminating the MD5 and allocation bottlenecks, this would become a more significant fraction of remaining CPU time [7]. This separation of concerns is a textbook example of the Unix philosophy applied at the function level: each function should do one thing well. The original Generate method was doing two things (data generation + checksumming), and the benchmark couldn't distinguish between them. By splitting these responsibilities, the assistant created the ability to measure each operation's true cost. The refactoring of the benchmark test file did not go smoothly. When the assistant edited loadtest_test.go to add new benchmark variants for the FillBuffer and GenerateData paths, the edit introduced a syntax error: a duplicate for loop was left behind, causing the Go parser to encounter a loop keyword where it expected a top-level declaration [8][9]. The LSP diagnostics caught this immediately: ERROR [216:2] expected declaration, found 'for'. The assistant read the file to diagnose the problem, identified the duplicate loop, and applied a minimal fix — deleting the offending code [10]. This small correction was the gate that allowed the benchmarks to compile and run, enabling the verification of the entire optimization effort.## The Three-Tier Performance Model: A Diagnostic Framework With the benchmarks running correctly, the assistant discovered three distinct performance tiers that mapped directly to the optimization decisions [11]: | Tier | Throughput | Bottleneck | |------|-----------|------------| | WithMD5 | ~700–800 MB/s | MD5 checksum (50% of CPU) | | DataOnly (with allocation) | ~3–6.5 GB/s | Memory allocation + zeroing | | FillBuffer (no allocation) | ~50–85 GB/s | Pure memory bandwidth | Tier 1: WithMD5 (~700–800 MB/s). This is the original benchmark result that triggered the investigation. The MD5 hash computation is a fixed cost per byte — Go's standard library implementation processes data in 64-byte blocks, and on this AMD Ryzen processor, it achieves roughly 1.4–1.6 GB/s per core. Since the benchmark was also doing data generation (shard assembly + copying), the combined throughput settled at ~700–800 MB/s. The MD5 was the ceiling [11]. Tier 2: DataOnly with allocation (~3–6.5 GB/s). By removing the MD5 computation, the assistant isolated the cost of data generation alone. The 4–8x speedup over the MD5-included case confirms that the hash was indeed the primary bottleneck. But even at 3–6.5 GB/s, the benchmark still showed allocation overhead: runtime.memclrNoHeapPointers accounted for 15% of the original profile. Every call to make([]byte, size) forces the Go runtime to zero the memory, which is a linear-time operation proportional to the allocation size. The DataOnly tier at 3–6.5 GB/s already exceeded the user's target of 2–3 GB/s per core [11]. Tier 3: FillBuffer with no allocation (~50–85 GB/s). This is the most informative benchmark. By pre-allocating a buffer once and reusing it across benchmark iterations, the assistant eliminated both allocation and zeroing costs. The resulting throughput of 50–85 GB/s is essentially the speed of memory bandwidth: the shard copy operations (memcpy) and the random index selection. At these speeds, the bottleneck has shifted from the CPU to the memory subsystem. This number is an upper bound on what the hardware can sustain for a pure copy workload [11]. The three-tier model also provided a valuable taxonomy for understanding where each bottleneck lived. The WithMD5 tier was CPU-bound on cryptographic hashing. The DataOnly tier was allocation-bound, limited by memory zeroing and garbage collection. The FillBuffer tier was memory-bandwidth-bound, limited only by how fast the CPU could copy data from the shard pool into the output buffer. Each tier had a different bottleneck, and each required a different optimization strategy.

From Discovery to Delivery: Adding Tests for the Optimized Path

With the benchmark results in hand, the assistant's next step was to update the loadtest worker to use the optimized methods. But before changing production code, the prudent step was to add tests for the new methods [12].

The message at index 1007 captures this moment: "Let me add a test for FillBuffer and GenerateData." The assistant examined the existing test file (loadtest_test.go) and added new test functions that verified:

FillBuffer correctly fills a pre-allocated buffer with random data of the requested size, without allocating new memory.
GenerateData produces correct-sized, unique data without computing MD5, verifying the allocation-based path works correctly [12]. This step is the quiet bridge between discovery and delivery. The benchmarks had already proven the performance characteristics in isolation, but unit tests serve a different purpose: they verify correctness across edge cases, ensure the methods handle various sizes correctly, and guard against regressions when the code is modified later. Tests also serve as executable documentation, showing exactly what inputs each method expects and what outputs it produces [12]. The assistant's reasoning followed a clear arc: 1. Accept the user's challenge — profile to find the real bottleneck rather than defending the original numbers. 2. Interpret profiling data — identify MD5 at 50% and memory clearing at 15% as the targets for optimization. 3. Design the fix — separate checksum from generation, add a pre-allocated buffer path, use faster random selection. 4. Benchmark to validate — run the updated benchmarks to confirm the optimization works, producing the three-tier hierarchy. 5. Test before integrating — before updating the loadtest worker to use the new methods, add tests to ensure correctness. 6. Proceed to integration — update the worker function to use FillBuffer with pre-allocated buffers, eliminating allocation overhead in the hot loop [12].

The Silence That Speaks Volumes

The final message in this segment — message 1008 — is from the user, and its content is empty. A pair of XML-like data wrapper tags containing nothing but a blank line [13].

On its face, this appears to be nothing. Yet in the context of the conversation, this empty response marks a significant turning point. The user's earlier message (997) was a demand for explanation: "profile why it's so extremely slow." The assistant's response was the answer, backed by hard data from pprof and benchstat. When the evidence is this clear — a CPU profile showing 50% of time in md5.block — there is no room for debate. The numbers were the message [13].

The most straightforward interpretation is that the user was satisfied. The benchmarks now showed exactly the performance the user had demanded: 3–6.5 GB/s for the allocation path, well above the 2–3 GB/s target. The FillBuffer path at 50–85 GB/s was a bonus. There was nothing to critique, nothing to correct, nothing to add. The empty message is the conversational equivalent of a nod — "good, proceed." [13]

This empty response illuminates several aspects of the human-AI collaboration dynamic. The user did not need to say "good job" or "approved" because the evidence was self-authenticating. The benchmark numbers were the validation. This reflects a mature collaboration where trust is built on reproducible data rather than verbal affirmation. The user's earlier critique was specific, quantitative, and actionable — it named a target (2–3 GB/s per core) and a method (profiling). The empty response is the closing bracket on that feedback loop: the problem was stated, investigated, solved, and the solution was verified [13].

The Commit That Told the Story

With the optimizations validated, the assistant committed the changes in two logical commits [14]. The first covered the S3 frontend buffer-pool changes, explaining the "why" and "how" of the optimization. The second covered the loadtest optimization and benchmarks, including a summary of benchmark results in the commit message. Each commit told a coherent story, making future code archaeology easier.

The assistant then updated the todo list, marking all four high-priority optimization tasks as completed [15]. This was not just administrative housekeeping — it was a signal that the optimization sprint had reached its conclusion, and the project could move on to the next phase.

What This Journey Reveals: Lessons in Performance Engineering

The optimization pipeline from 700 MB/s to 85 GB/s reveals several fundamental principles of performance engineering that extend far beyond this specific project:

Measure Before You Optimize

The assistant's initial optimization — the shard-based data generator — was clever and well-implemented, but it was solving the wrong problem. The profiling step revealed that MD5, not random number generation, was the dominant bottleneck. Without profiling, the assistant might have continued optimizing the wrong thing indefinitely. The shard-based generator was not useless — it eliminated the crypto/rand contention problem — but it was not the bottleneck that was limiting throughput.

Isolate What You Measure

The original Generate method bundled data generation with checksum computation, making it impossible to tell which part was slow. By separating these concerns, the assistant enabled accurate measurement of each component's true cost. The 120x "improvement" was not really an improvement at all — it was a correction of measurement. The FillBuffer path was always that fast; it was just hidden behind the MD5 computation.

Set Clear Performance Targets

The user's expectation of 2–3 GB/s per core provided a concrete goal that drove the entire diagnostic process. Without that target, the assistant might have accepted 700 MB/s as "good enough" and moved on. The target created the motivation to dig deeper, profile, and find the real bottlenecks. A vague goal like "make it faster" would not have produced the same result.

Use the Right Tools

Go's pprof toolchain — go test -cpuprofile followed by go tool pprof -top — provided the precise diagnostic data needed to identify the MD5 bottleneck. Without it, the assistant would have been flying blind. The LSP diagnostics caught syntax errors immediately, preventing the assistant from running invalid code. The read-edit-verify loop — read the file, apply the edit, check for errors — is the same disciplined approach a human developer would use.

Challenge Assumptions About What's "Necessary"

The MD5 checksum seemed like a natural part of data generation, but it was only needed for a specific use case (S3 integrity verification). By separating the concerns, the assistant freed 50% of CPU time for actual data generation. This principle applies broadly: every feature has a cost, and features that are "always on" can silently consume resources that belong elsewhere.

Pre-Allocated Buffers Are a Powerful Optimization

The 15% overhead from memclrNoHeapPointers was eliminated entirely by reusing buffers. This pattern — allocate once, reuse many times — is applicable to any high-throughput data pipeline. The go-buffer-pool library made this trivial to implement, but the concept is language-independent.

Conclusion

The journey from 700 MB/s to 85 GB/s is a dramatic illustration of how much performance can be hidden by the wrong measurement. What began as a user's blunt critique — "profile why it's so extremely slow" — became a systematic investigation that uncovered a hidden MD5 bottleneck, exposed the hidden costs of memory allocation, and produced a validated, high-throughput data generator suitable for load testing the S3 frontend proxy.

The three tiers of performance — WithMD5 at ~700–800 MB/s, DataOnly at ~3–6.5 GB/s, and FillBuffer at ~50–85 GB/s — serve as both a diagnostic framework and a roadmap for optimization. Each tier isolates a different cost, making it possible to identify where CPU cycles are actually going and which optimization strategy will yield the greatest benefit.

For the broader S3 storage system, this optimization ensures that the load test harness will never be the bottleneck — no matter how many concurrent workers are thrown at the proxy. The data generator can now produce test payloads at 50–85 GB/s per core, limited only by memory bandwidth. The real bottlenecks in end-to-end tests will be elsewhere: network I/O, S3 proxy processing, or the Kuri storage backend. And that is exactly where they should be.

The most important lesson, however, is not about MD5 or buffer pools or shard-based generation. It is about the discipline of performance engineering: profile first, target the real bottleneck, verify with isolated benchmarks, and never accept "good enough" without understanding what "good" actually means. The user's challenge — "profile why it's so extremely slow" — was not a criticism. It was an invitation to think more deeply, measure more carefully, and build something truly worthy of the hardware it runs on.## References

[1] Segment 3 analyzer summary — "Optimized the load test data generator by analyzing benchmarks and implementing pre-allocated buffers to eliminate allocation overhead and bypass MD5 bottlenecks."

[2] User message at index 950 — "Commit changes made so far. Optimizing: In all S3 http paths ensure all io.Copy / io.ReadAll use github.com/libp2p/go-buffer-pool..."

[3] Assistant message at index 996 — Summary of commits showing buffer-pool integration and shard-based generator implementation.

[4] Assistant message at index 990 — "The benchmarks show good performance. The sharded generator is slightly slower than crypto/rand for larger sizes..."

[5] User message at index 997 — "The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core."

[6] Assistant message at index 999 — CPU profile showing 49.75% in crypto/md5.block and 15.58% in runtime.memclrNoHeapPointers.

[7] Assistant message at index 1001 — Refactoring to separate data generation from checksum computation, add FillBuffer and GenerateData methods.

[8] Assistant message at index 1002 — Edit applied with LSP error: "expected declaration, found 'for'".

[9] Assistant message at index 1003 — Reading the test file to diagnose the duplicate loop error.

[10] Assistant message at index 1004 — Fix applied to remove the duplicate loop.

[11] Assistant message at index 1005-1006 — Benchmark results showing three tiers: WithMD5 (~700-800 MB/s), DataOnly (~3-6.5 GB/s), FillBuffer (~50-85 GB/s).

[12] Assistant message at index 1007 — "Let me add a test for FillBuffer and GenerateData."

[13] User message at index 1008 — Empty response, interpreted as tacit approval.

[14] Assistant messages at index 992-993 — Commit messages for buffer-pool and loadtest optimizations.

[15] Assistant message at index 995 — Todo list updated with all optimization tasks marked completed.