When 700 MB/s Is "Extremely Slow": A Lesson in Performance Expectations

The Message

User (msg 997): "The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core"

This single sentence, delivered after the assistant had proudly committed what it believed to be a fully optimized data generator, is a masterclass in the difference between "good enough" optimization and genuine performance engineering. It is not a complaint. It is a course correction — a demand to stop accepting surface-level results and dig into the real bottlenecks.

Context: What Led to This Moment

To understand why this message was written, we must examine what preceded it. The assistant had been working on a horizontally scalable S3-compatible storage system for the Filecoin Gateway. One component was a load testing utility (integrations/ritool/loadtest.go) that needed to generate random test data at high throughput. The assistant had already identified one bottleneck — crypto/rand — and implemented a ShardedDataGenerator that pre-generated random shards and assembled payloads by shuffling them. This was a legitimate optimization.

The assistant then committed this change with commit message 9cf8e07, reporting benchmark results of approximately 700–850 MB/s for data generation. From the assistant's perspective, this was a success. The commit message reads like a victory lap: "Performance optimizations," "Benchmark results show ~700-850 MB/s data generation throughput." The todo list was marked complete. The assistant was ready to move on.

Then the user read those numbers and responded: this is "extremely slow."

Why This Message Matters: The Performance Expectation Gap

The user's expectation of 2–3 GB/s per core reveals a fundamentally different mindset from the assistant's. The assistant saw 700 MB/s and thought "that's fast." The user saw 700 MB/s and immediately recognized that on modern hardware — specifically a 96-core AMD Ryzen Threadripper PRO 7995WX — this meant the code was leaving 90% of the available performance on the table. The assistant had optimized the wrong thing (avoiding crypto/rand contention) while missing the actual bottleneck entirely.

The user's framing is precise. They say "for just the benchmark" — meaning this is a pure data generation workload with no network I/O, no disk I/O, no S3 protocol overhead. In isolation, data generation should be memory-bandwidth-bound, not CPU-bound. 700 MB/s on a Threadripper is a red flag that something fundamental is wrong.

The Implicit Decisions in This Message

Although the message is only a single sentence, it encodes several decisions:

1. The decision to reject the current results as unacceptable. The assistant had already committed the code. The user could have accepted the numbers and moved on to the next task. Instead, they demanded investigation.
2. The decision to profile rather than guess. The user doesn't say "try X optimization" or "maybe Y is slow." They say "profile why." This is a methodological decision: don't optimize blind, measure first.
3. The decision to set a clear performance target. "Expect 2-3GB/s per core" gives the assistant a concrete goal. Without this, the assistant might have made incremental improvements and stopped early.

Assumptions Made

The user makes several assumptions in this message:

• That the benchmark results are real and reproducible. They don't question whether the benchmark is measuring the right thing — they accept the numbers and declare them unacceptable.
• That there is a bottleneck to find. The user assumes the code is not inherently limited by the hardware, but rather by a suboptimal implementation choice.
• That the assistant has the tools and knowledge to profile. The user assumes go tool pprof or similar profiling tools are available and that the assistant knows how to use them.
• That the hardware is capable of 2–3 GB/s per core. On a Threadripper PRO 7995WX with DDR5 memory, this is a reasonable assumption for memory-bound operations.

What Was Wrong: The Mistakes in the Prior Work

The assistant's prior work had two significant mistakes:

1. The MD5 bottleneck was invisible because it was embedded in the Generate method. The ShardedDataGenerator.Generate() function computed an MD5 checksum of every generated payload. MD5 is a cryptographic hash — it's computationally expensive. The assistant had combined data generation with checksumming into a single method, making it impossible to tell which part was slow.
2. Allocation overhead was ignored. The benchmark showed 4 allocations per operation and 1,052,325 B/op for 1MB payloads. The memclrNoHeapPointers function (zeroing newly allocated memory) consumed 15% of CPU time. The assistant had not considered pre-allocating buffers. The user's demand to profile forced these issues into the open. When the assistant ran go tool pprof, the results were unambiguous: 50% of CPU time in crypto/md5.block and 15% in runtime.memclrNoHeapPointers. The "optimized" data generator was spending most of its time doing something unrelated to data generation.

Input Knowledge Required

To fully understand this message, a reader needs:

• Knowledge of benchmark methodology. The user understands that benchmark results should be decomposed — a single throughput number hides the contributions of different sub-operations.
• Knowledge of modern CPU performance. The expectation of 2–3 GB/s per core for memory-bound operations comes from understanding that modern CPUs can saturate memory bandwidth at those rates.
• Knowledge of Go profiling tools. The user's instruction to "profile" implies familiarity with go tool pprof, CPU profiling, and flame graphs.
• Knowledge of the system architecture. The user knows this is a pure data generation benchmark, not an end-to-end test involving network or disk I/O.
• Knowledge of cryptographic overhead. The user likely suspected that MD5 or similar operations were the bottleneck, which is why they demanded profiling rather than suggesting specific code changes.

Output Knowledge Created

This message triggered a cascade of discoveries:

1. The three-tier performance model. The assistant separated data generation into three methods: Generate (with MD5, ~700–800 MB/s), GenerateData (data only with allocation, ~3–6.5 GB/s), and FillBuffer (pre-allocated buffer, ~50–85 GB/s). This taxonomy itself is valuable knowledge — it tells you exactly where each bottleneck lives.
2. The MD5 bottleneck quantification. Cryptographic hash functions are expensive, but knowing that MD5 consumes 50% of CPU time in a "data generation" benchmark is a specific, actionable finding.
3. The allocation overhead quantification. Memory zeroing consuming 15% of CPU time is significant. Pre-allocating buffers eliminates this entirely.
4. The corrected performance ceiling. The FillBuffer method achieving 50–85 GB/s confirms that the hardware is capable of far more than the original 700 MB/s — the bottleneck was entirely in the software design.

The Thinking Process Visible in This Message

The user's thinking process, though only visible through this single sentence, can be reconstructed:

First, the user read the assistant's summary message (msg 996) which reported "~700-850 MB/s data generation" as a success. The user's mental model immediately flagged this as wrong. They knew the Threadripper's memory bandwidth and per-core performance characteristics. They knew that pure data generation — copying bytes from pre-generated shards into an output buffer — should be memory-bandwidth-bound, not CPU-bound.

The user then considered possible explanations: maybe the benchmark includes checksumming, maybe there's allocation overhead, maybe the shard-shuffling logic is inefficient. But rather than guessing, they chose the most productive path: profile it. The instruction is concise because the methodology is standard: measure, identify the hotspot, fix it, repeat.

The user also implicitly communicated a standard: don't stop optimizing until you're within striking distance of the hardware's theoretical peak. 700 MB/s on a 96-core machine is approximately 0.2% of the system's total memory bandwidth. That's not "optimized" — that's barely started.

The Broader Lesson

This message is a case study in the importance of performance expectations. Without a target, any number looks good. The assistant's 700 MB/s seemed impressive until compared against the 2–3 GB/s per core that the hardware could deliver. The user's intervention wasn't about being harsh — it was about providing the compass that the optimization work needed.

The result speaks for itself. After profiling and refactoring, the FillBuffer method achieved 50–85 GB/s, and GenerateData with allocation hit 3–6.5 GB/s — meeting and exceeding the user's target. The difference between "done" and "actually done" was one profiling session, prompted by one sentence that refused to accept mediocrity.