The MD5 That Wasn't Supposed to Be There: A Case Study in Performance Profiling

Introduction

In the middle of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, a developer encountered a puzzling performance discrepancy. The load test data generator they had just optimized was running at approximately 707 MB/s — respectable by many standards, but far below the 2–3 GB/s per core they expected. Message 1000 in this coding session captures the exact moment when profiling data transformed confusion into clarity, revealing that the bottleneck was not where anyone had assumed it would be. This message is a masterclass in the discipline of measuring before optimizing, and it demonstrates how a single profiling run can completely reframe a performance problem.

The Subject Message

Here is the message in full, exactly as it appeared in the conversation:

[assistant] The bottleneck is clear: 50% of time is spent in crypto/md5.block - the MD5 checksum calculation is the main bottleneck, not the data generation itself!

>

Also memclrNoHeapPointers (15%) is from allocating the large byte slices.

>

Let me optimize: 1. Don't compute MD5 in the Generate function - compute it lazily or separately 2. Reuse buffers instead of allocating new ones each time [read] /home/theuser/gw/integrations/ritool/loadtest.go

The Context: What Led to This Message

To understand why this message was written, we need to trace the chain of events that preceded it. The developer had been working on a horizontally scalable S3 architecture for the Filecoin Gateway, a distributed storage system where multiple Kuri storage nodes operate independently while sharing object routing metadata via YugabyteDB. The architecture follows a three-layer design: stateless S3 frontend proxies on port 8078 route requests to Kuri storage nodes, which in turn store data with shared metadata in YugabyteDB.

One of the tools needed for this system was a load testing utility — something that could generate realistic S3 traffic to verify correctness, measure throughput, and identify bottlenecks before the system reached production. The developer had built this load test utility in integrations/ritool/loadtest.go, and it included a ShardedDataGenerator that pre-generated random data shards and assembled test payloads by shuffling them, avoiding the crypto/rand bottleneck that would otherwise occur under high concurrency.

The user, however, was not satisfied. In message 997, they wrote: "The performance is actually extremely slow for just the benchmark, profile why it's so extremely slow. Expect 2-3GB/s per core." This was a clear directive: the benchmark was underperforming expectations by a factor of 3–4x, and the developer needed to find out why.

The Profiling Run: Turning Assumptions into Data

The developer's response in message 998 was methodical. Rather than guessing at the cause, they ran a targeted benchmark with CPU and memory profiling enabled:

go test ./integrations/ritool/... -bench=BenchmarkShardedDataGenerator_Generate/Size_1024KB -benchmem -cpuprofile=cpu.prof -memprofile=mem.prof -run=^$

This is a textbook approach to performance investigation. The developer selected a single benchmark variant (1024KB size) to keep the profiling focused, and they captured both CPU and memory profiles simultaneously. The benchmark ran on an AMD Ryzen Threadripper PRO 7995WX with 96 cores — a machine with enormous computational capacity, making the 707 MB/s result even more puzzling.

The raw benchmark output confirmed the problem: 1481933 ns/op for 1MB of data, yielding 707.57 MB/s. The allocation profile showed 1052325 B/op with 4 allocs/op — meaning each call was allocating roughly 1MB of memory. This was the first clue.

The Profiling Revelation

Message 999 shows the developer running go tool pprof -top cpu.prof to examine the CPU profile. The results were stark:

• crypto/md5.block: 49.75% of CPU time (0.99s flat)
• runtime.memclrNoHeapPointers: 15.58% of CPU time (0.31s flat) Half the CPU time was spent computing MD5 checksums. Another 15% was spent zeroing out memory for newly allocated byte slices. Together, these two functions accounted for nearly two-thirds of all CPU time in the benchmark. This was the critical insight that message 1000 communicates: the bottleneck was not in the data generation logic at all. The ShardedDataGenerator was doing its job efficiently — shuffling pre-generated shards into output buffers. The problem was that the Generate method was also computing an MD5 checksum of the generated data, and that checksum computation was consuming 50% of the CPU cycles. The developer had inadvertently coupled two concerns — data generation and checksum verification — into a single method, and the checksum was dominating the performance profile.

The Thinking Process Visible in the Message

Message 1000 reveals several layers of reasoning. First, the developer immediately interprets the profiling data and draws the correct conclusion: "the MD5 checksum calculation is the main bottleneck, not the data generation itself!" The exclamation mark conveys a moment of discovery — this was genuinely surprising, because the developer had assumed the bottleneck would be in the shard shuffling or memory allocation.

Second, the developer identifies a secondary bottleneck: memclrNoHeapPointers at 15%. This is the Go runtime's memory clearing function, which zeroes out freshly allocated memory. Every time Generate creates a new byte slice of the requested size, the runtime must clear that memory to prevent information leakage between allocations. For a 1MB buffer, this clearing operation is non-trivial.

Third, the developer formulates a two-pronged optimization strategy:

1. Decouple checksum from generation: Don't compute MD5 in the Generate function. Instead, compute it lazily or provide a separate method for checksum calculation. This eliminates the 50% CPU overhead when checksums aren't needed.
2. Reuse buffers: Instead of allocating new byte slices on every call, provide a method that fills a pre-allocated buffer. This eliminates both the allocation overhead and the memclrNoHeapPointers cost. These two optimizations directly target the two largest items in the profile. The developer is thinking in terms of eliminating the dominant costs rather than micro-optimizing the existing code.

Assumptions and Their Consequences

The message reveals several assumptions that were operating beneath the surface:

Assumption 1: The bottleneck would be in data generation. The developer had spent considerable effort optimizing the ShardedDataGenerator to avoid the crypto/rand bottleneck, implementing a shard-based approach with pre-generated random data. The implicit assumption was that data generation would be the performance-critical path. The profiling proved this assumption wrong — the data generation was fast enough; it was the MD5 checksum bolted on top that was the problem.

Assumption 2: MD5 checksums are "free" or at least cheap enough to include by default. The Generate method returned both data and checksum, which was convenient for the read-after-write verification workflow. But this convenience came at a 50% performance tax. The developer had not considered that the checksum computation would dominate the runtime.

Assumption 3: Allocation overhead was acceptable. Each call to Generate allocated a fresh byte slice of the requested size. While Go's garbage collector handles this efficiently, the memclrNoHeapPointers cost was significant at 15% of CPU time. The developer had not considered the impact of zero-initialization on large allocations.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message 1000, a reader needs several pieces of background knowledge:

Go profiling tools: Understanding that go test -cpuprofile and go tool pprof are standard Go profiling tools, and that the output shows both flat (time spent directly in the function) and cumulative (time spent in the function and its callees) percentages.

The Go memory model: Understanding that memclrNoHeapPointers is the runtime function that zeroes newly allocated memory, and that this is mandatory in Go for safety reasons (preventing information leakage between heap allocations).

MD5 performance characteristics: Knowing that MD5 is a cryptographic hash function that operates on blocks of data, and that crypto/md5.block is the inner loop that processes those blocks. On modern hardware, MD5 throughput is typically in the range of 500–1000 MB/s per core, which aligns perfectly with the observed 707 MB/s bottleneck.

The architecture of the S3 system: Understanding that the load test generator is part of a larger horizontally scalable S3 storage system, and that the 2–3 GB/s per core target is driven by the need to saturate high-performance storage nodes during testing.

Output Knowledge Created by This Message

Message 1000 creates several pieces of actionable knowledge:

The MD5 bottleneck is quantified: 50% of CPU time in the benchmark is spent in MD5 computation. This is a precise, measured result, not a guess.

The allocation overhead is quantified: 15% of CPU time is spent in memory clearing. This is a secondary but significant cost.

A clear optimization plan is articulated: Two specific changes to the code — decouple checksum from generation, and add a buffer-filling method — are identified as the path to achieving the 2–3 GB/s target.

The profiling methodology is validated: The sequence of "benchmark → profile → interpret → plan" is demonstrated as an effective approach to performance debugging.

The Broader Significance

What makes message 1000 noteworthy is not just the specific optimization it describes, but the mindset it embodies. The developer did not respond to the user's complaint of "extremely slow" performance by guessing or tweaking parameters. They ran a controlled benchmark, captured a CPU profile, and let the data guide the investigation. This is the scientific method applied to software engineering.

The message also illustrates a common pitfall in performance-sensitive code: the tendency to bundle convenience features (like checksum computation) into core data paths without measuring their cost. The MD5 checksum was useful for verification, but including it in the Generate method by default meant that every benchmark run — and every actual load test — paid the 50% tax whether they needed verification or not.

Finally, the message demonstrates the value of having the right tools and knowing how to use them. The developer had pprof available, knew how to run it, and could interpret its output immediately. This is not trivial — interpreting profiling data requires understanding what each function does and whether its CPU share is appropriate for its role in the program.

Conclusion

Message 1000 captures a turning point in a performance optimization effort. Before this message, the developer had a benchmark running at 707 MB/s and a vague sense that something was wrong. After this message, they had a precise understanding of where the CPU cycles were going and a concrete plan to eliminate the two dominant costs. The MD5 checksum, which had been silently consuming half the CPU time, was identified and targeted for removal from the hot path. The allocation overhead, visible in the memclrNoHeapPointers profile, was targeted for elimination through buffer reuse. This is performance optimization at its best: measure, identify, plan, execute — in that order, never skipping the measurement step.