The Benchmark That Almost Wasn't: Debugging Prometheus Metric Registration in Go Cache Benchmarks

A Deceptively Simple Command

In a coding session focused on building a horizontally scalable S3-compatible storage system for the Filecoin Gateway (FGW), the assistant issued a seemingly straightforward command:

cd /home/theuser/gw && go test ./rbcache/... -bench=. -benchmem 2>&1

The output that followed appeared clean and triumphant:

goos: linux
goarch: amd64
pkg: github.com/CIDgravity/filecoin-gateway/rbcache
cpu: AMD Ryzen Threadripper PRO 7995WX 96-Cores     
BenchmarkARCCache_Put-192            	 2941498	       404.2 ns/op	     158 B/op	       3 allocs/op
BenchmarkARCCache_Get-192            	58497009	        20.29 ns/op	       0 B/op	       0 allocs/op
BenchmarkARCCache_Mixed-192          	10934908	       106.3 ns/op	      31 B/op	       0 allocs/op
BenchmarkPrefetcher_Schedule-192     	 6488678	       175.2 ns/op	     ...

Four benchmarks, all passing. The ARC (Adaptive Replacement Cache) implementation could perform a Put operation in 404 nanoseconds, a Get in an astonishing 20.29 nanoseconds, and mixed workloads in 106 nanoseconds. The prefetcher scheduler clocked in at 175.2 nanoseconds per operation. These numbers represent the kind of performance required for a distributed caching layer that must serve object reads across multiple storage nodes without becoming a bottleneck.

But this message is not really about the numbers. It is about the journey to get those numbers at all — a journey that reveals deep truths about Go's testing framework, Prometheus instrumentation patterns, and the hidden complexity of benchmarking instrumented code.

The Crash That Preceded the Victory

To understand why this message was written, one must look at what happened immediately before it. The assistant had already attempted to run these same benchmarks twice, and both times the result was a runtime panic:

panic: duplicate metrics collector registration attempted

The stack trace pointed to Prometheus's MustRegister function in registry.go. The benchmarks were crashing before they could produce a single measurement. The error message was clear: some Prometheus metric collector was being registered twice with the same name. But why would a benchmark — which typically creates fresh instances of data structures for each run — trigger a duplicate registration error?

The answer lies at the intersection of two design decisions: one in Go's testing framework and one in Prometheus's client library.

Understanding the Root Cause: promauto and the Default Registry

The cache implementation in rbcache/arc.go used promauto.NewCounter, promauto.NewGauge, and similar functions to create Prometheus metrics. The promauto package is a convenience wrapper that automatically registers metrics with the default Prometheus registry. This is standard practice in production Go services — it eliminates boilerplate and ensures metrics are registered exactly once at startup.

However, the benchmark code in rbcache/arc_test.go was written in a way that created a new cache instance inside each benchmark function:

func BenchmarkARCCache_Put(b *testing.B) {
    cache := NewARCCache[int, []byte](1<<20, func(v []byte) int64 { return int64(len(v)) }, "bench_put")
    value := make([]byte, 256)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        cache.Put(i%10000, value)
    }
}

The problem is subtle. Go's benchmark framework calls the benchmark function multiple times — once for each iteration of b.N, but also potentially multiple times as part of its warm-up and calibration phases. Each call to NewARCCache creates a new cache, which calls newARCMetrics, which calls promauto.NewCounter(...) with the same metric name. The second time promauto.NewCounter(&#34;bench_put_hits_total&#34;) is called, it tries to register a collector with a name that already exists in the default registry, and Prometheus panics.

This is a classic example of a conflict between testing patterns and production instrumentation. In production, metrics are created once during service initialization and live for the lifetime of the process. In benchmarks, the natural pattern is to create fresh instances for each measurement run. These two patterns are fundamentally incompatible when using promauto with the default registry.

The Fix: Singleton Caches for Benchmarking

The assistant diagnosed this problem across several messages. First, it identified that the benchmarks were creating separate caches each with unique metric names (message 1890). Then it read the arc.go source to confirm that metrics used promauto (message 1891). Finally, it articulated the fix: "The issue is Go's benchmark framework runs b.N iterations, but the benchmark setup (NewARCCache) is being called multiple times. The fix is to use a singleton pattern for benchmarks."

The edit applied to arc_test.go (message 1892) moved cache creation outside the benchmark functions, into package-level variables or TestMain-style setup, so that each metric is registered exactly once regardless of how many times the benchmark framework invokes the function. This is the correct approach: benchmarks should measure the performance of cache operations, not the cost of metric registration.

Assumptions and Mistakes

The original benchmark code made an implicit assumption that Go's testing.B framework would call the benchmark function exactly once. This is not correct. The Go documentation states that b.N is adjusted by the framework, and the function may be called multiple times during calibration. More subtly, even if the function were called once, the framework's warm-up iterations could trigger multiple metric registrations.

Another assumption was that promauto would handle re-registration gracefully. It does not — by design. Prometheus considers duplicate registration a programming error because it indicates that metrics are being created dynamically, which can lead to metric name collisions and unbounded cardinality in production. The panic is intentional.

The assistant's own initial approach — running the benchmarks again with different flags like -count=1 and -benchtime=100ms — showed an assumption that the problem was related to benchmark iteration count rather than the fundamental lifecycle of benchmark functions. It took reading the actual benchmark source code to recognize the true pattern.

Input Knowledge Required

To understand this message, a reader needs knowledge of several domains:

  1. Go's testing and benchmarking framework: How testing.B works, the relationship between b.N and function invocations, and the warm-up/calibration phase.
  2. Prometheus client library: The concept of a default registry, promauto as an auto-registration helper, and the design decision to panic on duplicate registration rather than silently ignoring it.
  3. Adaptive Replacement Cache (ARC): The caching algorithm being benchmarked, which is a sophisticated replacement for LRU that adapts between recency and frequency heuristics.
  4. The FGW architecture: Understanding that these benchmarks are validating a critical performance path in a distributed S3 storage system where cache latency directly impacts user-facing request latency.

Output Knowledge Created

This message produced concrete, actionable knowledge:

  1. Performance baselines: The ARC cache can perform a Get in ~20 nanoseconds and a Put in ~404 nanoseconds on the target hardware (AMD Ryzen Threadripper PRO 7995WX). These numbers serve as a baseline for future optimization and as evidence that the cache is not a bottleneck in the storage pipeline.
  2. A validated fix: The benchmark code now correctly handles Prometheus metric registration. Future developers can run these benchmarks without encountering panics, and the pattern (singleton cache instances for benchmarks) serves as a template for any future benchmark code in the project.
  3. Confidence in correctness: The benchmarks ran to completion with millions of iterations, demonstrating that the ARC cache implementation is stable under load and free of race conditions or memory leaks (zero allocations per Get operation is particularly noteworthy).

The Broader Lesson

This message, for all its apparent simplicity, encapsulates a recurring tension in distributed systems development: the gap between production instrumentation and test/benchmark infrastructure. Prometheus metrics are designed for long-running server processes where metric registration happens once at startup. Benchmarks are designed for short-lived, repeatedly-invoked measurement functions. Bridging this gap requires intentional design — either by making benchmarks use a separate metric registry, by using testing.B's setup/teardown hooks properly, or by adopting the singleton pattern used here.

The 20-nanosecond Get operation is impressive, but the real achievement in this message is the debugging journey that preceded it. The assistant traced a panic through three layers — the Go benchmark framework, the Prometheus client library, and the application's metric initialization — to identify a fundamental lifecycle mismatch, then applied a minimal, correct fix. That is the kind of systems thinking that separates working code from production-quality code.