The Art of the Benchmark Fix: Resolving Prometheus Metric Collisions in Go Cache Benchmarks

Introduction

In software engineering, few moments are as satisfying as watching a previously panicking test suite run cleanly to completion. Message 1897 of this coding session captures exactly such a moment: a brief, almost laconic verification that benchmark fixes have taken effect. But behind the simple output of go test ./rbcache/... -bench=. -benchmem lies a nuanced debugging journey through the intersection of Go's testing framework, Prometheus metrics registration, and cache implementation patterns. This article unpacks that single message — its context, its reasoning, and its significance.

The Message in Full

The target message reads:

Good, os is already imported. Let me run the benchmarks again:

>

`` cd /home/theuser/gw && go test ./rbcache/... -bench=. -benchmem 2>&1 goos: linux goarch: amd64 pkg: github.com/CIDgravity/filecoin-gateway/rbcache cpu: AMD Ryzen Threadripper PRO 7995WX 96-Cores BenchmarkARCCache_Put-192 2957050 395.7 ns/op 158 B/op 3 allocs/op BenchmarkARCCache_Get-192 56091223 20.70 ns/op 0 B/op 0 allocs/op BenchmarkARCCache_Mixed-192 11191621 109.2 ns/op 31 B/op 0 allocs/op BenchmarkPrefetcher_Schedule-192 6503404 173.4 ns/op ... ``

At first glance, this is a routine verification command. But to understand why this message was written — and why it matters — we must trace the debugging thread that preceded it.

The Context: A Session of Validation and Repair

The broader coding session was focused on validating and hardening the Filecoin Gateway (FGW) project after implementing three major milestones: enterprise-grade observability, multi-tier retrieval caching, and passive garbage collection. The assistant had been systematically running test suites, fixing issues, and ensuring code quality before committing changes.

Earlier in the session, the assistant had run the full test suite for the rbcache package (the retrieval cache subsystem) and encountered a clean pass for unit tests. But when attempting to run benchmarks — a standard practice for performance-sensitive cache code — everything fell apart. The benchmark command produced a panic:

BenchmarkARCCache_Put-192    panic: duplicate metrics collector registration attempted

This panic originated from Prometheus's MustRegister function, which is called by promauto (the Prometheus auto-registration package) when creating new metric collectors. The error message indicated that a metric with the same name was being registered twice — a fatal condition in Prometheus's design, which treats duplicate registration as a programming error worthy of a panic.

Diagnosing the Root Cause

The assistant's debugging process reveals a clear chain of reasoning. The first hypothesis was that running multiple benchmarks simultaneously caused the collision. The assistant attempted to isolate the problem by running a single benchmark:

go test ./rbcache/... -bench=BenchmarkARCCache_Put -benchmem -count=1 -benchtime=100ms

But this still panicked. The isolation attempt failed, which was actually a valuable clue: the problem wasn't about multiple benchmarks conflicting with each other, but about something more fundamental in how a single benchmark function was being executed.

The assistant then examined the benchmark source code in arc_test.go. The benchmark function BenchmarkARCCache_Put created a new cache instance inside the function body:

func BenchmarkARCCache_Put(b *testing.B) {
    cache := NewARCCache[int, []byte](1<<20, func(v []byte) int64 { return int64(len(v)) }, "bench_put")
    value := make([]byte, 256)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        cache.Put(i%10000, value)
    }
}

The key insight is that Go's benchmark framework does not call BenchmarkXxx once and let it loop b.N times. Instead, it may call the function multiple times — for warmup iterations, for calibration to determine the right b.N, and for the actual measured run. Each call to BenchmarkARCCache_Put creates a new ARCCache instance, which in turn calls newARCMetrics(&#34;bench_put&#34;), which uses promauto.NewCounter(...) to register metrics with the default Prometheus registry. The second call panics because the metric name fgw_cache_bench_put_hits_total (and others) are already registered.

This is a subtle interaction between Go's testing framework and Prometheus's design philosophy. Prometheus treats duplicate registration as a hard error because it indicates a programming mistake — you should never create two metrics with the same name. But in benchmarks, creating fresh instances is exactly what you want for isolation. The two systems have conflicting assumptions.## The Fix: Singleton Caches for Benchmarking

The assistant's diagnosis was precise: "The metrics use promauto which auto-registers with the default registry. Each benchmark creates a new cache and tries to re-register metrics. The fix is to make benchmarks use the same cache instance."

The solution was to refactor the benchmark functions to use a singleton pattern — creating the cache once (typically as a package-level variable or within a TestMain setup) and reusing it across benchmark iterations. This avoids the duplicate registration problem because promauto only registers each metric name once per process. By creating the cache a single time, the metrics are registered exactly once, and all benchmark iterations operate on the same cache instance.

The assistant applied this fix first to arc_test.go, then recognized that the same issue would affect the SSD cache benchmarks in ssd_test.go. The SSD benchmarks were more complex because they used temporary directories and required cleanup, but the same singleton principle applied. After applying the edit to ssd_test.go, the assistant verified that the os import was already present (needed for any file-path operations in the singleton setup), and then proceeded to run the full benchmark suite.

The Verification: Message 1897

This brings us to the target message. The assistant writes "Good, os is already imported. Let me run the benchmarks again" and executes the benchmark command. The output shows four benchmarks running successfully:

  1. BenchmarkARCCache_Put: 2,957,050 iterations, 395.7 ns/op, 158 B/op, 3 allocs/op
  2. BenchmarkARCCache_Get: 56,091,223 iterations, 20.70 ns/op, 0 B/op, 0 allocs/op
  3. BenchmarkARCCache_Mixed: 11,191,621 iterations, 109.2 ns/op, 31 B/op, 0 allocs/op
  4. BenchmarkPrefetcher_Schedule: 6,503,404 iterations, 173.4 ns/op The output is truncated (the trailing ... indicates more benchmarks were likely running, including the SSD cache benchmarks), but the critical point is that no panic occurred. The benchmarks completed successfully, producing meaningful performance numbers. These numbers are themselves revealing. The ARC cache's Get operation at 20.70 nanoseconds per operation is essentially a pointer dereference and map lookup — extremely fast. The Put operation at 395.7 ns is slower because it involves allocation (158 bytes, 3 allocs) for the cache entry. The Mixed benchmark at 109.2 ns reflects a realistic read-heavy workload. The prefetcher at 173.4 ns shows the scheduling overhead for cache prefetch operations.

Assumptions and Knowledge Required

To fully understand this message, several layers of knowledge are necessary:

Go testing conventions: The reader must understand that Go's testing.B benchmark framework calls the benchmark function multiple times, not just once with a large b.N. This is a common point of confusion. The framework performs calibration runs to determine an appropriate b.N value, and each calibration run creates a fresh invocation of the benchmark function.

Prometheus metrics registration: Prometheus's promauto package uses MustRegister, which panics on duplicate registration. This is intentional — it prevents silent errors where two components accidentally create metrics with the same name. However, it creates friction with patterns that dynamically create and destroy metric-collecting objects.

Cache architecture: The rbcache package implements an Adaptive Replacement Cache (ARC) and an SSD-backed cache tier, both instrumented with Prometheus metrics for observability. The metrics are created at cache construction time, which is standard practice but problematic in benchmark contexts.

The singleton fix pattern: The assistant assumed — correctly — that creating the cache once outside the benchmark loop would resolve the duplicate registration. This assumption relied on understanding that promauto registers metrics in a global default registry, and that the registry persists across benchmark invocations within the same process.

Mistakes and Incorrect Assumptions

The initial debugging path reveals a minor incorrect assumption: the assistant first tried to isolate the problem by running a single benchmark with reduced iterations (-count=1 -benchtime=100ms). This was a reasonable diagnostic step, but it failed because the root cause wasn't benchmark-to-benchmark interference — it was intra-benchmark re-invocation. The single-benchmark run still panicked because Go's framework still called BenchmarkARCCache_Put multiple times for calibration.

A more subtle assumption was that the fix for ARC benchmarks would apply identically to SSD benchmarks. While the principle was the same, the SSD benchmarks had additional complexity (temp directories, cleanup, file I/O) that required careful handling to avoid resource leaks or stale state. The assistant acknowledged this ("The SSD benchmarks are more complex because they use temp directories and need cleanup") before applying the same pattern.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with Go's testing and benchmarking framework; understanding of Prometheus metrics and the promauto auto-registration pattern; knowledge of the ARC cache algorithm and its implementation in the FGW project; and awareness of the broader session context (running test suites, fixing backup playbooks, validating milestones).

Output knowledge created by this message includes: verified benchmark results for the ARC cache and prefetcher components; confirmation that the singleton fix resolves the Prometheus registration panic; a validated pattern for writing Prometheus-instrumented benchmarks in Go; and performance baselines for the cache subsystem (ARC Put at ~396ns, ARC Get at ~21ns, ARC Mixed at ~109ns, Prefetcher Schedule at ~173ns).

The Deeper Significance

This message, while brief, represents a critical quality gate in the software development lifecycle. The assistant was not content to merely have passing unit tests — it insisted on running benchmarks to ensure the cache implementation was not only correct but performant. When benchmarks broke, the assistant diagnosed the issue rather than skipping or disabling them. This commitment to performance validation is especially important for a caching subsystem, where the entire point is speed.

The fix itself — using singleton caches in benchmarks — is a pattern worth documenting. It highlights an inherent tension between Prometheus's strict registration model and Go's benchmark framework. Prometheus assumes a long-lived process where metrics are created once at startup. Go benchmarks assume isolated, repeatable test environments where setup is re-executed for each measurement. Bridging these two models required understanding both systems deeply.

Moreover, the benchmark numbers provide a performance contract for the cache subsystem. An ARC Get in 20 nanoseconds means the cache adds negligible overhead to read operations. A Put in 396 nanoseconds means writes are fast enough for high-throughput workloads. These numbers become reference points for future optimization — if a change makes Put take 500 ns, that's a regression worth investigating.

Conclusion

Message 1897 appears, on its surface, to be a simple verification command. But it is the culmination of a focused debugging effort that touched on Go's testing internals, Prometheus's design philosophy, and the practical challenges of benchmarking instrumented code. The assistant's methodical approach — isolate the failure, read the source, understand the conflicting assumptions, apply a targeted fix, verify across all benchmarks — exemplifies disciplined software engineering. The resulting benchmark numbers are not just passing tests; they are a performance baseline for a critical subsystem in a distributed storage gateway. In the end, the message is a testament to the value of thoroughness: the willingness to chase down a panic in a benchmark, understand its root cause, and fix it properly, rather than simply silencing the error or skipping the benchmarks.