The Panic in the Benchmark: When Prometheus Metrics Refuse to Be Duplicated

The Message

On January 31, 2026, during a coding session for the Filecoin Gateway (FGW) distributed storage system, an assistant executed a seemingly routine command:

[bash] cd /home/theuser/gw && go test ./rbcache/... -bench=. -benchmem 2>&1
goos: linux
goarch: amd64
pkg: github.com/CIDgravity/filecoin-gateway/rbcache
cpu: AMD Ryzen Threadripper PRO 7995WX 96-Cores     
BenchmarkARCCache_Put-192            	panic: duplicate metrics collector registration attempted

goroutine 715 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xafa520, {0xc0002d8000?, 0x7e4fe4?, 0x5?})
	/home/theuser/.opt/go/pkg/mod/github.com/prometheus/client_golang@v1.22.0/prometheus/registry.go:406 +0x65
github.com/prometheus/c...

What appears at first glance to be a trivial benchmark crash is, in fact, a revealing moment in the development of a complex distributed storage system. This single message — a command that produced a panic instead of benchmark results — exposes assumptions about test infrastructure, the tension between observability and testability, and the subtle ways that production-oriented code patterns can break under testing conditions.

Context: The State of the Project

To understand why this message matters, one must understand what had just been accomplished. The assistant and user had been working on the Filecoin Gateway (FGW), a horizontally scalable S3-compatible storage system built on top of the Filecoin decentralized storage network. The project had recently completed three major milestones:

The Crash: What Actually Happened

The command go test ./rbcache/... -bench=. -benchmem is a standard Go incantation that runs all benchmarks in the rbcache package, measuring both timing and memory allocations. What the assistant expected was a clean table of benchmark results showing nanoseconds per operation and bytes allocated per call.

Instead, the very first benchmark — BenchmarkARCCache_Put — panicked with a stack trace pointing to Prometheus's MustRegister function. The error message was unambiguous: "duplicate metrics collector registration attempted."

The stack trace reveals the mechanism. The promauto package (Prometheus Auto) in client_golang v1.22.0 automatically registers metrics collectors with a global default registry. When NewARCCache is called, it creates a new set of Prometheus metrics — counters for hits, misses, and various cache statistics — each with a unique name derived from the MetricsName parameter. The benchmark function creates a new cache instance with the name "bench_put", and the metrics get registered. But Go's benchmark framework calls the benchmark function multiple times (with increasing values of b.N), and each invocation creates a new cache, which attempts to register the same metrics again. The second registration attempt panics.

The Assumptions Laid Bare

This failure reveals several interconnected assumptions, some correct and some incorrect.

Correct assumption: The cache system needed production-grade observability. The ARC cache was designed to be deployed in a live distributed storage cluster where operators would need to monitor hit rates, eviction counts, and memory usage through Prometheus dashboards. Using promauto for automatic registration was the standard, idiomatic approach for production Go services.

Correct assumption: The benchmark should measure realistic performance. Creating a fresh cache for each benchmark run is a common pattern to ensure clean state and reproducible measurements.

Incorrect assumption: The benchmark infrastructure would handle metric registration gracefully. The Go testing framework (testing.B) does not reset the Prometheus global registry between benchmark iterations. The testing.B type manages timing and iteration count, but it has no knowledge of external state like registered metrics. The assistant assumed that either (a) the metrics would be registered once and reused, or (b) the test framework would handle cleanup.

Incorrect assumption: promauto.NewCounter and similar functions would be safe to call multiple times with the same name. The promauto package's contract is that it registers metrics with the default registry, and MustRegister panics on duplicate registration. This is by design — in production, duplicate metric names indicate a bug. But in a benchmark context, where the same setup code runs multiple times, this safety mechanism becomes a liability.

The Reasoning Process Visible in the Message

The message itself is terse — a single bash command followed by truncated output. But the reasoning behind it is visible in the surrounding context. The assistant had just completed a systematic validation pipeline:

  1. Fix the backup playbook syntax error (completed)
  2. Run the full test suite (completed — all passed)
  3. Run cache benchmarks (in progress — this message) The assistant was following a deliberate, methodical approach to validation. The test suite verified correctness; the benchmarks would verify performance. The todo list management (todowrite) shows the assistant tracking progress explicitly, treating the benchmark run as a discrete, completable task. The choice of command flags is also revealing: -bench=. runs all benchmarks, -benchmem measures memory allocations. The assistant wanted a comprehensive view of cache performance — both speed and memory efficiency. The ARC cache's design goals included low-latency lookups and minimal allocation overhead, making both metrics critical.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. Go testing conventions: The testing.B type and how benchmarks work — specifically that b.N varies across iterations and the benchmark function is called multiple times.
  2. Prometheus client_golang: The promauto package and its auto-registration behavior, the global default registry, and the MustRegister panic-on-duplicate design.
  3. The FGW architecture: That the cache system was designed for a distributed storage gateway where Prometheus metrics are essential for operational monitoring.
  4. The project's current state: That this was a post-milestone validation phase, not initial development. The code was committed and supposedly complete.

Output Knowledge Created

This message created several pieces of valuable knowledge:

  1. A discovered defect: The benchmarks were broken. This wasn't a theoretical concern — the code literally panicked when run.
  2. A constraint revealed: The Prometheus metric registration pattern was incompatible with Go's benchmark framework. Any code using promauto with unique metric names per instance would fail under benchmark conditions.
  3. A debugging trace: The stack trace pointed directly to the registration point, making diagnosis straightforward.
  4. A priority shift: The assistant would need to fix the benchmarks before obtaining performance data, changing the task from "measure performance" to "fix benchmark infrastructure."

The Deeper Significance

This message is a microcosm of a broader challenge in distributed systems development: the tension between production robustness and testability. The Prometheus metrics were added to make the cache observable in production — a laudable and necessary goal for any serious distributed storage system. But the same pattern that makes production monitoring work (global registration, panic on duplicates) makes testing harder.

The assistant's response to this failure (visible in subsequent messages) is instructive. Rather than removing metrics from the benchmark or using a workaround like testing.Cleanup, the assistant examined the benchmark code, identified the root cause (multiple cache instances re-registering metrics), and restructured the benchmarks to use a singleton cache pattern. This preserved both the production metrics and the ability to benchmark.

This is the hallmark of experienced infrastructure engineering: when production patterns clash with testing patterns, the solution is not to weaken the production code but to make the tests smarter. The assistant chose to fix the benchmarks rather than compromise the observability design.

Conclusion

Message 1888 is a small moment of failure that speaks volumes about the development process. It shows a methodical engineer working through a validation checklist, hitting an unexpected obstacle, and having to pause and diagnose. The panic itself is mundane — a duplicate registration error in a benchmark — but the context transforms it into a case study in the challenges of building observable, testable distributed systems. The message captures the exact moment when a carefully planned validation step goes wrong, revealing hidden assumptions about how production code interacts with testing infrastructure. In the broader narrative of the FGW project, this is the moment where the assistant learned that even "completed" code needs validation, and that the gap between "tests pass" and "benchmarks run" can hide significant engineering work.