When Benchmarks Panic: Diagnosing Prometheus Metric Registration in Go Tests

The Message

In a coding session focused on validating an enterprise-grade distributed storage system for Filecoin, the assistant encountered a puzzling benchmark failure and delivered this diagnostic message:

The benchmarks create separate caches each with unique metric names. The issue is Go's benchmark framework runs b.N iterations, but the benchmark setup (NewARCCache) is being called multiple times. The fix is to use a singleton pattern for benchmarks. Let me update the benchmarks to handle metrics properly:

>

[read] /home/theuser/gw/rbcache/arc.go

>

`` 00071| t2Size prometheus.Gauge 00072| pValue prometheus.Gauge 00073| ghostHitsB1 prometheus.Counter 00074| ghostHitsB2 prometheus.Counter 00075| } 00076| 00077| func newARCMetrics(name string) *arcMetrics { 00078| return &arcMetrics{ 00079| hits: promauto.NewCounter(prometheus.CounterOpts{ 00080| Namespace: "fgw", 00081| Subsystem: "cache", 00082| Name: name + "_hits_total", 00083| Help: "Total cache hits", 00084| }), 00085| misses: promauto.New... ``

At first glance, this looks like a simple bug report. But beneath the surface, this message captures a moment of genuine debugging insight—a developer (or in this case, an AI coding assistant) piecing together the behavior of Go's testing framework, Prometheus's client library, and the architecture of a caching system to understand why a seemingly straightforward benchmark panics. This article unpacks that moment in detail.

Context: The Road to Benchmarks

To understand why this message matters, we need to step back. The session leading up to this point had been productive and wide-ranging. The assistant had just completed three major milestones for the FGW (Filecoin Gateway) project:

• Milestone 02: Enterprise-grade observability with Grafana dashboards, Loki logging, Prometheus metrics, and an AI support system
• Milestone 03: A multi-tier persistent retrieval cache system with L1 ARC (Adaptive Replacement Cache) in memory and L2 SSD cache with SLRU eviction
• Milestone 04: A passive garbage collection system with reference counting, reverse indices, and claim lifecycle management All three milestones had been committed. The assistant was working through a checklist of post-milestone recommendations: integration testing, Ansible role validation, load testing the cache, and documentation review. It had already fixed a YAML syntax error in the backup playbook and run the full test suite successfully. The next item on the list was running cache benchmarks. The assistant ran go test ./rbcache/... -bench=. -benchmem and got a panic: "duplicate metrics collector registration attempted." It tried a more targeted invocation with a single benchmark and -benchtime=100ms, but the panic persisted. Something fundamental was wrong with how the benchmarks interacted with Prometheus metrics.

The Diagnostic Leap

The subject message is the assistant's analysis after seeing two failed benchmark runs. It's a concise, three-sentence diagnosis that packs in several layers of reasoning:

First sentence: "The benchmarks create separate caches each with unique metric names." This acknowledges that the benchmark code appears to do the right thing—each cache instance gets a unique metrics name (like "bench_put"), which should avoid naming collisions in Prometheus. The assistant had already verified this by reading the benchmark source code in arc_test.go, where NewARCCache[int, []byte](1<<20, ..., "bench_put") passes a unique name string.

Second sentence: "The issue is Go's benchmark framework runs b.N iterations, but the benchmark setup (NewARCCache) is being called multiple times." This is the crucial insight. The assistant realized that Go's testing.B benchmark loop doesn't just call the benchmark function body repeatedly—it can call the entire function multiple times, including the setup code that creates the cache. Each invocation of NewARCCache calls newARCMetrics, which uses promauto.NewCounter and similar functions. And promauto auto-registers metrics with the global Prometheus default registry. Even though each cache has a unique name prefix, the benchmark framework's internal behavior causes multiple registration attempts for the same metric name.

Third sentence: "The fix is to use a singleton pattern for benchmarks." This is the proposed solution: instead of creating a fresh cache inside the benchmark function (which gets called multiple times), create the cache once at package level or in a TestMain function, so the metrics are registered exactly once.

What the Assistant Assumed—and Got Right

The assistant made several assumptions that turned out to be correct:

1. That promauto uses a global registry. The promauto package in Prometheus's Go client library provides convenience wrappers that auto-register metrics with the global default registry. This is by design for production code, but it creates problems in test scenarios where the same metric might be registered multiple times.
2. That the benchmark function is invoked more than once. Go's benchmark framework has a subtle behavior: the benchmark function is called multiple times by the framework to warm up, to calibrate, and to gather statistics. Each call to the function re-executes the setup code. This is documented behavior but easy to overlook.
3. That unique metric names alone aren't sufficient. The assistant correctly recognized that even though each cache instance uses a different metrics name (like "bench_put" vs "bench_get"), the issue is that the same name gets registered multiple times across repeated invocations of the same benchmark function.
4. That the fix belongs in the test code, not the production code. The assistant didn't propose changing newARCMetrics or the production cache code. The singleton pattern would be applied only to the benchmark functions, keeping production code clean.

What the Assistant Might Have Missed

The analysis is largely correct, but there's a subtlety worth examining. The assistant's diagnosis focuses on the benchmark framework calling the function multiple times. However, Go's testing.B framework actually calls the benchmark function once per benchmark, and the b.N loop is inside that single call. The panic happens because:

• When running -bench=., Go's test framework runs multiple benchmark functions sequentially (BenchmarkARCCache_Put, BenchmarkARCCache_Get, etc.)
• Each benchmark function creates a new cache with NewARCCache(..., "bench_put") or similar
• But if the same benchmark function is run multiple times (e.g., across different invocations of go test), the second run will try to re-register the same metrics The assistant's earlier attempt to run a single benchmark with -bench=BenchmarkARCCache_Put -count=1 still panicked, which suggests that even a single benchmark function, when run once, might trigger the issue. This could happen if the benchmark function itself calls b.N iterations and each iteration creates a new cache—but looking at the benchmark code, the cache is created before b.ResetTimer(), so it's created once per benchmark function invocation. The panic in the single-benchmark case might be from a previous test in the same binary that already registered metrics with the same names. The assistant's singleton fix addresses all these cases elegantly: by creating the cache once at package level, metrics are registered exactly once regardless of how many times the benchmark function is called or how many benchmarks share the binary.

Input Knowledge Required

To fully understand this message, one needs:

1. Go's testing framework: Understanding that func BenchmarkXxx(b *testing.B) is the benchmark signature, that b.N is set by the framework, and that benchmark functions can be invoked multiple times by the test runner.
2. Prometheus client library: Knowing that promauto.NewCounter and similar functions register metrics with a global default registry, and that duplicate registration panics rather than returning an error.
3. The ARC cache architecture: Understanding that NewARCCache takes a metrics name parameter and creates Prometheus metrics internally via newARCMetrics.
4. The singleton pattern: Recognizing that creating a shared, once-initialized instance avoids the re-registration problem.

Output Knowledge Created

This message produces several valuable insights:

1. A clear root cause analysis for the benchmark panic, connecting Go framework behavior with Prometheus library constraints.
2. A specific, actionable fix: use a singleton pattern for benchmark cache instances.
3. A reusable debugging pattern: when a test or benchmark panics with "duplicate registration," look for code paths that might be executed more than once, even if they appear to use unique names.
4. Documentation of a design tension: the convenience of auto-registration (promauto) conflicts with the repeatability requirements of benchmarking.

The Thinking Process

The assistant's reasoning unfolds in stages visible across the conversation:

Stage 1 — Observation: Running go test ./rbcache/... -bench=. -benchmem produces a panic. The error message points to Prometheus's MustRegister function, indicating a duplicate metric registration.

Stage 2 — Hypothesis testing: The assistant tries running a single benchmark with -bench=BenchmarkARCCache_Put -count=1 -benchtime=100ms. The panic persists, ruling out the hypothesis that the issue is simply multiple benchmarks colliding.

Stage 3 — Code inspection: The assistant reads the benchmark source code in arc_test.go and the metric creation code in arc.go. It confirms that each cache gets a unique metrics name via the MetricsName parameter.

Stage 4 — Framework behavior analysis: The assistant connects the dots: Go's benchmark framework can call the benchmark function multiple times (for warmup, calibration, or multiple runs), and each call re-executes the NewARCCache setup, which re-registers metrics.

Stage 5 — Solution design: The assistant proposes the singleton pattern, which is a clean, minimal change that doesn't alter production code.

The Broader Significance

This message is a microcosm of a common challenge in distributed systems development: the tension between observability and testability. Prometheus metrics are essential for production monitoring, but their global registration model creates friction in test environments. The assistant's solution—isolating test code from production registration patterns—is a pragmatic compromise that preserves observability without sacrificing test reliability.

The message also illustrates how debugging often requires synthesizing knowledge across multiple domains: understanding the testing framework's execution model, the metrics library's registration semantics, and the application code's initialization patterns. The assistant's ability to hold all three in working memory and identify the intersection point is the hallmark of effective debugging.

Finally, this moment captures the iterative nature of validation work. The assistant wasn't writing new features or fixing user-reported bugs. It was running benchmarks to validate existing code—and discovered a latent issue that would have surfaced the moment anyone tried to measure cache performance. Finding and fixing these kinds of "second-order" bugs is what separates robust engineering from code that merely passes unit tests.