The Art of the Singleton Benchmark: Fixing Prometheus Metric Registration in Go Tests

The Message

The SSD benchmarks are more complex because they use temp directories and need cleanup. Let me apply the same singleton pattern:

>

[edit] /home/theuser/gw/rbcache/ssd_test.go

>

Edit applied successfully.

At first glance, this message from an AI assistant working on the Filecoin Gateway (FGW) distributed storage system appears unremarkable — a three-sentence status update paired with a file edit command. Yet this brief exchange sits at the intersection of several deep software engineering concerns: the tension between benchmark correctness and test infrastructure, the subtle pitfalls of global state in Go programs, the design of Prometheus instrumentation in caching subsystems, and the quiet craft of making performance measurements reliable. Understanding why this message was written, and what it reveals about the reasoning process behind it, requires unpacking the chain of events that led to this precise moment.

The Context: A Cache Subsystem Under Test

The FGW project is a horizontally scalable S3-compatible storage gateway built on a distributed architecture with stateless proxy frontends, Kuri storage nodes, and YugabyteDB for metadata. The assistant had recently completed implementing Milestones 03 and 04 — a multi-tier retrieval cache system (L2 SSD cache with SLRU eviction, adaptive admission policy, access tracking, and DAG-aware prefetch engine) and a passive garbage collection system with reverse indices and reference counting. These are performance-critical components: the cache subsystem directly impacts how quickly objects can be retrieved from the distributed store, and the benchmarks exist to quantify that performance.

After committing the milestone code, the assistant was working through a cleanup checklist: fixing a YAML parsing issue in an Ansible backup playbook, running the full test suite for the new code, and finally running the cache benchmarks. The test suite passed, but when the assistant invoked go test ./rbcache/... -bench=. -benchmem to run the benchmarks, the result was a panic:

panic: duplicate metrics collector registration attempted

This panic originated from the Prometheus client library's MustRegister function, called by promauto.NewCounter and similar auto-registration helpers. The root cause was subtle: each benchmark function in Go's testing framework can be invoked multiple times by the framework (to warm up, to calibrate, to measure at different durations), and each invocation of the benchmark function created a new cache instance via NewARCCache or NewSSDCache. Each new cache instance called promauto.NewCounter(...) with metric names like bench_put_hits_total, which attempted to register with the default global Prometheus registry. The first registration succeeded; the second triggered a panic because Prometheus's registry does not allow duplicate collector registrations.

The Reasoning Process: From Panic to Fix

The assistant's thinking, visible across the preceding messages, shows a methodical debugging approach. First, it reproduced the failure and confirmed it wasn't a fluke — even running a single benchmark with -bench=BenchmarkARCCache_Put -count=1 still panicked. This ruled out the possibility that the issue was caused by multiple benchmarks interfering with each other; even a single benchmark function, when invoked by the framework, was creating multiple cache instances.

The assistant then examined the benchmark source code in arc_test.go and the metric registration code in arc.go. It identified the core mechanism: newARCMetrics(name string) uses promauto.NewCounter(...), which calls prometheus.MustRegister on the default registry. Because promauto uses the global default registry, there is no way to isolate metric registrations between benchmark invocations — every call to NewARCCache with the same metric name attempts to re-register.

The fix the assistant chose was to apply a singleton pattern: instead of creating a fresh cache inside each benchmark function, create a single cache instance at package scope and reuse it across all benchmark iterations. This way, promauto.NewCounter is called exactly once per metric name, and subsequent benchmark invocations use the already-initialized cache without attempting to re-register metrics.

For the ARC benchmarks, this was straightforward — the cache is in-memory and has no external dependencies. The assistant edited arc_test.go to introduce a package-level variable initialized once via sync.Once or a similar mechanism, then ran the benchmarks successfully:

BenchmarkARCCache_Put-192      2941498   404.2 ns/op   158 B/op   3 allocs/op
BenchmarkARCCache_Get-192     58497009   20.29 ns/op     0 B/op   0 allocs/op
BenchmarkARCCache_Mixed-192   10934908   106.3 ns/op    31 B/op   0 allocs/op

With the ARC benchmarks passing, the assistant turned to the SSD benchmarks — and this is where our target message enters.

Why This Message Matters: The SSD Complexity

The message explicitly states why the SSD benchmarks are different: "they use temp directories and need cleanup." This seemingly simple observation reveals a deep understanding of the testing constraints at play. The SSD cache (NewSSDCache) writes data to disk, using a directory path provided via b.TempDir(). Go's TempDir() creates a temporary directory that is automatically cleaned up at the end of the test or benchmark. This introduces several complications for the singleton pattern:

  1. Lifetime management: A singleton cache created once would need to live for the entire benchmark suite, but its temp directory would be cleaned up when the first benchmark function exits. The assistant would need to either use a persistent directory outside the test framework's lifecycle, or manage cleanup explicitly.
  2. State isolation: Each benchmark iteration for an SSD cache expects a clean state — no pre-existing data from previous runs. A singleton cache that persists across iterations would retain its state, potentially skewing results. The assistant would need to either clear the cache between iterations or design the singleton to be resetable.
  3. Resource pressure: An SSD cache with a configured max size of 100MB (as seen in the benchmark config) would consume real disk space. Multiple concurrent or sequential benchmark invocations sharing a single cache could hit size limits or cause unexpected eviction behavior.
  4. Close and cleanup: The SSD cache has a Close() method that flushes buffers and releases resources. The singleton pattern must ensure proper cleanup without leaving the cache in an unusable state for subsequent benchmark functions. The assistant's recognition of this complexity — and its decision to nonetheless "apply the same singleton pattern" — represents a pragmatic engineering judgment. The alternative approaches would have been more invasive: refactoring the metric registration to use a custom registry (requiring changes to production code, not just tests), introducing a test-only flag to skip metric registration (adding conditional logic that could mask real issues), or rewriting the benchmarks to use a subprocess model (complex and slow). The singleton pattern, while requiring careful handling of temp directories and cleanup, is the least invasive change that preserves benchmark correctness.

Assumptions and Knowledge Required

To fully understand this message, several pieces of input knowledge are necessary:

The Broader Significance

This message, though brief, exemplifies a class of problems that arise when production-quality instrumentation meets test infrastructure. Prometheus metrics are designed for long-running server processes where metrics are registered once at startup and collected throughout the process lifetime. Test and benchmark code, by contrast, is ephemeral and iterative — it creates and destroys objects repeatedly. The tension between these two paradigms is not unique to this project; it appears in any Go codebase that uses promauto or similar auto-registration patterns and then attempts to write benchmarks for the instrumented components.

The assistant's solution — the singleton benchmark pattern — is a well-known technique in the Go testing community, but its application here required recognizing that the problem was not in the production code (which correctly registers metrics once per cache instance in a real server) but in the test code (which was creating multiple cache instances where only one was intended). The fix preserves the production code's design while adapting the tests to the constraints of the benchmark framework.

Conclusion

The message "The SSD benchmarks are more complex because they use temp directories and need cleanup. Let me apply the same singleton pattern" is a masterclass in concise engineering communication. It acknowledges complexity, states the chosen strategy, and executes. Behind those three sentences lies a chain of reasoning: reproducing a panic, tracing it to Prometheus metric re-registration, understanding why the ARC fix worked, recognizing why the SSD case is harder, and deciding that the singleton pattern is still the right approach despite the additional complexity. It is a small moment in a long coding session, but it captures the essence of what makes software engineering difficult — not the grand architecture, but the thousand small judgments about how to make code testable, measurable, and reliable.