The Duplicate Metrics Collector: A Case Study in Prometheus Test Architecture

Introduction

In the midst of a sprawling implementation session for the Filecoin Gateway (FGW) project—where an assistant had just completed three major milestones encompassing enterprise-grade observability, multi-tier caching, and data lifecycle management—a single test failure message reveals a surprisingly deep architectural tension. Message 1865 captures the moment when the assistant, executing a follow-up recommendation to add test coverage for newly written metrics code, encounters a panic: duplicate metrics collector registration attempted error from the Prometheus client library. This seemingly small failure is a window into fundamental design decisions about how monitoring infrastructure interacts with testability, and it exposes assumptions about global state that many production systems silently carry.

The Message in Context

The message is deceptively brief. After writing three test files—gc_test.go, deal_metrics_test.go, and balance_metrics_test.go—the assistant runs the test suite with the command:

go test ./rbdeal/... -v -count=1 -run "GC|Metrics|Balance" 2>&1 | tail -50

The output shows TestNewBalanceMetrics failing with a panic originating from prometheus.(*Registry).MustRegister. The error message is truncated in the output, but the key phrase is clear: "duplicate metrics collector registration attempted."

This failure occurs because Prometheus's default global registry is a singleton. When multiple test cases within the same test binary attempt to register metric collectors with the same names—using promauto helper functions that call MustRegister under the hood—the second registration attempt panics. This is by design in Prometheus: metric names must be unique within a registry to ensure unambiguous querying. But this design choice creates a well-known friction point for testing.

The Reasoning and Motivation

Why was this message written? The assistant was executing a recommendation the user had explicitly approved: "Execute recommendations." The original recommendations included integration testing, Ansible role testing, load testing, and documentation review. The assistant chose to start with integration testing, specifically adding unit tests for the GC and metrics code that had been written as part of Milestone 02.

The motivation was thoroughness and validation. The assistant had just completed a massive implementation spanning three milestones. The verification step showed that while the core code compiled and the cache tests passed, the rbdeal package had no test files at all. Writing tests for the metrics and GC code was a natural quality assurance step—ensuring that the observability infrastructure that operators would rely on was actually working correctly.

However, the assistant made an assumption that proved incorrect: that writing straightforward unit tests for Prometheus-wrapped metrics would work without special handling. This assumption is understandable—many developers first encountering Prometheus metrics in Go make the same mistake. The promauto package is designed for convenience in production code, where the global registry is initialized once and metrics are registered at startup. But in a test environment, where the same binary may run dozens of test functions, the global registry becomes a liability.

The Thinking Process Visible

The message shows the assistant in a diagnostic loop. The tail -50 output captures the panic trace, but the assistant doesn't immediately show the fix—that comes in subsequent messages. What's visible is the moment of discovery: the test ran, it failed, and the failure mode is clearly identified in the panic message.

The assistant's reasoning at this point would be: "The tests I wrote are failing because of duplicate Prometheus metric registration. This is a common issue. I need to either use a local registry per test, or restructure the tests to avoid duplicate registrations, or use prometheus.NewRegistry() and pass it to the code under test."

The truncated output is also telling. The assistant used tail -50 to see the end of the test output, which captured the panic but not the full stack trace. This suggests the assistant expected to see pass/fail summaries and was surprised by the panic. The choice of tail rather than grepping for specific patterns indicates a "let me see what happened" approach rather than a targeted search for known failure modes.

Assumptions and Their Consequences

Several assumptions are embedded in this message:

  1. That production metric registration patterns work in tests. The balance_metrics.go file uses promauto.NewGauge and similar helpers that register against the default global registry. This is fine for production but problematic for tests.
  2. That test isolation is guaranteed by Go's testing framework. Go runs tests within the same package in the same binary, sharing global state. The assistant assumed each test function would be isolated, but Prometheus's global registry is shared across all tests.
  3. That the test files written were correct. The assistant wrote three test files in quick succession (messages 1860-1864), reading the source files to understand the actual API, then writing tests that matched. The tests themselves were structurally correct—they instantiated the metrics structs and verified they weren't nil. The failure was in the test infrastructure, not the test logic.
  4. That -count=1 would prevent caching issues. The -count=1 flag disables test result caching but doesn't affect global state isolation. The consequence of these assumptions was a test panic that halted the entire test suite. In Go, a panic in one test function can crash the entire test binary, preventing all subsequent tests from running. This is a severe failure mode—not just a test failure but a test crash.

Input Knowledge Required

To understand this message, a reader needs:

Output Knowledge Created

This message creates several forms of knowledge:

  1. A diagnostic signal: The panic message itself is knowledge—it tells the assistant (and anyone reading the logs) that the test infrastructure has a Prometheus registration conflict.
  2. A priority shift: The failure changes the assistant's task queue. Before this message, the assistant was planning to run through integration testing, Ansible testing, load testing, and documentation review. After this failure, the immediate priority becomes fixing the test infrastructure.
  3. A design insight: The failure reveals that the metrics code, as written, is not safely testable. This is a design smell—if production code can't be tested without panicking, there's a coupling issue between the code and its global dependencies.
  4. A documentation opportunity: The fix for this pattern (using local registries, or refactoring to accept a registry parameter) is a well-known pattern that deserves documentation in the project's testing guide.

Mistakes and Incorrect Assumptions

The primary mistake was not anticipating the Prometheus global registry conflict. This is an extremely common mistake—so common that the Prometheus documentation explicitly warns about it, and many projects have developed patterns to work around it. The assistant's error was one of inexperience with this specific testing pattern rather than a fundamental misunderstanding.

A secondary mistake was writing all three test files before running any of them. A more iterative approach—write one test, run it, see the failure, fix the pattern, then write the remaining tests—would have saved time. The assistant wrote three test files across messages 1860-1864, then ran them all at once in message 1865, discovering the failure only after all three were written.

A tertiary issue is the use of tail -50 to view test output. This truncated the full panic stack trace, potentially hiding useful diagnostic information. A command like go test ./rbdeal/... -v -count=1 -run "GC|Metrics|Balance" 2>&1 | grep -A 20 "panic" would have captured the full panic context.

The Broader Architectural Lesson

This message, though small, illustrates a fundamental tension in software architecture: the conflict between convenient global state (Prometheus's default registry) and testability. The promauto package exists to make production code concise—developers can write promauto.NewCounter("my_metric") without thinking about registry management. But this convenience comes at a cost: the code is coupled to a global singleton, making isolated testing impossible.

The standard resolution involves one of three patterns:

  1. Registry injection: Accept a *prometheus.Registry parameter in constructors, allowing tests to pass a fresh registry per test case.
  2. Test-only reset: Use prometheus.DefaultRegisterer = prometheus.NewRegistry() in test setup, though this is fragile and not thread-safe.
  3. Integration-level testing: Accept that unit tests for metrics code are impractical and test metrics at the integration level where the registry is initialized once. The assistant's subsequent messages (not shown in this article's scope) would need to implement one of these patterns. The choice has implications for the entire metrics architecture—it's not just a test fix but a design decision about how metrics are constructed throughout the codebase.

Conclusion

Message 1865 captures a moment that every Prometheus user in Go eventually encounters: the first time their tests panic because of duplicate metric registration. It's a rite of passage, a teachable moment, and a reminder that convenience abstractions often hide complexity that surfaces at the least convenient time—during testing. The assistant's response to this failure—diagnosing the issue, understanding the global registry problem, and restructuring the tests—would determine whether this becomes a frustrating blocker or a valuable learning experience that improves the entire project's test architecture.