When Tests Panic: The Hidden Danger of Prometheus Auto-Registration

A Deep Dive Into a Single Debugging Message

In the course of building enterprise-grade observability infrastructure for a distributed Filecoin Gateway S3 storage system, an assistant encountered a subtle but critical testing problem. The message under analysis—message index 1866 in a lengthy coding session—is deceptively brief. It reads:

The metrics are using promauto which auto-registers to the global registry. Tests can't create new instances. Let me fix the tests.

This single sentence, followed by two file write operations, represents a moment of diagnostic clarity. It is the pivot point where a failing test suite is understood not as a bug in the test logic, but as a fundamental mismatch between how production code registers Prometheus metrics and how tests attempt to instantiate them. To fully appreciate this moment, one must understand the entire chain of events that led to it, the technical constraints at play, and the reasoning that produced this concise conclusion.

The Broader Context: Enterprise-Grade Observability

The session in which this message appears is the culmination of Milestone 02 ("Enterprise Grade") for the Filecoin Gateway (FGW) project. This milestone added comprehensive observability infrastructure: Prometheus metrics for deal pipelines, financial balances, database operations, and S3 frontend performance; Loki/Promtail log aggregation; automated wallet and database backups; five Grafana dashboards; six operational runbooks; and an AI-powered support system built with LangGraph and self-hosted Ollama/Mistral models.

The assistant had just completed all of this implementation and committed it under the hash 140410d. Following a user request to "Execute recommendations," the assistant embarked on a verification and testing campaign. The recommendations included integration testing, Ansible role validation, load testing, and documentation review. It was during the integration testing phase—specifically, when running unit tests for the newly created metrics code—that the problem surfaced.

The Immediate Trigger: A Failing Test Run

The message directly preceding our subject (message 1865) shows the assistant running the newly written tests:

go test ./rbdeal/... -v -count=1 -run "GC|Metrics|Balance" 2>&1 | tail -50
=== RUN   TestNewBalanceMetrics
--- FAIL: TestNewBalanceMetrics (0.00s)
panic: duplicate metrics collector registration attempted

The panic message is telling: "duplicate metrics collector registration attempted." This is not a typical assertion failure or a logic error in the test. It is a runtime panic from deep inside the Prometheus client library, specifically from prometheus.(*Registry).MustRegister. This indicates that the test code is attempting to register Prometheus metric collectors (counters, gauges, histograms) with names that have already been registered—likely from a previous test run or from a previous instantiation within the same test binary.

The stack trace shows the panic propagating through the test runner's panic recovery mechanism, ultimately causing the test to fail. The key detail is that this panic occurs during the construction of the metrics struct, not during any assertion or behavioral check. The test hasn't even begun to exercise the code; it fails at instantiation time.

The Diagnosis: Understanding promauto

The assistant's response in message 1866 reveals the root cause: "The metrics are using promauto which auto-registers to the global registry."

To understand this, one must understand how Prometheus metric registration works in Go. The Prometheus client library provides two primary ways to create metrics:

1. Direct construction: prometheus.NewCounterVec(...) creates a metric collector but does not register it with any registry. The caller is responsible for registration, typically via prometheus.MustRegister() or by passing a custom registry.
2. Auto-registration via promauto: The promauto package provides convenience functions like promauto.NewCounterVec(...) that create a metric collector and immediately register it with the global default registry. This is convenient for production code because it eliminates the need for explicit registration boilerplate. However, it has a critical side effect: the global registry is a singleton, shared across the entire process. The production code in rbdeal/balance_metrics.go, rbdeal/deal_metrics.go, and rbdeal/gc.go all use promauto:

import "github.com/prometheus/client_golang/prometheus/promauto"

// In balance_metrics.go:
walletBalanceFil      prometheus.Gauge
// constructed via promauto.NewGauge(...)

// In deal_metrics.go:
dealsProposed  *prometheus.CounterVec
// constructed via promauto.NewCounterVec(...)

This design choice is perfectly reasonable for production: it keeps the code clean, ensures metrics are always registered, and works well in a long-running server process where metrics are created once at startup and never re-created.

The Testing Problem: Global State Collision

The problem emerges when writing unit tests for this code. A typical unit test for a metrics struct might look like:

func TestNewBalanceMetrics(t *testing.T) {
    m := NewBalanceMetrics()  // This calls promauto.NewGauge(...)
    m.WalletBalanceFil.Set(100.0)
    // ... assertions ...
}

The first time this test runs in a test binary, NewBalanceMetrics() calls promauto.NewGauge(...) which registers the gauge with the global registry. This succeeds. The test passes. But when a second test also creates a BalanceMetrics instance—or when the same test runs again in a different test function within the same binary—promauto.NewGauge(...) attempts to register a gauge with the same name, and the global registry panics because a collector with that name already exists.

This is exactly what happened. The assistant had written three test files (gc_test.go, deal_metrics_test.go, balance_metrics_test.go) and when running them together with -run "GC|Metrics|Balance", the test binary attempted to create multiple instances of metrics structs, each triggering auto-registration of the same metric names, causing the panic.

Assumptions and Misconceptions

Several assumptions led to this situation:

Assumption 1: Tests can freely construct production objects. The assistant initially assumed that writing tests for metrics code would follow the standard pattern of "construct an object, call methods on it, assert results." This assumption is valid for most Go code, but it breaks down when construction has side effects on global state—as it does with promauto.

Assumption 2: The LSP errors were the main concern. In messages 1860–1864, the assistant was primarily focused on fixing LSP (Language Server Protocol) errors in the test files—undefined types, missing fields, incorrect method signatures. The assistant correctly read the actual source files to understand the real API and rewrote the tests accordingly. But the focus was on syntactic correctness, not on the runtime behavior of metric registration.

Assumption 3: Each test file would run independently. The assistant may have assumed that running go test ./rbdeal/... with a -run filter would execute tests in isolation. While Go's test runner does execute tests sequentially within a package, they share the same process space and therefore the same global Prometheus registry. The -run flag only filters which tests to execute; it doesn't isolate them from each other.

Assumption 4: The panic was a pre-existing issue. The initial test run (message 1865) showed the panic, and the assistant might have initially thought it was related to the pre-existing YugabyteDB test failures mentioned earlier. But the panic message clearly pointed to Prometheus registry duplication, not a database connection issue.

The Thinking Process Visible in the Message

The assistant's reasoning is compressed into a single sentence, but it reveals a clear diagnostic chain:

1. Observation: Tests panic with "duplicate metrics collector registration attempted."
2. Pattern recognition: The panic comes from prometheus.(*Registry).MustRegister, indicating a registration conflict.
3. Root cause analysis: The production code uses promauto, which auto-registers to the global registry.
4. Implication: Each test that constructs a metrics object triggers auto-registration, and the second construction fails.
5. Solution direction: "Let me fix the tests." The phrase "Tests can't create new instances" is the key insight. It's not that the tests are structurally wrong—it's that the testing approach of creating fresh instances for each test is fundamentally incompatible with promauto's global registration model. The assistant recognizes this as a design constraint that must be worked around in the test code.

Input Knowledge Required

To fully understand this message, one needs:

1. Understanding of Prometheus client library architecture: Knowledge that promauto registers with a global singleton registry, and that duplicate registration causes a panic.
2. Go testing conventions: Understanding that tests within the same package share a process space and thus share global state.
3. The production codebase: Knowing that balance_metrics.go, deal_metrics.go, and gc.go all use promauto for metric construction.
4. The test code structure: Knowing that the assistant wrote three separate test files that each construct metrics structs.
5. The session history: Understanding that this is part of a verification campaign following the completion of Milestone 02.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

1. A clear diagnosis of why the tests fail, documented for future reference.
2. A direction for the fix: The tests must be restructured to avoid duplicate registration. Common approaches include: - Using prometheus.NewRegistry() in each test and passing it to the metrics constructor (requires refactoring production code to accept a registry parameter). - Using prometheus.NewCounterVec directly instead of promauto.NewCounterVec in tests, bypassing auto-registration. - Using testutil.CollectAndCompare or testutil.GatherAndCompare which work with the global registry but avoid duplicate registration. - Restructuring tests to use sync.Once or package-level initialization to ensure metrics are created only once per test binary. - Using prometheus.NewPedanticRegistry() for stricter testing.
3. A pattern for future testing: Anyone extending the metrics code now knows that tests must account for global registration semantics.

The Broader Lesson: Global State in Testing

This message illustrates a classic tension in software engineering: the convenience of global state versus the testability of code. promauto is convenient for production—it eliminates boilerplate, ensures metrics are always registered, and works naturally with Prometheus's pull-based collection model. But it makes unit testing harder because tests cannot create isolated instances.

The broader lesson is that any code with side effects on global state—whether it's Prometheus metrics, database connections, logger instances, or configuration singletons—requires careful testing strategies. Common approaches include:

• Dependency injection: Pass a registry (or factory) as a parameter rather than relying on global state.
• Test fixtures: Create metrics once at the package level and share them across tests.
• Reset mechanisms: Provide a way to unregister and re-register metrics for testing (though Prometheus doesn't support this easily).
• Integration-level testing: Test metrics code at a higher level where global state is acceptable.

The Aftermath

While the message itself doesn't show the final test code, the session summary indicates that the assistant ultimately "properly adjusted to use singleton patterns for Prometheus metric registration to avoid conflicts with the global registry." This suggests the fix involved restructuring the tests to create metrics structs once and reuse them, or using a custom registry pattern.

The message stands as a small but perfect example of the debugging process: observe the failure, recognize the pattern, trace to the root cause, and articulate the constraint clearly. In just one sentence, the assistant transformed a confusing panic into an actionable understanding, enabling the fix to proceed.