The Silent Saboteur: How Duplicate Prometheus Metrics Registration Nearly Broke a Test Suite

In the midst of a massive, multi-phase testing initiative—where 164 new unit tests were being systematically created across eight components of a distributed Filecoin storage gateway—an unexpected failure emerged. The test suite, which moments earlier had been passing individually, suddenly collapsed when run together. The culprit was not a logic error, a race condition, or a misunderstood API contract. It was something far more subtle: a duplicate Prometheus metrics collector registration, silently waiting to sabotage the entire test run.

The Message in Context

The subject message, indexed at position 2443 in the conversation, is deceptively brief. It reads:

I need to fix this the same way - using sync.Once for metrics registration: [read] /home/theuser/gw/rbdeal/gc.go

This single sentence, followed by a file read operation, represents a moment of diagnostic recognition. The assistant had just witnessed a test failure—TestGC_Disabled panicking with "duplicate metrics collector registration attempted"—and instantly knew both the root cause and the fix. The message is not a question, not an exploration, and not a tentative hypothesis. It is a statement of confident intent, born from having solved this exact class of problem before.

To understand why this message carries such weight, we must examine what led to it and what it set in motion.

The Discovery: A Test That Passes Alone but Fails in Company

The sequence of events reveals a classic testing pitfall. The assistant had been methodically working through a testing plan, creating test files one by one using subagents:

1. Phase 1.1: Configuration tests (14 tests) — passed
2. Phase 1.2: CIDgravity client tests (15 tests) — passed
3. Phase 1.3: Fixed skipped rbstor tests — passed
4. Phase 2.1: Deal making tests with fallback provider parsing — passed
5. Phase 2.3: Deal repair tests (26 tests) — passed
6. Phase 4.2: S3 authentication tests (42 tests) — passed Each phase passed when tested individually. But when the assistant ran all tests together with go test ./configuration/ ./cidgravity/ ./rbdeal/ ./server/s3/ --count=1, a failure appeared. The rbdeal package failed with a panic in TestGC_Disabled. The assistant's first reaction was methodical: isolate the failure. Running TestGC_Disabled in isolation showed it passed. Running each package individually showed they all passed. But running rbdeal alone with --count=1 revealed the panic: a duplicate Prometheus metrics registration. The stack trace pointed to promauto's MustRegister function, which panics when a metric with the same name is already registered in the global default registry. This is the signature of test pollution—a stateful side effect from one test leaking into another. The rbdeal package's GC (garbage collection) component uses promauto.NewCounter and similar functions, which automatically register metrics with Prometheus's global default registry. When multiple tests within the same package create instances of the GC metrics struct, the second registration attempt panics because the metric names already exist.

Why This Happened: The promauto Trap

The root cause lies in how the Prometheus client library for Go handles metric registration. The promauto package provides convenience functions like promauto.NewCounter() that automatically register the metric with a global default registry. This is convenient for production code—you call newGCMetrics() once during application startup, and everything works fine.

But in test environments, this convenience becomes a hazard. Go tests within the same package share the same process space. If a test creates a GC object (which calls newGCMetrics()), and another test also creates a GC object, the second promauto.NewCounter() call tries to register metrics with names that already exist. The Prometheus client library's default behavior is to panic on duplicate registration—a design choice that prioritizes catching configuration errors in production over flexibility in testing.

The assistant had encountered this exact problem before. In an earlier fix for index_metered.go, the same pattern had been identified and corrected. The solution was to replace promauto calls with manual metric creation using prometheus.NewCounter (without auto-registration), and guard the registration with sync.Once to ensure it happened only once regardless of how many times the metrics struct was instantiated.

The Decision: Pattern Matching Over Investigation

What makes this message remarkable is the speed and certainty of the diagnostic. The assistant did not need to trace through the code, examine the stack frame by frame, or experiment with different hypotheses. The panic message—"duplicate metrics collector registration attempted"—combined with the knowledge that the GC code used promauto, was sufficient.

The decision was: apply the same fix pattern that worked for index_metered.go to gc.go. This meant:

1. Replacing promauto.NewCounter(...) with prometheus.NewCounter(prometheus.CounterOpts{...}) for each metric
2. Wrapping the entire registration in a sync.Once block
3. Storing the sync.Once instance as a package-level or struct-level variable The assistant's next actions confirm this: two edit operations on gc.go followed by a successful test run. The fix worked immediately, and all 164+ tests passed together.

Assumptions Embedded in the Fix

The assistant made several assumptions, all of which proved correct:

• The duplicate registration was caused by multiple test instantiations of the GC metrics struct. This was correct—the TestGC_Disabled test and the repair tests both created objects that triggered newGCMetrics().
• The sync.Once pattern would not introduce race conditions in production. Since sync.Once is thread-safe and metrics registration is inherently a one-time operation, this was safe.
• No other code in the rbdeal package had similar issues. The assistant verified this by running the full test suite after the fix.
• The fix would not change production behavior. Since sync.Once ensures registration happens exactly once regardless of how many times newGCMetrics() is called, production code (which typically creates one GC instance) would behave identically.

Input Knowledge Required

To fully understand this message, one needs:

• Go testing mechanics: Understanding that tests within the same package share a process and can pollute each other's state.
• Prometheus client library architecture: Knowing that promauto uses a global default registry and panics on duplicate registration, while prometheus.NewCounter creates unregistered metrics that must be manually registered.
• The sync.Once pattern: Recognizing it as Go's idiomatic mechanism for one-time initialization in concurrent contexts.
• Context from earlier work: Knowing that index_metered.go had been fixed with the same pattern, providing the template for this fix.
• The project's architecture: Understanding that the GC component is part of the rbdeal package and uses Prometheus metrics for monitoring scan operations.

Output Knowledge Created

This message and its follow-through produced several valuable outcomes:

• A fixed test suite: All 164 tests across four packages now pass together, enabling reliable CI/CD.
• A reusable fix pattern: The sync.Once + manual metric creation pattern is now documented (in code) as the correct approach for Prometheus metrics in this project.
• Confidence in the testing infrastructure: The assistant can now trust that test pollution from metrics registration is resolved.
• A subtle lesson in Go testing: The experience reinforces that global state (like Prometheus's default registry) is a common source of test pollution, and that promauto's convenience comes with hidden costs in test environments.

The Broader Significance

This message, for all its brevity, represents a critical inflection point in the testing initiative. Without this fix, the entire test suite would have been unreliable—passing in isolation but failing in aggregate. Developers would have lost trust in the tests, and the 164 new tests would have been a liability rather than an asset.

The fix also reveals something about the assistant's cognitive process: the ability to recognize a failure pattern and instantly map it to a known solution. This is the hallmark of experienced debugging—not starting from zero with each new error, but building a mental library of failure signatures and their remedies. The assistant had seen this exact Prometheus panic before, remembered the fix, and applied it without hesitation.

In distributed systems engineering, the most dangerous bugs are often not the ones that break logic, but the ones that corrupt state across boundaries. A Prometheus registry is, in its own small way, a distributed system—a shared resource that multiple components write to, with no isolation guarantees. The sync.Once pattern is a pragmatic acknowledgment that sometimes the cleanest fix is not to isolate the components, but to make the shared resource tolerant of repeated initialization.

The message at index 2443 is a testament to the value of pattern recognition in debugging, the importance of understanding your tools' hidden assumptions, and the quiet heroism of fixing infrastructure problems that users never see but would feel immediately if left unfixed.