The Validation That Almost Wasn't: Debugging Prometheus Test Conflicts in a Distributed Storage System

The Message

The subject of this analysis is a single message from an AI-assisted coding session, in which the assistant runs a Go test suite and reports that all tests pass. The message reads:

[assistant] [bash] go test ./rbdeal/... -v -count=1 -run "GC|Metrics|Balance" 2>&1 | tail -60 === RUN TestGetBalanceMetrics_Singleton --- PASS: TestGetBalanceMetrics_Singleton (0.00s) === RUN TestBalanceMetrics_AttoFILToFIL --- PASS: TestBalanceMetrics_AttoFILToFIL (0.00s) === RUN TestBalanceMetrics_LargeDatacap --- PASS: TestBalanceMetrics_LargeDatacap (0.00s) === RUN TestDealPipelineMetrics_IncProposed --- PASS: TestDealPipelineMetrics_IncProposed (0.00s) === RUN TestDealPipelineMetrics_IncAccepted --- PASS: TestDealPipelineMetrics_IncAccepted (0.00s) === RUN TestDealPipelineMetrics_IncRejected --- PASS: TestDealPipelineMetrics_IncRejected (0.00s) === RUN TestDealPipelineMetrics_IncExpired --- PASS: TestDealPipelineMetrics_IncExpired (0.00s) === RUN TestDealPipelineMetrics_IncRenewed --- PASS: TestDealPipelineMetrics_IncRenewed (0.00s) === RUN TestGCStateConstants --- PASS: TestGCStateConstants (0.00s) === RUN TestGCStateTransitions --- PASS: TestGCStateTransitions (0.00s) === RUN TestGarbageCollector_Noop --- PASS: TestGarbageCollector_Noop (0.00s) === RUN TestGarbageCollector_WithGroups --- PASS: TestGarbageCollector_WithGroups (0.00s)

On its surface, this is a mundane sight: green checkmarks, passing tests, a job well done. But to understand why this message matters, we must understand the journey that led to it—the debugging spiral, the architectural discovery about Prometheus metric registration, and the broader context of what these tests actually validate in a distributed storage system for the Filecoin network.

Context: The Milestone That Was Complete But Untested

The session leading up to this message was the culmination of weeks of work on the Filecoin Gateway (FGW) project, a horizontally scalable S3-compatible storage gateway built on top of the Filecoin and IPFS networks. The assistant had just completed three major milestones:

• Milestone 02 (Enterprise Grade): Comprehensive observability with Prometheus metrics, Grafana dashboards, operational runbooks, automated backup roles, and an AI-powered support system using LangGraph and Ollama.
• Milestone 03 (Persistent Retrieval Caches): Multi-tier caching with L1 ARC cache, L2 SSD cache with SLRU eviction, adaptive admission policy, access tracking, and a DAG-aware prefetch engine.
• Milestone 04 (Data Lifecycle Management): Passive garbage collection with reverse indices, reference counting, claim extender modifications, and repair worker configuration. All three milestones had been committed to the repository. The assistant had produced a detailed verification summary showing every component was in place. But the user's simple instruction—"Execute recommendations"—triggered a cascade of work that would expose a subtle but critical testing pitfall. The recommendations the assistant had proposed were: integration testing, Ansible role testing in the Docker test harness, load testing the cache system, and documentation review. The assistant began executing these in order, starting with integration testing for the newly written code.

The First Attempt: Writing Tests That Don't Match Reality

When the assistant first attempted to add tests for the rbdeal package (which contains the garbage collection logic, deal pipeline metrics, and balance metrics), it wrote test files based on assumptions about the code structure. These initial tests failed immediately with LSP errors—undefined types, missing fields, incorrect method signatures. The assistant had guessed at the API surface of GCConfig, GCState, and other types without reading the actual source files.

This is a common failure mode in AI-assisted coding: the model generates plausible-looking code that doesn't actually match the real implementation. The assistant recognized this and pivoted, reading the actual source files (gc.go, deal_metrics.go, balance_metrics.go) to understand the real code structure before writing tests a second time.

The Second Attempt: Correct Tests, Wrong Assumptions About Prometheus

With corrected knowledge of the codebase, the assistant wrote tests that matched the actual API. These tests compiled without errors. But when run, they crashed with a panic:

panic: duplicate metrics collector registration attempted

The root cause was subtle. The production code used promauto (from github.com/prometheus/client_golang/prometheus/promauto) to create Prometheus metrics. The promauto package automatically registers metrics with the global Prometheus registry. When a test creates a new instance of BalanceMetrics or DealPipelineMetrics, it triggers registration of the same metric names that may already exist from previous test runs or other tests in the same binary. The global registry rejects duplicate registrations, causing a panic.

This is a well-known challenge when testing code that uses Prometheus metrics in Go. The standard approaches include:

1. Using a custom registry per test (not possible with promauto since it targets the global registry)
2. Using prometheus.NewRegistry() and manually registering metrics
3. Using singleton accessor functions that return the same instance
4. Using testing.Cleanup or prometheus.Unregister to clean up after tests The assistant chose the singleton approach, modifying the tests to use accessor functions that return the same metrics instance rather than creating new ones. This is a pragmatic solution that avoids the registration conflict while still validating that the metrics behave correctly.

The Subject Message: What It Actually Validates

The passing tests in the subject message validate several distinct components of the FGW system:

Balance Metrics Tests

TestGetBalanceMetrics_Singleton verifies that the singleton accessor pattern works correctly—that calling the function multiple times returns the same metrics instance. This is critical because the singleton pattern was the fix for the Prometheus registration conflict.

TestBalanceMetrics_AttoFILToFIL and TestBalanceMetrics_LargeDatacap test conversion functions that transform attoFIL (10^-18 FIL, the smallest unit on the Filecoin network) to human-readable FIL values, and validate that large datacap values are handled without overflow or precision loss. These are important because the Filecoin blockchain deals in tiny units, and balance display must be accurate for operators monitoring wallet health.

Deal Pipeline Metrics Tests

The five deal pipeline tests (TestDealPipelineMetrics_IncProposed, TestDealPipelineMetrics_IncAccepted, TestDealPipelineMetrics_IncRejected, TestDealPipelineMetrics_IncExpired, TestDealPipelineMetrics_IncRenewed) validate that each counter in the deal lifecycle works independently. Deals on the Filecoin network go through a lifecycle: they are proposed, accepted or rejected by storage providers, and eventually expire or get renewed. Each transition must be accurately counted for operators to understand pipeline health.

GC Tests

TestGCStateConstants verifies that the garbage collection state constants (GCStateActive, GCStateCandidate, GCStateConfirmed, GCStateInProgress) have the correct integer values matching the SQL schema. A mismatch between code constants and database schema would cause silent corruption of GC state.

TestGCStateTransitions validates that state transitions follow the expected order (Active → Candidate → Confirmed → InProgress → Active for retry). The GC algorithm is passive, meaning it doesn't actively scan for garbage but instead relies on reference counting to identify candidates. Incorrect state transitions could lead to data loss or retention of garbage.

TestGarbageCollector_Noop and TestGarbageCollector_WithGroups test the GC algorithm's behavior with empty and populated group sets. The "noop" test validates that the collector doesn't crash when there are no groups to process, while the "with groups" test validates the actual collection logic.

Assumptions Made and Lessons Learned

The assistant made several assumptions during this process, some correct and some incorrect:

Correct assumption: That the rbdeal package needed test coverage. The package had zero tests despite containing critical business logic for deal lifecycle management, balance tracking, and garbage collection.

Incorrect assumption: That tests could freely instantiate Prometheus metrics structs. The assistant initially wrote tests that created new BalanceMetrics and DealPipelineMetrics instances, not realizing that promauto registration would conflict across tests.

Corrected assumption: That the singleton pattern was the right fix. After encountering the panic, the assistant correctly identified that using accessor functions returning the same instance would avoid the global registry conflict. However, this approach has a tradeoff: it means tests share state, which can lead to flaky tests if not carefully managed.

Implicit assumption: That the existing production code's use of promauto was the right design. In a larger codebase, using promauto with the global registry is often considered an anti-pattern because it makes testing difficult and couples code to global state. A more testable design would use dependency injection of a custom registry. The assistant chose to work within the existing design rather than refactoring it.

Input Knowledge Required

To fully understand this message, one needs:

• Knowledge of Go testing conventions and the testing package
• Understanding of Prometheus client libraries in Go, particularly the difference between promauto (auto-registering to global registry) and manual registration with custom registries
• Familiarity with the Filecoin network's unit system (attoFIL) and deal lifecycle
• Understanding of passive garbage collection and reference counting concepts
• Knowledge of the FGW project architecture: S3 frontend proxies, Kuri storage nodes, and YugabyteDB backend

Output Knowledge Created

This message creates several important outputs:

1. Validated test coverage for the rbdeal package, which previously had zero tests
2. A reusable pattern for testing Prometheus metrics code in this project (singleton accessors)
3. Confidence that the GC algorithm, balance metrics, and deal pipeline metrics work correctly
4. A foundation for future testing of the enterprise-grade observability infrastructure

The Broader Significance

This message represents more than just passing tests. It represents the moment when the assistant closed the loop on Milestone 02's test coverage requirement. The enterprise-grade milestone wasn't truly complete until the metrics and GC code were tested. The subject message is the validation point that transformed "code that exists" into "code that is verified to work."

The debugging journey—from writing incorrect tests, to reading the actual source, to encountering the Prometheus panic, to implementing the singleton fix—is a microcosm of the challenges in testing distributed systems code. Prometheus metrics are a cross-cutting concern that touches every component, and testing them requires careful consideration of global state. The assistant's journey from "this should be simple" to "this is actually subtle" to "now I understand the pattern" mirrors the learning curve that any developer faces when adding tests to an existing observability infrastructure.