When Benchmarks Panic: The Prometheus Metrics Registration Problem

A Single Debugging Message in the FGW Distributed Storage Project

In the life of a complex software project, most messages are about building—adding features, wiring up components, making progress. But some of the most revealing messages are about breaking: encountering an unexpected failure, diagnosing it, and attempting a fix. Message 1889 from the FGW (Filecoin Gateway) distributed storage project is exactly this kind of message. It captures a moment where the assistant, having successfully implemented three major milestones and committed them to version control, turns to performance validation and immediately hits a wall. The message is short, technically dense, and rich with assumptions, reasoning, and a failed fix attempt. It is a microcosm of the debugging process itself.

The Full Text of the Message

The assistant writes:

The benchmark fails due to Prometheus metric registration. That's a known issue when running multiple benchmarks - metrics can only be registered once. Let me run a single benchmark iteration:

>

`` cd /home/theuser/gw && go test ./rbcache/... -bench=BenchmarkARCCache_Put -benchmem -count=1 -benchtime=100ms 2>&1 ``

>

The output shows the same panic: `` BenchmarkARCCache_Put-192 panic: duplicate metrics collector registration attempted ``

The message then shows the stack trace pointing to Prometheus's MustRegister function in registry.go:406.

Why This Message Was Written: Context and Motivation

To understand why this message exists, we need to look at what came before it. The assistant had just completed a marathon implementation session covering three milestones:

1. Milestone 02: Enterprise Grade — Grafana dashboards, Prometheus metrics, Loki logging, backup automation, and an AI support system.
2. Milestone 03: Persistent Retrieval Caches — An L1 ARC cache, L2 SSD cache with SLRU eviction, and a DAG-aware prefetch engine.
3. Milestone 04: Data Lifecycle Management — Passive garbage collection, reference counting, and claim extender modifications. All three milestones were committed. The assistant then moved into a validation phase: fixing a YAML syntax error in the Ansible backup playbook, running the test suites, and planning cache benchmarks. The test suites passed. The next logical step was performance benchmarking of the new cache system—specifically the Adaptive Replacement Cache (ARC) that lives in rbcache/arc.go. The assistant had already run go test ./rbcache/... -bench=. -benchmem and hit the panic. Message 1889 is the assistant's response to that initial failure: a diagnosis and an attempted fix. The motivation is straightforward: the assistant is methodically working through a validation checklist. The cache benchmarks are item #3 on the todo list, marked as "medium" priority but "in_progress." The assistant wants to verify that the ARC cache performs adequately before moving on to documentation review and committing the test files. The benchmark failure is a blocker, and the assistant is trying to clear it.

The Reasoning and Diagnosis

The assistant's diagnosis is stated concisely: "The benchmark fails due to Prometheus metric registration. That's a known issue when running multiple benchmarks - metrics can only be registered once."

This diagnosis reveals several layers of understanding:

1. The assistant knows what Prometheus MustRegister does. In the Prometheus client library, MustRegister adds a metric collector (like a counter or gauge) to the global registry. If the same collector is registered twice, it panics. This is by design—it prevents silent metric collisions.
2. The assistant understands how Go's benchmark framework works. The go test -bench=. command runs all benchmarks in the package. Each benchmark function is executed multiple times (controlled by -benchtime and -count). The assistant correctly identifies that running multiple benchmarks causes the Prometheus metrics to be registered multiple times, triggering the panic.
3. The assistant has a theory about the root cause. The assumption is that the metrics are being registered in a global scope (likely in a init() function or a package-level variable initialization), and that each benchmark invocation re-triggers this registration.

The Attempted Fix: A Single Benchmark Iteration

The assistant's fix attempt is to run a single benchmark iteration:

go test ./rbcache/... -bench=BenchmarkARCCache_Put -benchmem -count=1 -benchtime=100ms

This command uses several flags:

• -bench=BenchmarkARCCache_Put — run only the Put benchmark, not all benchmarks.
• -count=1 — run only one iteration of the benchmark.
• -benchtime=100ms — run each benchmark for only 100 milliseconds. The reasoning is clear: if the problem is that multiple benchmarks cause duplicate registration, then running a single benchmark with a single count should avoid the collision. The assistant is trying to isolate the benchmark to see if the problem is truly cross-benchmark contamination or something deeper.

The Failure: The Fix Doesn't Work

The output shows the same panic, even with a single benchmark. This is the critical moment in the message. The assistant's hypothesis was incorrect—or at least incomplete. The panic occurs even when running a single benchmark function with a single count.

This failure reveals a deeper truth about Go's benchmark framework: even a single benchmark run can trigger multiple iterations. The -benchtime=100ms flag doesn't mean "run once for 100ms." It means "run the benchmark function repeatedly until 100ms has elapsed." The Go testing framework calls the benchmark function in a loop, and each call to BenchmarkARCCache_Put presumably triggers the Prometheus metric registration again.

But there's an even more fundamental issue. Looking at the stack trace, the panic happens inside MustRegister. This suggests that the metrics are being registered inside the benchmark function itself, or inside code that the benchmark calls. If the benchmark function creates a new instance of the cache (which it likely does, to get a clean state for each measurement), and that cache initialization registers Prometheus metrics, then every iteration of the benchmark loop will try to register the same metrics again.

Assumptions Made by the Assistant

The message reveals several assumptions, some correct and some incorrect:

Correct assumptions:

• Prometheus metrics can only be registered once per process.
• The Go benchmark framework runs multiple iterations.
• The panic originates from MustRegister in the Prometheus client library. Incorrect or incomplete assumptions:
• That running a single benchmark with -count=1 would avoid the problem. The assistant assumed that the "multiple benchmarks" were the cause, when in fact the cause is "multiple iterations of the same benchmark."
• That the metrics registration happens at package initialization time (which would only happen once). The evidence suggests the registration happens inside code that is called per-iteration, such as cache constructor code.
• That -benchtime=100ms would limit execution to a single call. In Go's benchmark framework, -benchtime controls the minimum duration of the benchmark loop, not the number of iterations.

Input Knowledge Required

To fully understand this message, the reader needs:

1. Knowledge of Go's testing and benchmark framework — understanding that go test -bench runs benchmark functions in a loop, that -count controls the number of iterations, and that -benchtime controls the minimum duration.
2. Knowledge of Prometheus client_golang — understanding that MustRegister panics on duplicate registration, that the default registry is global, and that metrics are typically registered once at startup.
3. Knowledge of the FGW project's architecture — understanding that the cache system in rbcache/ uses Prometheus metrics for observability (as part of Milestone 02's enterprise-grade monitoring), and that these metrics are likely registered when cache instances are created.
4. Knowledge of the project's recent history — understanding that the assistant is in a validation phase after committing three milestones, and that the cache benchmarks are a planned validation step.

Output Knowledge Created

This message creates several pieces of knowledge:

1. A documented bug — The ARC cache benchmarks are broken due to Prometheus metric registration conflicts. This is now a known issue that needs to be fixed before performance validation can proceed.
2. A failed fix attempt — The approach of running a single benchmark iteration does not work. Future attempts will need a different strategy.
3. A diagnostic clue — The stack trace points to registry.go:406 in the Prometheus library, which is inside MustRegister. This tells future debuggers exactly where the panic occurs.
4. A constraint on the testing approach — The benchmark cannot be run in-process with the current code structure. Either the metrics registration needs to be refactored (e.g., using a per-instance registry instead of the global default), or the benchmarks need to be run in separate processes (e.g., using -exec or a separate test binary).

The Thinking Process Visible in the Message

The assistant's thinking process is visible in the structure of the message:

1. Observe the failure: The benchmark panics with "duplicate metrics collector registration attempted."
2. Form a hypothesis: "That's a known issue when running multiple benchmarks - metrics can only be registered once." The assistant assumes the problem is cross-benchmark contamination.
3. Design an experiment: Run a single benchmark with a single count and a short benchtime. If the hypothesis is correct, this should avoid the collision.
4. Execute the experiment: The command is constructed with specific flags to isolate the benchmark.
5. Observe the result: The same panic occurs.
6. Implicitly conclude: The hypothesis was wrong, or at least incomplete. The problem is not about multiple benchmarks but about multiple iterations within a single benchmark. The message ends at this point. The assistant does not propose a next fix—the message is a checkpoint, a record of what was tried and what failed. The thinking is visible in the gap between the diagnosis ("known issue") and the failed fix. The assistant correctly identified the symptom but underestimated its depth. The metrics registration is not just a cross-benchmark issue; it's a per-iteration issue that requires a more fundamental refactoring.

The Broader Significance

This message is significant beyond its immediate context because it highlights a common tension in Go projects that use Prometheus: the global registry pattern. Prometheus's default registry is a global singleton, which works well for production code (where metrics are registered once at startup) but creates problems for testing (where code is initialized and torn down repeatedly). Many projects solve this by:

• Using a test-specific registry that is reset between tests.
• Registering metrics lazily (only on first access).
• Using MustRegister with a sync mechanism to prevent duplicate registration.
• Separating metric initialization from cache construction. The FGW project's approach—registering metrics in what appears to be cache constructor code—is a design choice that prioritizes simplicity in production but creates friction in testing. The message captures the moment when this design tension surfaces.

Conclusion

Message 1889 is a snapshot of active debugging: a hypothesis formed, an experiment designed, a result observed, and a hypothesis revised. It is not a triumphant message—the fix didn't work—but it is a productive one. It narrows the problem space, eliminates one possible cause, and points toward the deeper issue. In the broader arc of the FGW project, this message is a small but meaningful step toward robust performance validation. It reminds us that debugging is not about getting it right on the first try; it is about learning from each attempt and refining our understanding until the solution becomes clear.